Free/Libre Software and Open Source refer to the actual implementation of a software project.
Open Format refers to the specification of a file format which has been licensed freely so that anyone can make their own software which understands or produces the format.
You can have FLOSS software which produces a non-open format. Often software that implements someone else's format is highly unreliable (compared to the original producer of the format), or only able to read/write a subset of the format. You can also have FLOSS software which is the original producer of a format, but the code is so convoluted that it's very difficult to understand the format, making it hard to re-implement in another language.
You can also have non-FLOSS software which produces an open format, such as Microsoft Word. If you get into trouble when you implement the format, it's hard to know if it's a bug in your code, or an error in the specification, because we can't compare our software to the reference implementation.
You could also publish an open file format without any implementation at all. But without any software that uses it no one will care about your file format.