The scenario is the following: a proprietary software can save files in its own file format. I want to build a free/open-source software capable of decoding those files for further processing.
Given that there is no documentation for the file format, I create a few files and reverse engineer them. After experimenting with them and after understanding the rules behind them, I write my software. Note that I only worked with the files, I didn't attempt to disassemble the proprietary software itself.
Now, supposing there are no patents covering the file format, can I release my software to the public? Are patents the only thing I should worry about? Can the software license restrict what I can do with the files it produces?