Using reverse-engineered closed protocol in open source software

Question

The question is a partial derivative of this one: Someone open-sourced an un-open-source project

Here is the situation:

There is a device and a proprietary software to manage that device. They are developed by some big international company. The proprietary software uses some kind of protocol to exchange information with the device. This protocol was reversed engineered partially via sniffering, partially by decompiling the proprietary software. The protocol itself looks quite huge, but only small part, say 5-10% of all functions was reversed and used in a new open source software.

Here are the questions:

1. Is it legal to publish the open source software, say on github, where this reversed part of the protocol is used?

Note 1: The protocol itself is a relatively small part of the new open source software and not clearly visible from the first glance

Note 2: Yes, there are words about banning "reverse engineer" and "produce derivatives" in the vendor terms of use

2. From the practical perspective: what can be the consequences from such an action and how it happens?

Note 3: Probable options could be: from noone notice, to simple ban of the project / developer account, to court invitation. E.g. a vendor employee notice the software on github and trigger the action

Note 4: The logic can also be the following: The vendor makes money on selling the devices. So appearing a software, which does not expose/exploit any vulnerabilities of a relatively old product will not hurt anyone, even can attract some attention to the products

3. Bonus question: What happens if the new software is published as closed source on some app store?

Vendor itself should decompile the new software to prove that the protocol was reversed. This part is totally confusing.

Answer 1

Depending on jurisdiction, there could be copyright exceptions available that allow reverse-engineering for the purpose of interoperability. Sniffing is particularly unproblematic because you're observing behaviour instead of reading copyright-protected material, but decompilation could be an issue because it's not clear that your software is your own creative work, rather than copying relevant parts from the original software. In a professional setting that would want to limit liability, a clean–room approach would be used to prevent the people writing the reimplementation from having access to the original software.

If you have a legal right to reverse engineer the software, then a private contract denying you this right might be unenforceable in that part – but again, this would depend on jurisdiction. In the US, a contract (including EULAs) would override your right to reverse-engineer the software. Thus, it might be preferable to have the reverse engineering being done in countries where the right overrules contracts – it is the reverse-engineering and not the reimplementation that would be the primary concern.

If the vendor of the original software finds a reimplementation, they could reasonably believe that the protocol was not reverse-engineered legally and that your software could be a derivative work. They could for example send you a cease and desist letter, or send a takedown request (e.g. DMCA) to any services that host the software to you. You could also be sued, and would then have to demonstrate in court that your software was created legally – that it was created independently, without copying parts of the original software, without using reverse engineering techniques that were not authorized. The same would hold for closed-source software, except that the vendor would have a harder time becoming aware of potential infringement.

Answer 2

If you aren't allowed to "reverse engineer" and "decompile", what you did was illegal. So no, you are definitely not allowed to share results of such activity. Probably not even use it yourself. But that may depend on the exact details of the case, your local legal environment, and so on.

What practical consequences it may have is anybody's guess. They might never notice, or notice and don't care, or go after you all lawyers blazing. It might also happen that current owners don't care, but future ones might.