The concept of an open system in engineering predates the concept of open source by a few decades. The use and popularity of the concept in software development occurred roughly the same time as, if a bit earlier than, open source - with the development and rise of Unix in the 1980s.
An open software is one that can be easily integrated with other software. An example of which is the SWIFT system for bank-to-bank transfer. All banks agree to how it should work and all banks can integrate with it. The fact that you need to pay SWIFT an annual fee to connect to their network does not make them closed.
An example of a closed system is the API banks develop for their own online banking portals - other banks cannot integrate with those APIs easily without working directly with the bank that owns the API.
Other examples of open systems include Adobe Photoshop (anyone can write plugins and scripts for it), Wordpress, the PCIe interface, USB and the MPEG4 media format. Some of which are open source, others are closed source, still others are patented.
In this usage, the concept of open is well defined in industry. People who are new to programming or who didn't grow up in the 80s where almost everything is incompatible with each other may not be familiar with this use of the term open. But throughout the history of computing the term open has been used to promote interoperability.
The are also systems, including software, that can be integrated but is not considered open. The main difference between the two is weather you need permission to integrate said system with yours. Payment is not an issue. If you are required to pay, for example to write apps for iOS, the system is still considered open. What is not open is if the owner of the system can refuse to allow you to integrate the system or integrate with the system or charge you a lot more to allow you to use their system compared to others.