Robert Bosch Engineering and Business Solutions® has published an open source driver for their ES58X products. Based on this published code, I want to create an open-source hardware which is compatible with that driver. I want to get new VID/PID of course. The device would then be compatible with their proprietary software after a small change of their windows software. (a registry key as an example which defines VID/PID). Is it legal for me to do that? In general, is it legal to create a compatible open-source hardware for a proprietary software? The hardware is created by just sniffing the USB traffic between PC and device and guessing the protocol.
However, your hardware might include some code (in a processor, ROM, FLASH or otherwise) for the implementation of the USB protocol, and this code might be a derivative work of the existing code, possibly code from the drivers mentioned above or from the proprietary software you mentioned. Your question does not mention which part of the protocol you plan to reverse-engineer based on the sniffing of USB traffic. If it is the part covered by the GPL-licensed drivers I would not bother and just re-use the source code of these drivers as much as possible, it will be a derivative work which you have to license under GPL anyhow.
If, however, you are trying to reverse engineer the protocol parts that are not under an open license, but instead are originating in proprietary software, then you will have to carefully check the license terms of this proprietary software. Many license agreements for proprietary software include a clause that forbids reverse-engineering.
If you want to license the hardware under an open license, then you might want to consider license terms which are suitable for hardware, such as the CERN Open Hardware Licence or another license suitable for hardware, as most OSS licenses for software are not particularly suitable.
I will not be guessing about the impact of you changing a registry key on the user's Windows system. If you receive the user's explicit consent for that it will likely be fine, as I do not think that the value of any specific registry key would be copyrightable. The user might just drop out of the warranty clause of the proprietary software, but that's another thing.