Unless I'm missing something, you're either describing reverse engineering, where someone has figured out how an existing product works and has created a competing product, or you're talking about someone releasing their own code under one license and then changing it down the road.
In either case, I fail to see where anything can be considered unethical. In the case of someone changing the license on their own work, that's entirely their call, they did all of the work that they're freely sharing with the world. How can anyone possibly point and wag their finger at what somebody does with their own hard work?
In the case of a competing product being made, isn't that a good thing that is in line with the very spirit of open source? You're not releasing open source software so that everyone in the world becomes so dependent on it that everyone must come to you to eat, you're releasing stuff freely into the wild so that it might benefit others, however that benefit works.
Granted, there are business models built around open source where, like in any capitalistic system, you'd have daydreams of the piles of cash that would result in everyone needing your software exclusively, but also like any reasonable business person in a capitalistic society, you accept the concept and challenge of competition.
Specifically to the point of open source and business, I don't see how the boundaries of what is and isn't unethical are any different than any other form of business. If Honda released a new car, but then someone realized that it was just a Honda logo cheaply plastered over top of what was actually a Hyundai car, everyone would think it's a sleazy move and it would backfire.
A famous recent example of this in software is the Console OS accusation of wholesale copying Android-X86 without attribution. It wouldn't have been so bad if they had of been honest, but if they had of been honest it would have simply appeared to be a cash grab anyway.
In your question you seem to suggest something about competing products copying the name also. While I can't comment on the legality of it, I'd say that it's unethical because it shadows the other product. If your product is so much better, it shouldn't need to shadow its competition. Shadowing is blatantly done for no other reason than to hide the original project and steal traffic/users from it. There may be exceptions where this is acceptable, such as shadowing an abandoned project (unofficially resurrecting and taking over maintenance).