To my considerable surprise, I recently discovered that somebody has created a so-called "BSD Protection License" which appears to be designed to specifically solve this problem.
The text of the license may be found here or here. It is quite lengthy, so I won't quote it in full, but I will summarize it as follows:
- It allows verbatim redistribution of the software "as you receive it" in much the same way as the GPL, the MIT license, and most other open source licenses. This clause does not specifically mention the disposition of any source code, so presumably if you receive binaries without source, you can redistribute them as such. You're required to include a copy of the license and a copyright notice. This is all fairly typical.
- It allows redistribution of modified software, under a license which "expressly prohibit[s] the distribution of further derivative works." This is highly unusual for an open source license, but by itself it does not make the license non-free, as long as the license provides an alternative option.
- It also allows redistribution of modified software under the same license (i.e. copyleft). The modifications must be clearly indicated, which the GPL also requires, and copyleft extends to "any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof" - no exception for "mere aggregation."
Opinions vary on whether this is a free license. The Fedora wiki page, linked above, tersely describes it as "Free, but GPL-incompatible," but Debian thinks it's non-free, because the scope of its copyleft is even broader than that of the GPL. They also didn't like the wording of the "implied acceptance" section and suggested reusing the GPL's wording instead. Both of those problems are arguably fixable.
As for how effective it would be... that's unclear to me. If you keep the arguably non-free expansive copyleft, then it's certainly never going to be incorporated into a GPL project in any way, but this copyleft will extend to and prohibit all sorts of perfectly reasonable conduct, which could be problematic. For example, if somebody packages together your software with a bunch of other, unrelated software (e.g. in a Docker container or Linux distro), the license would arguably infect the entire package, which is probably not what you want and might create gratuitous license conflicts. If the copyleft were weakened, then it might strike a more appropriate balance here.
Overall, however, I'm not really impressed by this license. The sole beneficiaries of the "take proprietary" option are, presumably, developers who want to get into the business of selling the software. But since the "take proprietary" option is extended to the world at large, those developers would just end up competing with each other. If they would have used the GPL with a CLA, they could've gone proprietary without any competition (other than the inevitable non-relicensed fork, of course). If you want to take parts of the software proprietary and release other parts under copyleft, you can do that with a CLA too. And, of course, it's much harder to get people to contribute to your open source project when your license is screaming at the top of its lungs that you intend to take their contributions and go proprietary. So the license does not appear to provide substantive advantages over the CLA model, from the self-interested (greedy) perspective of the person choosing a license for their software.