First of all, make sure that anybody who makes contributions to the open source project assigns copyright to you, or else your ability to use their contributions in the proprietary version will be restricted and any copyright lawsuit will become extremely complicated.
Next, make sure the "LICENSE" text file for your open source project is not included in the proprietary one to make sure nobody can claim the proprietary code is available under an open license.
Finally, if somebody uses a pirated copy of your code, make sure they are unable to contact you for technical support and unable to receive bugfix patches in a timely fashion.
You can do this by simply typing "XXX pro source code" into google and check if it's visible on the first page or two of results, then send a DMCA takedown notice: http://rising.blackstar.com/how-to-send-a-dmca-takedown-notice.html
If they do not take the content down in a timely manner then they are guilty of "wilful infringement", and you can try to find a lawyer who's willing to work "on contingency", basically you only pay them if they win the case for you.
Be sure to "register" the copyright for your source code, it's cheap and and makes you eligible for up to $150,000 in statutory damages if somebody distributes pirate copies of your work. This will make it much easier to get a lawyer on contingency, since they know they can take a share of the $150,000.
Some people will still pirate your work, but the vast majority of your customers will choose to pay instead as long as your price is "reasonable" and it's somewhat difficult to pirate the source code within a day or so of an important bugfix.
It is not possible to completely eliminate piracy, but it's also unnecessary. As long as "enough" customers pay, then your business will thrive.