The FSF includes Freedom #1 in their Free Software Definition:
The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
A software artifact is not inherently free or nonfree, but the terms under which it is possible (legally and practically) to use, modify, and redistribute it are free or nonfree.
Per Freedom #1, a binary distributed without source is a nonfree distribution. That is, for any particular person, either they have access to the source under free terms or they do not. For those recipients who do have the source code (with, as you say, full FSF-required freedoms), the software can be used, modified, and distributed freely. For those recipients that do not have access to the source code under free terms, the software is nonfree. Under your economic model, recipients of the binary may receive it under nonfree terms and then pay to receive it under free terms.
In practice, I don't know of any company doing this. There is some software, like grsecurity, that is sold under the GPL but only made available to purchasers (but can be shared by those purchasers broadly thereafter). I don't know of any vendor that offers a gratis public binary alongside paid freedom-supplying source code, but it's not an impossible business model. Considering that some companies have had success doing this with the binary-plus-source together, this could work similarly, if you have buyers who have a business need to modify your software.
As with selling binary-plus-source, you are likely to see quick propagation of your free source becoming available from other sources. You can mitigate this if you provide frequent updates (so few people have the latest version unless they got it from you) or your have a base of purchasers who all have a business interest in not sharing their source with competitors.
Note also that if you are not the sole copyright holder of the work and you make use of code under a copyleft license (like the GPL), you may not distribute a binary without making the source available to each recipient. For this system to work, the code must either be all yours, or all not-yours parts are available under permissive non-copyleft terms.
