The FSF includes Freedom #1 in their Free Software Definition:
The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
A software artifact is not inherently free or nonfree, but the terms under which it is possible (legally and practically) to use, modify, and redistribute it are free or nonfree.
Per Freedom #1, a binary distributed without source is a nonfree distribution. That is, for any particular person, either they have access to the source under free terms or they do not. For those recipients who do have the source code (with, as you say, full FSF-required freedoms), the software can be used, modified, and distributed freely. For those recipients that do not have access to the source code under free terms, the software is nonfree. Under you economic model, recipients of the binary may receive it under nonfree terms and then pay you to receive it under free terms.
In practice, I don't know of any company doing this. There is some software, like grsecurity that is sold under the GPL but only made available to purchasers (but can be shared by those purchasers broadly thereafter). I don't know of any vendor that offers a gratis public binary alongside paid freedom-supplying source code, but it's not an impossible business model.