Once you release something under GPL, you cannot take this back. If anybody has downloaded your program before, they are free to put it into they own website for download. I am not sure if this is a "restriction" but may be important to consider.

Also, I am not sure if you can distribute even your own binaries only "under GPL" and which legal consequences this would have.

When you violate a license, somebody must sue you first, otherwise nothing happens. Who could sue you if you are the only owner of the code?

This is the reason why organizations like FSF require to assign the ownership of the contributed code to them. Otherwise they cannot enforce the license.