Assuming you're the sole rightsholder in the library, you can do this, but it's pointless. You are not bound by your own license grant, so can refuse source to those who ask for it (there might be arguments to made about promissory estoppel, but I'd hate to guess which way that would go).
But the recipients do not have that freedom. They have no power to redistribute source alongside the binary, because you have not given it to them, so they may not redistribute the binary at all (see, eg, GPLv3 s12). So what you've released is unredistributable software. I can't imagine anyone would have the slightest interest in using it, so I wouldn't bother if I were you.
If you're not the sole rightsholder in the library, but are releasing it GPL because you used upstream code in it which you received under GPL, you may not do this.