Law is complicated and varies by jurisdiction. In the United States (and probably many other jurisdictions), a court generally cannot compel you to release your source code simply because you did not abide by the terms of a copyleft copyright license. In this situation, you have committed copyright infringement (based on the facts you've presented), but the remedies for copyright infringement are monetary damages and injunctions to cease future distribution.
Part of what complicates the case is that the GPL says you may exercise certain permissions pursuant to certain obligations. If you exercise the permissions without meeting the obligations, the question is: have you simply acted outside the license altogether, or have you acted within the license but failed to meet your obligations? In the latter case, how should the law resolve the situation? Different jurisdictions may answer this question quite differently, and even within a jurisdiction the answer could possibly turn on case-by-case particulars.
Certainly the best thing you can do immediately is to cease distribution of the infringing version (as you say you've already done in your hypothetical). From there, it is up to the copyright holder whose copyrighted work you infringed whether to take legal action against you, or ask for a settlement in lieu of legal action, or do nothing at all. Regardless of what the law may do, it is not impossible that the copyright holder could ask that you release your source code or else they will pursue a lawsuit. In that case, you can decline their offer and leave it to the court whether and how your infringement will be punished under the law.
Eben Moglen (a lawyer who has advised the FSF substantially) offers the following explanation (quoted from this article):
The claim that a GPL violation could lead to the forcing open of proprietary code that has wrongfully included GPL'd components is simply wrong. There is no provision in the Copyright Act to require distribution of infringing work on altered terms. What copyright plaintiffs are entitled to, under the Act, are damages, injunctions to prevent infringing distribution, and--where appropriate--attorneys' fees. A defendant found to have wrongfully included GPL'd code in its own proprietary work can be mulcted in damages for the distribution that has already occurred, and prevented from distributing its product further. That's a sufficient disincentive to make wrongful use of GPL'd program code. And it is all that the Copyright Act permits.
Though he speaks from the perspective of a United States lawyer -- indeed, the Copyright Act he references is an American law, so the answer may differ internationally. See MadHatter's answer for some international particulars.
Note also that GPLv3, section 8, explicitly grants a copyright holder the right to permanently terminate your rights under the GPL if you violate those terms. If that happens, it doesn't impact your proprietary project, since you've already removed its dependency on the GPL-licensed code, but it would prevent you from using that dependency in other correctly-licensed projects in the future.