The GPL gives you a few options for how to distribute source code. Typically, you distribute source at the time you distribute the binary. However, section 3(b) of the GPLv2 allows you to distribute a work based on another GPL-licensed work, in binary form, as long as you also
Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange
The approach will work for you so long as you are prepared to respond to requests for source code for at least three years after you stop distributing a version of your software that includes GPLv2-licensed work. Since this offer is only usable by someone who has obtained your GPLv2-derived software, for three years after you distribute it, your obligation to supply source will naturally end three years after you switch to dependencies that do not include GPL-licensed code. This obligation only extends to the version(s) of your software that include GPLv2 components: fully non-GPL versions may be distributed with no source, even during the three years that you are required to share your the source of your old GPL'd version(s).
Note that this would require you to send the source, in physical media, to anyone in possession of your binary, upon request. Probably anyone would be willing nowadays to receive it via download instead, but recipients do have that right under the GPL.
Note also that this distribution of source still happens under the GPL, so any recipient in the world can take the source you give and upload it to a public website, forever. You might not get any requests for source code after that (unless you release a new version) because your source code is now freely out in the world.
In view of those factors, it's not clear to me what benefit you'll see by using a written offer, and you'll be incurring a three-year obligation to potentially mail out source code. If you're worried that everyone who downloads your binary will have to download your large source code as well, rest easy that the GPL expressly does not require users to download the source if they do not want it (emphasis mine):
If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code.