You wrote the program, and it's not a derivative of someone else's program. This means, you are not in trouble, and you can actually stop reading now.
It's unfortunate that you lost the sources, but that is it. You may tell those people that you're sorry but you lost the source code (in lack of a time machine, what else would you say!). End of story.
As you can read on gnu.org:
Note that the GPL, and other copyleft licenses, are copyright licenses. This means that only the copyright holders are empowered to act against violations.
The copyright holder(s), that's you. So nobody but yourself could act against your "violation". I'm rather sure you won't sue yourself.
Also note the wording of the license:
Each licensee is addressed as "you".
That means none more and none less than "you" in the GPL refers to the licensee, not you, the author. Apart from the fact that the original author (with very few exceptions) automatically holds the copyright and originator-right (and cannot even transfer the former in some countries, except under very specific conditions), that should also be evident from the fact that many GPL-licensed programs are dual licensed while the GPL states:
You may not copy, modify, sublicense [...] except as expressly provided under this License
Ouch! That would mean every company dual-licensing their software would place themselves in serious legal trouble. Surely, someone would have gotten the idea of sueing Oracle for GPL violation in the mean time?
But even if you cannot be pursued for the violation, is it strictly a violation in the first place? I daresay no. The GPL is a license where you document your goodwill intent of releasing your software as "free", which among many other things means people should have access to the source code. A number of possibilities how this could happen is given in the license, but it is only compulsatory to those redistributing your program.
Now, let's assume (hypothetically) the obligations of the license did indeed apply to you as well. It is entirely reasonable to assume that you acted in good faith, and that you indeed would send them the source code if only you could. Alas, you lost the sources. That's an accident, not a violation.
Sure, the net effect is that something was promised, and you can't deliver, but the spirit of the GPL is not to place silly obligations and liabilities on the authors, but to prevent someone from taking away the rights of the users, and to protect the authors.
But accident or not, you are failing to comply with the terms, so you possibly owe someone damages? No. The very license that you've chosen, and which is the only base for the user's right to obtain the source code explicitly denies warranties of any kind (paragraph 11 in v2 / 15 in v3) and damages (paragraph 12 in v2 / 16 in v3). This includes implied warranties and (explicitly) denies damages due to the inability to use the program.
I could imagine that someone might construct a (weak) case against you if you dual-licensed the program and had charged them money for the software together with the promise of getting access to the source code. In that case, they might argue having paid for the source code, which you now owe them (though after over 10 years, it's fair to assume the right has forfeited). Paragraph 17 of GPLv3 basically says the same thing, backwards, too ("unless... in return of a fee").
Well, you didn't do that. No fee, no sweat.