With respect to the other answers, I disagree with them.
You have developed some code, which you have released under a non-copyleft free licence; let's pick BSD 3-clause (BSD3). The choice doesn't affect the answer all that much, but it's easier than dealing with a list of possible licences.
At some point in the development of your project, you have linked in a library that is covered by GPLv2. Now, when you distribute the resulting project, GPLv2 s2b requires that you do so under GPLv2. This doesn't violate BSD3, which has no requirement that you not do this. Some would hold that this is re-licensing, but as I argue elsewhere it it not: it is adding additional conditions to the conveyance, over and above those which BSD3 imposes, which BSD3 does not forbid.
The FSF are clear that you may still continue to release your code on its own under BSD3. But when you release the whole project (including the library) you must do so with the conditions of GPLv2 applying to all of it. If you do not, then yes, you are violating the library's copyright licence.
The question of removal of the library then arises. Some (including me) would argue that by Ship of Theseus the version with the library removed is a derivative of the previous version with the library in and thus still subject to GPLv2. Others, such as Brandin in his excellent comments above, would disagree, and claim that once the GPL library was removed, the GPL obligations on derivatives would lapse. I'm not aware of any jurisprudence on the subject that can help us say which is the most likely interpretation, and it would depend on your jurisdiction anyway.
Of course, if you're still publishing the original, pre-GPL-library version under BSD3 (see above), and also your new-non-GPL library patches under BSD3, someone else could combine them and produce a version of the project that had never seen the GPL library, which would have a very strong claim to be releasable under BSD3 in its entirety.
