If I fork a library that uses the GPL3 license and heavily modify it, can I then release the new library with an ApacheV2 license
No, you may not. Your library is still, by your own admission, a derivative of the original GPLv3 code. GPLv3 s5c says that if you convey a derivative work, you must do so under GPLv3.
I've also contacted the original project's developer for express permission, would that cover any legal issues for my users?
If you were to obtain the permission of all the rightsholders in the work you modified (which may well be more than just the original developer, if the project accepted contributions, or if it was in turn built on another GPL work), then yes, you could relicense as you have done.
Now however I'm not so sure I was allowed to do this or what I need to do to comply with the license rules.
Your suspicions are justified. Right now, from what you've written, you're committing copyright infringement, and have been doing so for two years. At the very least, you should immediately stop distributing your code. In my opinion, you should promptly contact everyone who you believe may have taken a copy of your code, and tell them that they have no right to use it. If you have a project website, take it down, and replace it with a clear statement of the problem. And if I were you, I'd also be looking at talking to a lawyer.