Unfortunately, this is a tricky case. ;)
Does your program interact with GPL-incompatible code? And if so, how? If your program only interacts with GPL-compatible code then obviously you're good to go but I guess you wouldn't have asked the question if that were the case.
It depends on what your software is doing but there's a risk that it's a "GPL wrapper":
I'd like to incorporate GPL-covered software in my proprietary system.
Can I do this by putting a “wrapper” module, under a GPL-compatible
lax permissive license (such as the X11 license) in between the
GPL-covered part and the proprietary part? (#GPLWrapper)
No. The X11
license is compatible with the GPL, so you can add a module to the
GPL-covered program and put it under the X11 license. But if you were
to incorporate them both in a larger program, that whole would include
the GPL-covered part, so it would have to be licensed as a whole under
the GNU GPL.
The fact that proprietary module A communicates with GPL-covered
module C only through X11-licensed module B is legally irrelevant;
what matters is the fact that module C is included in the whole.
Obviously the wording here is aimed at a case where a single nefarious person is trying to circumvent the GPL but the FSF would probably feel the same way if a separate person with good intentions wrote the intervening module.
Using dynamic loading doesn't change anything. The wording used in the GPL is "combine" which intentionally avoids naming a specific technology. The FSF doesn't address dynamic loading in their FAQ but they do mention plugins and essentially say they aren't special.
Even the fact that you yourself don't distribute GPL'd code might not shield you from all liability. There's precedent for successful copyright infringement lawsuits when the infringement was only facilitated and not committed directly. The concept is called secondary liability and the GPL specifically mentions it.
Keep in mind this is all just what the FSF intends the GPL to mean. We won't know much about what courts think of the GPL for a long time because there simply aren't many GPL violations being brought to court.
All that said, the FSF clearly states that the GPL is not for punishing well meaning individuals who unintentionally violate its terms and their policy is to first notify and try to get voluntary compliance. The GPLv3 even added special accommodation for this case. If you want any certainty your best bet is probably to email the library author and see if they're ok with your usage. You could also try sending the FSF an email and getting this added to their FAQ since it's a good question for any software that does dynamic loading.
I am not a lawyer and have no training in law whatsoever. I came up with this answer mostly based on Google searches in the last ~20 minutes. This isn't legal advice and you shouldn't believe things you read on the internet.