This depends on exactly what they've said. If it's
We'd rather you didn't use non-stable libraries with this product. If you do this, we won't help you.
that's absolutely fine, and very similar to the position the Linux kernel community take with non-open source modules in the kernel. Even something like
We'd rather you didn't use non-stable libraries with this product. If you do this, we'll block your account on our forum and will never reinstate it.
is fine (legally anyway). They can make a modified version of the GPL which forbids this - but in that case it may not be called the "GNU General Public License" and it would be incompatible with the GPL itself, so they couldn't use any GPL code in their product.
What they can't do is something like:
You may not use non-stable libraries with this product. If you do this, you are in breach of the license and we will sue you.
or at the very least, such a term can be ignored in accordance with Section 7 of the GPL:
If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term.