I found a GPL v3 licensed library, that I can compile for various architectures and use in the code for a BSD licensed project. This means that I probably have to distribute the binaries with it.
You can not distribute anything linking against GPL code as anything other than GPL. At least, that is the FSF's position, there is no firm legal determination one way or the other. In case of doubt, use GPL if you don't want to volunteer for the fun of being at the center of the lawsuit deciding this. Maybe there are non-GPL alternatives?
In general you should fulfill the license of every code you use, so if you are distributing a GPL library or a modified form of it (e.g. as binary), you need to fulfill the GPL license, which has some requirements on the final package redistribution.
About linking: this is a gray spot. Some judges have ruled that also API is copyright-able, OTOH some people think that simply linking doesn't make a project derivable.
IMHO, if you code some functionalities around a library, your code is also derived from such library. If you link to a generic library, and then you change the library, it doesn't make it derivative (assuming: you are distributing separately the two things, the generic library is not derived from the other library [e.g. a stripped down version]).
As the other answer suggests, linking against a GPL library by non-GPL code is not a legit way. There are precedents where GNU GPL has been legally enforced in the past. So basically, here are your options:
Try and see if the BSD-licensed code has any GPL alternative. If possible, ask the author if you can create a GPLed fork out of it. Though the BSD license terms are liberal enough that you can do it yourself, its still better to ask if possible.
Alternatively, try and see if the GPL library can be additionally licensed under LGPL. Unlike GPL, LGPL licensed libraries can be dynamically linked to by non-GPL code. If you go this route, you should obviously take care not to link it statically from your BSD licensed code.