Firstly note that while linking to a GPL library means you have to comply with the GPL terms, that doesn't mean that your code has to be licensed under the GPL, only that the license you use cannot prevent users complying with the GPL terms. This is where GPL compatible licenses come into play, which includes Apache 2.0 and Modified BSD (3-clause). Note in the compatible licenses list, that the version matters, you will see that Apache 2.0 is compatible with GPLv3 but not GPLv2, also the BSD mentioned is the modified BSD license which has no advertising clause, so be sure to look closely at what is used when looking at compatible licenses.
From the examples I have seen, it is considered a matter of what code is compiled/linked into the final binary rather than the method of retrieving the code that is used. I would suggest using a build option or only searching for installed libraries rather than making your build system download and build the libraries, that leaves it to the user to install and enable the use of the GPL libraries.
An example to look at would be the eigen library, where older versions were released under LGPL, it is moslty licensed under MPL now. It uses the preprocessor symbol
EIGEN_MPL2_ONLY to not compile LGPL licensed code into the library. See example in v3.3 src/OrderingMethods/Ordering.h.
Similarly the quadriflow project using eigen has a build option to only use eigens MPL algorithms, you can see in src/optimizer.cpp that it chooses different classes based on the licensing choice.