Before actually using it, the clients have to download and install the libraryprogram on their own as a dependency. So in that regard we're in the clear selling our software without any GPL restrictions.
That is not correct. The determination if your software is a derived work of the GPL libraryprogram does not depend in any way on the number of parts you divide it in or from how many locations your customer needs to download the various parts.
If your software can only work with this particular GPL libraryprogram, then it is most likely a derived work and needs to be GPL licensed itself as well.
On the other hand, if your software can work with any package that provides the needed functionality and can be invoked in <this> way, and if that package can be created without knowing the source of the GPL libraryprogram, then the two are most likely a mere aggregation and unrelated in copyright terms.
For the determination if an alternative to the GPL libraryprogram can be used, only the technical possibility is needed, not the economical feasibility.
Taking all that into consideration, it makes no difference if you distribute the complete installation as a docker image, or if you let your customers download from multiple locations.