If there is a statement in a legally binding contract between two parties that says that one party is going to retain all source code and is not required to provide source code as part of the contract, but it turns out that parts of that source code are GPL-licensed, is that a direct conflict between GPL and the contract, especially if the other party is requesting the source? In that case, which takes precedence?
Let A be the party who is distributing the binary (you don't say this is what's happening, but you say that A is not distributing source, and there's not much else that they can be distributing except the binary) and B be the party who is receiving the binary, but not (as per contract with A) the source.
To my mind (and IANAL/IANYL) it depends on who the rightsholder for the GPL-licensed parts is. If A is the rightsholder on the GPL-licensed portions of the code, they are entirely within their rights to do this, because the copyright holder is never beholden to the rules of the holder's own license grant.
If, however, A is not the sole rightsholder in the GPL-licensed portions of the code, then they have a problem, because if they distribute a binary without complete corresponding source, they are violating the copyrights of the other rightsholders (let us call them "C"). GPLv3 s12 is very clear that
If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all.
So the point is not that the GPL "trumps" the contract between A and B, but that the contract between A and B doesn't trump the conditions on A's distribution licence from C.
C would be entirely within his/her rights to sue A for breach of copyright if A was distributing C's GPL-covered works in a binary-only form, regardless of why they were doing so. Whether B could sue for source access in such a case is jurisdictionally-dependent, and moreover complicated by B's having previously agreed not to receive source.