This boils down to “what is a derivative work?” and the answer is “we don't really know”.
However, there seems to be general industry consensus that a server program and a client program that behave as separate programs and just communicate over some IPC mechanism such as HTTP messages are separate works, not derivative works of each other. If that is the case, distributing the client cannot trigger any GPL requirements in the server code.
This means your understanding seems correct.
As long as you don't distribute/convey your server software you do not have to disclose the source code.
You are correct that the GPL does not trigger when merely running the software.
Exception: if you include modified AGPL-covered software then that already triggers when users can interact with that software, i.e. when you publicly perform the software.