You are re-distributing an entire Debian build with your software (licence unspecified) aggregated inside it.
If I run a cron job to update the vm from time to time am I correct that I would have to provide the source code of the updated gpl programs as well ?
You are. We have a question on re-distributing unmodified GPLv2 binaries (they are unmodified in the sense that you're not modifying what Debian puts out, simply passing it along to others) which confirms that you are responsible for source distribution to your end-users. GPLv3 has similar obligations.
In case I provide an admin software so that updates can be made easily by my customers, am I correct that in this case his would not count as distribution ?
It depends where the software is getting the updates from. If it's getting them from a repository under your control, as it seems it is, then that counts as distribution. You're making GPL software available online for others to access, and you therefore have obligations; whether you or someone else provides the tool that allows end-users to access your repository makes no difference to those obligations.
Since the docker containers are downloaded from my private repository when running an update I would need to provide the source code on my website for each version ?
Generally, yes. If any of the software is GPLv3, then for the binary-only software you distribute on USB key you are performing a s6b conveyance, and that requires you to give a written offer of source, valid for at least three years. Gratis network access to the source satisfies this, but you have a prescribed time for which you must continue to offer it, and you must inform USB stick recipients about it in writing.
And, of course, IANAL/IANYL.