I plan to build an app based on a public repo with a copyleft license (AGPL-3.0 license). I don't want to open-source my own code (my own code will just server and front-end code, which are built on top of that repo's codebase). I wonder what the potential consequences are? Theoretically can the original repo's owner sue me? But if I don't publish my code, how can they even prove whether I am using their repo or not? Thanks
A copyleft license has the requirement that when you distribute a derivative work, then you are required to do so under the same copyleft license.
The AGPL license is one of the strictest licenses regarding what triggers the copyleft nature of the license and regarding what acts are considered to be distribution.
In simplified terms, if the program you execute contains code that is under the AGPL license (even if it is just a single line), then the AGPL license requires that the complete program is released under the AGPL license. Also, just giving people remote access to the program, like the server backend of a web-app, triggers a requirement that you must give your users access to the source code under the terms of the AGPL license.
Failure to follow the requirements in the license you received from a third party means that you are infringing upon their rights under copyright law and the copyright holders of the third-party code have the right to start legal action to stop the infringement and to seek damages.
Even without access to the source code or even a binary, similarities in behavior, especially for edge-cases, can be sufficient for a suspicion of a copyright violation. I am not a lawyer, but I can imagine that if a strong enough case of such suspicion can be made, that a judge could order you to give an independent auditor access to your code to assess if you actually make use of the code you are thought to infringe upon. And as copyright violations are civil law, the standard of proof isn't as high as for criminal law.
It's about distribution with most licenses.
If you build a web app based on software that is copyleft-ed, but if you don't distribute it, and only make it network-accessible, then you don't have to make available the source code to your app or any changes to that copyleft-ed software, unless the license in question is the AGPLv3.
AGPLv3 requires all linked software that is network accessible, to have their source codes made available to those accessing and using the app by a network interface.
This is based on section 13 of the AGPLv3:
Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph.