I am aware that the licenses of e.g. code on Github are currently not observed when training AIs and that there are licenses such as AGPL that require the code to be disclosed. But is there already a license that explicitly prevents use (for training) by an AI, and if not, should one strive for something like that? Would that make sense?
The question here remains "is the AI model a derivative work of the training inputs?"
If it is, then all the AI companies are already in violation of copyright and you don't need a specific license for this.
If it's not - i.e. it is fair use - then a license saying "you may not use this to train an AI model" is irrelevant.
As of April 2023, the answer to the question is very much unknown; it will need to be decided on by courts, who may make different decisions in different jurisdictions.
If the output of an AI is a derivative work of its training input, then compliance with most existing licenses is likely to be difficult or impractical, because of the need for attribution, but this only affects distribution of the output and/or the model (it's currently unclear whether a problem with the output would also affect the model itself). It does not prevent you from creating a model for your own private use. Furthermore, a future AI might be "smart" enough to be able to attribute things correctly, and such an AI would be fully compliant with the relevant licenses. If a license wants to prohibit the use of private models and/or fully compliant models, then it would have to prohibit the training process itself (see next paragraph).
If the output of an AI is not a derivative work of its training input, then such a limitation would require the license to prohibit training, which is not compatible with our usual understanding of Free and Open Source Software. It violates freedom #1 of the Free Software Foundation's four freedoms ("The freedom to study how the program works, and change it so it does your computing as you wish") and OSD/DFSG requirement #3 ("The license must allow modifications and derived works..."), and possibly also OSD #10 ("No provision of the license may be predicated on any individual technology or style of interface"). This is not because the AI is "learning" anything, but rather because the AI could serve as a tool for a human to modify the code. By prohibiting AI, you are prohibiting that human from modifying that code with that tool, which is simply Not Done in FOSS licensing (imagine, for example, a license prohibiting the use of a particular IDE, compiler, or version control system).
We currently do not know whether the output of an AI is a derivative work of its input, so we don't know which of those two paragraphs applies.
There is this repo, which modifies popular open-source licenses to explicitly forbid the use of the source code in AI-training datasets:
Distribution of the model is a legal question on its own. It's not obviously computer code, or a work of literature, or any of the other things that can be copyrighted. It might be similar to perfumes, something that by omission simply is not covered by Intellectual Property laws. And if the model is not a work in the sense of copyright law, then the output cannot be a derivative work either.
You might try Copy Far "AI" (full disclosure: I'm the current maintainer).