Maybe possible, but could be tricky.
As the only copyright holder to the GPL-covered components of the software, you are free to add exceptions and additional terms to the GPLv3, as described in section 7 of that license. In fact, the LGPLv3 is just such a GPLv3 section 7 additional permission, allowing the component to be linked to proprietary code.
Writing your own additional permissions is tricky, just like writing your own license is tricky. If in any way possible, strongly consider whether the existing LGPLv3 or other widely accepted GPL exceptions would be sufficient for your purposes.
But yes, in principle you could write your own exception that only grants permissions to link with modules that use a specific plugin API. It is tricky to define that well, since it would be permissible (under the GPL) to change that plugin API, possibly widening it more than expected. It could be necessary to tie the license exception to versions published by you, but that would raise the question whether the resulting license is Open Source in practice (while it would certainly be GPL-compatible, the power asymmetry introduced by the exception could be argued to violate the Open Source definition).
There is prior art for a GPL-covered program that loads non-GPL extensions, but only permits access to specific APIs: the Linux Kernel. All kernel modules must declare a license. If the module declares a GPL-compatible license, the module has full access to the kernel's APIs. For other licenses, only access to a restricted subset of APIs is granted. The enforcement here is primarily legal, but the kernel also uses linker tricks to enforce proper licensing on a technical level. The kernel also does not expect eBPF programs to have a GPL-compatible license, since these are essentially user-space programs executed in a virtual machine.
In practice, it is probably best to side-step the in-process plugin question entirely. For example: