There is a lot of confusion regarding modification. You claim that "The lib's code is just used, not modified." That's a very narrow interpretation of modification. In court, integrating the library into your desktop application also counts as modification, hence you also need to open source the code of your desktop application.
As Amon indicates, your use of the AGPL corresponds with use under the GPL. The FAQ from the Free Software Foundation on gnu.org is clear:
I'd like to incorporate GPL-covered software in my proprietary system. Can I do this?
You cannot incorporate GPL-covered software in a proprietary system. The goal of the GPL is to grant everyone the freedom to copy, redistribute, understand, and modify a program. If you could incorporate GPL-covered software into a nonfree system, it would have the effect of making the GPL-covered software nonfree too. A system incorporating a GPL-covered program is an extended version of that program. The GPL says that any extended version of the program must be released under the GPL if it is released at all. This is for two reasons: to make sure that users who get the software get the freedom they should have, and to encourage people to give back improvements that they make.
However, in many cases you can distribute the GPL-covered software alongside your proprietary system. To do this validly, you must make sure that the free and nonfree programs communicate at arms length, that they are not combined in a way that would make them effectively a single program.
The difference between this and “incorporating” the GPL-covered software is partly a matter of substance and partly form. The substantive part is this: if the two programs are combined so that they become effectively two parts of one program, then you can't treat them as two separate programs. So the GPL has to cover the whole thing.
If the two programs remain well separated, like the compiler and the kernel, or like an editor and a shell, then you can treat them as two separate programs—but you have to do it properly. The issue is simply one of form: how you describe what you are doing. Why do we care about this? Because we want to make sure the users clearly understand the free status of the GPL-covered software in the collection.
If people were to distribute GPL-covered software calling it “part of” a system that users know is partly proprietary, users might be uncertain of their rights regarding the GPL-covered software. But if they know that what they have received is a free program plus another program, side by side, their rights will be clear.
In short, suppose that you have created an accounting app on the desktop, and that you need to create PDF invoices and PDF reports from this app. Suppose that you use an AGPL PDF library to create the PDF.
In this example, your own code defines the GUI, and it allows people to enter data that is stored in a database. When they click a button, PDF's are created using the PDF library.
That PDF library is an essential part of the accounting app because if you remove the PDF part, you remove an essential part of the application. You can't "work around" it by saying: we separate the GUI part from the PDF generation part. It's all connected.
The difference that is explained in the FAQ is for instance: when you compile your app with a specific compiler, then that compiler doesn't have to be open source because there's a clear separation between the application and the compiler. When you run the application on an operating system, then that operating system (or specific components of that operating system, such as the code that defines visual controls) doesn't have to be open source.
To make a long story short: your conclusion was wrong, because modification has a broader meaning than you initially assumed.