There is a lot of confusion regarding modification. You claim that "The lib's code is just used, not modified." That's a very narrow interpretation of modification. In court, integrating the library into your desktop application also counts as modification, hence you also need to open source the code of your desktop application.
As Amon indicates, your use of the AGPL corresponds with use under the GPL. The FAQ from the Free Software Foundation on gnu.org is clear:
I'd like to incorporate GPL-covered software in my proprietary system. Can I do this?
You cannot incorporate GPL-covered software in a proprietary system. The goal of the GPL is to grant everyone the freedom to copy, redistribute, understand, and modify a program. If you could incorporate GPL-covered software into a nonfree system, it would have the effect of making the GPL-covered software nonfree too. A system incorporating a GPL-covered program is an extended version of that program. The GPL says that any extended version of the program must be released under the GPL if it is released at all. This is for two reasons: to make sure that users who get the software get the freedom they should have, and to encourage people to give back improvements that they make.
However, in many cases you can distribute the GPL-covered software alongside your proprietary system. To do this validly, you must make sure that the free and nonfree programs communicate at arms length, that they are not combined in a way that would make them effectively a single program.
The difference between this and “incorporating” the GPL-covered software is partly a matter of substance and partly form. The substantive part is this: if the two programs are combined so that they become effectively two parts of one program, then you can't treat them as two separate programs. So the GPL has to cover the whole thing.
If the two programs remain well separated, like the compiler and the kernel, or like an editor and a shell, then you can treat them as two separate programs—but you have to do it properly. The issue is simply one of form: how you describe what you are doing. Why do we care about this? Because we want to make sure the users clearly understand the free status of the GPL-covered software in the collection.
If people were to distribute GPL-covered software calling it “part of” a system that users know is partly proprietary, users might be uncertain of their rights regarding the GPL-covered software. But if they know that what they have received is a free program plus another program, side by side, their rights will be clear.
In short, suppose that you have created an accounting app on the desktop, and that you need to create PDF invoices and PDF reports from this app. Suppose that you use an AGPL PDF library to create the PDF.
In this example, your own code defines the GUI, and it allows people to enter data that is stored in a database. When they click a button, PDF's are created using the PDF library.
That PDF library is an essential part of the accounting app because if you remove the PDF part, you remove an essential part of the application. You can't "work around" it by saying: we separate the GUI part from the PDF generation part. It's all connected.
The difference that is explained in the FAQ is for instance: when you compile your app with a specific compiler, then that compiler doesn't have to be open source because there's a clear separation between the application and the compiler. When you run the application on an operating system, then that operating system (or specific components of that operating system, such as the code that defines visual controls) doesn't have to be open source.
To make a long story short: your conclusion was wrong, because modification has a broader meaning than you initially assumed.
Update
This update is rather off-topic, but I'm adding it to stop the flow of comments by 3D1T0R who, for some unknown reason, doubts my expertise in these matters.
The example of the accounting software and the PDF invoices wasn't chosen at random. I'm the original developer of iText, an open source PDF library released as AGPL software.
In June 2015, I discovered that iText was used in a closed source module that served as an add-on to closed source accounting software. This is the first thing I did: I went to a bailiff to document the infringement (June 12, 2015):
Then I went to a lawyer, who sent a cease and desist letter:
The infringing party stalled, and came back with false arguments (similar to the arguments made by 3D1T0R in the comments).
Hence we went to court (July 8, 2015):
Our case was very clear, and it didn't take long for the court to issue a ruling (July 17, 2015):
Eventually, the infringing party accepted the ruling of the court:
These slides were taken from my JavaOne talk IANAL: What Developers should know about IP and Legal (I won a JavaOne Rockstar Award with this talk).
In his comments, 3D1T0R claims that this isn't an illustrative example as creating invoices isn't an essential part of accounting software. Landsgericht Köln didn't agree with that point of view. All software using the infringing module was taken from the market because the infringing party didn't want to open source it. Notice the word v e r b o t e n in the rulling. It was forbidden to distribute software that uses iText as a whole or in part without permission of the copyright owners of iText, given the fact that the copyleft of the AGPL wasn't respected.
Actually, "being an essential part" isn't a criterium. In Artifex versus Hancom, the South-Korean company Hancom used Artifex's Ghostscript to create PDF output from documents created with their office suite. Creating PDF was an optional feature that wasn't essential for the rest of the office functionality to work. However, Artifex went to court, and a Californian judge issued a ruling that the GPL could indeed be enforced even in cases where the GPL code is only used for a small feature that isn't essential in the context of the complete work.
In his comments, 3D1T0R claims that there are workarounds, but that's wishful thinking. A judge usually takes into account the technical effect rather than the technical implementation.
If user A pushes a button in product X, and by doing a technical effect is triggered that results in an action in (A)GPL library Y, then product X needs to be released under the same license as library Y to user A no matter how product X is technically implemented!
If you don't understand the difference between technical effect and technical implementation, consider this: suppose that person A wants to murder person X, but he hires person B to kill person X, then person A isn't technically the murderer of person X, but the effect is the same: person X is killed because of person A, and for a judge person A will be guilty all the same (and usually get a higher sentence than person B).
Use this comparison to a judge in court to explain why he should discard the technical details of a workaround, and you'll win your case. Actually, the infringing party might get a more severe sentence because of the intentional attempt to avoid compliance with the open source license.
If you do not agree with the above, please document with existing cases that went to court. Please be aware of the fact that allegations that have not been proven in court have no value whatsoever.
DISCLAIMER:
Regarding the first example: I am the original developer of iText, an AGPL PDF library, and I wrote most of the Affidavits that were used in the case documented with the screen shots above.
Regarding the second example: I was consulted by the CEO of Artifex for advice in December 2016, and by Hancom in the Summer of 2017. Eventually, the case was settled.
I have quite some experience in both cases, but unfortunately, I can't disclose more information than what is disclosed in this update. However, I think this is already more information than what is provided by 3D1T0R to support his (or her) claims.
It's quite disappointing to see how many developers are interpreting the law in a way that best suits the open source user, going against the rights of the open source producer.