There is a lot of confusion regarding modification. You claim that "The lib's code is just used, not modified." That's a very narrow interpretation of modification. In court, integrating the library into your desktop application also counts as modification, hence you also need to open source the code of your desktop application.
As Amon indicates, your use of the AGPL corresponds with use under the GPL. The FAQ from the Free Software Foundation on gnu.org is clear:
I'd like to incorporate GPL-covered software in my proprietary system. Can I do this?
You cannot incorporate GPL-covered software in a proprietary system. The goal of the GPL is to grant everyone the freedom to copy, redistribute, understand, and modify a program. If you could incorporate GPL-covered software into a nonfree system, it would have the effect of making the GPL-covered software nonfree too. A system incorporating a GPL-covered program is an extended version of that program. The GPL says that any extended version of the program must be released under the GPL if it is released at all. This is for two reasons: to make sure that users who get the software get the freedom they should have, and to encourage people to give back improvements that they make.
However, in many cases you can distribute the GPL-covered software alongside your proprietary system. To do this validly, you must make sure that the free and nonfree programs communicate at arms length, that they are not combined in a way that would make them effectively a single program.
The difference between this and “incorporating” the GPL-covered software is partly a matter of substance and partly form. The substantive part is this: if the two programs are combined so that they become effectively two parts of one program, then you can't treat them as two separate programs. So the GPL has to cover the whole thing.
If the two programs remain well separated, like the compiler and the kernel, or like an editor and a shell, then you can treat them as two separate programs—but you have to do it properly. The issue is simply one of form: how you describe what you are doing. Why do we care about this? Because we want to make sure the users clearly understand the free status of the GPL-covered software in the collection.
If people were to distribute GPL-covered software calling it “part of” a system that users know is partly proprietary, users might be uncertain of their rights regarding the GPL-covered software. But if they know that what they have received is a free program plus another program, side by side, their rights will be clear.
In short, suppose that you have created an accounting app on the desktop, and that you need to create PDF invoices and PDF reports from this app. Suppose that you use an AGPL PDF library to create the PDF.
In this example, your own code defines the GUI, and it allows people to enter data that is stored in a database. When they click a button, PDF's are created using the PDF library.
That PDF library is considered being a part of the accounting app because the functionality of the library is triggered by an action of the end user in the accounting application (e.g. a PDF invoice is created when the end user clicks a button). You can't "work around" the copyleft license by saying: we separate the GUI part from the PDF generation part. It's all connected.
The difference that is explained in the FAQ is for instance: when you compile your app with a specific compiler, then that compiler doesn't have to be open source because there's a clear separation between the application and the compiler. When you run the application on an operating system, then that operating system (or specific components of that operating system, such as the code that defines visual controls) doesn't have to be open source.
To make a long story short: your conclusion was wrong, because modification has a broader meaning than you initially assumed.
Update in answer to comments asking for actual proof of my claims:
Note: most of the comments I refer to were removed by their author.
The example of the accounting software and the PDF invoices wasn't chosen at random. I'm the original developer of iText, an open source PDF library released as AGPL software.
In June 2015, I discovered that iText was used in a closed source module that served as an add-on to closed source accounting software. This is the first thing I did: I went to a bailiff to document the infringement (June 12, 2015):
Then I went to a lawyer, who sent a cease and desist letter:
The infringing party stalled, and came back with false arguments (similar to some arguments made in the comments).
Hence we went to court (July 8, 2015):
Our case was very clear, and it didn't take long for the court to issue a ruling (July 17, 2015):
Eventually, the infringing party accepted the ruling of the court:
These slides were taken from my JavaOne talk IANAL: What Developers should know about IP and Legal (I won a JavaOne Rockstar Award with this talk).
In the comments, I read claims that this isn't an illustrative example because creating invoices isn't an essential part of accounting software. Landsgericht Köln didn't agree with that point of view. All software using the infringing module was taken from the market because the infringing party didn't want to open source it. Notice the word v e r b o t e n in the rulling. It was forbidden to distribute software that uses iText as a whole or in part without permission of the copyright owners of iText, given the fact that the copyleft of the AGPL wasn't respected.
Actually, being an essential part isn't even a criterium. In Artifex versus Hancom, the South-Korean company Hancom used Artifex's Ghostscript to create PDF output from documents created with their office suite. Creating PDF was an optional feature that wasn't essential for the rest of the office functionality to work. However, Artifex went to court, and a Californian judge issued a ruling that the GPL could indeed be enforced even in cases where the GPL code is only used for a small feature that isn't essential in the context of the complete work.
Often you read about thought experiments that tend to work around the (A)GPL, but such thought experiments are usually based on wishful thinking. In my (real-world) experience, a judge takes into account the technical effect rather than the technical implementation.
If user A pushes a button in product X, and by doing a technical effect is triggered that results in an action in (A)GPL library Y, then product X needs to be released under the same license as library Y to user A no matter how product X is technically implemented!
If you don't understand the difference between technical effect and technical implementation, consider this: suppose that person A wants to murder person X, but he hires person B to kill person X, then person A isn't technically the murderer of person X, but the effect is the same: person X is murdered because of person A, and for a judge person A will be guilty all the same (and usually get a higher sentence than person B).
If a counter-party tells the judge that the AGPL doesn't apply when you access an AGPL library from a closed source system through a service that is open sourced under the AGPL, feel free to use the "murder" comparison in court to explain why the judge should discard the technical details of such a workaround. You'll increase your chances at winning your case. Actually, the infringing party might get a more severe sentence because of the intentional attempt to avoid compliance with the open source license.
If you do not agree with the above, please document with existing cases that went to court. Please be aware of the fact that allegations that have not been proven in court have no value whatsoever.
DISCLAIMER:
Regarding the first example: I am the original developer of iText, an AGPL PDF library, and I wrote most of the Affidavits that were used in the case documented with the screen shots above.
Regarding the second example: I was consulted by the CEO of Artifex for advice in December 2016, and by Hancom in the Summer of 2017. Eventually, the case was settled.
I have quite some firsthand experience in matters like these, and it's quite disappointing to see how many developers are interpreting the law in a way that best suits the open source user, going against the rights of the open source producer. I would expect developers to stand up for the rights for their fellow open source developers. In my experience however, decision makers at a company have a better understanding than the company's developers. Management usually makes much less of a problem of the dual license system that allows their company to use software under a commercial license in cases where the company can't comply with the copyleft license.