I'm not a lawyer, but let's talk about what the AGPL requires, and how it forms derivative works.
If your server uses the AGPL libraries in a way that creates a new work, then your entire work that uses the libraries is covered under the AGPL. Since, in that case, your server is a modified version of AGPL software, you must offer the source code for the entire collective work when you make the software accessible via a network.
Unfortunately for you, you don't have the source to offer. After you complete the initial physical transfer of the software, you can't compel the author to surrender the source code to you purely based on your position as a licensee of the software under the AGPL. The AGPLv3 is irrevocable, which means the licencor can't take away your right to use/modify/distribute the code, but the licencor isn't under a continuous long-term obligation to supply the source to you again if you lose it or declined to receive it in the first place.1
This is addressed, in part, in the GPL FAQ's advice to authors for making the source available:
How can I make sure each user who downloads the binaries also gets the source?
You don't have to make sure of this. As long as you make the source and binaries available so that the users can see what's available and take what they want, you have done what is required of you. It is up to the user whether to download the source.
Our requirements for redistributors are intended to make sure the users can get the source code, not to force users to download the source code even if they don't want it.
When you received the binaries, the licencor satisfied their responsibility to offer you the source by advertising the source code's availability in the Github repository. Unfortunately, you elected not to receive the source code (or did receive it and later lost it). Now, the author has chosen to stop offering the program, and you can't get it anymore. Since they no longer offer the binaries, they no longer have an obligation to offer you the source. (This is true as long as there is no written offer involved; see the footnote for an uncommon special case.)
Since you don't have a copy of the source from when it was available, you can't fulfill the requirements necessary to offer the combined AGPL work as a network service. That does not mean you must delete the AGPL binaries, but it does mean that you can no longer publicly run a network service using a program that includes AGPL components for which you cannot offer the source. You are free to continue doing things that do not require you to offer the source, such as modification, running locally for private use, etc.
This is likely your case. Any further consideration gets a bit murky; see below the bar.
Finally, offering decompiled source does not seem sufficient to satisfy the GPL's definition of "source code" as "preferred form of the work for making modifications to it." It's certainly better than the binary, but it's certainly worse than the actual code that the original author wrote.
1 As a special exception, this is not true in the rare case that the licencor gave you a formal written offer of source code under distribution option 6(b) or 6(c). In that case, the person who issued the offer of source code must honor it for at least three years from the date of issue. Since your case deals with a Github repository, it's vastly more likely that distribution was done under option 6(d), or else was not subject to the GPL at all because it was done by the original author, who is not bound by the license.
If your server's use of the AGPL libraries does not create a combined work, then your server and the plugin are separate works. In that case, you might possibly be in the clear but it's a subjective decision at what level of interaction two pieces of software combine into a single work. See the relevant GPL FAQ entry, but replace "non-free" with "GPL" and replace "GPL" with "AGPL":
Can I release a non-free program that's designed to load a GPL-covered plug-in?
It depends on how the program invokes its plug-ins. For instance, if the program uses only simple fork and exec to invoke and communicate with plug-ins, then the plug-ins are separate programs, so the license of the plug-in makes no requirements about the main program.
If the program dynamically links plug-ins, and they make function calls to each other and share data structures, we believe they form a single program, which must be treated as an extension of both the main program and the plug-ins. In order to use the GPL-covered plug-ins, the main program must be released under the GPL or a GPL-compatible free software license, and that the terms of the GPL must be followed when the main program is distributed for use with these plug-ins.
...
Since the AGPL would not apply to your entire server if it doesn't form a combined work with the AGPL library, your server does not need to share its source code. For example, this is exactly what happens with MongoDB, which is AGPL-licensed but positively asserts that a program which interacts with a MongoDB database does not form a combined work with the database, so the AGPL does not apply to any program that uses it.
