Suppose I have a program that uses a library released under the AGPL license. Can the program be released as GPL, or does it have to be released as AGPL too ?
Yes, you may perform this combination, and the works technically retain their respective GPL/AGPL licenses. However, your concern that the combination may "have to be released as AGPL too" has some truth to it: the AGPL requirement to share source over a network does indeed apply to the combination.
This is explicitly handled in section 13 of the GPLv3 (and correspondingly in section 13 of the AGPLv3), which articulates the interaction between GPLv3- and AGPLv3-licensed code when combined:
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such.
The last sentence here means that the most significant requirement of the AGPL -- disclosing source to users over a network -- applies to a combination of GPL and AGPL work.
It is technically true, though, that the AGPLv3 copyleft requirements do not extend into the GPLv3-licensed material (nor vice versa), so the two licenses are compatible. The GPL and AGPL components may be freely intermixed and each retains its original AGPL/GPL licensing status.
In https://www.gnu.org/licenses/agpl-3.0.txt, section 5:
The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices".
So, no GPL linking to an AGPL library.