This is a messy situation created by software patents.
GPLv3 has anti-patent clauses; it explicitly forbids the licensor from using software patents to sue or limit the freedoms of licensees. This is an important difference between GPLv3 and GPLv2, the latter which does not mention patents.
Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free.
However, libfdk_aac is under a license that does not grant a patent license. Users are required to obtain a patent license separately, if they haven't done so already.
- NO PATENT LICENSE
NO EXPRESS OR IMPLIED LICENSES TO ANY PATENT CLAIMS, including without limitation the patents of Fraunhofer,
ARE GRANTED BY THIS SOFTWARE LICENSE. Fraunhofer provides no warranty of patent non-infringement with
respect to this software.
You may use this FDK AAC Codec software or modifications thereto only for purposes that are authorized
by appropriate patent licenses.
This is why the two licenses are incompatible. Media formats are often patent-encumbered, which is a problem that affects projects like FFmpeg, a big collection of incompatible codecs. This is why FFmpeg themselves generally refrain from distributing binaries at all, and give you options to compile a legal subset of their software for your own use (e.g.
How can Handbrake/FFmpeg HI distribute binaries?
Good question. While some projects like FFmpeg HI probably operate in a legal grey area, there are situations where software patents are not an issue,
which probably describes what Handbrake is doing1:
- The applicability of software patents globally is very uneven. Unlike copyright which is virtually consistent across the whole world, software patents either aren't recognised or are very limited in many places, such as the EU. Handbrake is hosted in France, so if you are also located somewhere that doesn't recognise software patents and download Handbrake, neither you nor Handbrake are affected by them.
- You probably already have a patent license for those codecs, because they are subsidised by your OS and/or hardware vendor, like Microsoft, Apple or Google (Android). Part of the money you paid for your devices went to patent holders. This also explains why many free software projects have historically suffered from poor codec choice; because these projects (e.g. FFmpeg, Linux, Firefox) are freely distributed, they cannot guarantee that their users have the required patent licenses, whereas someone distributing, say, OS X software can assume their users have licenses for the HE-AAC format via Apple's CoreAudio.
1: Actually, Handbrake were unaware of the licensing incompatibility, and have removed libfdk_aac. Thanks for the update.