There seems to be consensus that linking to a GPL licensed library constitutes a derivative work and triggers the GPL requirement to license the code that links to the GPLd code under the terms of the GPL as well.
Weird, because that's obviously wrong. Linking to a library is how you use it in the ordinary way. Just like you don't need a license to read a book, you don't need a license to use a library.
This is not a controversial point and it's very likely your comment is based on a misunderstanding. Here's how GNU puts it:
"Merely agreeing to the GPL doesn't place any obligations on you. You are not required to agree to anything to merely use software which is licensed under the GPL. You only have obligations if you modify or distribute the software." GPL FAQ
There also seems to be consensus that merely calling a GPL licensed standalone executable (like GPLd FFMPEG) does not constitute a derivative work and does not trigger the requirement to license the calling code under GPL.
Right, again it's ordinary use. Under US law, the right to the ordinary use of a work is a right every lawful possessor of a work has unless they give it up through some agreement they actually agreed to.
But: Does distributing GPLd software together with closed source software also trigger GPL obligations in respect of my program's source code, even without linking?
Yes and no.
Distribution requires a license. If you could copy and distribute works without a license, the music and movie industries would be very different. So you do have whatever obligations the GPL imposes on you.
However, the GPL defines a "covered work" as, effectively, a derivative work. So the question is whether the closed source software is a derivative work or not. But if the closed source software was a derivative work, you'd have GPL obligations even if you didn't distribute it together with the closed source software.
In sum, distributing an aggregate work doesn't change your GPL obligations over distributing the pieces of the work separately because the GPL only extends to derivative works and whether a work is a derivative work isn't changed by aggregate distribution.