If I use a GPLed image to illustrate a blog post, does that make the text of the blog post part a derivative work, thus "infecting" it with the GPL?
First, GPL does not "infect" anything. This is not a virus. That important point being cleared, I would consider reusing an image in a blog post as side-by-side redistribution. Only the image has GPL obligations.
As a side note, the GPL is a poor choice for media in general as this was devised for code more than for data. I therefore may have some challenges to meet the GPL obligations for an image. I would consider a comment with credits to the author and a link to the GPL to be good enough for me and the author in most cases.
Or do only changes made to the image itself count as a derivative work?
Yes, in light of my comment above.
Or, I suppose the most likely answer is: It depends. So, on what does it depend?
I cannot fathom how it would depend on anything else assuming my blog post is HTML'ish.