Let's say I am installing a distro, which comes with two pieces of firmware: free firmware for chip A, and nonfree firmware for chip B. I install it on a machine that has chip A and chip B, but I choose to only activate chip A. If my system still has the nonfree firmware for chip B, but I never use it, does my system still count as being all free software?
Freedom is not a property that inheres in software; it's a set of rights that attach to software through the act of conveyance. Even less does it inhere in an installed system, powered-down, on a table. So it's not very meaningful to speak of your system as being free, or non-free, in isolation.
If you have the rights and capability to exercise all four freedoms with respect to all the software that's installed on your system that you use and care about, I'd say it's free. If there are a couple of additional files full of random numbers on the disc, no harm is done, even if they later turn out to be firmware instructions for some piece of hardware you're not using. Delete those files if you feel bad, or don't.
In my opinion, as long as you can fully exercise the four freedoms with your system, it's free.