I don't think your question admits of an answer better than "you are right, it's not a great idea".
But it may help to know you're not the only person who thinks this way. I heard Georg Greve, ex-president of the FSFE, talk about the hardware-trust problem, and possible solutions from the Open Power Foundation, at FOSDEM 2017.
We need hardware to run our software on. If we want control over our software, we had better have trust in that underlying hardware. There are two routes to that trust; one is faith-based, and the other is through verifiability. At the moment, the CPUs at the heart of the desktop and server equipment that most of us run our free software on are generally made by Intel, and its track record for both justifiable good faith and verifiable openness is not all that good
The talk writeup was published in LWN (from which the above quote comes) but in retrospect I've been impressed at how prescient Greve was. He drew attention to the problem of Intel's Management Engine, the mysterious CPU-behind-the-CPU, and only two months later I was reporting on a zero-day bug that bypassed all access controls in the ME's Active Management Technology.
So yes, it's a problem, and at the end of the day if you don't run your own CPU fabrication plant you will have to trust someone. That said, you can at least try to avoid giving trust to people with a demonstrable record of not deserving it. You can diversify your risk portfolio by building systems from components instead of ordering closed-box solutions from single vendors; any individual component may still be malicious, but its malice may be contained by the lack of assistance from its neighbour components.
My personal colocated server has two hardware RNGs attached to it, to supply strong entropy; one of them comes from a vendor who openly and clearly addresses the issues of checking that your hardware and associated firmware is as they shipped it, and as you would wish it. When you find vendors who want you to be able to trust them, who give you clear pathways to do so, reward them by buying their stuff. Tell them why you like it.
So there are partial solutions; they will require you to do more research and spend more time and money, but then, freedom was never free.