It's risky to use any computer. Zero-day exploits are found all the time, and we're all essentially defenceless against them until they're disclosed and fixed. But it's risky to cross a road, and yet we all do it; the question is always are the benefits worth the risk, and sadly, nobody can answer that for you, because nobody can tell you how much you'll value the benefit, or deplore the downsides if the risk bites.
So, binary firmware blobs: what benefit, and how risky? Much modern hardware, particularly that dealing with wireless signal transceiving and processing, tends to use closed-source firmware blobs that the drivers load upon hardware initialisation. It's a great shame, but it's a fact of modern life, and if you want to do WiFi, or 3+G wireless data, you will find that your choices of usable hardware are much reduced if you're not prepared to do it. So the benefit is being able to use most modern wireless hardware, as you've found out.
The downside is that you simply can't know what the firmware is making the hardware do. Since most modern hardware communicates via DMA, your firmware could be getting your hardware to exfiltrate chunks of your main memory under cover of normal operations. It probably isn't, but it's hard to know for sure. You rather have to trust the vendor.
You can still get things done if you're not prepared to run firmware blobs, but you'll be spending a lot of time vetting and sourcing hardware, and you won't be keeping up with the latest and greatest. Then again, if you're using non-free hardware - hardware where you don't have the schematics, the board layouts, the details of what's inside all the silicon packages, and the like - you're going to have to trust the vendor even if the firmware is completely open. Malicious hardware attacks are by no means unknown.
Should you use firmware blobs? Are you happy to do that? Only you can say.