I want to build an Android ROM for myself and the ROM i picked has an "official maintainer". On the website it states that if a device has an "official maintainer", it is forbidden to share an "unofficial build". If someone builds and shares it, they will send a dmca request. is that possible with open source software?
Not according to the definition of open source we use here, which is the OSI's. That requires that
The license shall not restrict any party from selling or giving away the software
The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software
So if I can't modify the program to make my own "unofficial build", and then share it, we don't consider the software to be open source. That said, the term "open source" isn't a restricted one, so anyone can distribute their software under any old hideous proprietary licence and yet claim it's "open source". They can even start DMCA takedowns for violation of their proprietary licence, and still claim with a straight face that their software is open source. To my mind, they're wrong, and their claim is a lie, but they're not forbidden from making it.
However, the idea that you can restrict the use of a trademarked name is much more normal, and acceptable within the free software community. So a site that said, eg, "we distribute the official build of FooBarOSTM for the Google BazBar Developer Phone; if you distribute your own build for this phone, you must not call it FooBarOS" is fine.