Lots of software is focused on solving a particular problem (e.g., performing calculations, handling point of sale transactions, visually rendering and editing pictures/graphs). In those cases, the software is a means to an end: the end user has a need, and they use the software to fill that need. In such cases, there is often an obvious path to monetization: pay to add new features to fill unmet needs, pay to get better guides on how to use the software to meet users' needs.
In a video game context, using the software is the end goal. There's no deeper objective external to the software, other than to be entertained. This cuts out a lot of monetization options. Furthermore, it's often inherently fun to create modifications to a game (versus making modification to, say, a point-of-sale program or a spreadsheet program), so you'll have a lot of hobby-based competition for implementing features. You could still possibly do it, if your for-pay modifications have a reputation for being head-and-shoulders above your hobbyist competitors, but it's still difficult to convince people to shell out for your mod when there's an extant gratis alternative.
The main options I can see are pre-release crowdfunding and non-free game content.
If you can create sufficient pre-release hype for your game, either by attractive marketing or by a proven history of previous good games, then you can get people to pay for your game before it comes out. You can gather funds from a crowdfunding website to pay for development of the game. In this case, you don't get any residual income after the game is released. People pay for the game to exist and be released into the wild; after that happens, you lose monopoly leverage over the game to monetize it.
Another approach is to release the source code of the game's engine under a free license, but keep the content under a non-free license. When you sell the game, you're really selling the content for the levels, since the open-source executable game is probably available elsewhere for free. It's a bit of a cheat to call this an "open source business model", since you entire game isn't really open source: you've merely divided it into free and non-free components and charge for the non-free part. Still, this gets you many of the benefits of open source: notably, others can fix bugs in your engine and you can pull back downstream improvements (provided downstream changes make their source code available). As an example, several games sold through the Humble Bundle have employed this model.
As a less viable third option, if you're really big, you might be able to get away with leveraging your trademark and selling merchandise. As a far-fetched example, if Valve chose to release the next Half Life game under an open source license, they could still sell a lot of Half Life branded t-shirts, etc. As a less far-fetched example, if the developers for the open-source first-person shooter Tremulous chose to release a limited-edition run of branded stuffed animals modeled after the game's aliens, I might buy one.
