Commercial projects often do this when the assets are the valuable parts. Games are a good example: often what makes a game worth purchasing is not the engine, but the level design, the graphics, etc. Another category of examples is having an open source reader for closed content; in this case the content may be distributed in a standard format for which open source readers are already available, such as PDF or EPUB.
The benefit of not making the asset open source is of course to make money by charging per user.
The advantage of making the engine open source is that you can get others to do maintenance for you. A bug occurs? Some of your tech-savvy users will send a debug trace or even a patch, so it's less work for you to fix it. A new platform comes out? Someone may do the job of porting the engine to that platform. Your engine might be picked up by some free software distributions: free advertising.
Usually some assets are available to everyone, e.g. demo levels. But in any case the people who will be most motivated to work on the open source engine are the ones who have purchased the assets. People can test their modifications on the demo levels or on the commercial levels.