How do you work with open core code efficiently in Git?

Question

MPL 2.0 license allows for using open source code in part of a 'larger work' (the closed source), where the 'larger work' is contained in separate files.

What is an efficient way to maintain an open core project in Git, where closed code is only contained in separate files?

Ideally, the solution also satisfies these needs:

• contribution to open source code can be done in the open
• close source code files do not end up in the open source repository, but still have revision control
• working on the entire project (open + closed source code)

Answer 1

Most version control systems (like git) don't allow to split the repository into "visible" and "invisible" parts. What you can do with git is to have the public part in one repository, publicly accessible, and the private part in another one. The second, closed, one can then include the first, public, one as a submodule (see git-submodule(1)). The submodule has to be a subdirectory, which shouldn't be a restriction, and probably helps with wider use. The parent (user package) can request a specific version of the (independently developed) submodule, so it isn't at the mercy of random changes in that one either.

Answer 2

A common solution I've seen is to design the core for extensibility and put the closed-source elements into extensions. Since the closed-source portions of the product interact only through the extension interface, the source code for them doesn't need to be stored anywhere near the core's source code.

For example, an open-core web application might come with a basic database-based authentication system, while the paid version adds the ability to authenticate against LDAP or Active Directory. The basic system would be part of the public Git repository, while the code for the paid extensions would be stored in a private repository.

Answer 3

This situation is common at large software companies which contribute to open source projects, so there are some tools you can find specifically to migrate between open and closed repositories. I have used these tools. They generally work like this:

• You have your internal repository. This could be a monorepo containing an open-source subproject, or it could be just an internal fork with some confidential parts to it.

• You have your external repository. This could be a public repository on GitHub, containing the open-source contributions.

• You use software to migrate commits from the internal repository to the external one, and vice versa. Metadata may get added to these migrated commits so the tools can recognize the original commit from a migrated commit. For example, Copybara will add GitOrigin-RevId: <hash> to the bottom of a commit message.

I’m aware of two tools that do this.

• Copybara: Written by Google.

• fbshipit: Written by Facebook.

These systems allow you a lot of flexibility in pushing internal changes to public repositories, and accepting public contributions and integrating them into your internal repositories. They are, perhaps, flexible to a fault—you may need to spend some time experimenting with them and trying different configurations.

Also note that these tools are not super well-supported tools with discussion forums. For example, you can’t download a pre-built binary for Copybara, you have to get Bazel to compile it. And you can’t just run fbshipit, you need Hack (Facebook’s PHP derivative, which doesn’t run on all platforms). There is no “Copybara” tag on Stack Overflow.

However, these tools do work and I use them.