How do you maintain an open-source and an internal version of the same code?

Question

This question is often asked from the licence perspective, but my issue is more about the practical aspects.

I have some code I would like to open source, but there is in the repository some "internal" code (gitlab CI yml configuration, Maven internal repo settings, SonarQube integration, etc.)

I would like to be able to maintain two versions of the repo, with some files pushed to my internal Gitlab instance, but not to the open-source repo.

I don't think this is possible, but it would work like having a different .gitignore file for the two remotes.

How is this typically done?

Answer 1

IMO that's a fairly clear-cut use case for submodules. Basically, you structure it like this:

internal:

Gitlab CI
Maven settings
...

Submodule public:

Source code
...

Glue code / directory info / symlinks to access public sources in the internal project

This way, you can work on the public code while compiling it according to the internal configuration.

• After making internal-only changes, you just commit them in the internal repo.
• After making public-only changes, you commit them in the public repo and then also make a simple commit in internal that does nothing but update the submodule to the new public HEAD.
• After making changes that involve both parts of the code, you commit everything public in the public repo and everything internal in the internal repo along with the change of the submodule.

Answer 2

There isn't really a satisfactory solution.

Git was designed for a distributed group of people to work on Open Source software, so it doesn't really feature things like access controls. In particular, branches just describe different strands of development history, and aren't useful for describing different editions of the same software.

One approach is to work on an internal repository, then regularly export snapshots to a separate, public repository. During this export step, you could ignore some files. But the two repositories must not share their Git history. Note that this approach will complicate dealing with outside contributions.

It's also possible to to the inverse: work mostly on the public repository, but occasionally merge it into the internal repository. In principle, the internal view could just be another branch that you don't push to be a public repository.

For a lot of projects that have some sensitive parts, it is more appropriate to define extension points in the Open Source version, and to implement the internal features as plugins. These additions might either be inserted at runtime, or during your internal build process.

Your examples relate to more to tooling around the software, not to the software itself.

• Consider whether they should be part of the repository in the first place. Secrets should be never committed, not even for purely internal repos. Instead, CI environments offer tools to manage secrets such as API keys.

• Consider whether you actually have to remove this configuration from a public version. For example, if you develop the project in an Open Source first manner, then you will likely want to run CI jobs on that version as well. Even if not, it can be OK to include files that don't have value outside of your organization (if they don't disclose anything sensitive).

• Quite often, such issues can be avoided by keeping environment-specific details out of the repository. For example, instead of hardcoding a Maven URL, you might require developers to set an appropriate environment variable. Instead of having a complex CI pipeline, you might have a simple pipeline definition that just calls into an environment-independent build script that contains the actual steps.

I've linked to the 12 Factor App here, because it is a great (albeit dated) condensation of best practices mostly for web apps, SaaS, and cloud-native software. Many of those lessons are also applicable in an Open Source context, and also for different kinds of software.

Answer 3

I suggest to really look at the "exceptional files":

• gitlab CI yml configuration: Can be in a different git repository. You can configure gitlab to read it from there
• Maven internal repo settings: Should be in settings.xml, outside your git repo
• Other things: Look at the docs if you can separate it out, or consider using environment variables (as amon already suggested)

Probably, not much really sensitive or bothering is left after thoroughly inspecting the relevant files.

Answer 4

Google's tool for this is Copybara. It's used within Google for exactly the cases you suggest -- as the readme says, "A common case is a project that involves maintaining a confidential repository and a public repository in sync." It's also handy for periodically importing changes from the public repository to the private one.