13

This question is often asked from the licence perspective, but my issue is more about the practical aspects.

I have some code I would like to open source, but there is in the repository some "internal" code (gitlab CI yml configuration, Maven internal repo settings, SonarQube integration, etc.)

I would like to be able to maintain two versions of the repo, with some files pushed to my internal Gitlab instance, but not to the open-source repo.

I don't think this is possible, but it would work like having a different .gitignore file for the two remotes.

How is this typically done?

3
  • 2
    I have dealt with this in various forms over the years. I'll avoid another answer, but if requested, can provide more details. If it is only additional configurations, I can imaging Git Submodules may provide a solution here. If not, then a separate, local 'mirror' of the upstream which is periodically carefully merged into and pushed may work.
    – GCon
    Feb 17, 2023 at 12:25
  • Thinking about this question, and the answers already present, makes me wonder if it is time to think differently about it. The problem is trivially solved with overlay filesystems, which you get built-in to container systems (like Docker). And we do a lot of work with containers as a target to run the software we're building in. Maybe it's time to also use containers as a host to run your build system and IDE in? As a matter of daily practice. (And to the extent it isn't suitable yet for that: fix it so it is.) (I know some people do this, but I'm talking about ubiquity.)
    – davidbak
    Feb 18, 2023 at 16:18
  • Since you're using GitLab, maybe look how they are doing it.
    – AndreKR
    Feb 19, 2023 at 17:44

4 Answers 4

14

IMO that's a fairly clear-cut use case for submodules. Basically, you structure it like this:

internal:

Gitlab CI
Maven settings
...

Submodule public:
Source code
...

Glue code / directory info / symlinks to access public sources in the internal project

This way, you can work on the public code while compiling it according to the internal configuration.

  • After making internal-only changes, you just commit them in the internal repo.
  • After making public-only changes, you commit them in the public repo and then also make a simple commit in internal that does nothing but update the submodule to the new public HEAD.
  • After making changes that involve both parts of the code, you commit everything public in the public repo and everything internal in the internal repo along with the change of the submodule.
24

There isn't really a satisfactory solution.

Git was designed for a distributed group of people to work on Open Source software, so it doesn't really feature things like access controls. In particular, branches just describe different strands of development history, and aren't useful for describing different editions of the same software.

One approach is to work on an internal repository, then regularly export snapshots to a separate, public repository. During this export step, you could ignore some files. But the two repositories must not share their Git history. Note that this approach will complicate dealing with outside contributions.

It's also possible to to the inverse: work mostly on the public repository, but occasionally merge it into the internal repository. In principle, the internal view could just be another branch that you don't push to be a public repository.

For a lot of projects that have some sensitive parts, it is more appropriate to define extension points in the Open Source version, and to implement the internal features as plugins. These additions might either be inserted at runtime, or during your internal build process.

Your examples relate to more to tooling around the software, not to the software itself.

  • Consider whether they should be part of the repository in the first place. Secrets should be never committed, not even for purely internal repos. Instead, CI environments offer tools to manage secrets such as API keys.

  • Consider whether you actually have to remove this configuration from a public version. For example, if you develop the project in an Open Source first manner, then you will likely want to run CI jobs on that version as well. Even if not, it can be OK to include files that don't have value outside of your organization (if they don't disclose anything sensitive).

  • Quite often, such issues can be avoided by keeping environment-specific details out of the repository. For example, instead of hardcoding a Maven URL, you might require developers to set an appropriate environment variable. Instead of having a complex CI pipeline, you might have a simple pipeline definition that just calls into an environment-independent build script that contains the actual steps.

I've linked to the 12 Factor App here, because it is a great (albeit dated) condensation of best practices mostly for web apps, SaaS, and cloud-native software. Many of those lessons are also applicable in an Open Source context, and also for different kinds of software.

3

I suggest to really look at the "exceptional files":

  • gitlab CI yml configuration: Can be in a different git repository. You can configure gitlab to read it from there
  • Maven internal repo settings: Should be in settings.xml, outside your git repo
  • Other things: Look at the docs if you can separate it out, or consider using environment variables (as amon already suggested)

Probably, not much really sensitive or bothering is left after thoroughly inspecting the relevant files.

3
  • How do your other internal team members get the maven settings?
    – mmmmmm
    Feb 18, 2023 at 13:17
  • 1
    @mmmmmm I have a custom Docker Maven container for the build that contains the settings.xml. This is built from a Docker file in a separate Git repository. Feb 18, 2023 at 14:40
  • 1
    You should include that in your answer otherwise you just give part of the solution
    – mmmmmm
    Feb 18, 2023 at 14:45
2

Google's tool for this is Copybara. It's used within Google for exactly the cases you suggest -- as the readme says, "A common case is a project that involves maintaining a confidential repository and a public repository in sync." It's also handy for periodically importing changes from the public repository to the private one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.