3

I'm currently involved in a project to produce a tool for carrying out certain types of standardised performance assesments on a specific type of industrial plant, that will soon be going open source. We have a potential problem because certain features of our software need real data, or very realistic synthetic data, to test against (i.e. the integration tests that validate that the results produced for a given plant configuration are correct given the assesment standard we are applying). We have access to real data from a whole range of different plants, but we only have permission to publsih a small sample of data from one plant, and very little chance of getting permission to publish more. The public data we have is enough to validate a lot of our core functionality, but we will still have features that cannot be tested without access to data which we can't publish.

Our options seem to be to not validate certain functionality at all, or to have a private test suite that we run as part of code review before accepting contributions, which has the obvious problem that we are asking contributors to make sure their changes pass tests which we can't share with them. Unfortunatly we're not really in a position to just not support plant configurations where no public data is available, or to demand that operators publish their data before we support their plants.

Is this something other open source projects have encountered? Are there examples of best practices for dealing with this kind of problem?

Improve this question
4
  • Is there any way you could get enough confidence using unittests with synthesized or otherwise non-sensitive data? Where is the problem with synthesizing data for your tests? How do you deal with negative tests (tests that verify that data is properly marked as incorrect)?
    – Bart van Ingen Schenau
    Oct 26, 2022 at 14:37
  • @BartvanIngenSchenau, the problem is that generating synthetic data that is similar enough to the real data to be meaningful while being different enough so as not to break our NDAs is hard (e.g. geographic location and weather conditions matter). We would probably have to create several entierly fictional plants and do physical modelling, which is expensive. The assesments we're doing aren't pass fail, they produce various KPIs, and we have validated the correct values of these for a selection of real data. I wouldn't be happy without end-to-end tests on realistic data.
    – user11369247
    Oct 26, 2022 at 15:52
  • 1
    You could wait until you actually get someone who is making a candidate contribution that would cause your private test suite to fail, and then address it in an appropriate manner (e.g. maybe try to explain to that developer what needs to be changed without violating NDAs). Another possibility -- if you want to accept the change, but the incoming change needs a bit of work before it's compliant -- then you could simply do that additional work yourself. So, then the incoming change becomes 2 changes, one by the outside developer, and another one by you to make the feature 100% compliant.
    – Brandin
    Oct 27, 2022 at 8:49
  • 1
    Also, maybe it's too technical, but often in software development you have different branches, e.g. a "stable branch" and a "feature branch" or "beta branch" and so on. So, if you're using that organisation, you could conceive that you would accept all contributions that pass all of the publicized tests into the feature branch (i.e. that's the branch with all of the latest and greatest added features) but then you carefully select only the changes that are 100% compliant into your stable branch. Then you offer end users the choice, do you want the stable compliance-tested version, or the beta?
    – Brandin
    Oct 27, 2022 at 8:55

1 Answer 1

Reset to default
6

This is only my opinion, but I suspect you won't have as much of a problem as you think. Your mental model seems to be that once you release this codebase, developers will flock from far and wide with contributions that they can't test because they don't, themselves, have an industrial plant. But free-software development isn't so much a bunch of solvers looking for problems; it's much more a bunch of people with itches, scratching them.

People who do not have their own industrial plants aren't very likely to be working with your codebase, and those who do will be in a position to test fixes. It's definitely possible that big chunks of your codebase will be abstracted into other people's projects, but those will have different use cases, and different test data sets, accordingly. Yes, it's also possible that in doing this, bugs will be discovered in your codebase, which may even be accompanied by fixes which are unusable because they won't pass the private test data, but even then, you'll have a bug logged in your tracker, and hopefully examples to trigger it.

If you are minded to open up this code, I'd say: good for you. Go for it. Cross any later bridges as they appear.

Improve this answer
2
  • It wasn't so much that we were assuming developers would flock from far and wide, more that we were concerned about devs working for organisations who do own plants submitting changes that support some wierd quirk of their setup, and accidentally breaking things for other plant configs.
    – user11369247
    Dec 1, 2022 at 11:38
  • I'm going to mark this as the answer, because it seems like the general consensus from the comments here is "deal with this problem when it actually becomes a problem". Thank you all,
    – user11369247
    Dec 1, 2022 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.