I'm currently involved in a project to produce a tool for carrying out certain types of standardised performance assesments on a specific type of industrial plant, that will soon be going open source. We have a potential problem because certain features of our software need real data, or very realistic synthetic data, to test against (i.e. the integration tests that validate that the results produced for a given plant configuration are correct given the assesment standard we are applying). We have access to real data from a whole range of different plants, but we only have permission to publsih a small sample of data from one plant, and very little chance of getting permission to publish more. The public data we have is enough to validate a lot of our core functionality, but we will still have features that cannot be tested without access to data which we can't publish.
Our options seem to be to not validate certain functionality at all, or to have a private test suite that we run as part of code review before accepting contributions, which has the obvious problem that we are asking contributors to make sure their changes pass tests which we can't share with them. Unfortunatly we're not really in a position to just not support plant configurations where no public data is available, or to demand that operators publish their data before we support their plants.
Is this something other open source projects have encountered? Are there examples of best practices for dealing with this kind of problem?