How to benchmark GPL software?

Question

I would like to create a benchmark that compares several open-source implementations of a particular functionality. Some of those implementations carry a GPL license, while others are more open: Apache, MIT, BSD-3.

The benchmark itself is relatively simple: prepare data, run the function in a loop, measure time/memory footprint, repeat for every implementation. Even though I can run this on my own computer and publish just the results, it is considered a good practice to also provide the source code for the benchmarks. Assuming that I'm only allowed to publish the benchmarking code as Apache v2, what options do I have? The code that I'm testing has to be modified slightly in order to ensure uniformity of interfaces across all implementations.

• Claim fair use exception and include GPL code into the benchmark? It is my understanding that GPL is copyright-based, and the fair use doctrine specifically exists to allow reuse of small portions of copyrighted text in reviews (which benchmarking essentially is).
• Include the GPL code, but have it commented out, with instructions saying that whoever wants to benchmark the GPL implementations will have to manually uncomment and recompile? In this case the GPL code would be propagated, but without being included in the "final product" - would this count as distributing an "aggregate" (from GPL standpoint)?
• Do not include the GPL code at all, stating that the terms of GPL license prevent me from doing so? Accompanied by a link where one can download this code on GitHub, and an explicit instructions how to incorporate it into the benchmarking framework.

Answer 1

Communicate at Arm's Length

What you can do is communicate only with the open-source licensed code "At arm's length". See here.

Communicating at arm's length, e.g. via command-line call, ensures that rather than your work being a derivative of the software you are benchmarking, it is simply code distributed along side it.

Now, most code you want to benchmark won't have a suitable commandline interface for your purposes. That is fine. You create for each package a boiler-plate that is licensed the same as that package, and which provides a command-line interface.

Then you create your benchmarking framework that triggers each of the boiler-plates in turn, over the command line (eg python's popen, julia's run), and reads back in the key data. They run in a seperate process, and the framework only interacts with the boiler-plates like a user would. So the framework is not a derivative of the boilerplate, so you can license it however.

Your overall benchmarking suite, comprised of Framework + Boilerplate, is not under any one license. It is not a derivative (though parts of it are), it is a collection.

Now depending on the exact situation:

• how similar each boilerplate is
• how much has to be done in the same process
• how much should be done in the framework (e.g graphs, data-gen etc)

You might want to generate the boilerplates eg with a templating language. The licensing on that feels a bit weird, but it should be fine.

Or you could have the common functionality shared by all the boiler-plates into another library which you license as CC/0 (compatible with everything), and then each of your boilerplates just import benchmarkcommons; runbench(loaddll("SOFTWARE.so"))

Answer 2

When you are using a GPL library in your product, the main requirement is that the product as a whole is distributed under the terms of the GPL license.

For the code you write, this can be accomplished most easily by using the GPL license as well for your own code, but that is not a requirement. The only requirement is that your license is compatible to the GPL.

As the Apache license is compatible with the GPL, you can put the benchmark code itself and any "adapter code" that is needed to benchmark the GPL library, under the Apache license.

As long as the GPL library itself and the corresponding "adapter code" is not part of the same distribution as the benchmark framework, then the framework as a whole can be distributed under the Apache license.
Only if the GPL library, or code closely dependent on it, is included do you need to distribute the complete package under the GPL license.