Open Source Stack Exchange is a question and answer site for people organizing, marketing or licensing open source development projects. It only takes a minute to sign up.
Sign up to join this community
When submitting a contribution to an open source project, whether a bug fix or a new feature, is testing this new code the responsibility of the person contributing it, or can testing be left to whoever decides whether to accept the new code?
Is there a division of labour with some people contributing code and others taking care of testing, or must every contributor also be a tester?
I would believe that testing the project is the responsibility of all collaborators. It should fall under the hands of the contributor, as well as the maintainer. For example, a contributor should never let this go through (code in Python):
import os
os.system("rm -rf /") # Mua ha ha
Contributors should be wary of what they submit, but they make mistakes. That's why testing should fall onto the entire group. Each person makes code, and each contributor and person involved in the project are responsible for integrating their contributions within the rest of the project. The project is a collaborative effort, and all contributors should strive to make it as bug-free as possible.
First and foremost, testing the contribution is responsibility of the contributor himself, before submitting a patch, to see if it breaks something or otherwise causes problems when combined with the rest of the codebase. This is true for every programing job, not just open source, testing their own changes is the developer's responsibility.
That said, once submitted, each project handles it differently. Smaller projects tend to have few developers who share testing and reviewing tasks and everyone reviews everything. Larger projects sometimes have a group that does more reviewing. What each one actually does varies widely according to each one's customs.
And in every case, there is always the group review, every person involved with the project can look at commits, review code quality, standards, possible bugs introduced and the like. Not just people that actively contribute, just anyone that is used to see the code can spot problems before they go into a release.
I think that isn't the same for each project. A project which maintainers are burdened already with a lot of bugfixing and seldom find time is probably bad of with another patch adding functionality but probably also adding more bugs that is mostly untested. A project which is more or less stable for a long time and has a lot of contributors with enough spare time can handle that a lot better. So, it is overall a question of the policy the project sets.
That's said, more thorough testing on the side of contributor of the patch can never hurt.
When contributing to an open source project you should always make sure your solutions / additions work. This is not only out of courtesy (for the people working on the project) but also for the sake of the project. What is the point of uploading your patch if it is only going to cause problems?
This being said if you test your code and contribute it to the project.. The other people can HELP you solve the problems with your addition.
It's a team effort, with testing needed by both the contributor and integrator. But a high quality project begins with high quality contributions.
It is too easy to confuse contributing to a FLOSS project with a 'hobbyist' approach. But contributors should remember that any contribution they make is also a very public entry in their own resume - whether they work in the field or not. Even if applying for a non-programming job, you might want to cite your involvement in project 'X' as a showcase of your other abilities and interests - and employers are likely to look at it to judge your skills in teamwork, communication, and diligence.
With that in mind, it behooves us to treat our FLOSS contributions with the same seriousness that a professional developer would. That means that the contributor should consider themselves responsible for (at a minimum)
In short everything about each contribution should be 'ready to go'.
On the other side of the coin, the project committers are also responsible for integrating contributions. That means they need to test the release under preparation as a whole, and deal with ensuring that contributions from disparate contributors 'play well together'. They are responsible for generating the release notes and creating and testing the binary distributions (if any).
The person who decides to accept the code is required to make sure the code has been properly tested.
They chose to accept the code, so the buck stops with them.
But that doesn't mean they have to actually test the code, they can delegate it to somebody else who is deemed competent.
As somebody contributing code, you should also make sure your work has been properly tested so you don't waste anybody else's time with obvious bugs.
Exactly who does the testing will depend on the people involved and the specific project. When I'm accepting code, if it's from somebody I don't know I will test it myself thoroughly every time. But if it's someone I know and they have a history of writing reliable and tested code, then I might only do a quick skim read over their code before accepting it.
There are more complex scenarios. For example a difficult to reproduce bug might require a binary sent to an end user who is the only person ever to see the problem. In that case you're getting the user to do your testing, but it is still the responsibility of the person accepting the code to make sure those tests were carried out properly.
A typical policy is to require a contribution to come with comprehensive automated tests. If it fixes a bug, the contributor should start by creating a test that repros the problem, and then passes once the problem is fixed. If it's a feature, it must test the feature.
