Are there established guidelines or best practices for checking open source software for security flaws prior to using it in a business environment?
Is this down to the individual business to decide for themselves or are there approaches that apply universally?
Having the source code available makes it easier to review it for security problems. However, this is mostly of theoretical interest in the context of a business (or a non-profit) deciding to use a piece of software. In practice, reviewing code for security is a long process requiring rare expertise, so people don't do it. When an organization wants to have an assessment of the security of the software, they typically get professionals to do it; mid- to high-assurance assessments tend to require that the assessor has access to the source code (by special contract with the author, if the source code is not normally available). Typical prices charged by professionals are in the $10k–1m range for a few months spent to review and test a medium-sized project with a few security objectives.
Having the development history and the review history is actually more valuable than having the source code. Seeing that a project follows good development practices and has a good review process gives confidence in its security, and that's relatively easy to assess, compared with directly assessing the security of the software. Professional security assessment schemes typically verify that the development process followed good practices, in addition to the source review and penetration testing.
Having the source available makes it easier to explore specific potential security issues. If you worry about, say, the way a program generates cryptographic keys, then having the source code can give you an answer in a few minutes, whereas this particular topic is hard to assess without the source.
Obviously you can apply all the checks that you can also apply to proprietary software.
But there are a few advantages you can use to check the software.
You have access to the source code. That allows you to check the source: Make an audit. Also you can use tools for static code analysis with patterns for security related flaws. You can add logging of metrics to the source and use that in your test-environment to gain more insight into possible problems.
