How trustworthy a project is may be reflected in its reputation, but the existing user base simply not care beyond the first question of
Does it even try to do what it says on the tin? Reviews and other evidence of an active user base such as community mailing lists will give you a hand in this area. Some critical thinking about whether you trust the reviews may be required. Inclusion in downstream packages such as a Linux distribution would be a stronger indication in this area.
In case you care more than the current users you could ask a few more questions:
Does the installer/binary only include only what's in the code repository? The most effective way to ensure that nothing unpleasant got injected during packaging is to build from source. If that's too high a hurdle to jump, get the installer from a trusted packager (like you Linux distro) or the project's own site.
Is the code clean/securely built? If you can't or won't review the code base in full you may be able to get a piece of software to do it for you (Fortify, Coverity, ...). This may still not be practical for many people. In which case the next best thing is evidence that the project cares about security. If the project has signed up with the Coverity Scan initiative they have at least looked for issues in the code base, even if the issues have not been addressed. Quick response to security issues raised by community and evidence of code review are also good signs.