An open source license that satisfies the OSI definition of "Open Source" will allow anyone to fork a project (that is, creating their own version of it). This includes people you may not want to have anything to do with - for instance a group of people you politically disagree with.
You can refuse to incorporate any pull requests or patches submitted by such people into your code base (the one you and your three friends distribute) for any reason. (Bad code quality of probably being the most common ground for rejection.)

(Source: xkcd).
But you can not stop them from creating a derivative by changing the source code, and then go on and distribute their version to the general public.
However, if you're are good maintainer of your project (and that includes accepting sensible patches and pull requests in a timely fashion), a fork has small chances of attracting a following - but you cannot prevent them from happening.
In a comment,
ArtOfCode says:
Not that political disagreements are a good reason to refuse pull requests, of course...
That depends: Let's say I create an open source educational simulation software that allows K-12 pupils to run through simulations of historical events, such as WWII. Let's also assume a group of holocaust deniers modifies the WWII part of this simulation to show their alternate version of WWII and Nazi-Germany's treatment of jews - and submits it as a pull-request. If I was the author of such a simulation program, I am pretty darn sure I would reject it, simply because I disagree (to put it mildly) with the politics of holocaust deniers.
However, If I released this (hypothetical) software as Open Source or Free Software, I could not stop this group of holocaust deniers from distributing their own fork of it.
PS: According to Godwin's law, this question can now be closed :-)
.