Well, without source code it isn't open source.
The availability of source code allows for others to change the code, one of the main reasons to have open source in the first place. Without source code it isn't longer possible.
If you only intend to use the software, the differences between using precompiled binaries and compiling yourself are:
- You can check the source code for unwanted code and compile it to be sure it has no backdoors. You can't be sure with the binary.
- Compiled specifically to your platform (CPU, OS-version etc.) the code might perform a bit better than a generic binary.
- You can apply patches from sources other than the maintainer.
- You can fix minor bugs that affect your usage.
- You can remove parts you don't need and complicate usage, add security risks or hamper performance.
The last points already go in the area that you change the code, but still under the premise that you want to use the software. It seems if the code is available for change the line between using software and working on software can become unclear.