If others have started contributing code and/or documentation to the project, then the copyright of the project is shared between all contributors and if there is no explicit license, then technically you are in violation of each other's copyrights.
The technical copyright violation is not a problem on the short term (although it would prohibit you from making a public release), but it does indicate that now is the time to discuss with everyone what license should be used for the project.
When changing the license (and that includes moving away from the "all rights reserved" status if there is no explicit license), all copyright holders need to agree on the new license. As not everyone has the same motivations for contributing to the project, this can lead to the need for a compromise.
Going by your preferences
to let [the project] be used by other open source libraries/applications
This can be satisfied by any open-source license, but it also depends on what license terms appeal to the developers of that other library/application. They might want a less restrictive license than what you use.
if someone finds the app useful, it's more beneficial to contribute to the project, than make private modifications.
Completely private modifications, as in changes that are not distributed at all, can't be prevented by any license. There is no license in existence that forces distribution of changes.
The best you can get is that modifications are shared under the same license as the original app, when they are shared. This is achieved by using a copyleft license, like LGPL, GPL or AGPL.
I also want to prevent it to be used by closed source, commercial software.
There is a huge difference between closed-source software, where you don't have the right to change, build upon or even see the source code, and commercial software, where the software is used by someone to make money.
All open-source licenses allow the software to be used to make money, but it may require a different business model than just simply selling copies of the same software over and over again, because the open-source license grants that same right of selling to those who buy from you.
For protection against inclusion in closed-source software, you again need to turn to the copyleft GPL licenses.
- The LGPL can be used in projects that also used closed-source parts, as long as the recipient of the larger project is able to replace the LGPL-licensed code with a different version (potentially one that they modified themselves).
- The GPL license requires that all code in the complete project is subject to the freedoms that open-source gives you. Any recipient of the project must be able to modify any part of it.
- The AGPL license is specifically designed for SaaS situations to give users of a SaaS service the right to obtain a copy of the (source code of) application that provides the service. In all other respects, it is the same as the GPL license.
