Note: I am not a lawyer, this is my understanding as a long-time participant in the opensource world.
Open source licenses and contributor agreements serve different purposes.
Open source licenses are a grant of permission from the copyright holder(s) of the work to the general public allowing them to use, modify and redistribute the work. Depending on the particular license there may be conditions attached such as requiring derivative works to be released under the same license and requiring source code to be released alongside binaries.
In the absence of evidence to the contrary the default assumption is that copyright of contributions remains with their creators (or their creators employers in the event of a work for hire) and that the contributors are released under the same license as the project that is contributed to. I don't know if this assumption has ever been tested in court but it is the assumption that a substantial proportion of the opensource world works on.
The result is, in a project with lots of contributors and no contributor agreement, the license can become effectively "locked in". To change the license terms would require getting permission from all the copyright holders, some of which may disagree or may simply be impossible to contact.
Contributor agreements are typically used create an asymmetry in the legal relationship. Contributors grant extra permissions to the originators of the project or even transfer copyright completely. This gives the originators of the project the right to release those contributions under other licenses either as part of an overall project re-licensing or in some cases to sell licenses to use the code in ways not allowed by the open-source license.
Such agreements are controversial, on the one side they can make it easier for a project to move with the times and can provide a means of financing development by selling licenses to downstream users who do not want to accept the open-source license terms. On the other side many find the asymmetry goes against the principles of open-source/free software.