I came across this question which answers what should be done if you find that your open source project infringes a license.
One of the answers states it's ok to simply remove the infringing code. However, the repository history will always have the infringing code and anyone will have access. Is it really enough to simply remove the code and from now on live a non-gangster-criminal-code-thief-thug life?