If downstream users of your project distribute it without including attribution for the CC-BY-licensed work, they are in violation of the terms of that CC-BY-licensed work. This is an obligation that copyright holder of the work has asked for as part of allowing redistribution of their copyrighted work. If someone redistributes the work without meeting that obligation, the copyright holder of that work could sue them and probably win.
The fact that the work is incorporated into a single project alongside your MIT-licensed code does not change CC-BY's obligation to attribute the author.
I also read something about as long as the software is considered a new and not derivative work then it's fine to not attribute if using my software.
If you (or anyone else) distributes the author's CC-BY-licensed work without attribution, that is prima facie copyright infringement. By default, redistribution of someone else's copyrighted work is not allowed, and in this case, redistribution without attribution ignores the terms under which the author has allowed redistribution of their work to occur.
There are many jurisdiction-specific defenses to prima facie copyright infringement, but they are complicated and situation-specific (how are you using the work? is it sufficiently transformative? is your use critical or academic in nature?) but it takes close reading of a jurisdiction's past case law on the subject to know whether any such defense applies.
You may have also read that CC BY works can be included within a larger, differently-licensed work. This is true: a work that includes CC BY work within it need not be licensed under CC-BY as a whole; you can choose different terms for your downstream modifications. However, the CC BY requirement still applies to the CC-BY-licensed bit, and you must continue to attribute its author. By contrast, CC licenses with a ShareAlike element (like CC BY-SA) require derivative works to be licensed as a whole under those same terms. That requirement does not apply to CC BY works without an SA component.
This particular resource is very small/minor compared to my work and could easily be replaced.
So be it: if a downstream re-user actually does modify your project such that it no longer includes that CC-BY licensed work, then they have no obligations to the author of the replaced work anymore, because they are not distributing the work. Until such a replacement actually occurs, they may only legally redistribute the work if obeying the author's CC BY requirements.