It seems like the repo could just disappear if the author decides to delete it and if I don't have it cloned locally, I would lose access unless I had previously made a fork.
Yes, that's correct. Then you would be scrambling to find a copy of the software somewhere, or else you'd be unable to recreate your software accurately.
If that's not OK to you, then yes, it makes sense to keep local copies of all the software you use to build your projects.
There are even projects devoted to helping developers do this. For example, in the Perl world, all the modules are stored in the CPAN. There's a project called Pinto that lets you make your own local mirror of the CPAN, using exactly the versions you specify, so that you always have correct copies of the software. That way, if the version you need gets deleted, or even if the entire CPAN is down for some reason, you'll still have copies. See http://perlmaven.com/pinto-tutorial for more about Pinto.