Open source licenses operate by licensing the rights that exist under copyright. If those rights did not exist (just as they cease to do when a copyright expires), then any terms imposed by the license are moot: recipients of the work can do largely whatever they please with it, because there are no copyright rules telling them they can't. This is mildly crippling to permissive licenses (they can no longer require attribution, but otherwise they were giving most rights away anyway) and massively crippling to copyleft licenses, since they derive their power from the default restrictions of copyright.
So, open source licenses couldn't exist (because there's no copyright rights to license), but software could still satisfy (or fail to satisfy) the definition of open source. The bulk of the Open Source Definition specifies what permissions must exist on open source software:
- "The license shall not restrict any party from selling or giving away the software..."
- "The license must allow modifications and derived works..."
- "The license must not discriminate against any person..."
Since the loss of copyright means the loss of ability to impose restrictions via licensing, all of these permissions naturally exist in the absence of copyright. But that doesn't mean that all software is suddenly open source. You need to satisfy the critical second requirement: source code! If an author releases a binary but doesn't publish their source code, then the binary can be freely distributed, but the lack of copyright does not force the author to release their source code. If they choose not to release their source code, then obviously their work is not open source.
Note that patents would still be relevant in a post-copyright world, so licenses that include a patent grant (such as the Apache License) would still be relevant in that regard.