- Is this even really Open Source software?
The parts that have source code available and are licensed under an open source license are open source software, according to the OSI. The parts that do not have source code available or are not licensed under an open source license are not open source software. Since the software includes some components that fail to satisfy the definition of open source software, it is correct to say that the entire project, considered in totality, is not completely open source software. Of course, the components that do satisfy the definition are open-source, though they have the weakness (from a software-freedom perspective) of being subject to a problem similar to the Java Trap (i.e., free code designed to run in a non-free environment).
- By bundling in a closed-source core is it violating its own license?
Absolutely not. There are several points the would independently be sufficient to show that there is no license violation:
If the same copyright holder owns both the core internal
closed-source package and the supporting open-source components, there cannot be any violation. The copyright holder doesn't need a license to do anything with their own code; they may offer a license to others to perform some subset of copyright-protected actions. It's impossible for a copyright holder to violate their own license, because licenses are only offered to other people.
If the closed-source and open-source components are owned by the different people, then the only possible violation here is against the owner of the closed-source component, if it was not licensed in a way that allows reuse or redistribution alongside Apache-licensed code. If it was licensed in a way that allows reuse (e.g., if the binaries were licensed under the Apache 2.0 license), then there is no violation. If the closed-source component were licensed under the GPL, then there would indeed be a violation because the downstream author didn't include the source code of the upstream author's code.
You probably think they might be a violation because you expect the open source license to require the totality of the project's source code to be available. However, this is only true for copyleft licenses like the GPL (not permissive licenses like Apache).
If so, is there any recourse? I'm not trying to sue anyone, I just don't think it's fair to act like your project is open source, and release critically incomplete source code.
The author of software may license it however they please. If they want to make the binaries freely available but only release source code for half of the files, or none of them, that's perfectly legal.
Your complaint appears to be the author's advertising the project as open source while it includes some non-open components. The only possible recourse I can think of would be some kind of consumer protection law against false advertising, if the author makes claims that the project is open source in a way that is demonstrably false. However, since "open source" is a tremendously nebulous term (e.g., the OSI failed to secure a trademark on the term), I think it is unlikely such a suit would prevail, even if you could identify a relevant statute to pursue legal action.