To my understanding, if I modify the library, I need to give the source of the library with the changes highlighted.
No, you have no such obligation. The Apache 2.0 license does not contain any copyleft requirements that would require you to share your modified source code.
From the Apache License FAQ:
I HAVE MADE CHANGES TO AN APACHE PACKAGE AND I WANT TO DISTRIBUTE THEM. DO I NEED TO CONTRIBUTE THEM TO THE APACHE SOFTWARE FOUNDATION?
No. You can keep your changes a secret if you like. Maybe your modifications are embarrassing, maybe you'll get rich selling those improvements. Whatever. But please seriously consider giving your changes back! We all benefit when you do.
The only obligations you have when redistributing the binaries of an Apache-licensed project are detailed in Section 4, subsection a, b, and d:
a. You must give any other recipients of the Work or Derivative Works a copy of this License; and
b. You must cause any modified files to carry prominent notices stating that You changed the files; and
d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: [...] within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. [...]
Sections (a) and (d) are clear enough: preserve the license text and any NOTICE file to downstream recipients. You may include the NOTICE text within the application itself or in a separate file. I am not sure whether or how to indicate changes in a binary-only distribution, as apparently required by section (b), but I suppose you could include that information in an "About" screen in your application. (e.g., "This executable includes software licensed under the Apache License 2.0, modified by John Smith.")