I am not going to distribute the binary ever. The program or the library is not mine. Will it be a problem?
The X11 license permits virtually anything, as long as you retain copyright notices and mention that X11 permissions apply to some portion of your software. This is compatible with virtually all other licenses, including the GPL and even proprietary licenses.
Since you are not distributing the software, compatibility with the GPL does not even matter, since GPL obligations only apply upon distribution of the software. However, rest assured that with the MIT/X11 license, you will be free to distribute your software as well, if you choose to do so in the future.