Your question implies four things that you want to do. I will list each one you want to do and analyze whether they are allowed by the license(s):
- Release GPL_APP with full source-code.
This is completely in accordance with the GPL.
- Patch GPL_APP to look for a proprietary library and use it if available.
This is possibly allowed. See the GPL FAQ "Can I release a program under the GPL which I developed using nonfree tools?"
if you link nonfree libraries with the [GPL program], that would be an issue you need to deal with. It does not preclude releasing the source code under the GPL, but if the libraries don't fit under the “system library” exception, you should affix an explicit notice giving permission to link your program with them
If you are the author of the complete GPL_APP, then it will basically be allowed because you can always choose to attach the permission if you desire. If you are not the author of all of the GPL_APP code, then you seem to have two possible options to overcome this: (a) You need to get permission from the contributors to GPL_APP to add such a permission to the license. (b) You need to develop and distribute your own Operating System under which your library qualifies as a "system library" according to the GPL.
- Write a library (let's call it TREEVIEW_LIB) that is a subclass of QTreeView, and release that library as closed source.
Qt is available as LGPL and under a commercial license. If you use the LGPL version of Qt, then your library will be considered as a modification or derivative work of Qt, so you will need to release the TREEVIEW_LIB source code to comply with the LGPL. If you get a suitable Commercial license from the Qt company, and their Commercial license allows it, then in principle you could release TREEVIEW_LIB without releasing the source code. Essentially your TREEVIEW_LIB would be seen as a modified version of Qt. Note that whether releasing TREEVIEW_LIB would actually comply with the Qt Company's commercial license agreement is a closed source licensing question that you would have to discuss with the Qt Company.
- "I know in first place that users will combine [GPL_APP and TREEVIEW_LIB]."
If you do choose to release TREEVIEW_LIB as closed source (e.g. by purchasing a suitable commercial license from Qt, or by releasing closed source code derived from an LGPL version and possibly violating the LGPL outright), then the situation will be that users will be allowed to combine them on their own machine by running GPL_APP with your TREEVIEW_LIB library (See GPLv2 Term 0), but the combination itself that is installed on their machine will no longer be GPL. I.e. a user combining your TREEVIEW_LIB library with a GPL program will "taint" the GPL program and so the user will no longer be allowed to redistribute that combination according to the GPL (See GPLv2 Term 1).
Note that this opinion seems to be at odds with the GPL FAQ "If I write a plug-in to use with a GPL-covered program, what requirements does that impose on the licenses I can use for distributing my plug-in?" (https://www.gnu.org/licenses/gpl-faq.en.html#GPLAndPlugins):
If the main program and the plugins are a single combined program then this means you must license the plug-in under the GPL or a GPL-compatible free software license and distribute it with source code in a GPL-compliant way.
According to the GPL FAQ, the above quotation seems to mean that you must license TREEVIEW_LIB in a GPL-compatible way, because of the technical way that a library will combine with another program while running ('dynamic linking' as opposed to 'fork and exec'). However, this Answer in the FAQ is an opinion and should not be seen as the final arbiter of what you are allowed to do. Your actual permissions under law and the licenses are determined by relevant Copyright Law and the license(s) of the software you are actually distributing (e.g. TREEVIEW_LIB).
If you distribute TREEVIEW_LIB only, and if that TREEVIEW_LIB distribution does not include any LGPL licensed code (e.g. because you used the commercial version of Qt instead of the LGPL version), then the GPL cannot pose any requirements on how your distribute TREEVIEW_LIB, since that software does not include anyone's ((L)GPL) licensed code. The Free Software Foundation may disapprove, and the decision arguably violates the 'spirit' of the GPL, but in the end, only the copyright holders of the software that you are deriving from (e.g. Qt) can legally complain about how you distribute their software.
A main program that is separate from its plug-ins makes no requirements for the plug-ins.
is applicable when I release them separately and application can work separately, while my goal is that users download and combine them?