Licence compatibility arises when works, say A and B, under two different licences, say N and P respectively, are combined to form a new work C. Because C is a derivative work of A it must honour any conditions N applies to derivative works, and because it is also a derivative of B it must honour any conditions P applies to derivatives.
If it possible to honour the conditions of both N and P, then C can be distributed. If it is not possible to honour the conditions of both N and P, then C cannot legally be distributed.
For example, the Boost licence and the 3-clause BSD licence are compatible because the former requires preservation of the licence text and the copyright notices in the code, and the latter requires preservation of the licence text and the copyright notices in the code and that one refrain from claiming any endorsement by the original code's author. These two sets of requirements can be honoured simultaneously, so the licences are compatible, and a derivative of both is distributable.
The GNU GPLv3 and the Jabber Open Source Licence v1 are incompatible, because the former requires that any derivative be released under the GPLv3 (s5b) and the latter requires that any derivative be released under JOSL1 (s4a). GPLv3 further forbids the application of any additional requirements in s10; a requirement to release under (eg) JOSL1 would be such a forbidden additional requirement. It is therefore impossible to honour the requirements of both GPLv3 and JOSL1 with respect to a particular body of code, and thus the licences are incompatible, and any such derivative work cannot be distributed.
Licence compatibility is commutative (or as you put it, symmetric); if N is compatible with P, then P is compatible with N. Compatibility is not a transitive property; that is, just because licences N and P are a compatible pair, and P and Q are a compatible pair, it does not follow that N and Q are compatible.