Competing emergent Potts orders and possible nematic spin liquids in the Kagome $J_{1}-J_{3}$ Heisenberg model

The extreme frustration in the Kagome antiferromagnet makes it possible to realize a large number of closely competing magnetic phases either at the classical or the quantum level. Motivated by recent neutron scattering study on the Kagome antiferromagnet vesignieite BaCu$_{3}$V$_{2}$O$_{8}$(OH)$_{2}$, we have made systematic investigation of the phase diagram of the classical and quantum $J_{1}-J_{3}$ antiferromagnetic Heisenberg model defined on the Kagome lattice(KAFHM). While it is shown previously that a large antiferromagnetic exchange between the third-neighboring spins can drive an emergent $q=4$ Potts order through the order-by-disorder mechanism, it is elusive what will happen in the so called "grey region" where the Luttinger-Tisza criteria fails to predict the classical ordering pattern. Through extensive Monte Carlo simulation, we find that the "grey region" is characterized by competing emergent $q=3$ and $q=4$ Potts order, whose emergence are beyond the description of the conventional order-by-disorder mechanism. Our Schwinger Boson mean field calculation in the "grey region" confirms the existence of the Potts-3 order in the ground state of quantum $J_{1}-J_{3}$ KAFHM. The predicted Potts-3 state is found to take two different forms distinguished by their projective symmetry group character(PSG or quantum order), although they have exactly the same symmetry. A comparison with the exact diagonalization result on a 36-site cluster shows that the "grey region" may indeed host a nematic spin liquid ground state featuring an anisotropic ring structure around the $\mathbf{q}=0$ point in its spinon dispersion relation. We find that such "grey region" of extremely frustrated magnets should better be taken as the playground to study the rich competition between exotic emergent phases, rather than a burden on their theoretical analysis.Comment: 16 pages and 16 figure