Extended degeneracy and order by disorder in the square lattice J(1)-J(2)-J(3) model

The square lattice antiferromagnet with frustrating next-nearest-neighbor coupling continues to generate tremendous interest, with an elusive quantum disordered phase in the vicinity of J(2) = J(1)/2. At this precise value of frustration, the classical model has a very large degeneracy, which makes the problem difficult to handle. We show that introducing a ferromagnetic J(3) coupling partially lifts this degeneracy. It gives rise to a four-site magnetic unit cell with the constraint that the spins on every square must add to zero. This leads to a two-parameter family of ground states and an emergent vector order parameter. We reinterpret this family of ground states as coexistence states of three spirals. Using spin wave analysis, we show that thermal and quantum fluctuations break this degeneracy differently. Thermal fluctuations break it down to a threefold degeneracy with one Neel phase and two stripe phases. This threefold symmetry is restored via a Z(3) thermal transition, as we demonstrate using classical Monte Carlo simulations. On the other hand, quantum fluctuations select the Neel state. In the extreme quantum limit of spin 1/2, we use exact diagonalization to demonstrate Neel ordering beyond a critical J(3) coupling. For weak J(3), a variational approach suggests an s-wave plaquette-RVB state. Away from the J(2) = J(1)/2 line, we show that quantum fluctuations favor Neel ordering strongly enough to stabilize it within the classical stripe region. Our results shed light on the origin of the quantum disordered phase in the J(1)-J(2) model