Phase diagram of a bilinear-biquadratic spin-1 model on the triangular lattice from density matrix renormalization group simulations

We investigate a highly frustrated spin-1 model on the triangular lattice, with nearest- and next-nearest-neighbor antiferromagnetic $S\cdot S$ interactions and nearest-neighbor $(S\cdot S)^2$ interactions. Using the density matrix renormalization group (DMRG) technique, we find three magnetically ordered phases, namely 120$^\circ$ spiral order, stripe order, and tetrahedral order, as well as two spin nematic phases: ferroquadrupolar and antiferroquadrupolar. While our data could be consistent with a spin liquid phase between the 120$^\circ$ spiral and antiferroquadrupolar orders, the more likely scenario is a direct continuous transition between these two orders.Comment: 13 pages, 8 figures, plus 13 pages (17 figures) Supplemental Materia