Emergence of Unconventional Phases in Quantum Spin Systems

In this thesis, we investigate strongly correlated phenomena in quantum spin systems. In the first part of this work, we study geometrically frustrated antiferromagnets (AFMs). Generalizing the SU(2) Heisenberg Hamiltonian to Sp(N) symmetry, we obtain, in the large-N limit, the mean-field phase diagrams for the planar pyrochlore and cubic AFMs. We then use gauge theories to consider fluctuation effects about their respective mean-field configurations. We find, in addition to conventional Neel states, a plethora of novel magnetically disordered phases: two kinds of spin liquids, Z2 in 2+1D and U(1)in 3+1D, and several valence bond solids such as two and three-dimensional plaquette and columnar singlet states. We use the same approach to study the diamond lattice AFM which possesses extended classical ground state degeneracy. We demonstrate that quantum and entropic fluctuations lift this degeneracy in different ways. In the second part of the thesis, we study ultracold spinor atoms confined in optical lattices. We first demonstrate the feasibility of experimental realization of rotor models using ultracold spin-one Bose atoms in a spin-dependent and disordered optical lattice. We show that the ground state of such disordered rotor models with quadrupolar interactions can exhibit biaxial nematic ordering in the disorder-averaged sense, and suggest an imaging experiment to detect the biaxial nematicity in such systems. Finally, using variational wavefunction methods, we study the Mott phases and superfluid-insulator transition of spin-three bosons in an optical lattice with an anisotropic two dimensional optical trap. We chart out the phase diagrams for Mott states with n = 1 and n = 2 atoms per lattice site. We show that the long-range dipolar interaction stabilizes a state characterized by antiferromagnetic chains made of ferromagnetically aligned spins. We also obtain the mean-field phase boundary for the superfluid-insulator transition, and show that inside the superfluid phase and near the superfluid-insulator phase boundary, the system undergoes a first order antiferromagnetic-ferromagnetic spin ordering transition.Ph