C © Owned by the authors, published by EDP Sciences, 2013 Quantum metrology with cold atomic ensembles

Abstract. Quantum metrology uses quantum features such as entanglement and squeezing to improve the sensitivity of quantum-limited measurements. Long established as a valuable technique in optical measurements such as gravitational-wave detection, quantum metrology is increasingly being applied to atomic instruments such as matter-wave interferometers, atomic clocks, and atomic magnetometers. Several of these new applications involve dual optical/atomic quantum systems, presenting both new challenges and new opportunities. Here we describe an optical magnetometry system that achieves both shot-noise-limited and projection-noise-limited performance, allowing study of optical magnetometry in a fully-quantum regime [1]. By near-resonant Faraday rotation probing, we demonstrate measurement-based spin squeezing in a magnetically-sensitive atomic ensemble [2–4]. The versatility of this system allows us also to design metrologically-relevant optical nonlinearities, and to perform quantum-noise-limited measurements with interacting photons. As a first interaction-based measurement [5], we implement a non-linear metrology scheme proposed by Boixo et al. with the surprising feature of precision scaling better than the 1/N “Heisenberg limit ” [6]. 1