Two-Tone Optomechanical Instability and Its Fundamental Implications for Backaction-Evading Measurements

While quantum mechanics imposes a fundamental limit on the precision of interferometric measurements of mechanical motion due to measurement backaction, the nonlinear nature of the coupling also leads to parametric instabilities that place practical limits on the sensitivity by limiting the power in the interferometer. Such instabilities have been extensively studied in the context of gravitational wave detectors, and their presence has recently been reported in Advanced LIGO. Here, we observe experimentally and describe theoretically a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements, a protocol designed to overcome the standard quantum limit. We demonstrate the effect in the optical domain with a photonic crystal nanobeam cavity and in the microwave domain with a micromechanical oscillator coupled to a microwave resonator. In contrast to the well-known parametric oscillatory instability that occurs in single-tone, blue-detuned pumping, and results from a two-mode squeezing interaction between the optical and mechanical modes, the parametric instability in balanced two-tone optomechanics results from single-mode squeezing of the mechanical mode in the presence of small detuning errors in the two pump frequencies. Counterintuitively, the instability occurs even in the presence of perfectly balanced intracavity fields and can occur for both signs of detuning errors. We find excellent quantitative agreement with our theoretical predictions. Since the constraints on tuning accuracy become stricter with increasing probe power, the instability imposes a fundamental limitation on BAE measurements as well as other two-tone schemes, such as dissipative squeezing of optical and microwave fields or of mechanical motion. In addition to identifying a new limitation in two-tone BAE measurements, the results also introduce a new type of nonlinear dynamics in cavity optomechanics