Feedback enhanced sensitivity in optomechanics: Surpassing the parametric instability barrier

Parametric instability of an optomechanical system occurs when radiation pressure is strong enough to cause regenerative amplification of mechanical modes. In measurements using optical cavities or interferometers parametric instability severely degrades the signal to noise ratio (SNR), hence limiting the ultimate sensitivity. Here we characterize this degradation in sensitivity and apply active feedback to the unstable mechanical mode to suppress regenerative amplification. This enables us to overcome the lower limit on sensitivity set by parametric instability as demonstrated by a 4 fold increase in sensitivity to a final value of ~1 × 10 mHz -1/2