Nonlinear mode coupling in rotating stars and the r-mode instability in neutron stars

We develop the formalism required to study the nonlinear interaction of modes in rotating Newtonian stars in the weakly nonlinear regime. The formalism simplifies and extends previous treatments. At linear order, we elucidate and extend slightly a formalism due to Schutz, show how to decompose a general motion of a rotating star into a sum over modes, and obtain uncoupled equations of motion for the mode amplitudes under the influence of an external force. Nonlinear effects are added perturbatively via three-mode couplings; the formalism is easy to extend to higher order couplings. We describe a new, efficient way to compute the coupling coefficients, to zeroth order in the stellar rotation rate, using spin-weighted spherical harmonics, and compute specific examples to illustrate the technique. We apply this formalism to compute coupling coefficients relevant to the nonlinear interactions of unstable r-modes in neutron stars, postponing numerical integrations of the coupled equations of motion to a later paper. From an astrophysical viewpoint, the most interesting result of our paper is that many couplings of r-modes to other rotational modes (modes with zero frequencies in the non-rotating limit) are small: either they vanish altogether because of various selection rules, or they vanish to lowest order in the angular velocity. In zero-buoyancy stars, the coupling of three r-modes is forbidden entirely and the coupling of two r-modes to one hybrid rotational modes vanishes to zeroth order in rotation frequency