The ultimate bound to the accuracy of phase estimates is often assumed to be given by the Heisenberg limit. Recent work seemed to indicate that this bound can be violated, yielding measurements with much higher accuracy than was previously expected. The Heisenberg limit can be restored as a rigorous bound to the accuracy provided one considers the accuracy averaged over the possible values of the unknown phase, as we have recently shown. Here we present an expanded proof of this result together with a number of additional results, including the proof of a previously conjectured stronger bound in the asymptotic limit. Other measures of the accuracy are examined, as well as other restrictions on the generator of the phase shifts. We provide e...