Calculations of ohmic decay of neutron star dipolar magnetic fields which are confined to the neutron star crust are carried out. It is shown that the field does not decay exponentially as has been assumed in most analyses of pulsar observations and that if it occupies the entire crust it decays by less than a factor of order 100 in a Hubble time. Fields that are confined to the outer regions of the crust decay faster and again non-exponentially. We also show that the surface magnetic fields of young pulsars may fluctuate and even change polarities on time-scales much shorter than their secular decay time-scale. As a result the pulsar braking indices can significantly deviate from the value of 3, expected if the fields are constant. Through...