How should we respond to evidence when our evidence indicates that we are rationally impaired? I will defend a novel answer based on the analogy between self-doubt and memory loss. To believe that one is now impaired and previously was not is to believe that one’s epistemic position has deteriorated. Memory loss is also a form of epistemic deterioration. I argue that agents who suffer from epistemic deterioration should return to the priors they had at an earlier time. I develop this argument regarding memory loss then extend it to cases of self-doubt