Intelligent Learning Rate Distribution to Reduce Catastrophic Forgetting in Transformers

Kenneweg P, Schulz A, Schroeder S, Hammer B. Intelligent Learning Rate Distribution to Reduce Catastrophic Forgetting in Transformers. In: Yin H, Camacho D, Tino P, eds. Intelligent Data Engineering and Automated Learning – IDEAL 2022. 23rd International Conference, IDEAL 2022, Manchester, UK, November 24–26, 2022, Proceedings. Lecture Notes in Computer Science. Vol 13756. Cham: Springer International Publishing; 2022: 252-261.Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on a variety of tasks. In this paper, we investigate the problem of catastrophic forgetting in transformer neural networks and question the common practice of fine-tuning with a flat learning rate for the entire network in this context. We perform a hyperparameter optimization process to find learning rate distributions that are better than a flat learning rate. We combine the learning rate distributions thus found and show that they generalize to better performance with respect to the problem of catastrophic forgetting. We validate these learning rate distributions with a variety of NLP benchmarks from the GLUE dataset. The source code is open-source and free software, available at https://github.com/TheMody/NAS-CatastrophicForgetting