Towards Large Scale Continual Learning on Modular High Performance Computers

In the talk I will outline the opportunities and challenges towards removing current severe limitations in training robust generic transferable models from large data streams and progress towards neural architectures that are capable of continual learning. Continual learning posits set of abilities to receive streams of incoming, unlabeled data without any clear task boundaries and digest them into a progressively growing generic model without functional collapse. Using this generic model, learning network should be able to deal with variety of multiple tasks and diversity of specific domains without any additional external supervision or necessity to freeze or otherwise manually tune learning, showing increasingly better learning performance across tasks and domains as learning progresses (learning to learn). Apart from the core algorithmic challenge in designing learning systems of this kind, learning of such a versatile model will require active generation of large amounts of highly variable data and will thus put high computational demand on both data generation and training of networks. HPC facilities will therefore become indispensable for growing such general AI