Sparse Parameterisations For Efficient Machine Learning Algorithms

Progress in Machine Learning is being driven by continued growth in model size, training data and algorithmic innovations relying on access to high-performance computing clusters. While this paradigm has dominated with the advent of modern Deep Representation Learning, concerns over practical limitations are becoming increasingly common: Independent and identically distributed training on large datasets is prohibitively expensive for all but a handful of institutions, reducing participation and the pace of innovation. Even in cases where advanced training hardware is readily available, inference must often be carried out on limited hardware, posing the challenge of reducing model capacity once the functional relationship of a learning problem has been extracted. In addition, this paradigm conflicts with the inherent nature of real applications, where data is collected sequentially and thus Continual Learning is required. Hence, focus on efficiency must not merely be a burden or constraint but can instead both incentivise and benefit from knowledge transfer, leading to better generalisation. Finally, in an age of increasing concern about the environmental footprint of technology, reductions in computational requirements are not merely a cost-saving endeavour but critical to the long-term progress of the field. In this thesis, we develop algorithmic approaches towards increasing the efficiency of Machine Learning by utilising sparse parameterisations and formalising our techniques as operating on the explicit or tacit notion of several tasks. These principles will allow us to devise tractable computational procedures ranging from identifying the most informative subsets of data over drastic reductions in model size without performance loss to finding parameters with the highest plasticity, allowing the rapid adaptation to a task through Meta Learning. Finally, while covering various practical problems throughout the thesis, we will emphasise applications to data compression, a high-impact problem uniquely encapsulating both requirements and promises of Efficient Machine Learning