Reducing data path from storage to GPUs for Deep Learning

Deep learning is an emerging workload in the field of HPC. This powerful method of resolution is able to tackle problems which have been out of reach for traditional algorithmic approaches. However, before being able to solve an instance of the problem, Deep Learning has to go through a learning phase involving a huge volume of data. From the computational standpoint the profile of Deep Learning applications is specific enough to be run almost exclusively on dedicated architectures such as GPUs. Most of the research in the field of deep learning has been focused in the numerical side and architectural aspects of the GPUs. Implicitly, the data access was taken as granted. However, the ever increasing amount of data to ingest for the learning phase has shifted the bottleneck from the compute to the storage part. The contribution of this thesis is an analysis of existing Deep Learning solutions, detailed performance measurements, identifying the bottlenecks in the system and a proposition of a new scheme to shorten the data path thus accelerating the Deep Learning process as a whole