Multi-gpu training of convnets

Convolutional neural networks [3] have proven useful in many domains, including computer vi-sion [1, 4, 5], audio processing [6, 7] and natural language processing [8]. These powerful models come at great cost in training time, however. Currently, long training periods make experimentation difficult and time consuming. In this work, we consider a standard architecture [1] trained on the Imagenet dataset [2] for classifi-cation and investigate methods to speed convergence by parallelizing training across multiple GPUs. In this work, we used up to 4 NVIDIA TITAN GPUs with 6GB of RAM. While our experiments are performed on a single server, our GPUs have disjoint memory spaces, and just as in the distributed setting, communication overheads are an important consideration. Unlike previous work [9, 10, 11], we do not aim to improve the underlying optimization algorithm. Instead, we isolate the impact of parallelism, while using standard supervised back-propagation and synchronous mini-batch stochas-tic gradient descent. We consider two basic approaches: data and model parallelism [9]. In data parallelism, the mini-batch is split across several GPUs as shown in fig. 3. Each GPU is responsible for computing gradients with respect to all model parameters, but does so using a subset of the samples in th