The Impact of Recurrent Neural Network Quantization on Federated Learning

Federated Learning consists of a network of distributed hetoregeneous devices that learn a centralized model in a collective and collaborative manner. We survey current state-of-the-art Neural Network compression techniques and elect "Alternating Quantization" as a quantization technique to be applied during the federated learning process to reduce a neural network model size. We propose "Federated Quantization", a theoretical methodology for carrying out federated learning, in which Alternating Quantization is applied both at the server and at the client level to reduce the downlink data transmitted from the central server to the clients and the uplink data transmitted from the clients to the server during the federated learning process, while aiming to maintain a high model quality. We validate our methodology experimentally by carrying out Federated Quantization of LSTMs and GRUs on the task of next-word prediction over a sample of the WikiText2 dataset. We compare our "Federated Quantization" methodology against 1) "Local Full Precision" training 2) "Federated Full Precision" training and 3) state-of-the-art "Local Quantization" training by "Alternating Multi-Bit Quantization of Recurrent Neural Networks". Experimental results show that the accuracy of federated quantized models feature up to 1.3% accuracy decrease w.r.t. federated full precision model accuracy, up to 1% accuracy decrease w.r.t. local full precision model accuracy, up to 2.6% accuracy decrease w.r.t. local quantization model accuracy. Such slight decrease in accuracy is compensated by a convergence to a minimum test loss in fewer epochs using federated quantization than with local quantization. Furthermore, our approach attains a data reduction up to n * 8.70x when transmitting quantized models from the server to n clients and from n clients to the server using the ZSTD compression technique, when compared to the data transmitted in federated full precision learning. We believe such results are promising and pave the way for further investigation of neural network compression techniques during the federated learning process to reduce the data transmitted, while at the same time preserving a high model accuracy