Joint Optimization via Deep Reinforcement Learning in Wireless Networked Controlled Systems

This paper proposes a deep Reinforcement Learning (RL) based co-design approach for joint-optimization of wireless networked control systems (WNCS) where the co-design approach can help achieve optimal control performance under network uncertainties, e.g. delay and variable throughput. Compared to traditional and modern control methods where the dynamics of the system are important for predicting a system’s future response, a model-free approach can adapt to many applications of stochastic behaviour. Our work provides a comparison of how the control performance is affected by network uncertainties such as delays and bandwidth consumption under an unknown number of devices. The control data is transmitted under different network conditions where several applications transmit background traffic data using the same network. The problem contains several sub-optimization problems because the optimal number of devices is non-deterministic under network delay and channel capacity constraints. The proposed approach seeks to minimize control errors in wireless network control systems in order to improve Quality of Service and Quality of Control. This proposed approach is used and compared using three model-free RL Q-learning algorithms for high-throughput flow control in a double emulsion droplets formation application. The results show that the allowable number of devices for reliable network communication under bounded network constraints is 10 when using binary search. The control performance of the system without considering network effect in the reward function (Scenario 1) was good with the C51 algorithm; when including OMNet++ based network effect in the reward function (Scenario 2), the best performance was achieved with all three algorithms (C51, DQN, DDQN) with an exponential reward function, and only with C51 in the case of a linear reward function. Finally, under random network conditions (Scenario 3), C51 and DDQN performed well, but DQN did not converge. Comparisons with other machine learning and non-machine learning algorithms also highlight the superior performance of the utilized algorithms.This research is funded in part by the Estonian Science Agency ETAg grant numbers PRG620, PUT1435, and TAR16013 Center of Excellence 'EXCITE IT', "TTU development program 2016-2022" project code 2014-2020.4.01.16-0032, EU H2020 Grant 668995 COEL. The work was also conducted in the context of the SUITED project funded by the Parrot Franco-Estonian Hubert Curien Partnership