Reinforcement Learning for Voltage Control-based Ancillary Service using Thermostatically Controlled Loads

Advances in the demand response for energy imbalance management (EIM) ancillary services can change the future power systems. These changes are subject for research in academia and industry. Although an important/promising part of this research is the application of Machine Learning methods to shape future power systems domain, the domain has not fully benefited from this application yet. Thus, the main objective of the presented project is to investigate and assess opportunities for applying reinforcement learning (RL) to achieve such advances by developing an intelligentvoltagecontrol-basedancillaryservicethatusesthermostaticallycontrolled loads (TCLs). Two stages of the project are presented: a proof of concept (PoC) and extensions. The PoC includes modelling and training of a voltage controller utilising Q-learning, chosen due to its efficiency that is achieved without unnecessary sophistication. Simplest relevant for demand response power system of 20 TCLs is considered in the experiments to provide ancillary service. The power system model is developed with Modelica tools. ExtensionsaimtoexceedPoCperformancebyapplyingadvancedRLmethods: Qlearning modification that uses a window of environment states as an input (WIQL), smartdiscretisationstrategiesforenvironment’scontinuousstatespaceandadeepQnetwork(DQN)withexperiencereplay. Toinvestigateparticularitiesofthedeveloped controller , modifications in an experimental setup such as controller testing longer than training, different simulation start time are considered. The improvement of 4% in median performance is achieved compared to the competing analytical approach - optimal constant control chosen using whole time interval simulation for the same voltage controller design. The presented results and corresponding discussions can be useful for both further work on the RL-driven voltage controllers for EIM and other applications of RL in power system domain using Modelica models