APPLICATION OF REINFORCEMENT LEARNING IN MANAGED PRESSURE DRILLING

Automation in any industry has a control system as its base, and control systems are composed of a controller. In recent years an area of machine learning known as reinforcement learning (RL) has been focused on solving control problems for engineers and scientists. RL methods are actively applied to design control mechanisms for various industrial applications and in this study, the focus will be on designing such algorithms and modeling the given control problem into a structure where these RL algorithms can be applied. In the oil and gas industry, there has been a push to expand operations into areas where usual drilling methods are not successful mainly because of narrow operational windows, and technologies such as Managed Pressure Drilling (MPD) are found to be very successful in solving this issue. MPD is a control technique that is aimed at controlling the bottom hole pressure between narrow operational windows. The standard technique used for automating MPD is a proportional-integral-derivative (PID) controller, but many other non-linear control systems have also been employed to do the same task. This study seeks to add value to the drilling process by developing an Reinforcement Learning (RL) based agent to tune the PID controller. After tuning the PID controller, the system dynamics will be optimized and kept under boundary conditions of the drilling environment. The goal is to provide a reference bottom hole pressure set point and tuning parameters to the PID controller so that the optimum pressure can be reached safely at a certain depth. During the study, the most important features were the depth of the drilling bit, the fracture pressure, and pore pressure at that depth. The RL agent first proposes a suitable reference bottom hole pressure based on the fracture and pore pressure and then tunes the PID controller to achieve the desired pressure set point. The task of training this RL agent is handled in a specialized simulator environment which can calculate the bottom hole pressure at every simulation step and give feedback to the agent about the status. The agent uses a policy gradient method called Proximal Policy Optimization (PPO) and then later on Multi-armed bandit algorithms. PPO is implemented using Mathwork’s Reinforcement Learning Toolbox, and after some tuning of hyperparameters, the agent is able to narrow down to an optimal policy for various depth scenarios, whereas the latter is developed in python. At the end of this study, the agent is able to replace the decision maker and automatically suggest reference bottom hole pressure and tune the PID accordingly