DECISION-ANALYTIC MODELS USING REINFORCEMENT LEARNING TO INFORM DYNAMIC SEQUENTIAL DECISIONS IN PUBLIC POLICY

We developed decision-analytic models specifically suited for long-term sequential decision-making in the context of large-scale dynamic stochastic systems, focusing on public policy investment decisions. We found that while machine learning and artificial intelligence algorithms provide the most suitable frameworks for such analyses, multiple challenges arise in its successful adaptation. We address three specific challenges in two public sectors, public health and climate policy, through the following three essays. In Essay I, we developed a reinforcement learning (RL) model to identify optimal sequence of testing and retention-in-care interventions to inform the national strategic plan “Ending the HIV Epidemic in the US”. The large dimensions of the solution space along with the computational complexity of the simulations over long analytic horizons create compounding computational challenges not suitable for solution algorithms. We show that reformulation of the problem by solving for proxy decision-metrics significantly reduces the solution space and ensures convergence to optimality. In Essay II, we developed a deep RL decision-analytic model for effective early control of infectious disease outbreaks, focusing on new or emerging outbreaks that do not yet have pharmaceutical intervention options. Using the COVID-19 pandemic as a test case, we evaluated the question of whether a lockdown is necessary, and if so, when it should be initiated, to what level (proportion lockdown), and how this should change over time, such that it minimizes both epidemic and economic burdens. A key component of this problem is decisions are jurisdictional, i.e., limited in geographical authority, but occurring in interacting environments, i.e., actions of one jurisdiction can influence the epidemic in other jurisdictions. We evaluated the above question in the context of two-geographical jurisdictions which make autonomous, independent decisions, cooperatively or non-cooperatively, but interact in the same environment through travel. In Essay III, focusing on the climate policy sector, we defined a cost-effectiveness metric called Levelized Cost of Carbon (LCC) that carefully accounts for the time-value of money and the time-value of emissions reduction. This metric is a simple tool that local government agencies can use to evaluate climate change projects alongside other issues they may face, such as safety, congestion, pollution, and political considerations. We also investigated the theoretical and practical implications and limitations of using a cost-effectiveness metric as an approach to rank projects