Unbounded-rate Markov decision processes : structural properties via a parametrisation approach

This research is interested in optimal control of Markov decision processes (MDPs). Herein a key role is played by structural properties. Properties such as monotonicity and convexity help in finding the optimal policy. Value iteration is a tool to derive such properties in discrete time processes. However, in queueing theory there arise problems that can best be modelled as a unbounded-rate continuous time MDP. These processes are not uniformisable and thus value iteration is not available. This thesis builds towards a systemic way for deriving properties, for both disounted and average cost. The procedure that is proposed consist of multiple steps. The first step is to make the MDP uniformisable a truncation needs to be made that keeps the properties intact, we have some recommendations for suitable truncations. In the second step, value iteration can be used to prove the desired structure, we have developed a list of results that can be used for these proofs. As the third step, taking the limit of the truncation to infinity, we have provided conditions that the structures for the truncated processes hold for the unbounded process as well. Applications of this method include the competing queues problem and a server farm problem.