Delay-optimal dynamic mode selection and resource allocation in device-to-device communications - part I: optimal policy

In this paper (Part I and Part II), we investigate the optimal dynamic mode selection and resource allocation to minimize the average end-to-end delay under a dropping probability constraint for an orthogonal frequency-division multiple-access (OFDMA) cellular network with device-to-device (D2D) communications. Different from the previous studies, which mostly focus on an infinite-backlog traffic model, we consider dynamic data arrival with nonsaturated buffers and formulate the resource control problem in D2D communications into an infinite-horizon average-reward constrained Markov decision process (CMDP) in Part I. The CMDP characterizes the dynamic interference between D2D links and cellular links based on their varying backlogged states, the dynamic route selection, and the coupled interactions between uplink and downlink resource allocations. We propose the general form of the optimal policy. In particular, it is proved that the optimal delay, respective of all feasible randomized policies, is attained by either a deterministic policy or a simple mixed policy, which randomizes between two deterministic policies. Therefore, the determination of an optimal randomized policy essentially becomes the determination of one or two deterministic policies, which can be obtained by an equivalent Bellman's equation with reduced state space. Simulation results show that the optimal policy based on the CMDP model outperforms the conventional channel-state-information-only scheme and the throughput-optimal scheme in stability sense