Evolutionary Dynamics of Cancer Cell Populations under Immune Selection Pressure and Optimal Control of Chemotherapy

Abstract. Increasing experimental evidence suggests that epigenetic and microenvironmental factors play a key role in cancer progression. In this respect, it is now generally recognized that the immune system can act as an additional selective pressure, which modulates tumor develop-ment and leads, through cancer immunoediting, to the selection for resistance to immune effector mechanisms. This may have serious implications for the design of effective anti-cancer proto-cols. Motivated by these considerations, we present a mathematical model for the dynamics of cancer and immune cells under the effects of chemotherapy and immunity-boosters. Tumor cells are modeled as a population structured by a continuous phenotypic trait, that is related to the level of resistance to receptor-induced cell death triggered by effector lymphocytes. The level of resistance can vary over time due to the effects of epigenetic modifications. In the asymp-totic regime of small epimutations, we highlight the ability of the model to reproduce cancer immunoediting. In an optimal control framework, we tackle the problem of designing effective anti-cancer protocols. The results obtained suggest that chemotherapeutic drugs characterized by high cytotoxic effects can be useful for treating tumors of large size. On the other hand, less cytotoxic chemotherapy in combination with immunity-boosters can be effective against tumor