Effect of update rule transition triggered by Q-learning algorithm in evolutionary prisoner's dilemma game involving extortion

Most studies have shown that the heterogeneity of update rules has an important impact on evolutionary game dynamics. In the meanwhile, Q-learning algorithm has gained attention and extensive study in evolutionary games. Therefore, a mixed stochastic evolutionary game dynamic model involving extortion strategy is constructed by combining imitation and aspiration-driven updating rules. During the evolution of the model, individuals will use the Q-learning algorithm which is a typical self-reinforcement learning algorithm to determine which update rule to adopt. Herein, through numerical simulation analyses, it is found that the mixed stochastic evolutionary game dynamic model affected by the Q-learning algorithm ensures the survival of cooperators in the grid network. Moreover, the cooperators cannot form a cooperation cluster in the grid network but will form a chessboard-like distribution with extortioners to protect cooperators from the invasion of defectors. In addition, a series of results show that, before the evolution turns into steady state, our model increases the number of nodes utilizing the average aspiration-driven update rule, thereby promoting the emergence of chessboard-like distribution. Overall, our study may provide some interesting insights into the development of cooperative behavior in the real world