A Comparative Study of Multi-agent Reinforcement Learning Dynamics

Multi-agent learning plays an increasingly important role in solving complex dynamic problems in to-day’s society. Recently, an evolutionary game theoretic approach to multi-agent reinforcement learning has been proposed as a first step towards a more general theoretical framework. This article uses the evolu-tionary game theory perspective to link behavioral properties of learning algorithms to their performance in both homogeneous and heterogeneous games, thereby contributing to a better understanding of multi-agent reinforcement learning dynamics. Simulation experiments are performed in the domain of 2 × 2 normal form games with the learning algorithms Lenient and non-lenient Frequency Adjusted Q-learning, Finite Action-set Learning Automata and Polynomial Weights Regret Minimization. The results show that evolutionary game theory provides an efficient way to predict the behavior, convergence properties and performance of reinforcement learners. In general, leniency is found to be the preferable choice in cooperative games. Furthermore, the non-lenient learning algorithms do not show significant differences when their intrinsic learning speed is compensated for.