A selection-mutation model for q-learning in multiagent systems

Although well understood in the single-agent framework, the use of traditional reinforcement learning (RL) algorithms in multi-agent systems (MAS) is not always justified. The feed-back an agent experiences in a MAS, is usually influenced by the other agents present in the system. Multi agent envi-ronments are therefore non-stationary and convergence and optimality guarantees of RL algorithms are lost. To better understand the dynamics of traditional RL algo-rithms we analyze the learning process in terms of evolu-tionary dynamics. More specifically we show how the Repli-cator Dynamics (RD) can be used as a model for Q-learning in games. The dynamical equations of Q-learning are de-rived and illustrated by some well chosen experiments. Both reveal an interesting connection between the exploitation-exploration scheme from RL and the selection-mutation mech-anisms from evolutionary game theory