From Transfer to Scaling: Lessons Learned in Understanding Novel Reinforcement Learning Algorithms

A major drawback of reinforcement learning (RL) is the slow learning rate. We are interested in speeding up RL. We first approached this problem with transfer learning where we have two domains. We developed a method to trans-fer knowledge from a completely trained RL domain to a partially trained related domain (where we want to speed up learning) and this helped increase the learning rate suf-ficiently. While trying to come up with a theoretical justi-fication we found that our method of transfer of knowledge was actually scaling the Q-values, which was the main rea-son for the effects seen. We then scaled the Q-values with an appropriate scalar value in the RL domain after partial learning and saw similar results. Empirical results in a vari-ety of grid worlds and a multi-agent block loading domain that is exceptionally difficult to solve using standard rein-forcement learning algorithms show significant speedups in learning using scaling