Growing Reservoir Networks Using the Genetic Algorithm Deep HyperNEAT

Typical Artificial Neural Networks (ANNs) have static architectures. The number of nodes and their organization must be chosen and tuned for each task. Choosing these values, or hyperparameters, is a bit of a guessing game, and optimizing must be repeated for each task. If the model is larger than necessary, this leads to more training time and computational cost. The goal of this project is to evolve networks that grow according to the task at hand. By gradually increasing the size and complexity of the network to the extent that the task requires, we will build networks that are more optimal and efficient for the task. We also hypothesize that such evolved networks will exhibit modularity. The type of ANN we use in this research is an Echo State Network (ESN), a type of Reservoir Computer (RC). ESNs have lower training complexity than a typical neural network, only requiring output weights to be trained. While a traditional ESN has random connections between nodes in its reservoir, recent research has shown that creating sub-reservoirs, or modularity, increases performance. We generate and optimize minimal network architectures using a genetic algorithm called Deep HyperNEAT (DHN). The resultant architectures from various tasks are analyzed using graph-theoretical measures to see how information is processed. We hypothesize that reservoirs evolved with DHN will be smaller, more efficient, and exhibit more modularity than randomly generated reservoirs. Multitasking, or training on multiple tasks, will be performed to investigate whether structures within the evolved architecture are shared between tasks