Deep Evolvable-Substrate HyperNEAT - Extending ES-HyperNEAT with Multiple Substrates in an Evolving Topology

Abstract Neuroevolution is a technique that evolves artificial neural networks through evolutionary algorithms, inspired by the natural evolution of biological brains. HyperNEAT is one such method, evolving patterns to determine neural networks’ weights based on their geometry within a substrate. Evolvable-Substrate HyperNEAT (ES-HyperNEAT) has extended the method to additionally evolve the network’s geometry. Multi-Spatial Substrate (MSS) extends HyperNEAT in another direction, evolving separate patterns to determine the weights of a network constructed across multiple substrates. This thesis introduces the new framework Deep Evolvable-Substrate Hyper- NEAT (DES-HyperNEAT), combining the characteristic features of Deep Hyper- NEAT, ES-HyperNEAT and MSS. The principal novelty of DES-HyperNEAT is the extension of ES-HyperNEAT, from network construction within a single substrate, to network construction across multiple substrates. The new approach essentially evolves deep neural networks by evolving and combining numerous substrates. DES-HyperNEAT separates complexity over multiple substrates and CPPNs, while also having the advantage of evolving node positions within each substrate. Additionally, it dynamically evolves deeper networks by inserting new substrates throughout evolution. The combination of these properties makes DES-HyperNEAT unique. Three implementations of the framework are proposed. Through statistical analysis, the implementation Layered DES-HyperNEAT is selected, and its properties optimized. It evolves multiple unique CPPNs, one for each substrate and each connection between them. Its performance is evaluated with the datasets Iris, Wine, and Retina. It is shown that Layered DES-HyperNEAT consistently outperforms HyperNEAT and ES-HyperNEAT both when comparing fitness to runtime and number of generations