Copyright © 1998, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. Optimizing Initial Configurations of Neural Networks for the Task of Natural Language Learning

One approach used to develop computer systems for natu- identifying phrases, e.g.<the boy> is a noun phrase; and (3) ral language processing (NLP) is that of Artificial Neural Net- identifying the relationships among phrases, e.g. <the boy> is works (NNs). Because of the large number of parameters a the agent of <runs>. This is similar to Jain (1991). NN has (e.g. network topology, learning algorithm, transfer The NN has 75 hidden nodes, divided into between 1 and functions) and the way they interact, finding optimal parame- 30 layers, as determined by the GA. The GA also determines ter values for any particular task can be extremely difficult. how these layers are connected to each other, and which Topology can greatly affect the performance of a NN. learning algorithm and transfer functions to use. Stolcke (1990) found that simple sentences could be proce- The NN is tested on a language of 508 sentences with three ssed by NNs with one hidden layer. Performance degraded, different complexity levels. The basic training set is 20 % of however, when the NN was presented with embedded senten- the complete language. The GA makes three major decisions ces, for which more complex topologies were needed. What about training: (1) whether to train with sentences from all topology to use for any particular task is still an open ques- complexity levels at once, or with sentences from the simpler tion. Most decisions regarding NN topology are based on ide- level first and more complex sentences later, (2) whether to as of how the problem should be tackled. Jain (1991) used a train with sentences from all complexity levels in the same NN that first decomposed a sentence into phrases and later proportion, or with more sentences from one level or another