IMPLICIT COMPOSITIONAL STRUCTURE IN THE VECTOR REPRESENTATIONS OF ARTIFICIAL NEURAL NETWORKS

What type of computational system is the mind? I focus on this question from the perspective of language. For millennia, linguists have viewed language as a symbolic system, in which discrete units (e.g., words) are combined in structured ways (e.g., in syntax trees). In recent years, however, artificial intelligence has seen tremendous progress with a very different type of system: neural networks. These models encode information in vectors of continuous numbers and then process those vectors using mathematical operations. The fact that they are not designed to process symbols might make them seem poorly suited for language, yet neural networks are the state of the art for a range of linguistic tasks (e.g., machine translation), far outperforming approaches motivated by symbolic linguistic theories. At first blush, the success of neural networks seems to refute traditional symbolic conceptions of language. In this dissertation, I consider a different possibility: perhaps neural networks are implicitly implementing symbolic structure. To test this hypothesis, I introduce an analysis method called DISCOVER (DISsecting COmpositionality in VEctor Representations) which seeks to approximate a neural network’s vector representations with an explicit compositional structure. I first apply DISCOVER to models trained on synthetic data. DISCOVER is able to closely approximate these models, in many cases replicating their behavior with over 99% accuracy. I also apply DISCOVER to BERT, a large-scale language model trained on natural text. DISCOVER captures about 75% of the variance in BERT representations of templatically-generated sentences, and it replicate BERT’s output for these sentences with an output similarity score of about 80% to 100% (depending on the layer in BERT). I close by discussing how the type of structure illuminated by DISCOVER provides a unifying perspective on an array of prior observations about neural network models. Overall, these experiments support the hypothesis that neural networks learn structured, compositional representations. Therefore, despite appearances to the contrary, the success of neural networks in artificial intelligence may not challenge symbolic conceptions of cognition after all