From Visual Experience to Perceptual Symbol Systems

Consider a human-like agent, looking at the world, and constructing models for the visual experience. Later these models are associated with labels to cre-ate symbols. In this work we use this symbol for-mation process for the construction of a perceptual symbol system. The first task is to analyze a set of video sequences and create a perceptual symbol framework for the objects belonging to multiple cat-egories by creating the modal symbols of the objects. The modal symbols are the image-based analog rep-resentations of the perceptual experience. Later we use unsupervised learning, to create multiclass cat-egorization of these objects to classify semantically. The next step of the work would be to look at the clustering of the different classes of the same ob-ject. For example, when we consider common dogs and Havanese dogs they are classified into differ-ent categories due to the vast differences in their modal symbols. Therefore we need to do cluster-ing of these two modal symbol classes. The final part of the work would be to analyze how linguis-tic symbols develop together with their associated modal symbols and help in better categorization of objects. For example, when we just consider the modal symbols of calves and goats they are classifie