The dynamics of dimensionality reduction for information retrieval: a study of latent semantic indexing using simulated data

The effects of dimensionality reduction on information retrieval system performance are studied using Latent Semantic Indexing (LSI) on simulated data. The author hypothesize that LSI improve retrieval by improving the fit between a linear language model and non-linear data such as natural language text. The study analyzes how the values of three variables bear on optimizing k, the system's representational dimensionality. The variables studied are: the correlation between terms, the dimensionality of the untransformed termspace, and the number of documents in the collection. Using multinormally-distributed, stochastic matrices as input, precision/recall and average search length (ASL) are computed for differently modeled retrieval situations. Results indicate that optimizing k relates to the type and degree of correlation found in the original termspace. Findings also suggest that a more nuanced simulation model will permit more robust analysis