Online information retrieval: identification of measures that discriminate among users with different levels and types of experience

The major goal of this research was to discover those behaviors associated with the process of online bibliographic searching that are correlated with success. In the assumption that more experienced searchers are more successful, the major research objectives were to identify (1) the differences among the searches of users of online systems who have different amounts of overall experience and (2) the differences between the searches of persons with and without experience on the data base being searched. Subobjectives were to identify behaviors that contribute to success in searching when success is defined by standard measures such as recall and precision, and to classify errors made in searching. Five groups of searchers each performed two of four preselected searches on the DIALOG system using ONTAP, the 1975 subset of the ERIC (Educational Resources Information Center) data base. The groups were novices, moderately experienced searchers without ERIC experience, moderately experienced searchers with ERIC experience, very experienced searchers without ERIC experience, and very experienced searchers with ERIC experience. Data were collected on the education, online training, online experience, institutional setting, personal characteristics, and attitudes and opinions of the searchers. Variables that describe the search process (e.g., number of commands used) and variables that describe the outcome of searchers (e.g., recall) were measured by examination of the search transcripts. The results showed that, compared to the experienced subjects, the novices performed surprisingly well. Although, as a group, they searched more slowly than the experienced subjects, made more errors, and scored lower on most (but not all) outcome measures, the differences were not as great as might be expected. Three meaningful patterns were found among the searches of the four groups of experienced searchers. The group with the greatest overall experience and the greatest ERIC data base experience achieved the highest recall and had the highest values of a set of search process variables designated "search effort" variables (e.g., number of commands and descriptors, connect time), suggesting a relationship between search effort and recall. This result was supported by the finding that several of the search effort variables correlated significantly with recall. In general, the moderately experienced searchers with ERIC experience performed the briefest, most cost effective searches (when cost effectiveness is measured in terms of time per relevant reference retrieved). This pattern is attributed to the fact that over 90 percent of this group work in academic libraries that charge individual users for online connect time. In this situation pressure to keep costs low appears to be great. The subjects with ERIC experience used more thesaurus terms than the subjects without ERIC experience. The subjects without ERIC experience tended to prefer free text to thesaurus terms. In regard to outcome, only slight evidence was found to support the hypothesis that ERIC data base experience leads to greater success in searching. Enormous individual variability in searching behavior was found; in searches of persons in the same experience groups, values of the search process and search outcome measures sometimes varied by a factor of ten or more, and there was a great deal of overlap among the groups. For this reason observed differences among the groups are not clear cut. The classification of search errors showed that the overall error rate was low - 1.3 non-typographical errors per search for the experienced subjects and about twice as many for the novices. Most of the errors were noticed by the searchers and corrected immediately; very few actually affected the search results. The simplicity of a large portion of the experienced subjects' searches was striking. In half the initial strategy was not modified and it was uncommon to find use of any but the most basic techniques of selecting and combining terms. A major value of this research is its contribution to the methodology of evaluating online systems. This is an important area in which further research is needed. The thought and planning that went into systematically identifying all possible sources of variation that could influence the results provides useful guidance for future researchers.Ph.D., Information Studies -- Drexel University, 197