Evaluation of Web-Based Search Engines from the End-User\u27s Perspective: A Pilot Study

Due to the rapid development of the World Wide Web (WWW), Web-based search engines have become the most important tools for accessing global information resources on the Internet. Although these search engines are easily accessible and highly popular, systematic evaluation of their performance based on real users and their information problems has not been conducted. This study describes a pilot study using a systematic methodology proposed by Su (1997) to gather such data. Measures used to assess four Web-based search engines are based on five evaluation criteria: relevance, efficiency, utility, user satisfaction, and connectivity. Eleven participants, who were faculty, staff, and graduate students at the University of Pittsburgh, searched for their individual information topics on four search engines using simple search form. They tested two randomly selected hyperlinks from each set of twenty items retrieved for each engine. They made relevance assessments of the first twenty items retrieved by their best strategy on each engine. They chose and ranked five most relevant items from each set of the twenty items in relation to their individual information problems. Preliminary data analysis reveals that none of the four engines dominates in every aspect of the multidimensional evaluation. The study results shows that even though Lycos has the best relevance ranking performance and the highest precision ratio, Alta Vista has the best connectivity, the highest user\u27s satisfaction ratios for precision and for three components of system features, and the highest value of the search result as a whole. One-way ANOVA test indicates that user\u27s satisfaction ratings with online documentation (F=9.97 & p=0.000) and value of search results as a whole (F=3.65 & p=0.023) are significantly different among the four search engines