On retrieval system evaluation of search engines

Evaluation of Web search brings challenges into the traditional evaluation methods of information retrieval systems. In this paper, the query set with different user's information categories is constructed by analyzing the query log of Tianwang search engine. In the evaluation experiments for three popular search engines, the differences of indexed document sets are reduced by filtering the query results on the InfoMall Web archive. Experiments show that: (1) Significant differences are found in voluntary assessors, but the results of evaluation keep stable, (2) Continuous relevant scores and corresponding measures have better distinction capability than the binary ones, and (3) Query set with size of 50 is enough for the evaluation measure DCG in the Web search evaluation.EI0101706-17114