The Effect of Document Retrieval Quality on Factoid

INTRODUCTION A widely-used architecture for factoid question answering (QA) involves the use of a multi-step pipeline consisting of: 1) initial question analysis, 2) document and/or passage retrieval, and 3) answer extraction. In this study, we examine the relationship between the quality of document retrieval and the overall accuracy of QA systems. We evaluate two QA systems using TREC 2002 test set questions [9]: Carnegie Mellon's JAVELIN system [7] and Waterloo's MultiText QA system [2]. We adapt the two QA systems in order to use di#erent sets of documents as input, and seven different document retrieval methods to create the list of documents including a combination of di#erent systems. The set of known relevant documents was used as a baseline to compare the di#erent retrieval methods. Documents with exact or inexact judgments are considered relevant. Our main hypothesis for this study is that there is a positive relationship between improved document retrieval and QA accurac