Extending test collection pools without manual runs

Information retrieval test collections traditionally use a combina-tion of automatic and manual runs to create a pool of documents to be judged. The quality of the final judgments produced for a collection is a product of the variety across each of the runs sub-mitted and the pool depth. In this work, we explore fully auto-mated approaches to generating a pool. By combining a simple voting approach with machine learning from documents retrieved by automatic runs, we are able to identify a large portion of rele-vant documents that would normally only be found through manual runs. Our initial results are promising and can be extended in fu-ture studies to help test collection curators ensure proper judgment coverage is maintained across complete document collections