TREC: Topic engineeRing ExerCise

In this work, we investigate approaches to engineer better topic sets in information retrieval test collections. By recasting the TREC evaluation exercise from one of building more effective systems to an exercise in building better topics, we present two possible ap-proaches to quantify topic “goodness”: topic ease and topic set predictivity. A novel interpretation of a well known result and a twofold analysis of data from several TREC editions lead to a re-sult that has been neglected so far: both topic ease and topic set predictivity have changed significantly across the years, sometimes in a perhaps undesirable way