Training Efficient Tree-Based Models for Document Ranking

Abstract. Gradient-boosted regression trees (GBRTs) have proven to be an effective solution to the learning-to-rank problem. This work pro-poses and evaluates techniques for training GBRTs that have efficient runtime characteristics. Our approach is based on the simple idea that compact, shallow, and balanced trees yield faster predictions: thus, it makes sense to incorporate some notion of execution cost during training to “encourage ” trees with these topological characteristics. We propose two strategies for accomplishing this: the first, by directly modifying the node splitting criterion during tree induction, and the second, by stage-wise tree pruning. Experiments on a standard learning-to-rank dataset show that the pruning approach is superior; one balanced setting yields an approximately 40 % decrease in prediction latency with minimal re-duction in output quality as measured by NDCG.