Risk Minimization Models for Technology-Assisted Review and their Application to e-discovery

In several subfields of data science, the term “review” refers to the activities, carried out by one or more human annotators, of checking the correctness of the class labels attributed by an automatic process to unlabelled data items, and of replacing wrong labels with the correct labels. Given a set D of unlabelled items that are automatically classified, only a subset of such items are usually reviewed (otherwise, the previous automatic classification step would be useless). This is due to the fact that reviewing comes at a cost, and is even more true when the size of the automatically classified dataset is large, which is increasingly often the case in many application domains. The amount of data items that are reviewed depends (among other factors) on the annotation budget, i.e., the maximum amount of data items that the annotator is willing / has time to review, or the maximum cost one is willing to pay an annotator for having the data reviewed. In order to support the activity of human annotators, it would be important to identify which data items in D should be reviewed and which should not, in such a way that overall costs are minimized and the accuracy of the resulting labels is maximized. Exactly identifying these data items is the task of technology-assisted review (TAR) algorithms. Essentially, these algorithms attempt to strike an optimal tradeoff between the contrasting goals of minimizing the cost of human intervention and maximizing the accuracy of the resulting labelled data, by focusing on those data items that are most likely to have been misclassified by the automatic classifier, and/or those data items for which review would bring about the highest benefit. In this work we introduce and test three major modifications of MINECORE, a recently proposed TAR algorithm whose goal is that of minimizing the expected costs of review for topical relevance (also known as “responsiveness”) and sensitivity (also known as “privilege”) in e-discovery, where the latter is defined as the task of determining, within a civil lawsuit in legal systems based on common law, which documents need be produced by one party to the other party in the lawsuit. The first modification consists in attempting to increase the quality of the “pos- terior” probabilities that are input to MINECORE. We attempt to do this via an instance of the EM algorithm which leverages the transductive nature of e-discovery, i.e., the fact that the set of documents that must be classified is finite and available at training time. The second modification relaxes a simplifying assumption on which MINECORE relies, i.e., that we should “trust” the posterior probabilities output by our machine- learned classifiers. The third modification relaxes yet another simplifying assumption on which MINECORE relies, i.e., the assumption that human annotators always attribute the correct class label to the data items they review. We report experimental results obtained on a large (≈ 800K) dataset of textual documents, on which we experimentally compare the original version of MINECORE with our three variant