Challenges and opportunities in applied machine learning

What do citation screening for evidence-based medicineand generating land-cover maps of the Earth have incommon? Both are real-world problems for which we have applied machine-learning techniques to assist human experts, and in each case doing so has motivated the develop-ment of novel machine-learning methods. Our research group works closely with domain experts from other disciplines to solve practical problems. For many tasks, off-the-shelf methods work wonderfully, and when asked for a collaboration we sim-ply point the domain experts to the myriad available open-source machine-learning and data-mining tools. In other cases, however, we discover that the task presents unique challenges that render traditional machine-learning methods inadequate, necessitating the development of novel techniques. In this article we describe two of our collaborative efforts, both of which have addressed the same question: if you do not initially obtain the classification performance that you are look-ing for, what is the reason and what can you do about it? The point we would like to make is that much of the research in machine learning focuses on improvements to existing tech-niques, as measured over benchmark data sets. However, in our experience, when applying machine learning, we have found that the choice of learning algorithm (for example, logistic regression, SVM, decision tree, and so on) usually has only a small effect on performance, and for most real-world problems it’s easy to try some large subset of methods (for example, using cross-validation) until one finds the best method for the task at hand. In our view, the truly interesting questions arise when none of the available machine-learning algorithms performs adequately