LOCAL TEXTUAL INFERENCE: IT’S HARD TO CIRCUMSCRIBE, BUT YOU KNOW IT WHEN YOU SEE IT – AND NLP NEEDS IT

Technology for local textual inference is central to producing a next generation of intelligent yet robust human language processing systems. One can think of it as Information Retrieval++. It is needed for a search on male fertility may be affected by use of cell phones to match a document saying Startling new research into mobile phones suggests they can reduce a man’s sperm count up to 30%, despite the fact that the only word overlap is phones. But textual inference is useful more broadly. It is an enabling technology for applications of document interpretation, such as customer response management, where one would like to conclude from the message My Squeezebox regularly skips during music playback that Sender has set up Squeezebox and Sender can hear music through Squeezebox, and information extraction, where from the text Jorma Ollila joined Nokia in 1985 and held a variety of key management positions before taking the helm in 1992, one wants to extract that Jorma Ollila has served as the CEO of Nokia, a relation that might be more formally denoted as role(CEO, Nokia, Jorma Ollila). Textual inference is a difficult problem (as the results from early evaluations have shown): current systems do statistically better than random guessing, but not by very much. Nevertheless, it is also an area where there is promising developing technology and a good deal of natural language community interest. In other words, it is an ideal research problem. To further this research agenda, data sets have been constructed to assess textual inference systems. This paper examines how the task of textual inference has been and should be defined and discusses what kind of evaluation data is appropriate for the task.