Improving Literature-Based Discovery with Advanced Text Mining

Abstract. Automated Literature Based Discovery (LBD) generates new knowl-edge by combining what is already known in literature. Facilitating large-scale hypothesis testing and generation from huge collections of literature, LBD could significantly support research in biomedical sciences. However, the uptake of LBD by the scientific community has been limited. One of the key reasons for this is the limited nature of existing LBD methodology. Based on fairly shallow methods, current LBD captures only some of the information available in litera-ture. We discuss how advanced Text Mining based on Information retrieval, Nat-ural Language Processing and data mining could open the doors to much deeper, wider coverage and dynamic LBD better capable of evolving with science, in par-ticular when combined with sophisticated, state-of-the-art knowledge discovery techniques. 1 Scientific Background The volume of scientific literature has grown dramatically over the past decades, par-ticularly in rapidly developing areas such as biomedicine. PubMed (the US National Library of Medicine’s literature service) provides now access to more than 24M cita-tions, adding thousands of records daily5. It is now impossible for scientists working in biomedical fields to read all the literature relevant to their field, let alone relevant adjacent fields. Critical hypothesis generating evidence is often discovered long after it was first published, leading to wasted research time and resources [20]. This hinders the progress on solving fundamental problems such as understanding the mechanisms 5 PubMed