Searching for new breakthroughs in science: How effective are computerised detection algorithms?

In this study we design, develop, implement and test an analytical framework and measurement model to detect scientific discoveries with 'breakthrough' characteristics. To do so we have developed a series of computerized search algorithms that data mine large quantities of research publications. These algorithms facilitate early-stage detection of 'breakout' papers that emerge as highly cited and distinctive and are considered to be potential breakthroughs. Combining computer-aided data mining with decision heuristics, enabled us to assess structural changes within citation patterns with the international scientific literature. In our case studies we applied a citation impact time window of 24--36 months after publication of each research paper. In this paper, we report on our test results, in which five algorithms were applied to the entire Web of Science database. We analysed the citation impact patterns of all research articles from the period 1990--1994. We succeeded in detecting many papers with distinctive impact profiles (breakouts). A small subset of these breakouts is classified as 'breakthroughs': Nobel Prize research papers; papers occurring in Nature's Top-100 Most Cited Papers Ever; papers still (highly) cited by review papers or patents; or those frequently mentioned in today's social media. We also compare the outcomes of our algorithms with the results of a 'baseline' detection algorithm developed by Redner in 2005, which selects the world's most highly cited 'hot papers'.The detection rates of the algorithms vary, but overall, they present a powerful tool for tracing breakout papers in science. The wider applicability of these algorithms, across all science fields, has not yet been ascertained. Whether or not our early-stage breakout papers present a 'breakthrough' remains a matter of opinion, where input from subject experts is needed for verification and confirmation, but our detection approach certain helps to limit the search domain to trace and track important emerging topics in science.