On Evaluation of Automatically Generated Clinical Discharge Summaries

Abstract. Proper evaluation is crucial for developing high-quality com-puterized text summarization systems. In the clinical domain, the spe-cialized information needs of the clinicians complicates the task of eval-uating automatically produced clinical text summaries. In this paper we present and compare the results from both manual and automatic evalu-ation of computer-generated summaries. These are composed of sentence extracts from the free text in clinical daily notes – corresponding to indi-vidual care episodes, written by physicians concerning patient care. The purpose of this study is primarily to find out if there is a correlation between the conducted automatic evaluation and the manual evalua-tion. We analyze which of the automatic evaluation metrics correlates the most with the scores from the manual evaluation. The manual eval-uation is performed by domain experts who follow an evaluation tool that we developed as a part of this study. As a result, we hope to get some insight into the reliability of the selected approach to automati