Evaluating the Effects of Natural Language Generation Techniques on Reader Satisfaction

We are witnessing the emergence of a new technology for dynamically creating stories tailored to the interests of particular readers. Narrative prose generators offer much promise for literacy education, but designing them for maximal effectiveness requires us to understand their effect on readers. This article describes the evaluation of StoryBook, an implemented narrative prose generation system that produces original fairy tales in the Little Red Riding Hood domain. StoryBook creates two to three pages of text consistently represented at the deep linguistic structure level. Because of this, we can formally evaluate multiple versions of a single story and be assured that the content is identical across all versions. We produced five such versions of two separate stories which were compared by a pool of twenty upper division students in English and analyzed with an ANOVA test. While the results are most informative for designers of narrative prose generators, it provides important baselines for research into natural language systems in genera