Transcription conventions and evaluation techniques for spoken language system research

Abstract: "We describe the transcription conventions currently in use for spontaneous speech at Carnegie Mellon University. Two sets of conventions are described, a detail-rich system for wizard experiments, and a more rigid evaluation system designed for purposes of SLS evaluation. The latter is suitable for automatic scoring using the existing NBS (now NIST) scoring software. A simple wizard transcription is included as well as a sample of live-system transcription together with system output. Transcripts can be used to generate a number of diagnostic metrics useful for system evaluation.