Reliability of Listener Ratings Using Selected Rating Scales in Auditory-Perceptual Voice Evaluation

Auditory-perceptual evaluation is an important method used to rate voice quality both clinically and in research. Findings in previous studies disagree on which type of rating scale is more reliable in making auditory-perceptual ratings. The objective of the present study was to compare intra- and inter-rater reliability of three rating scales: a 10 cm, 11-point equal-appearing interval (EAI) scale, a 10 cm visual analog (VA) scale, and a 10 cm hybrid visual analog scale modeled after the CAPE-V scale. Eighteen graduate students in speech-language pathology were asked to rate the degree of breathiness and roughness of 16 voice samples. After an initial 30-minute training session, raters made ratings on each of the scales during two listening sessions separated by one week. It was found that for intra-rater reliability, though each of the three rating scales demonstrated strong Pearson\u27s correlation coefficients between observations of week one and two (\u3e0.79), there was no significant difference between any of the scales. Similarly, there were no significant differences between scales for inter-rater reliability as determined by intra-class correlation coefficients (ranging from 0.63-0.80) and average variability scores (ranging from 2.03-3.52) for both breathiness and roughness. Though these average intra- and inter-rater reliability scores showed no significant difference between the rating scales, there was variability across individual raters and how they responded to individual voice samples, with a greater range of ratings among listeners evidenced for mild and moderate voice samples than normal and severe voice samples