Towards an Automatic Caption Quality Assessment Model Reflecting the Subjective Views of Deaf, and Hard of Hearing Audiences

Closed Captioning (CC) was designed to serve Deaf and Hard of Hearing viewers, but the quality of CC has been assessed without much inclusion of these primary consumer groups. Currently, caption quality is mostly measured using arithmetic counts based on “errors” between the original spoken words and the caption text. A method to assess the quality of CC that includes the subjective perspective of D/HoH viewers could provide descriptive evaluations and ratings reflective to hearing conditions. Furthermore, using machine learning algorithms based on subjective ratings from D and HoH audiences could automate the process of assessment while reflecting human subjective assessment. Towards the goal of automatic caption quality assessment based on target consumer group perspectives, three studies were conducted to: 1) determine the potential of neural network prediction model feasibility; 2) generate a probability model of human caption quality assessment using the Signal Detection Theory framework; 3) construct and train neural networks to represent human quality assessors using Active Learning; and 4) carry out a benchmarking analysis to compare the newly developed assessment system with existing measures. The main findings from the studies include: 1) statistically significant difference in subjective assessment between D and HoH audiences; 2) correlation between caption error sensitivity and the perceived quality ratings on captions; 3) empirical evidence of the relationship between the values of the four quality factors (Delay, Speed, Number of missing words, and Caption paraphrasing) and the subjective quality rating; 4) use of Signal Detection Theory for modeling D and HoH audience error detection behaviour from watching CC; 5) implementation of Caption quality Assessment Intelligence System (CAIS) which can automatically rate the quality of CC by replicating human subjective assessment on various captions. The main contributions of this research are: 1) the development of a quality assessment neural network model for Closed Captioning which includes Deaf and Hard of Hearing viewer’s subjective opinions; 2) the development of probability decision models for Deaf and Hard of Hearing viewers based on caption error detection; and 3) provision of additional quantitative evidence of subjective difference between Deaf and Hard of Hearing groups.Ph.D