Hemami.‖Tradeoffs in Subjective Testing Methods for Image and Video Quality Assessment

An objective quality estimator for either still images or video should accurately estimate the perceived quality scores of a collection of stimuli. New applications and processing techniques will introduce novel distortions that will need to be quantified in terms of perceived quality in order to confidently evaluate an objective quality estimator. The subjective testing method used to obtain the perceived quality scores affects the accuracy and reliability of the data collected. Two common methods used to collect perceived quality scores are absolute categorical rating (ACR) and subjective assessment methodology for video quality (SAMVIQ). The ACR test method presents stimuli in a random order and uses a coarse resolution rating scale for evaluation. The SAMVIQ test method allows the observer to freely view several stimuli multiple times and uses a fine resolution rating scale for evaluation. Ease of implementation typically influences the adoption of ACR over SAMVIQ, since ACR accommodates more stimuli per testing session. This paper investigates the tradeoffs of these two subjective testing methods using three different subjective databases that have perceived quality scores corresponding to the ACR and SAMVIQ test methods. Results are 1) the fine resolution rating scale used by SAMVIQ is superfluous, 2) SAMVIQ scores have greater accurac