Creation & Assessment of a Video Quality Ruler

Image quality has always been a matter of great importance for anyone working with or consuming images and videos. Images and videos are, in general, for human consumption. Thus, image quality is essentially a subjective attribute of an image. Though there exist both objective and subjective approaches to assessing image quality, objective results may not correlate well with subjective results. On the other hand, pre-existing subjective methods for evaluating video quality can be time-consuming, resource-consuming, or unreliable. The aim of this thesis was to create and analyse a method for sub- jectively determining video quality. Based on a previously established method for assessing still image quality prescribed in ISO 20462, this method, the Video Quality Ruler, establishes an absolute scale of per- ceptive video quality. The Video Quality Ruler consists of a series of 31 ordered video clips, varying solely in sharpness. Each of the 31 video clips are one perceptual unit apart. Users are able to determine the overall level of quality of a video through comparison against the Ruler. The method provides an easy, universal analysis of video quality that correlates well to human opinions, and is not as time- nor resource- consuming as many other subjective methods. The creation of the Video Quality Ruler was, on the whole, successful. The calibration of the ruler videos differs slightly from the calibration of the Image Quality Ruler from ISO 20462, particularly for images on the blurrier end of the Ruler. Assessment of the Video Quality Ruler deter- mined that a few levels of the Ruler may need to be adjusted, and that the software and laboratory used for the experiment should be improved slightly, to eliminate bias and reduce the variance of results.Image quality has always been a matter of great importance for anyone working with or consuming images and videos. There exist both objective and subjective approaches to assessing image quality. Example metrics of the former include PSNR and SSIM, both of which mathematically compare pixels of the original image with the equivalent pixels of the distorted or changed image. Objective methods are fast and easy to implement, but may not always be well correlated with human perception of image quality. Images and videos are, in general, for human consumption. Thus, image quality is essentially a subjective attribute of an image. Pre-existing subjective methods for evaluating image quality can be time- and resource-consuming, or unreliable. Examples of such methods include the Mean Opinion Score (MOS) test, whereby participants are asked to rate an image along a scale, which may be numerical or descriptive. The results of all participants are averaged to obtain the mean opinion score, and thus the quality level of the image. A problem with the MOS test is the lack of frame of reference, resulting in ambiguous and unreliable scores. A subjective method which solves this particular issue is the Paired Comparison test, in which participants select the preferred image out of a pair of images. However, this method can only provide information about relative quality, and not absolute quality. A subjective method that eliminates the problems of both of these methods is the Image Quality Ruler (IQR) method, outlined in ISO 20462. An IQR is a series of 31 images, varying only sharpness. Each image is separated in one perceptual unit of quality, where one perceptual unit is defined as 75% agreement amongst participants of the preferred image, between a pair of images. The unit is known as a just noticable difference, or JND. Sharpness is chosen as the varying attribute, as it is easily changed using image processing, has low scene and observer variability, and a strong influence on image quality . Sharpness can be measured by a function known as the Modulation Transfer Function (MTF). The MTF of the images in the IQR mimic the MTF of a diffraction-limited lens. The aim of this thesis was to develop the IQR method for video purposes, thus establishing a Video Quality Ruler (VQR). The method of creating a wellcalibrated VQR followed roughly four steps. Firstly, capture an adequate scene to be used for the video clips of the VQR. A 12 second scene containing a variety of information, including text, people, and buildings was recorded. Secondly, the sharpness of the video clip was altered to 62 different levels, spanning the 1 same range of sharpness of the IQR. The MTFs of the 62 distorted video clips mimicked MTFs of a diffraction-limited lens, as in the IQR. The third step was to implement a Paired Comparison test to determine which of the video clips were one JND apart. All possible permutations of video clips that were one JND apart were found. The weighted average of the permutations was computed, where the weights were dependent on the empirical percentage agreement amongst participants in each pair of videos. The fourth and final step was to implement the VQR to validate the results found in the Paired Comparison test in the previous step, and to analyse the VQR methodology itself. The calibration of the VQR differed slightly from that of the IQR, particularly on the blurrier end of the spectrum. Here, it was determined that smaller changes in objective sharpness were needed to create the same perceived change in quality. The validation experiment found that the levels of the VQR were calibrated accurately, with the exception of a few clips which are in need of adjustment. Implementation of the VQR also determined parts of the methodology that should be adjusted for reliability and accuracy. These include randomisation of the initial positioning of the ruler to avoid bias and changing of the non-uniform computer screen used to display the video clips. The VQR does not cover the same range of quality as the IQR, and thus it is advised to extend the range of the former to match that of the latter