TID2008 – A database for evaluation of full-reference visual quality assessment metrics

In this paper, a new image database, TID2008, for evaluation of full-reference visual quality assessment metrics is described. It contains 1700 test images (25 reference images, 17 types of distortions for each reference image, 4 different levels of each type of distortion). A comparative analysis of TID20008 and its nearest analog LIVE Database (only 5 types of distortions, less statistical accuracy) is presented. Mean Opinion Scores (MOS) for TID2008 database have been obtained as a result of more than 800 experiments. During these tests, observers from three countries (Finland, Italy, and Ukraine) have carried out about 256000 individual human quality judgments. The obtained MOS can be used for effective testing of different visual quality metrics as well as for the design of new metrics. Using the designed image database, we have tested several known quality metrics. The values of rank correlations of Spearman and Kendall with the considered metrics and Mean Opinion Score obtained by exploiting TID2008 in experiments are presented. The metrics are verified for both full set of distorted test images in TID2008 and for particular subsets of TID2008 that include distortions most important for digital image processing applications. It is demonstrated that for the considered wide range of possible distortion types no existing metric performs well enough. In aggregate, the best results are provided by Multiscale Structural Similarity Index Metric (MSSIM). For this metric Spearman Correlation with MOS is 0.853, Kendall correlation with MOS is 0.654. Analysis has been also performed for particular subsets of distortion types. This analysis has shown that for most typical practical applications like image filtering and compression the metrics PSNR-HVS and PSNR-HVS-M produce reasonably good results (Spearman correlation is 0.929, Kendall correlation is 0.765). The designed test image database is available for downloading and utilization in scientific investigations