基於深度學習之3D視訊品質評估技術;No-reference 3D video quality assessment Based on convolutional neural network

[[abstract]]近年來，隨著 3D 電視、電影的興起，3D技術受到學術界和業界的廣泛關注，越來越多人認同未來將是3D立體視訊技術快速成長的階段。 然而，由於構建3D觀看環境的困難，以及解決諸如視覺不適等各種3D質量因素，3D-VQA (3D Video Quality Assessment)的研究相對較少，因此，評估3D視訊品質的要求是迫切需要的。本論文係針對彩色加深度格式的3D視訊品質進行評估。現今深度估測演算法眾多，深度估測好壞對3D立體影像品質的影響大，例如當深度邊緣與彩色影像的邊緣無法對齊，或是深度圖估測錯誤時，會造成3D立體影像在觀看時前景物凹向畫面內或背景物凸出畫面的不良效果，所以要評斷所產生深度圖造成的3D立體影像主觀品質好壞為本論文主要目的。本論文針對數種深度估測方法，首先利用DIBR技術獲得左右眼立體影像，接著利用主觀評分方式取得立體影像的主觀分數，不同深度估測方法獲得不同主觀分數。接著利用四種視覺特徵做為輸入，例如:運動向量圖、彩色及深度邊緣圖…等，接著利用深度學習中的卷積神經網絡架構來做主觀評估分數的訓練及預測。深度學習部分採用卷積神經網絡是因為卷積神經網絡是一種由一個或多個卷積層和全連通層組成的學習架構，對於圖像處理或分類有出色表現。本論文整體架構為無參考式的客觀品質評估，意欲使深度學習架構所預測的3D視訊品質能接近人眼主觀品質，藉以判斷特定已訓練深度估測方法所產生深度圖的好壞或未經訓練深度估測方法的優劣。經實驗結果發現:本論文所提出的架構在實驗一，也就是評估已訓練深度估測方法所產生深度圖的好壞上，準確率可以達到70%以上。在實驗二，未經訓練深度估測方法的評估，其準確度可以達到80%~90%。不過也發現了一些問題，主觀分數樣本太過集中而導致樣本少的分數區間判斷錯誤。 In recent years, with the rise of 3D TV, the film, 3D technology by the academic community and the industry's attention, more and more people agree that the future will be 3D stereo video technology rapid growth stage. However, due to the difficulty of building a 3D viewing environment and solving various 3D quality factors such as visual discomfort, 3D-VQA’s (3D Video Quality Assessment) research is less, so the assessment of 3D video quality requirements is urgently needed.This paper evaluates the 3D video quality of the color plus depth format. The depth of the current estimation algorithm, the depth of the assessment of the quality of 3D stereo image quality, such as when the depth edge and color edge can not be aligned, or the depth map estimate error, it will cause 3D stereoscopic images When the foreground material concave to the screen or the background of the protrusion of the adverse effects of the screen, so to assess the depth of the 3D subjective quality score is our main purpose of this paper.In this paper, several kinds of depth estimation methods are used, we obtain the left and right eye stereoscopic images by using DIBR technique. Then, the subjective scores of stereo images are obtained by subjective scoring method. Different subjective scores are obtained by different depth estimation methods. And then use the four visual feature as input, such as: motion vector, color and depth edge map ... and so on, and then use the deep learning about convolution neural network architecture to do the subjective assessment of the scores of training and prediction. The deep learning part uses the convolution neural network because the convolution neural network is a learning architecture composed of one or more convolution layers and a fully connected layer, which is excellent for image processing or classification. The overall structure of this paper is a non-reference objective quality assessment, which means that the 3D video quality predicted by the deep learning architecture can be close to the subjective quality of the human eye in order to judge whether the depth map of the trained depth estimation method is good or bad Advantages and disadvantages of training depth estimation method.The experimental results show that the proposed scheme is more than 70% on the experimental one, that is, the depth map of the trained depth estimation method. In experiment 2, the accuracy of the untrained depth estimation method can be as high as 80% to 90%. But also found some problems, subjective scores of the sample too concentrated and lead to less sample interval judgment error