A Study on Error concealment for Image and Video Communications

本論文討論了在三種現今應用於發展中的影像及視訊壓縮標準中錯誤修補的理論。 數位影像及視訊傳輸是被廣泛的使用在現今的消費產品上，例如：數位相機、數位電視及一些手持系統上。而新發展的壓縮標準JPEG 2000的目標是高壓縮率並運用於網路及無線傳輸。然而JPEG 2000是以一個bitplane一個bitplane的編碼及解碼，當有任何的錯誤在bitstream時都將影響到接下來的bitplane，甚至會影響整張畫面，為了處理這個問題，本論文提出了應用在解碼bitstream時立即處理的兩個錯誤修補的方法，一個是依據二個 inter subband之間資料關係去估計遺失的位元資訊，然而此方法無法回復最低層的DWT係數，因此本論文提出另一個方法是利用在同一個subband中感興趣的方向性去預估遺失的位元資訊。所提出之理論不但在硬體上是非常容易實現，而且在影像及時的傳輸上是非常有效率。 在視訊的傳輸上，現行發展的系統有MPEG-4以及H.264/AVC，而MPEG-4是以物件為導向的視訊傳輸，有別於其他傳輸的視訊壓縮標準，他的bitstream分為shape及texture，為了因應MPEG-4在傳輸上發生錯誤的情形，本論文提出二個方法解決，第一個是針對shape的資訊有錯誤時，以模糊理論來處理interpolation後邊緣會模糊的問題，另一方面為了同時可以處理shape及texture遺失的資訊，特徵比對方法是利用來來修復錯誤的資訊，其中包含了三個步驟，第一是截取出特徵點後以cross-radial search 去比對出相對應的特徵點，第二個部驟是以LSE來估計出affine parameter，藉由此參數將遺失的資訊在前一張畫面中找尋相對之pixel回復回來。提出的方法在物件劇烈的轉動上有很明顯的改善。 另一個正在發展中的視訊標準H.264/AVC中最大的特徵是提供不同區塊大小及可參考多張畫面的移動估計，然而在參考軟體中的錯誤修補方法中並未探討到此問題，因此我們提出了一個判斷法則判別出遺失的資訊中區塊的形式為何，並能利用此區塊的形式進一步節省下多張尋找的計算量。 以上這些方法對於影像及視訊傳輸上的畫質都有很大的助益。This dissertation presents some error concealment algorithms for three current developing Image and Video standards that are essential for real-time digital Image/Video transmission systems. Digital Image/Video transmission is widely used in consumer products, e.g., digital camera, digital TV and hand-held system, etc. The newly defined JPEG 2000 delivers image with lower bit rate in the internet and wireless communications. However, JPEG 2000 decoding is processed bitplane by bitplane. Any data loss in bitstream will affect the consequent bitplanes and even possibly destroy the whole picture. We proposed two new algorithms to recover the damaged bitplanes data. One is according to the correlation information between cross-subbands and undamaged bitplane information. The other method is according to the interested directional sets (IDS) of in-subband. The proposed algorithms are quite simple, but are very efficient in concealing the loss data of any subband in streaming real-time video data in the internet and wireless communications. The simulation results show that the proposed algorithm has 1.5~3dB improvement than previously presented error resilient mechanism. In a subjective view, the proposed concealment algorithm can achieve much smoother edges on the reconstructed images. In video transmission, two new error concealment algorithms for MPEG-4 object-based video are presented. An algorithm based on fuzzy set theory is proposed to repair damaged portions of the shape information. The feature based error concealment is for MPEG-4 lost shape and texture data. The algorithm consists of a feature matching step to identify temporally corresponding features between video frames and an affine parameter estimation step to find the motion of the feature points. In the feature matching step, an efficient cross-radial search (CRS) method is developed to find the best matching points. In the affine parameter estimation step, a non-iterative least squares estimation algorithm is developed to estimate the affine parameters. An attractive feature of the algorithm is that the shape data and texture data are handled by the same method. Unlike previous methods, this unified approach works for the case where the video object undergoes a drastic movement, such as a sharp turn. Experimental results show that the proposed algorithm performs much better than previous approaches by about 0.3~2.8 dB for shape data and 1.6~5.0 dB for texture data. In H.264/AVC low bit rate data transmission, a new error concealment algorithm for the new coding standard H.264 is presented. The algorithm consists of a block size determination step to determine the size type of the lost block and a motion vector recovery step to find the lost motion vector from multiple reference frames. The main features of this algorithm are as follows. In the block size determination step, we propose a criterion to determine the size type of the lost block from the current frame. In the motion vector recovery step, the optimal motion vector for the lost block chosen from multiple previous reference frames with the minimum value of the side match distortion. The proposed algorithm not only can determine the most correct mode for the lost block, but also can save much more computation time for motion vector recovery. Experimental results show that the proposed algorithm achieves 0.4~7 dB improvement than conventional VM method does. These techniques can greatly improve the image/video quality and be suitable implemented for image/video transmission systems. The proposed algorithms are applied in decoder when error is detected in bitstream domain for image coding by JPEG 2000, and are applied in image domain for video coding by MPEG-4 or H.264/AVC.CONTENTSLIST OF FIGURES 1 LIST OF FIGURES 4 LIST OF TABLES 7 ABSTRACT 8 CHAPTER 1 INTRODUCTION 10 1.1 IMAGE/VIDEO CODING SYSTEM 10 1.2 INTRODUCTION REAL-TIME IMAGE/VIDEO COMMUNICATION SYSTEM 11 1.3 CHALLENGE AND APPROACH 12 1.3.1 Effects of Bit Loss 13 1.3.2 Effects of Packet Loss 13 1.3.3 Error Handling 14 1.4 OVERVIEW ERROR RESILIENCE TOOLS IN CURRENT STANDARD 14 1.4.1 Error Resilience Tools in JPEG 2000 15 1.4.2 Error Resilience Tools in MPEG-4 16 1.4.3 Error Resilience Tools in H.264 18 1.5 RESEARCH MOTIVATION AND APPROACH 19 1.5.1 Error Concealment Tools in JPEG 2000 20 1.5.2 Error Concealment Tools in MPEG-4 21 1.5.3 Error Concealment Tools in H.264 21 1.6 DISSERTATION ORGANIZATION 22 CHAPTER2 ERROR CONCEALMENT ALGORITHM FOR JPEG 2000 IMAGE TRANSIMISSION 23 2.1 INTRODUCED TO JPEG 2000 IMAGE STANDARD 23 2.1.1 The JPEG 2000 Image Standard 23 2.1.2 Error Robustness Consider in JPEG 2000 24 2.1.3 The Problem Descrepition 25 2.2 ERROR CONCEALMENT ALGORITHM BASED ON CROSS-SUBBAND CORRELATION 29 2.2.1 The Correlation of DWT Coefficients between Cross-Subband 29 2.2.2 Error Concealment Based on Cross-Subband Correlation 29 2.2.3 Simulation Results 32 2.3 USING INTERESTED DIRECTION FOR JPEG 2000 IMAGE TRANSMISSION 38 2.3.1 The Interested Direction of In-Subband 38 2.3.2 Error Concealment Algorithm Using Interested Direction 39 2.3.3 Simulation Results 42 2.4 SUMMARY 49 CHAPTER 3 ERROR CONCEALMENT ALGORITHM FOR MPEG-4 VIDEO TRANSIMISSION 50 3.1 INTRODUCTION 50 3.1.1 MPEG-4 Standard Overview 51 3.1.2 Error Robustness Consider in MPEG-4 52 3.1.3 Error resilient coding of arbitrary shaped video object 54 3.2 ERROR CONCEALMENT ALGORITHM FOR SHAPE DATA 56 3.2.1 Overview Error Concealment of Shape Data in MPEG-4 56 3.2.2 Proposed Spatial Interpolation with Defuzzification (SIDF) 57 3.2.3 Simulation Results 61 3.3 FEATURE-BASED ERROR CONCEALMENT FOR OBJECT-BASED VIDEO 67 3.3.1 Feature based error concealment 67 3.3.2 Feature Points Extraction 69 3.3.3 Feature Points Matching 72 3.3.4 Parameters Estimation 73 3.3.5 Damaged Pixels Replacement 75 3.4 THE FBEC ALGORITHM FOR MPEG-4 76 3.4.1 The FBEC for MPEG-4 Coded Bit Stream 76 3.4.2 Simulation and Analysis 80 3.4.3 Performance Evaluation for P-VOPs. 81 3.4.4 Performance Evaluation for I-VOPs. 86 3.5 SUMMARY 87 CHAPTER 4. ERROR CONCEALMENT ALGORITHM FOR H.264 VIDEO 89 4.1. INTRODUCTION 89 4.2. VARIABLE BLOCK SIZE ERROR CONCEALMENT WITH MULTIPLE-HYPOTHESIS 91 4.2.1. Size type selection of the lost block 92 4.2.2. Error concealment with multiple reference frames 95 4.3. SIMULATION RESULT 99 4.4. SUMMARY 101 CHAPTER 5. CONCLUSIONS 103 5.1 PRINCIPLE CONTRIBUTIONS 103 5.2 FUTURE DIRECTIONS 104 BIBLIOGRAPHY 10