Video enhancement using content-adaptive least mean square filters

The current television system is evolving towards an entirely digital, high resolution and high picture rate broadcasting system. As compatibility with the past weights heavily for a popular consumer service, this evolution progresses rather slowly. The progress is particularly slow compared to the revolution we observe in modern display technology, which in the last decade eliminated the traditional bottleneck in television picture quality. Methods, that can bridge the resulting gap between broadcast and display picture quality, are therefore increasingly important and may profit from the rapid developments in semiconductor technology. This last trend enables a steadily increasing processing power at a given price level, and makes the implementation of complex video enhancement algorithms feasible in consumer equipment. We started our exploration documented in this thesis by reviewing image and video resolution enhancement techniques proposed in scientific papers and patent literature. This review revealed two interesting approaches, both derived from image restoration theory, i.e. the Least Mean Square (LMS) or Wiener filtering approach, and the Bayesian image restoration approach. We focused on the LMS-filtering methods as particularly Kondo’s classification based LMS filtering approach for video resolution up-conversion method attracted our special attention due to its high performance and relatively simple hardware implementation. Li’s localized LMS filtering approach for resolution up-conversion initially served in a comparative study as an alternative. We recognized though that Kondo’s classification based LMS filtering outperforms Li’s localized LMS-filtering and, moreover, can be used in a broad range of video enhancement applications. Examples of those applications elaborated in this thesis include resolution up-conversion, de-interlacing, chrominance resolution up-conversion and coding artefact reduction. Furthermore we showed that although the LMS filtering approaches are based on objective metrics, particularly the MSE criterion, it is still possible, for video up-scaling, to design subjectively more pleasing filters using the LMS metric. We had to conclude that a direct mapping of the content-adaptive LMS filtering to the problem of intra-field de-interlacing does not outperform all existing intra-field methods. However, the classification based LMS filtering approach was shown to successfully replace the heuristics in two other de-interlacing designs, including the vertical temporal filtering and the data mixing for hybrid deinterlacing combining motion-compensated and edge-adaptive techniques. Finally, we investigated the classification based LMS filtering approach for chrominance resolution enhancement and coding-artifact reduction. In the chrominance up-conversion problem, we combined luminance and chrominance information in an innovative classification of the local image patterns leading to an improved performance compared to earlier designs. In coding-artifact reduction, the relative position information of the pixel inside the coding block along with the local image structure for classification was shown to give an interesting performance and elegant optimization. Looking back, this thesis reveals a more general usage of LMS filtering for video enhancement, which can avoid many heuristic optimizations that are commonly seen in this area