Graph-based Mix-out Networks for Video Restoration

The video restoration task aims to generate clear and high-quality videos from those noisy or blurry low-quality videos. Different from image restoration tasks, the temporal information is an additional data dimension in videos and it plays a key role in video restoration task. Therefore, how to effectively take the full advantage of the inter-frame temporal information is a challenging problem for recovering video quality. Current approaches attempt to either warp the features based on the estimated motion modality before post-processing or fuse the features (e.g. concatenation) only once at a specific place during the process. In the former solution, it relies on the accuracy of estimating motion representation very much, but the existing algorithms for estimating motion have problems of being either too slow or inaccurate. As for the latter solution, too few times (usually only once) of feature fusion will lead to the loss of the intra-frame information after fusion and insufficient utilization of the inter-frame information. Therefore, to fully utilize the inter-frame information, we design a network with the mix-out mechanism. Our Mix-out network can simultaneously refine the inter-frame and intra-frame information instead of fully relying on only one of them. Besides, we devise a novel Graph-based Pixel Associating module (GPA module) to better infer the pixel-wise correlation of adjacency frames, which to our knowledge is the first pixel-wise GCN module that can be practically applied on low-level tasks. Extensive experiments show that our method can outperform the state-of-the-art methods on video compression artifact removal task (codec JPEG2000 and codec HEVC (x265)) by 0.74dB (q=20), 0.83dB (q=40) for codec JPEG2000 and 0.46dB (qp=37), 0.48dB (qp=32) for codec HEVC(x265) in terms of PSNR. Besides, we further extend our method to other video restoration tasks like video super resolution and video Gaussian denoising, and also achieve the state-of-the-art performance