Probabilistic Optical Flow and its Image-Adaptive Refinement

Optical flow estimation, i.e. the prediction of motion in an image sequence, is an essential problem in low-level computer vision. Optical flow serves particularly as an input for many other tasks such as navigation, object tracking, or image registration. In the estimation of flow fields, certain image regions are particularly challenging due to task-inherent difficulties such as illumination changes and occlusions as well as common prediction mistakes, e.g. for large displacements or near motion boundaries. Therefore, the reliability of optical flow estimates varies heavily across the image domain. The first part of this thesis thus focuses on probabilistic optical flow methods, which predict a posterior distribution over the flow field conditioned on the input images. The first proposed method obtains probabilistic estimates by using variational inference to approximate a posterior derived from energy-based optical flow formulations. With ProbFlow, a fully probabilistic optical flow approach shows for the first time competitive results on popular benchmark datasets. The model-inherent confidence measure performs superior in comparison to previous work and the uncertainties are beneficially applied to improve optical flow estimates and a subsequent motion segmentation. In a follow-up work, SVIGL is developed to combine stochastic approaches for variational inference with gradient linearization - a frequently used procedure in optical flow energy methods due to its good optimization properties. SVIGL shows faster convergence and higher robustness than standard approaches for stochastic variational inference of complex posteriors. Moreover, it provides probabilistic optical flow without the tedious derivation of update equations required in ProbFlow while maintaining comparable performance. Although confidence measures detect unreliable regions, they do not directly improve the estimated flow fields. The second part of this thesis thus targets the refinement of optical flow in the context of neural networks. Here, the input images guide the post-processing as they provide valuable information about the structure of correct predictions. The first approach builds on an existing method for image-adaptive convolutions in a high-dimensional space. This space is spanned by feature dimensions that are now learned from data to improve the concept of pixel similarity used in the filtering operation. When applying the so-called semantic lattice to replace the bilinear upsampling step of state-of-the-art deep networks, one sees a clear improvement of the predictions, in particular at motion boundaries. In the last contribution, the two goals of this thesis are combined and per-pixel confidence estimates are leveraged for the image-adaptive refinement of deep optical flow predictions. As such, the proposed probabilistic pixel-adaptive convolutions (PPACs) do not only weigh pixels in a neighborhood according to learned similarity characteristics but also based on their individual reliability. The proposed PPAC refinement networks lead to substantial improvements in comparison to the underlying optical flow estimates. The obtained results are state-of-the-art on several benchmarks and show smooth flow fields with crisp boundaries as well as improved results in unreliable regions