Space-time joint multi-layer segmentation and depth estimation

Video-based segmentation and reconstruction tech-niques are predominantly extensions of techniques devel-oped for the image domain treating each frame indepen-dently. These approaches ignore the temporal information contained in input videos which can lead to incoherent re-sults. We propose a framework for joint segmentation and reconstruction which explicitly enforces temporal consis-tency by formulating the problem as an energy minimisation generalised to groups of frames. The main idea is to use op-tical flow in combination with a confidence measure to im-pose robust temporal smoothness constraints. Optimisation is performed using recent advances in the field of graph-cuts combined with practical considerations to reduce run-time and memory consumption. Experimental results with real sequences containing rapid motion demonstrate that the method is able to improve spatio-temporal coherence both in terms of segmentation and reconstruction without introducing any degradation in regions where optical flow fails due to fast motion. 1