Probabilistic models for motion segmentation in image sequences

Motion segmentation is the task of assigning a binary label to every pixel in an image sequence specifying whether it is a moving foreground object or stationary background. It is often an important task in many computer vision applications such as automatic surveillance and tracking systems. Depending on whether the camera is stationary or moving, different approaches are possible for segmentation. Motion segmentation when the camera is stationary is a well studied problem with many effective algorithms and systems in use today. In contrast, the problem of segmentation with a moving camera is much more complex. In this thesis, we make contributions to the problem of motion segmentation in both camera settings. First for the stationary camera case, we develop a probabilistic model that intuitively combines the various aspects of the problem in a system that is easy to interpret and extend. In most stationary camera systems, a distribution over feature values for the background at each pixel location is learned from previous frames in the sequence and used for classification in the current frame. These pixelwise models fail to account for the influence of neighboring pixels on each other. We propose a model that by spatially spreading the information in the pixelwise distributions better reflects the spatial influence between pixels. Further, we show that existing algorithms that use a constant variance value for the distributions at every pixel location in the image are inaccurate and present an alternate pixelwise adaptive variance method. These improvements result in a system that outperforms all existing algorithms on a standard benchmark. Compared to stationary camera videos, moving camera videos have fewer established solutions for motion segmentation. One of the contributions of this thesis is the development of a viable segmentation method that is effective on a wide range of videos and robust to complex background settings. In moving camera videos, motion segmentation is commonly performed using the image plane motion of pixels, or optical flow. However, objects that are at different depths from the camera can exhibit different optical flows, even if they share the same real-world motion. This can cause a depth-dependent segmentation of the scene. While such a segmentation is meaningful, it can be ineffective for the purpose of identifying independently moving objects. Our goal is to develop a segmentation algorithm that clusters pixels that have similar real-world motion. Our solution uses optical flow orientations instead of the complete vectors and exploits the well-known property that under translational camera motion, optical flow orientations are independent of object depth. We introduce a non-parametric probabilistic model that automatically estimates the number of observed independent motions and results in a labeling that is consistent with real-world motion in the scene. Most importantly, static objects are correctly identified as one segment even if they are at different depths. Finally, a rotation compensation algorithm is proposed that can be applied to real-world videos taken with hand-held cameras. We benchmark the system on over thirty videos from multiple data sets containing videos taken in challenging scenarios. Our system is particularly robust on complex background scenes containing objects at significantly different depths