Exploiting temporal structures in computational photography

Despite the tremendous progress in computer vision over the last decade, images captured by a digital camera are often regarded as the instantaneous 2D projection of the scene for simplification. This assumption poses a significant challenge to machine perception algorithms due to the existence of many imaging- and scene-induced artifacts in the camera's measurements such as the hand-shake blur and time-of-flight multi-path interference. In this thesis we introduce time-resolved image formation models for color and depth cameras by exploiting the temporal structure within their raw sensor measurements of the scene. Specifically, we present our efforts on leveraging the inter-scanline and cross-frame content correlations for image and video deblurring, as well as utilizing the complementary components of time-of-flight frequency measurements in collaboration for improved depth acquisition. By tackling these limitations we also enable novel imaging applications such as direct material classification from raw time-of-flight data. In addition, we devise post-processing algorithms with temporal structure awareness so that the hidden information can be decoded efficiently with existing off-the-shelf hardware devices. We believe that the proposed time-resolved modeling of the encoding-decoding process of a digital camera opens the door to many exciting directions in computational photography research.Science, Faculty ofComputer Science, Department ofGraduat