Towards Motion-Aware Light Field Video for Dynamic Scenes

Current Light Field (LF) cameras offer fixed resolution in space, time and angle which is decided a-priori and is independent of the scene. These cameras either trade-off spatial resolution to capture single-shot LF [20, 27, 12] or tradeoff temporal resolution by assuming a static scene to capture high spatial resolution LF [18, 3]. Thus, captur-ing high spatial resolution LF video for dynamic scenes re-mains an open and challenging problem. We present the concept, design and implementation of a LF video camera that allows capturing high resolution LF video. The spatial, angular and temporal resolution are not fixed a-priori and we exploit the scene-specific re-dundancy in space, time and angle. Our reconstruction is motion-aware and offers a continuum of resolution trade-off with increasing motion in the scene. The key idea is (a) to design efficient multiplexing matrices that allow resolu-tion tradeoffs, (b) use dictionary learning and sparse repre-sentations for robust reconstruction, and (c) perform local motion-aware adaptive reconstruction. We perform extensive analysis and characterize the per-formance of our motion-aware reconstruction algorithm. We show realistic simulations using a graphics simulator as well as real results using a LCoS based programmable camera. We demonstrate novel results such as high resolu-tion digital refocusing for dynamic moving objects. 1