Video Frame Interpolation for High Dynamic Range Sequences Captured with Dual-exposure Sensors

Video frame interpolation (VFI) enables many important applications thatmight involve the temporal domain, such as slow motion playback, or the spatialdomain, such as stop motion sequences. We are focusing on the former task,where one of the key challenges is handling high dynamic range (HDR) scenes inthe presence of complex motion. To this end, we explore possible advantages ofdual-exposure sensors that readily provide sharp short and blurry longexposures that are spatially registered and whose ends are temporally aligned.This way, motion blur registers temporally continuous information on the scenemotion that, combined with the sharp reference, enables more precise motionsampling within a single camera shot. We demonstrate that this facilitates amore complex motion reconstruction in the VFI task, as well as HDR framereconstruction that so far has been considered only for the originally capturedframes, not in-between interpolated frames. We design a neural network trainedin these tasks that clearly outperforms existing solutions. We also propose ametric for scene motion complexity that provides important insights into theperformance of VFI methods at the test time.