ADAPTIVE FRAME AND QP SELECTION FOR TEMPORALLY SUPER-RESOLVED FULL-EXPOSURE-TIME VIDEO

In order to allow sufficient amount of light into the image sen-sor, videos captured in poor lighting conditions typically have low frame rate and frame exposure time equals to inter-frame period— commonly called full exposure time (FET). FET low-frame-rate videos are common in situations where lighting cannot be improved a priori due to practical (e.g., large physical distance between camera and captured objects) or economical (e.g., long duration of night-time surveillance) reasons. Previous computer vision work has shown that content at a desired higher frame rate can be recovered (to some degree of precision) from the captured FET video using self-similarity-based temporal super-resolution. For a network streaming scenario, where a client receives a FET video stream from a server and plays back in real-time, the following practical question remains, however: what is the most suitable rep-resentation of the captured FET video at encoder, given that a video at higher frame rate must be constructed at the decoder at low com-plexity? In this paper, we present an adaptive frame and quantization parameter (QP) selection strategy, where, for a given targeted rate-distortion (RD) tradeoff, FET video frames at appropriate temporal resolutions and QP are selected for encoding using standard H.264 tools at encoder. At the decoder, temporal super-resolution is per-formed at low complexity on the decoded frames to synthesize the desired high frame rate video for display in real-time. We formulate the selection of individual FET frames at different temporal resolu-tions and QP as a shortest path problem to minimize Lagrangian cost of the encoded sequence. Then, we propose a computation-efficient algorithm based on monotonicity in predictor’s temporal resolution and QP to find the shortest path. Experiments show that our strat-egy outperforms alternative naı̈ve non-adaptive approaches by up to 1.3dB at the same bitrate. Index Terms — Video compression, temporal super-resolution 1