Optimization of transform coefficient selection and motion vector estimation considering interpicture dependencies in hybrid video coding

Considering inter-picture dependencies when selecting transform coefficient levels in hybrid video coding can be done via formulating the decoding process as a lin-ear signal model and solving a quadratic program. The basic method assumes motion estimation and quanti-zation parameters as being given and then selects the transform coefficient levels. However, when motion vectors are determined in advance, motion estimation must be conducted using uncoded reference pictures which is known to deliver inferior results compared to motion estimation on decoded reference pictures. In this work, we expand the basic method to incorpo-rate the case where the motion estimation is consid-ering decoded reference pictures. We propose an ap-proach that iterates between transform coefficient se-lection and motion estimation. We find that a simple two-pass iteration works reasonably well. Our sim-ulation results using an H.264/AVC-conforming en-coder show coding gains up to 1 dB in comparison to the quantization method specified in the test model of H.264/AVC. 1