A novel rate estimation model for mode decision of H.264/AVC

To acquire the optimal coding mode of each macroblock, the H.264/AVC encoder exhaustively calculates the rate-distortion cost for all available modes and chooses the minimum one as the best mode. Therefore, the mode decision process is very computationally demanding. To reduce the computation complexity of the rate-distortion cost, in this paper, we propose a novel rate estimation model for the mode decision in H.264/AVC. By modeling the transform coefficients with Generalized Gaussian distributions (GGD), a direct relationship between the magnitude and the information bits of the quantized transform coefficients is deduced. Based on this deduction, the weighted sum of quantized transform coefficients is proposed as an efficient bit-rate estimator of the residual blocks. Extensive experiments show that the proposed algorithm can save up to 30 % of total encoding time with ignorable degradation in coding performance for both inter- and intra-mode decision