OPTIMAL MODE SELECTION VIA MAXIMUM LIKELIHOOD RATE ESTIMATION: APPLICATION IN FAST MODE-SELECTION WITHIN H.264

In this paper a novel method to estimate the required bits for representing the coded (quantized) coefficients within a block of natural video sequences is proposed. The proposed method assumes a parameterized probabilistic model for coded data and utilizes a maximum likelihood parameter estimation technique to estimate the model’s parameters. The proposed method achieves a robust estimation of the rate based on the estimated probability of each coefficient based on zero order entropy. The rate estimation via maximum likelihood parameter estimation enjoys lower computational complexity, compared to the actual entropy coding of the data. For the purpose of rate estimation we consider a number of options to estimate the parameters of the probabilistic model of coded data. In the context optimal mode selection within the H.264/AVC video codec, the rate estimation method is used to estimate a Lagrangian rate-distortion cost which would be minimized for the optimal mode. The experimental results show that the proposed method considerably reduces the computational complexity while maintaining almost the same rate-distortion coding efficiency compared to mode selection using the actual rate. Index Terms — Rate estimation, maximum likelihood