Abstract1 terframe motion estimation/compensation can select the most suitable encoding mode from seven modes (16×16, 16×8, 8×16, 8×8, 8×4, 4×8 and 4×4) by the method of Lagrangian rate-distortion (RD) optimization. The La-grangian function is shown in eq.(1), To improve the coding efficiency, many optional macroblock encoding modes have been provided in H.264/AVC video coding standard. Exhaustively searching over all possible modes to choose the best coding mode for each macroblock is the most reliable one, but it causes a lot of computational complexity. This paper proposes a fast algorithm to select the coding mode such that the number of candidate modes to be searched in motion estimation is signifi-cantly reduced and the coding picture...