Early Determination of Zero-Quantized 8 × 8 DCT Coefficients

Abstract—This paper proposes a novel approach to early determination of zero-quantized 8 × 8 discrete cosine transform (DCT) coefficients for fast video encoding. First, with the dynamic range analysis of DCT coefficients at different frequency posi-tions, several sufficient conditions are derived to early determine whether a prediction error block (8 × 8) is an all-zero or a partial-zero block, i.e., the DCT coefficients within the block are all or partially zero-quantized. Being different from traditional methods that utilize the sum of absolute difference (SAD) of the entire prediction error block, the sufficient conditions are derived based on the SAD of each row of the prediction error block. For partial-zero blocks, fast DCT/IDCT algorithms are further developed by pruning conventional 8-point butterfly-based DCT/IDCT algorithms. Experimental results exhibit that the proposed early determination algorithm greatly reduces computational complexity in terms of DCT/IDCT, quantization, and inverse quantization, as compared with existing algorithms. Index Terms—Butterfly-based DCT, computational complexity, discrete cosine transform, sum of absolute difference, zero-quantized coefficients. I