Video coding is the process of compressing and decompressing a 2D digital video signal, which is widely used in broadcasting TV, terrestrial TV, video streaming, video conferencing and DVD applications. The benefits of improving the coding efficiency are obvious in the sense that it will reduce the bandwidth for transmitting the compressed signal, save the storage memory size and improve the visual quality under the same bitrate. To improve the coding efficiency, rate-distortion optimization techniques are widely used in image and video compression systems, which are concerned with the task of representing a source with as few bits as possible, for a given fidelity constraint. In practice, the constrained rate-distortion optimization problem is formulated as an unconstrained L...[ Read more ]

Video coding is the process of compressing and decompressing a 2D digital video signal, which is widely used in broadcasting TV, terrestrial TV, video streaming, video conferencing and DVD applications. The benefits of improving the coding efficiency are obvious in the sense that it will reduce the bandwidth for transmitting the compressed signal, save the storage memory size and improve the visual quality under the same bitrate. To improve the coding efficiency, rate-distortion optimization techniques are widely used in image and video compression systems, which are concerned with the task of representing a source with as few bits as possible, for a given fidelity constraint. In practice, the constrained rate-distortion optimization problem is formulated as an unconstrained Lagrangian minimization problem, which is used to evaluate the performance of each possible coding mode.

In this thesis, we address the problems of intra prediction, intra transform coding and quantization in the traditional 2D video coding framework, and interview prediction in the 3D video coding framework from the perspective of rate distortion optimization. First, an edge-based adaptive directional intra prediction is proposed to reduce the residue energy for edge rich regions. Starting from the nature of traditional block based intra prediction, the residue characteristics are experimentally analyzed. It is pointed out that traditional block based intra prediction fails to predict the edge region with high prediction accuracy. To solve this problem, an edge model is established within a block and the model parameters are determined using the rate distortion optimization criteria.

Next, the intra transform coding is investigated. After intra prediction, residue is formed by subtracting the predictor from the original signal and transform coding is applied to the residue. To further improve the coding efficiency of intra transform coding, a rate-distortion cost function is established to estimate the practical rate-distortion cost. Through minimizing the proposed cost function, we put forward a novel rate-distortion optimized transform (RDOT) scheme which allows a set of specially trained transforms to be available to all modes, and each block can choose its preferred transform to minimize the rate-distortion cost. The optimization can be solved using the Lloyd-type algorithm (a sequence of transform optimization and data reclassification alternately) to find the optimal set of transforms.

Further, we address the scalar quantization in intra frame coding. After transform, the transformed coefficients are quantized using scaler quantizer. We examine and analyze the statistical properties of transformed coefficients of different intra prediction modes, and derive an optimal quantization matrix, such that the total rate is minimized under a distortion constraint for each mode.

Besides the intra frame coding, the inter-view coding is discussed in the context of 3D video coding from the perspective of rate distortion optimization. To improve the coding efficiency of 3D coding systems, view synthesis prediction (VSP) is proposed as an alternative non-translational disparity compensated predictor. And it is argued that when we choose the depth derived disparity vector as the motion vector and motion vector predictor, the lower bound of the Lagrangian cost is achieved. And this technique is adopted in the 3D video coding standard 3D-HEVC.

To further increase the usage of VSP, an improved depth coding is proposed by allowing different quantization parameters (QP) for each block according to their content characteristics. And the decision of the depth QP is made according to the rate distortion minimization criteria, where the distortion is measured in terms of both synthesis distortion and depth coding distortion, rather than depth coding distortion alone.