A new postprocessing method for the block-based DCT coding based on the convex-projection theory

(1)

1054 IEEE Transactions on Consumer Electronics, Vol. 44, No. 3, AUGUST 1998

CESSING METHOD FOR THE BLOCK-BASED DCT CODING

BASED ON THE CONVEX-PROJECTION THEORY

Chi-Min Liu, Member, IEEE, Chung-Neng Wang and Jine-Yi Lin Department and Institute of Computer Science and Information Engineering

National Chiao Tung University, Hsinchu, 30050, Taiwan E-mail: cmliu @csie.nctu.edu.tw

Abssfracf- The block-based DCT coding has been adopted in image compression standards such as JPEG and MPEGs. This type of coding causes some noticeable artifacts known as the blocking effect. Various post processing methods have been devoted to removing the blocking effect; however, these methods also unavoidably blur the decoded image. Removing the blocking effect while avoiding blurring is the main issue of the postprocessing. This paper presents a new postprocessing method based on convex-projection theory. This method is shown to have keep good tradeoff for the blocking effects and the blurring effects.

I. INTRODUCTION

HE block-based discrete cosine transform (DCT) coding is one of the most popular transform techniques for image and video compression, and has been adopted in compression standards such as JPEG and MPEGs. Fig. 1 illustrates the block diagram of the block-based DCT coding algorithm. An image is partitioned into non-overlapping 8x8 blocks and each block is individually processed through the DCT and the quantization. Since that the DCT and the quantization are individually applied to each block, quantization errors between adjacent blocks are discontinuous. Such an encoding process will magnify the difference between neighboring blocks. This

C"mprr*lrd ymbol Bitrtrulm

t

Fig. 1. The block diagram illustrating the processing of the block-based DCT encoders and decoders.

artifact which is obvious at -very low bit rate coding has been referred to as the blocking effect. Fig. 2 demonstrates the blocking effect of the Lena picture that is coded at JPEG standard with quantization table scaled by 3.

Recently, there have been many postprocessing methods [ 11- [ 171 proposed to reduce the blocking effect. We can group the existing methods on postprocessing into three categories: space- variant lowpass filtering (SVLP) [1]-[7], block-difference

Fig. 2 . The decoded image of the JPEG with quantization table scaled by 3.

minimizing (BDM) [lo], [I 11, and convex-projection (CP) [4], [SI, [9], [13], [15]-[17]. All these methods try to reduce the blocking effect through a smoothing manner; however, the smoothing also unavoidably blurs the image. Hence, removing blocking effect while preserving the edges is the main issue of the postprocessing. This paper presents a new postprocessing method. The method has been shown to keep balance on the blocking effect and the blurring effect as compared with the SVLP, BDM, and the existing CP methods through the subjective measure and the objective measure which we have developed [12].

The information available for removing blocking effect can be considered from the spatial domain and the frequency domain. The spatial-domain information is from the discontinuity among blocks. The space-variant lowpass filtering (SVLP) [ 11-[7] and the block-difference minimizing (BDM) [lo], [ l l ] are to remove the discontinuity through smoothing or minimizing manners. The frequency domain information is from the quantization process where the coefficients are quantized through a scaled quantization table. The scaled quantization table provides the range of the unquantized coefficients and can be used as an effective information to reconstruct the original coefficients and hence the original image. However, it is not an easy way to collaborate the frequency information and the spatial information for postprocessing. The convex projection (CP) theory provides an iterative approach to integrate various processing schemes for a processing objective. The postprocessing

(2)

Liu, Wang and Lin: A New Postprocessing Method for the Block-Based DCT Coding Based on the Convex-Projection Theory 1055

methods in [4], [SI, [9], [13], [15]-[17] have been developed based on the CP theory. The key issue of the CP theory [18], [19] is the way to develop the schemes in each iteration to achieve the processing objective and simultaneously meet the constraints from CP theory for convergence. Sometimes, to meet the constraints of the convergence, the scheme in each iterative process has to be developed less effective for easing the blocking effect. However, the theoretical framework of the convex projection is good in bridging the various information. This paper presents a new postprocessing method based on the CP theory and shows that the method is better than other methods in the tradeoff between the blocking effect and the blurring effect.

The rest of the paper is organized as follows: presents the new postprocessing method based on projection theory. Section I11 demonstrates the results. Section IV gives a concluding remark.

Section I1

the convex experiment

11. NEW POSTPROCESSING METHOD

The CP theory has been adopted in [4], [8], [9], [13], [15]-[17] to process the decoded images. The key issue of the CP theory [IS], [I91 is the way to develop the schemes in each iteration to achieve a processing objective and simultaneously meet the constraints from CP theory for convergence.

By the CP theory, the known features of the processing objective are expressed as the closed convex sets. On each convex set, the associated projection operator is defined. The projection operators alternatively map the input vector onto the convex sets. The restriction of non-expansive of the projection operators can guarantee the convergence of the iterative projections.

When applying the CP theory for postprocessing, we consider two critical issues. The first is on the selection of the convex sets and associated projector satisfying the desired features of images. In the postprocessing method based on the convex- projection theory, we first define some closed convex sets. Each set is designated to bind the images matching one property of the original image. On the various convex sets, we define the associated projection operators. If all sets can satisfy the requirements of the theory, the image will converge to the intersection set in iteration. That is, the resultant image will satisfy all properties. However, it is not easy to find the ideal closed convex sets. If the set is large, you may find the solution in the intersection set; but the intersection solution can not guarantee the good image quality. Inversely, the set of smaller size may lead to an empty intersection set.

Since that there are difficulty in defining the convex sets, the image in the intersections of sets does not guarantee the better image quality. Hence, the initial image will play an essential role in CP iteration.

This paper presents postprocessing method based on the CP theory with five features, First, we develop a window-adaptive lowpass filter to smooth the blocking image according to image contents. The smoothed image can provide an initial image for

The second issue is on the initial images of the CP.

the CP iteration a better initial condition for the final image, Second, the edge preservation set and the associated projector are defined to preserve or reconstruct edges of an image in iteration. Third, the texture areas have been kept in each CP iteration because that preserving edge may also magnie the blocking effect in the processed image, especially in the texture areas where the intensities of neighboring pixels vary slightly (for example, the hair areas in Lena image). Fourth, to reconstruct the original coefficients and hence the original image, the de-quantized DCT coefficients of the processed image shall be retained in the range of the scaled quantization table. Finally, the intensity the pixels of gray-level image is restricted to be between 0 and 255.

A . Processing for the Initial Image

The initial image is obtained by processing the decoded image. The processing has two considerations. First, the pixels near the block boundaries are processed to keep good tradeoff between blocking effects and the blurring effects. Second, the pixels off the block boundaries are processed to preserve edge information. On the two considerations, this paper derives the window-adaptive method to process the image. The method consists of two steps.

The first processing step is to classify the pixels into four types: uniform, sparse, texture, and edge. The classification is based on the Sobel operator defined as

for

0 5 i , j S M - l (1)

where M is the image size,

vh(i,

j ) is the output value of the Sobel operators for detecting horizontal edges and v V ( i , j ) for detecting vertical edges. The classification types are decided according to

sparse,

r,

< V,? I

pixel type(i,j) =

where the threshold

T,

is a very smaller value to ensure no pixel with blocking effects included, and the others are dependent on the histogram of the edge image in Eq. (9). is the mean of

the edge image and It is

feasible that the pixels among the different classes have mutual exclusive properties in the space domain while those in separate class have similar properties. The pixels in each the four mutual exclusive classes is processed individually.

The second step has two processing methods separately for pixels on the block boundaries and off the block boundaries. For those on the block boundaries, two low pass filtering is adopted. For a pixel having the same pixel type as the majority of the nearest 24 pixels, the pixel are processed through a lowpass/bandlimit filter:

(3)

1056 E E E Transactions on Consumer Electronics, Vol. 44, No. 3, AUGUST 1998 where

[$I'

i = O a n d j = O (4) ww = (1 - l"j0,O) otherwise

This filter is designed to reserve the edges through the two factors

11,

over

7;

. Therefore, to gradually decrease the discontinuities between adjacent blocks, the ratio of

Tl,

over

7;

is used. are the thresholds extracting from a two dimensional (2-D) image, the squared-root operation is required for I-D threshold.

For a pixel not having the same pixel type as the majority of the nearest 24 pixels, the pixel are processed through a 3 x 3 equal-weight average filter:

i

8

Since the

Tu

and

In extreme cases of this specific filter in Eq. (3), as approaches to one, F becomes all-pass filter; inversely, the pixel approaches to a equal-weight average filter. Namely, the separate filters of size 3 x 3 are separately employed for gradually controlling the blurring effect by considering the correlation of gradients in the local area.

For pixels off the boundaries, the window adaptive lowpass filters are adopted to smooth the areas. The applying lowpass filters having three different sizes: 7 x 7 , 5x5, and 3 x 3 . The maximum window size is determined according to whether or not the pixels in the window have been classified as edge type. The pixels of edge type remain unfiltered. These filters are defined in the following form

where N=3, 5, and 7

B. Five Convex Sets

For the postprocessing based on the CP theory, the known properties have been expressed as the closed convex set. Given the decoded image, therefore, this paper selects four closed convex sets CE, CT, CQ ,and CI for iteration. Besides, the decoded image can be found in each of the four sets, the method ensures (CEnCTnCQnCI)=C&O. In this paper, CE preserves the edges with the introducing of the psychovisual effects and is the edge preservation set. CT keeps the textures and is the texture preservation set. CQ retains the dequantized DCT coefficients of the processed image in the range of the scaled quantization table.

The associated projector for the CE in the space-domain is CI is the intensity normalizing set.

defined as

a , x ( i , j ) < a

I

b, x ( i , j ) > b

P,l'nce [ x ( i , j ) ] = x(i, j ) , a I x ( i , j ) I b

The bounds a and b are determined by pixel classes:

a = x D ( i , j ) - A,, b = x D ( i , j ) .t A, (71,

where

x D ( i , j )

denotes the decoded image and

xAs(i,j)

the initial image.

(8)

JND, Ah

x ( i , j ) is off block border

x ( z , j ) is on block border

+{Y

where JND, is the threshold of the k-th block based on the model for contrast sensitivity [20]. A,, is decided by the pixel type as listed in the following table.

Similarly, PI is the associated projector for CI and has the The associated operator PQ is for CQ: 0

I

U,

v I

A4

-

1

.

bounds to be 0 and 255.

A,

X(U,V)

<: A

PT[X(u,v)]=

1 X(u,v),

A

<

X(U,V)

5 B (9)

B ,

X(u,v)>B

where

X(U,V)

is the DCT coefficient of the currently processed image. The bounds of PQ are dependent on the scaled step size:

where X , ( ~ , V ) denotes the DCT components of the decoded image and

XS(u,v)

the DCT components of the initial image.

q is the quantization scale factor and A L! (U, v ) is the step size on the (U, v)-position of the quantization table of JPEG.

The associated operator PT is for CT and is applied just to the texture blocks. The P, is formulated as P, in Eq. (9) , The

upper and lower bounds of PT are determined from the DCT coefficients of the initial image and the decoded image as follows:

(1 1). A = min{x,s ( i , j ) , - Y D ( i , j ) l ,

B = " { x S ( 4 / ) , x D ( i j j ) }

It is feasible to prove that the operators are nonexpansive. Therefore, CP theory guarantees the convergence of the iteration.

C. Zteration

In brief, the new postprocessing method is formulated as

i =

m+m l i r r l f P , ~ ~ ~ , . ~ , ) ~ ~ ~ ~

(4)

Liu, Wang and Lin: A New Postprocessing Method for the Block-Based DCT Coding Based on the Convex-Projection Theory 1057

4.111HI

c 6.111HI

''.*111 ISIIHI I.~IHI Z.IIIMI ~.IIIMI ~.IIIIII s.iun1 ~.IIIIII

Q%de

Fig. 4. The performances of various postprocessing methods on the Lena image.

To seek for the maximum iterations required for convergence, the metrics such as BMR[12] and PSNR are used. As the difference between the outputs of the selected metric is smaller than a threshold, the iteration stops. That is, the iterations will be terminated as the convergence conditions are satisfied. If the BMR is used, the blocking strength and the blurring strength must be converged into a fixed range. By BMR, the resultant image is a satisfactory solution with better perceptibility.

111. EXPERIMENT RESULTS

This section demonstrates the experiment results through two sample images, which are called as Lena and Ken image. Figs. 4 and 5 illustrate the performance by the new method and other existed methods. The resultant images are demonstrated in Figs. 7 and 8. In both illustrations, the values of the measures are arranged in an ordered form as udB = (SNR, P S N R , [ B N ~ , B ~ ~ ~ ] ) ,

where [B/k,Blur] is the measure we present as BMR in [12]. The Unprocessed indicates the BMR value of the unprocessed image. One method which employs only a 3 x 3 average filter is labeled as LP. The space-variant lowpass filtering [3], which applies four equal-weight average filters into the low frequency areas, are labeled as SVLP [3]. The filters used have the size 3x3, 5x5, 7x7, and 9x9. The iterative method based on CP theory, which includes the smoothing operator in iterations [8], is then denoted as CP(k), where k means the k-th iteration as illustrated in [SI. The perceptual based postprocessing is denoted as newCP. For each method, the contrast between the blocking strength and the blurring strength provides the subjective quality of the reconstruction [12]. In Figs. 4 and 5 ,

the Unprocessed case has the strongest blocking strength and is assumed to have no blurring. Due to the nature of the lowpass filter, LP case has little or even no blocking effect but the strongest blurring strength. Since, in the SVLP case, the edge

information was extracted from the decoded image and only the low frequency areas were smoothed, the blurring strength is conducted. However, as the compression rate is increasing, the

d r S V L P I C P ( 2 ) ILMNI

CMI 1

h

Fig. 5. The performances of various postprocessing methods on the Ken image.

edge information is more and more difficult to extract. Thus, the blocking strength of the SVLP case increases. The iterative method CP(1) performs similar as CP(2) in total; but the CP(2) is slightly blurred due to applying the lowpass filter to the resultant image of CP(1) once again. Both CP(k) cases smooth

all areas except for the edges to get a slightly blocky reconstruction. However, serious distortion of edges as the compression being increased leads CP(k) cases to be oversmoothed. In the case of new postprocessing method, the perceptually optimized image is obtained that the blocking strength and the blurring strength are shown to be balanced.

The convergence rates of iterative methods, CP(k) in [8] and the proposed method, for Lena image are illustrated in Fig. 6. It demonstrates that the proposed method converges to a satisfactory solution after about 2-4 iterations in terms of SNR and PSNR and after about 4-5 iterations based on BMR [12]. In Fig. 6, there is an abrupt change at the third iteration for the

CP(k). This abrupt change is due to smoothing the image again

such that the blurring strength of BMR is increased and the PSNR is decreased. To avoid oversmoothing is, thus, to exclude the smoothing operators out of the CP iterations.

In addition, the details like at the left hat and the hair areas are more similar to the original image in Fig. 7. The recovery of the Ken image demonstrates the balance of the blocking strength and the blurring effects by the new CP method. This can be shown by the clearness of the objective edges and the smoothness of the flat areas. All these environments indicate the feasibility of the new method.

IV. CONCLUSIONS

This paper has presented a new postprocessing based on the CP theory. The new method is distinct in the initial processing and the convex iteration. For the initial processing, the decoded image was smoothed according to the space-domain psychovisual effects of the image. The perceptual-based initialization has been shown to provide an initial image for a

(5)

better condition to converge to the final image. For the convex iteration, this paper has presented four convex sets and the associated operators. Because the visibility thresholds were introduced into the edge preserving set, the final image which fits with all constraints simultaneously can be claimed to be the perceptually optimized or acceptable reconstruction. The simulations have shown that the iterations converge to the desired image in a very efficient way. By comparing with the existing methods, all the environments indicate the feasibility of this new method.

REFERENCES

[ l ] B. Ramamurthi and A. Gersho, “Nonlinear space-variant postprocessing of block coded images, ‘I IEEE Transactions

on Acoustics, Speech, and Signal Processing, pp. 1258-1268,

vol. ASSP-34, no. 5 , Oct. 1986

[2] Yo-Sung and A . Gersho, “Contour-based postprocessing of coded images, ” Visual Communications and Image

Processing IV, 1989

[3] W. E. Lynch, A. R. Reibman and B. Liu, “Postprocessing transform coded images using edges”, in Proc JCASSP’95, [4] J. Luo, C. W. Chen, K.. J. Parker and T. S. Huang, “A new method for block effect removal in low bit-rate image compression, ” in Proc. ICASSP‘ 94, pp. V-341-344, 1994.

[5] H. C. Reeve and J. S . Lim, “Reduction of blocking effects in image coding, ” Optical Engineering, vol. 23, no. 1,

Jan./Feb. 1984.

[6] T. S. Liu and L. W. Chang, “An adaptive temporal-spatial filter for MPEG decoded video signals, ” Multidimensional

Systems and Signal Processing, pp.25 1-262, vo1.6, 1995

[7] J. M. Boyce,“Noise Reduction of Image Sequence Using Adaptive Motion Compensated Frame Averaging, ” in Proc.

[8] J. K . Su and R. M. Mersereau, “Postprocessing for artifact

reduction in JPEG-compressed images, ” in Proc ICASSP ’95,

[9] Y. Yang, N. P. Galatsanos, and A. K. Katsaggelos, “Iterative projections algorithms for removing the blocking artifacts of block-DCT compressed images, ” in Proc

[lo] S. Minami and A. Zakhor, “An optimization approach for removing blocking effects in transform coding, “ IEEE

Transactions on Circuits and Systems for Video Technology,

pp. 74-82, vo1.5, no.2. April, 1995

[ 1 11 R. Horng and A. J. Ahumada, “A fast DCT blocksmoothing algorithm, ” SPIE Visual Communication, 1995.

[12] C. M. Liu , J. Y. Lin, K. G. Wu, and C. N. Wang, “Objective image quality measure for block-based DCT coding, ‘‘ IEEE Transactions on Consumer Electronics, pp.

51 1-516, vo1.43, no.3, 1997.

[ 131 R. L. Stevenson, “Reduction of coding artifacts in transform image coding, ” in Proc ICASSP ’93, (Minneapolis, MN), pp. pp. 2323-2326, 1995. ICASSP ’92, ~01.111, pp.461-464. PP. 2363-2366, 1995. ICASSP’93, pp. V-405-408, 1993. V-40 1-404,1993 (a) BMR=[Blk, Blur] 33.InnI 1 *,.I“”, 16.01111 1 2 1 1 1 6 itC,%“Cm“ (b) SNWPSNR

Fig. 6. The illustration of the performance on convergence rates for two images with quantization table scaled by 3.

[14] Y. F. Hsu, and Y. C. Chen, “A new adaptive separable median filter for removing blocking effects, “ IEEE

Transactions on Consumer Electronics, pp.5 10-5 13, vo1.39,

no.3, Aug. 1993.

[15] A. Zakhor, “Iterative procedures for reduction of blocking effects in transform image coding, “ IEEE Transactions on

Circuits andsystems for Video Technology, pp. 91-95, v01.2,

no.1, Mar. 1992.

[16] S. J. Reeves, and S. L. Eddins, “Comments on “Iterative procedures for reduction of blocking effects in transform image coding”, ’’ IEEE Transactions on Circuits and Systems for Video Technology, pp. 439-440, vo1.3, no.6, Dec.

1993.

[17] Y. Yang, and N. P. Galatsanos, and A. K. Katsaggelos,

“ Regularized reconstruction to reduce blocking artifacts of

block Discrete Cosine Transform compressed images, “IEEE

Transactions on Circuits and Systems for Video Technology,

pp. 421-432, vo1.3, no.6, Dec. 1993.

[ 181 D. C. Y oula, “Generalized image restoration by the method of altemating orthogonal projections, ‘‘ IEEE Transactions

on Circuits and Systems for video Technology, pp. 694-702,

vol. 2 5 , Sept. 1978.

[19] D. C. Youla, and H. Webb, “Image restoration by the method of convex projections: Part 1 - Theory, “ IEEE

Transactions on Medical Imaging, pp. 81-94, vol. 1, no. 2,

Oct. 1982.

[20] C. H. Chou and Y. C. Li, “A perceptually tuned subband image coder based on the measure of just-noticeable- distortion profile,” IEEE Trans. Circuits and Systems for

(6)

Liu, Wang and Lin: A New Postprocessing Method for the Block-Based DCT Coding Based on the Convex-Projection Theory ₁₀₅₉

(a) The original image _{(b)UnProcessed~dB}₌_{(25.67, 30.84 ,[12.94,}_0.001)

(C) LPu,, = (25.68, 30.85 ,[O.OO, 12.121) (d) SVLpu, =(26.01, 31.18 ,[8.51, 4.481)

(e) CPU,, = (26.31, 31.48 ,[6.52, 5.751) (f)NEWCPu,, = (26.18, 31.35 ,[5.95, 3.371) Fig. 7. The recovered images of the decoded image named as Lena by JPEG with quantization table scaled by 3. The comparisons of the postprocessing methods is based on the measures SNR, PSNR, and BMR[12].

(7)

(a) The original image (b)UW'ocessedu,, = (28.03, 32.37 ,[10.75, 0.001)

(C) Lpudn = (26.52, 30.86 ,[O.OO, 5.251) (d) SVLpu,, = (27.67, 32.02 ,[7.29, 0.891)

(e) cpudB = (28.63, 32.97 ,[7.40, 0.841) ( f ) NEWCPu,, = (28.48, 32.82 ,[5.12, 0.611) Fig. 8.

by 3.

The recovered images of the decoded image named as Ken by JPEG with quantization table scaled The comparisons of the postprocessing methods is based on the measures SNR, PSNR, and BMR[12].

(8)

Liu, Wang and Lin: A New Postprocessing Method for the Block-Based DCT Coding Based on the Convex-Projection Theory 1061

Chi-Min Liu received the B.S. degree in electrical engineering from Tatung Institute of Technology, Taiwan, R.O.C. in 1985, and the M.S. degree and Ph.

D.

degree in electronics from National Chiao Tung University, Hsinchu, Taiwan, in 1987 and 199 1, respectively.

He is currently an Associate Professor of the Department of Comouter Science and Information Engineering, National Chiao Tung University, Hsinchu, Taiwan. His research interests include video/audio compression, speech recognition, radar processing, and application-specific VLSI architecture design.

and video compression.

Chung-Neng Wang was born in Pingtung, Taiwan, in 1972. He graduated from the Department of Computer Science and Information Engineering, National Chiao Tung University (NCTU), Taiwan, R.O.C., in 1994. He is now pursuing his ph.D. degree in the Institute of Computer Science and Information Engineering, NCTU. His current research interests are videohmage

Jine-Yi Lin graduated from the Department of Computer Science and Information Engineering, National Chiao Tung University (NCTU), Taiwan, R.O.C., in 1995. He received his M.S. degree in the Institute of Computer Science and Information Engineering, NCTU. He is currently in military service. His current research interests are image