Optimal tilings for image and video compression

(1)

OPTIMAL TILINGS FOR IMAGE AND VIDEO COMPRESSION K.-L. Hua, I. Pollak, and M. Comer

Purdue University

School of Electrical and Computer Engineering West Lafayette, IN 47907

ABSTRACT

We build on our prior work on best basis selection algorithms in large tree-structured dictionaries, previously used to con- struct effective image coders based on block and lapped trans- forms, as well as to improve the motion compensation step in video compression. In the present paper, we use such al- gorithms to select the optimal transform tiling for video com- pression. We illustrate the effectiveness of this approach by showing that our tiling selection method, combined with our optimal tiling selection for the motion compensation stage, re- sults in up to 30.5% savings in bit rate as compared to the stan- dard H.264/AVC tiling selection, for several standard video se- quences.

1. INTRODUCTION

Algorithms for best basis search in tree-structured dictionaries have been effectively used for many signal processing prob- lems, including noise removal and other estimation tasks [3, 8, 11], image compression [5, 6, 9, 15], audio compression [13, 14], and image segmentation [12]. An important class of best basis algorithms are methods that search for an optimal rect- angular tiling of an image [1, 2, 5, 6, 9, 15]. Such methods can signiﬁcantly improve image coding strategies based on block or lapped transforms, through adapting the sizes and shapes of transform blocks to the structure of an image. The utility of us- ing variable block sizes in video compression has recently been recognized, as well. For example, the H.264/AVC video com- pression standard [10] incorporates variable block sizes both in the motion compensation stage [16] and in the transform stage [17].

In a previous paper [4], we demonstrated that, by using optimal search over two larger dictionaries of tilings than the one allowed in the motion compensation stage of H.264/AVC, compression performance can be improved. In the present paper, we show that compression performance can be further improved by combining this with the selection of the optimal

This work was supported in part by a National Science Foundation (NSF) CAREER award CCR-0093105, an NSF grant IIS-0329156, and a MediaTek fellowship.

tiling for the transform stage. We use the efficient tiling algo- rithm of [6] to determine the tilings for motion compensation and transform which minimize a rate-distortion cost. Our ex- periments show bit rate reductions of up to 30.5% at typical PSNRs for several video sequences commonly used to evalu- ate the performance of video coders. This is a significant result since, in general, the use of finer tilings results in more over- head bits and therefore may increase the overall bit rate. We show, however, that our optimal tiling search is so effective that the increase in the number of overhead bits is more than com- pensated for by the reduction in the number of bits required to encode the motion vectors and transform coefficients.

The rest of this paper is organized as follows. Section 2 overviews the tiling selection for the motion compensation stage in H.264/AVC and our improved tiling selection of [4]. In Section 3 we introduce optimal tiling selection method for the transform stage. Section 4 presents experimental results.

2. BACKGROUND 2.1. Motion Compensation in H.264/AVC

There are two basic ways of encoding a frame of video: intra

mode and inter mode. Intra-frame compression only uses the

information contained within the current frame. Inter-frame

mode predicts the current frame from one or several reference

frames, and encodes the error between the predicted frame and

the actual one. The ability to accurately predict the current

frame is therefore crucial to the success of inter-frame com-

pression. The prediction is typically done by motion estima-

tion methods which partition a frame into blocks and match

every block with a similar block in the reference frame. Con-

ventional methods use square blocks of ﬁxed size. The choice

of the block size is problematic for such methods: if the size

is too small, too many bits are spent on encoding the motion

vectors; whereas if it is too large, the prediction of complicated

motion sequences is poor. Generally speaking, areas with no

motion should use large block sizes and areas with complicated

motion should use small block sizes. Therefore, H.264/AVC

incorporates variable block sizes in the motion compensation

(2)

stage.

Speciﬁcally, H.264/AVC allows partitioning each 16 × 16 macroblock into rectangular subblocks and performing motion estimation separately for each subblock. The speciﬁc tilings allowed by the standard are shown in Figs. 1 and 2. One of the four tilings in Fig. 1 may be used for a 16 × 16 macroblock. If the tiling of Fig. 1(d) is used, then each of the four 8 × 8 sub- macroblocks can be further tiled using the four tilings of Fig. 2.

The encoder in the H.264/AVC reference software [7] selects a tiling for each macroblock based on a cost function. Given a motion vector v for such a subblock, [7] defines the distor- tion d(v) as the sum, over all the pixels in the subblock, of the absolute differences between the pixel luminance and the luminance of the corresponding pixel in the reference frame, for integer pixel search; and the sum of absolute transformed differences for fractional pixel search. The rate r(v) is de- fined as the number of bits required to encode the difference between the motion vector and the predicted ¹ motion vector, as tabulated in [7]. The cost of a motion vector v is defined as

d(v) + λ m r(v), where the Lagrange multiplier λ m is

λ m =

0.85 · 2 ^{(QP −12)/3} , (1) and QP is the quantization parameter.

After determining the optimum motion vectors for each subblock, the cost of each tiling of a macroblock into sub- blocks is deﬁned as a weighted sum of the tiling’s distortion D and rate R: ^COST ≡ D + λR. Both the distortion and the rate are assumed to be additive over the subblocks in the tiling, i.e., it is assumed that the cost can be calculated as follows:

COST =

P

(D P + λR P ), (2)

where D P and R P are the distortion and rate, respective, for the subblock P , and the summation is taken over all subblocks in the tiling. Following [7], we define the distortion D P to be the sum of the squared differences between the pixels ² in the subblock P and its reconstruction. We also define the rate R P as the the sum of three terms: the number of bits for the se- lected tiling, the number of bits for the predicted motion vector error for subblock P , and the number of bits for the quantized transform coefficients for subblock P . Finally, the Lagrange multiplier in Eq. (2) is defined as λ ≡ λ ² _m where λ m is given by Eq. (1).

In our implementation of an H.264/AVC-compliant tiling scheme, we also follow [7]: the costs of the four tilings of Fig. 1 are evaluated; if one of the tilings (a), (b), or (c) has the lowest cost, this tiling is selected; if (d) has the lowest cost, the possibilities shown in Fig. 2 are evaluated for each of the four 8 × 8 sub-macroblocks. It is easily seen that the dictionary of tilings that this scheme selects from, contains a total of 3 +

1

A motion vector for a block is predicted based on the neighboring blocks, and then the difference between the prediction and the actual motion vector is encoded.

2

Here, each pixel value is motion-compensated, i.e., it is the difference between the pixel value in the original frame and its prediction.

16 16

8 8

16 16

8 8

(a) (b) (c) (d)

Fig. 1: Possible tilings of a 16 × 16 macroblock for the H.264/AVC motion estimation and compensation.

8

8 4

4

8 4 4

8

4 4 4 4

Fig. 2: Possible tilings of a 8 × 8 sub-macroblock for the H.264/AVC motion estimation and compensation.

4 ⁴ = 259 tilings. Note also that the procedure that selects a tiling out of these 259 tilings is not globally optimal, since the global minimum of the cost (2) may be achieved by a tiling which is ﬁner than (d) even if tiling (a), (b), or (c) has a lower cost than (d).

2.2. Optimal Tiling Algorithm

In [4], we proposed to select globally optimal tilings from large dictionaries, using the algorithm developed in [6]. The pro- posed dictionaries are not H.264/AVC-compliant but result in improved compression ratios. We presently review the opti- mal tiling algorithm of [6], and then discuss its adoption to our motion compensation and transform tasks.

The algorithm constructs optimal tree-structured tilings of a rectangular image domain into smaller rectangles via recur- sive bipartitioning. During this process, a rectangle may ei- ther be used as a single tile or split further into two subrect- angles. For example, the tiling of Fig. 3(a) may be obtained through such a recursive bipartitioning process, as illustrated in Fig. 3(c,d). In Fig. 3(c,d), a vertical (horizontal) line through a tree node signiﬁes a vertical (horizontal) split of the corre- sponding rectangle into two subrectangles. Note that in this case, two different trees correspond to the same tiling. On the other hand, the rectangular tiling of Fig. 3(b) cannot be ob- tained through such a recursive binary splitting process.

Referring back to Fig. 3(c,d), we point out that in some applications the important object is the tiling produced by the leaves of the tree, and thus there is no distinction between the different trees that may have produced the tiling. In our ap- plication, however, the various tilings will be encoded by en- coding the corresponding trees and therefore we will select the tree that corresponds to the most efﬁcient encoding. This moti- vates deﬁning the following cost function for a tree with leaves P 1 , . . . , P d and intermediate nodes Q 1 , . . . , Q d−1 :

C(tree) =

d i=1

e(P i ) + ^d−1

i=1

s(Q i ), (3)

where e and s are cost functions for individual tiles and inter- mediate tree nodes, respectively.

The smallest tile size that we use is 4 × 4. We deﬁne two

dictionaries of trees: the dyadic dictionary and the multitree

(3)

(a) A tree-structured tiling. (b) A non-tree-structured tiling.

(c) A tree of splits. (d) Another tree of splits.

Fig. 3: An illustration of tilings and trees of splits. (a) An admissible tiling—

i.e., a tiling that can be obtained via recursive binary splitting. (b) An inadmis- sible tiling. (c) A tree of splits that leads to the tiling in (a). (d) Another tree of splits that leads to the tiling in (a).

8 4 4

16 8 8

8 8

16 4 8 4

4 12

12 4 Dyadic tiling Multitree tiling

Fig. 4: Example of possible dyadic and multitree tilings.

dictionary. The former contains all trees that produce tilings by only splitting rectangles in the middle, horizontally or ver- tically. The latter allows splits at arbitrary locations that are multiples of four. A tiling from the dyadic dictionary and a tiling from the multitree dictionary are illustrated in Fig. 4.

An efficient algorithm for finding the globally optimal tree is developed in [6]. Specifically, this algorithm performs the following bottom-up optimization:

C _P ^∗ = min{e(P ), min(C _P ^∗

+ C _P ^∗

) + s(P )}, (4) where C _P ^∗ is the cost of the optimal tree with root P , and the inner minimization is performed over all pairs of subblocks P , P which partition P . This efﬁcient search algorithm ex- ploits the fact that although the number of possible trees and tilings is very large, the number of rectangular tiles is much smaller and manageable, as shown in Table 1.

In [4], when adopting the optimal tiling algorithm to the motion compensation stage, cost function e(P _i ) of Eq. (3) is deﬁned as Eq. (2) with R _P

_i

being the sum of the number of bits for the predicted motion vector error for subblock P i , and the

H.264/AVC Dyadic Multitree number of tilings 259 6857 68480

number of tiles 41 49 100

Table 1: Number of tilings and tiles

Original H.264/AVC Dyadic Multitree

Fig. 5: Example of foreman sequence with different tiling selection methods for the motion compensation stage.

0 100 200 300

30 35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

Multitree Dyadic Fixed 4x4

30 35 40 45

100 105 110 115 120

Y−PSNR [db]

RATE, % OF THE MULTITREE RATE

Multitree=100%

Dyadic Fixed 4x4

0 50 100 150 200 250

30 35 40 45 50

Bit Rate [Kbps]

Y−PSNR [db]

35 40 45

100 105 110 115 120 125

Y−PSNR [db]

Multitree=100%

Dyadic Fixed 4x4

0 50 100 150 200 250

35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

34 36 38 40 42 44 46

100 110 120 130

Y−PSNR [db]

Multitree=100%

Dyadic Fixed 4x4

Fig. 6: The rate-distortion curves for compression with motion compensation using the H.264/AVC tiling scheme (dashed lines with triangles) in conjunc- tion with ﬁxed 4 × 4 transform, and our optimal dyadic and multitree trans- forms. The right column shows bit rates as percentages of the bit rate for the multitree algorithm. The results are shown for frames 100 through 210 of

“foreman” (top), “carphone” (middle), and “highway” (bottom).

number bits for the quantized transform coefﬁcients for sub- block P i 3 ; the cost function s(Q i ) is deﬁned as λR i , where R i

is the number of bits for encoding the split of the rectangle Q i . Fig. 5 shows an example of the different tiling selection methods for the motion compensation stage.

3. OPTIMAL TRANSFORM TILINGS

In this section, we will ﬁrst overview the transform stage in H.264/AVC and then explain how to apply the optimal tiling algorithm described above to extract the best transform tiling.

3.1. Transform Stage for H.264/AVC

Typically, a ﬁxed 4×4 integer transform is used in H.264/AVC encoders. Adaptive block size transforms (ABT) are also sup- ported by H.264/AVC. The basic idea of ABT is to associate

3

This quantity is estimated using a histogram extracted from training data.

(4)

the block size of the transform matrix with the block size of motion compensation subblocks. In general, for a rectangular m × n image B

m,n

, the transform C

m,n

is deﬁned by

C

_m,n

= T

_v

· B

_m,n

· T

_h^T

, (5) where T

_v

and T

_h

are the m × m and n × n transform matri- ces in the vertical and horizontal directions, respectively. The maximum allowed block size in H.264/AVC is 8 × 8. If larger subblocks (i.e., 16 × 8, 8 × 16, and 16 × 16) are encoded, 8 × 8 transform matrix is used.

3.2. Optimal Transform Algorithm

We improve the compression algorithm further by using dyadic or multitree dictionary of tilings for the transform stage. We extract the optimal sub-tiling for each motion compensation subblock into transform subblocks, and jointly optimize the tiling for the motion compensation stage and the sub-tiling for the transform stage. Note that we use the 2D-DCT for all block size transform matrices in this transform.

For each motion compensation subblock, we select the sub- tiling from the corresponding dictionary that minimizes

COST

_T

=

^k

q=1

(D

q

+ λR

q

) +

^k−1

s=1

λR

s

, (6)

where the ﬁrst summation is over all the transform subblocks (T-subblocks) q in the tiling, D

_q

is the sum of the squared dif- ferences (SSD) for the T-subblock q, λ is the Lagrange multi- plier, R

_q

is the number of bits for encoding the quantized trans- form coefﬁcients for the T-subblock q, and R

_s

is the number of bits to encode the selected transform tiling.

The set of all valid dyadic transform splits is the set of all splits of a valid T-subblock into 2 congruent valid T-subblocks.

When using a multitree dictionary at the motion compensation, two extra types of splits are allowed in the dyadic transform dictionary:

• For a 12 × m T-subblock, the horizontal split of it into 8 × m and 4 × m T-subblocks where m is 4, 8, 12, or 16.

• For a m×12 T-subblock, the vertical split of it into m×8 and m × 4 T-subblocks where m is 4, 8, 12, or 16.

4. EXPERIMENTAL RESULTS

The coding algorithms are evaluated using Foreman, Carphone, and Highway video sequences with 110 frames and ± 16 search range[QCIF ( 176 × 144), luminance only]. We use a 15-frame GOP with coding pattern IPPPPPPPPPPPPPP. To focus on the performance with our algorithm adopting to inter-frames, the rate and PSNR are only computed for the luminance compo- nent of inter-frames.

Experiment 1: H.264/AVC Motion Compensation Tiling with Dyadic and Multitree Transforms. In this experiment, conventional H.264/AVC tree-structured motion compensation

0 100 200 300

30 35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

Multitree Dyadic Fixed 4x4 Fixed 4x4(H.264)

30 35 40 45

100 110 120 130

Y−PSNR [db]

Multitree=100%

Dyadic Fixed 4x4 Fixed 4x4(H.264)

0 50 100 150 200 250

35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

35 40 45

100 110 120 130

Y−PSNR [db]

Multitree=100%

0 50 100 150 200

34 36 38 40 42 44 46

Bit Rate [Kbps]

Y−PSNR [db]

34 36 38 40 42 44 46

90 100 110 120 130 140 150

Y−PSNR [db]

Multitree=100%

Fig. 7: The rate-distortion curves for compression with motion compensa- tion using the dyadic tiling scheme in conjunction with ﬁxed 4 × 4 transform, and our optimal dyadic and multitree transforms. Our baseline is an H.264- compliant scheme which uses ﬁxed 4 × 4 transform (dotted lines with tri- angles). The right column shows bit rates as percentages of the bit rate for the multitree algorithm. The results are shown for frames 100 through 210 of

“foreman” (top), “carphone” (middle), and “highway” (bottom).

is used. We evaluate the coding efﬁciency of our implemented H.264/AVC and proposed algorithms adopted to the transform stage. Fig. 6 shows the rate-distortion curves for three test se- quences. The second row of the ﬁgure shows the rate-distortion curves for three schemes with the bit rates displayed as per- centages of the multitree bit rate.

When we apply our optimal tiling algorithm to the trans- form stage, the running times of our dyadic and multitree al- gorithms are only about a factor of 1.15 and 1.19, respec- tively, slower than our implementation of the H.264-compliant scheme.

Experiment 2: Dyadic Motion Compensation Tiling with Dyadic and Multitree Transforms. In this experiment, dyadic motion compensation is used. Our baseline is an H.264-compliant scheme which uses fixed 4×4 transform. We evaluate the cod- ing efficiency of the proposed algorithms adopted to the com- bination of motion compensation and transform stages. Fig. 7 shows the rate-distortion curves for the test sequences. The second row of the figure shows the rate-distortion curves for four schemes with the bit rates displayed as percentages of the multitree bit rate.

Experiment 3: Multitree Motion Compensation Tiling

with Dyadic and Multitree Transforms. In this section, mul-

titree motion compensation is used. Our baseline is an H.264-

(5)

0 100 200 300 30

35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

30 35 40 45

100 110 120 130 140

Y−PSNR [db]

Multitree=100%

0 50 100 150 200 250

35 40 45

Bit Rate [Kbps]

Y−PSNR [db]

35 40 45

100 110 120 130 140

Y−PSNR [db]

Multitree=100%

0 50 100 150 200

34 36 38 40 42 44 46

Bit Rate [Kbps]

Y−PSNR [db]

34 36 38 40 42 44 46

100 120 140 160

Y−PSNR [db]

Multitree=100%

Fig. 8: The rate-distortion curves for compression with motion compensation using the multitree tiling scheme in conjunction with ﬁxed 4 × 4 transform, and our optimal dyadic and multitree transforms. Our baseline is an H.264- compliant scheme which uses ﬁxed 4 × 4 transform (dotted lines with tri- angles). The right column shows bit rates as percentages of the bit rate for the multitree algorithm. The results are shown for frames 100 through 210 of

“foreman” (top), “carphone” (middle), and “highway” (bottom).

compliant scheme which uses ﬁxed 4×4 transform. We evalu- ate the coding efﬁciency of the proposed algorithms adopted to the combination of motion compensation and transform stages.

Fig. 8 shows the rate-distortion curves for the test sequences.

The second row of the ﬁgure shows the rate-distortion curves for four schemes with the bit rates displayed as percentages of the multitree bit rate.

5. CONCLUSION

We developed an algorithm to search for the optimal transform in a dictionary of transforms and applied it to the transform stage of a video coder, demonstrating improvements in video compression performance.

6. REFERENCES

[1] N. N. Bennett. Fast algorithm for best anisotropic Walsh bases and rela- tives. J. of Appl. and Comput. Harmonic Analysis, 8(1):86–103, Jan. 2000.

[2] D. L. Donoho. CART and best-ortho-basis: A connection. Ann. Stat., 25(5):1870–1911, Oct. 1997.

[3] D. L. Donoho and I. M. Johnstone. Ideal denoising in an orthonormal basis chosen from a library of bases. Comptes Rendus Acad. Sci., Ser. I, 319:1317–1322, 1994.

[4] K.-L. Hua, I. Pollak, and M. Comer. Optimal image tilings with applica- tion to video compression. In Proc. ICIP, Atlanta, Georgia, Oct. 2006.

[5] Y. Huang and I. Pollak. MLC: A novel image coder based on multitree lo- cal cosine dictionaries. IEEE Sig. Proc. Lett., 12(12):843-846, Dec. 2005.

[6] Y. Huang, I. Pollak, M.N. Do, and C. A. Bouman. Fast search for best representations in multitree dictionaries. IEEE Trans. Im. Proc., 15(7), Jul. 2006.

[7] JVT reference software version 10.2.

iphome.hhi.de/suehring/tml/download/jm10.2.zip [8] H. Krim and J.-C. Pesquet. On the statistics of best bases criteria. In

A. Antoniadis, editor, Wavelets and statistics, Lecture Notes in Statistics, pages 193–207. Springer-Verlag, 1995.

[9] R. Leonardi and M. Kunt. Adaptive split-and-merge for image analysis and coding. In Proc. SPIE, volume 594, pages 2–9, Dec. 1985.

[10] A. Luthra, G.J. Sullivan, and T. Wiegand, Eds. Special Issue on the H.264/AVC Video Coding Standard. IEEE Trans. Ckts. Syst. Vid. Tech., , 13(7), Jul. 2003.

[11] P. Moulin. Signal estimation using adapted tree-structured bases and the MDL principle. In Proc. IEEE-SP Int. Symp. TFTS, pages 141–143, Paris, Jun. 1996.

[12] U. Ndili, R. D. Nowak, and M. A. T. Figueiredo. Coding theoretic ap- proach to image segmentation. In Proc. ICIP-2001, Thessaloniki, Greece, Oct. 2001.

[13] O.A. Niamut and R. Heusdens. Flexible frequency decompositions for cosine-modulated ﬁlter banks. In Proc. ICASSP-2003, Hong Kong, Apr. 2003.

[14] O.A. Niamut and R. Heusdens. RD optimal time segmentations for the time-varying MDCT. In Proceedings of European Signal Processing Con- ference (Eusipco), Vienna, Austria, Sep. 6-10, 2004.

Optimal tilings for image and video compression