FAST CLOSEST CODEWORD SEARCH ALGORITHM FOR VECTOR QUANTIZATION

(1)

Fast

closest codeword search algorthm for vector

quantisation

C.-H. Lee

L.-H. Chen

Indexing terms: Codebook design, Mean value, Variance, Vector quantisation

Abstract: One of the most serious problems for vector quantisation is the high computational complexity of searching for the closest codeword in the codebook design and encoding phases. The authors present a fast algorithm to search for the closest codeword. The proposed algorithm uses two significant features of a vector, mean value and variance, to reject many unlikely codewords and saves a great deal of computation time. Since the proposed algorithm rejects those codewords that are impossible to be the closest codeword, this algorithm introduces no extra distortion than conventional full search method. The results obtained confirm the effectiveness of the proposed algorithm.

1 Introduction

Vector quantisation (VQ) is a very efficient approach to low-bit-rate image compression [l]. It is defined as a mapping Q from a k-dimensional Euclidean space Rk to a finite subset Y of

Rk.

That is

Q : R ~ - + Y

where Y = { y i

I

i = 1, 2,

. . .

, N } is called the codebook, and N is the size of the codebook. Each yi = (yil. y i 2 ,

. .

.

,

y i x ) in Y is called a codeword. For the compression purpose, a VQ consists of three phases: codebook design phase, encoding phase and decoding phase. The objective of codebook design is to find a codebook Y which con- tains the most representative codewords. This codebook will be used by encoder and decoder. In the encoding phase, the encoder designs a mapping Q and assigns an index i to each input vector x = (xlr x 2 , ..., .xk) with Q(x) = y i

.

In this paper, we will consider the mapping

Q,

which is designed to map x to y i with y i satisfying the following condition

(1) where d2(x, y,) is the distortion of representing the input vector x by the codeword y,, as measured by the squared

d2(x, y i ) = min dZ(x, y,) f o r j = 1, 2,

. . .

, N

j

0 IEE, 1994

Paper 1140K (E4), first received 15th February 1993 and in revised form 5th January 1994

The authors are with the Department of Computer and Information Science, National Chiao Tung University, 1001 Ta Hsueh Road, Hsinchu, Taiwan 30050, Republic of China

I E E Proc.-Vis. Image Signal Process., Vol. 141, N o . 3, June 1994

Euclidean distance, i.e.

k

When encoding an image, the encoder first divides the image into several blocks, usually in square. Each block x containing k entities is considered as a k-dimensional vector. Hence, for each input vector x (i.e. a block), the encoder only needs to transmit or store the index i assigned to x. The decoder has the same codebook as the encoder. In the decoding phase, for each index i, the decoder merely performs a simple table look-up oper- ation to obtain y i , and then uses y i to represent the input vector x.

From the above description, we see that the compres- sion ratio is determined by the codebook size and the vector dimension; the distortion is dependent on the codebook size and selection of codewords. Therefore, designing a good codebook is the main task of VQ. Many algorithms for codebook design have been proposed [l-31. Among these algorithms, the most popular one was developed by Linde, Buzo and Gray [I, 21 and is referred to as the LBG algorithm. This algorithm iter- atively minimises the total distortion of representing the training vectors by their corresponding codewords. It divides the training vectors into several classes, each class is represented by the centroid of that class. The set of all centroids forms a codebook, and each centroid is referred to as a codeword. The algorithm is iterative, it first takes an initial codebook Yo with predetermined size N and then starts iterating. In each iteration, for each training vector x, it searches the current codebook exhaustively to find the closest codeword y i and assign x to class i. After all training vectors have been classified, the distortion between the set of training vectors and their corresponding codewords is calculated. The distortion difference between the current iteration and the previous iteration is then calculated. The ratio of the distortion difference to the distortion of the current iteration is checked to see if it is greater than a preset value E. If it is true, yi is then

replaced by the centroid of the new class i, and a new iteration starts; otherwise, the algorithm stops. Note that the LBG algorithm uses a full codebook search to find the closest codeword for each training vector. If the codebook size is N , the full codebook search (i.e. to

This research was supported in part by the National Science Council of ROC under contract NSC-82-0408-E009-063

(2)

evaluate eqns. 1 and 2, requires Nk multiplications, N(2k - 1) additions, and N - 1 comparisons for each k- dimensional training vector).

As mentioned above, N determines the accuracy of VQ. The larger

N

is, the more accurate VQ. However, when N is large, the computational complexity problem for full codebook search will occur. This problem is crit- ical in the codebook design and encoding of VQ. To avoid such an exhaustive search through the codebook, many fast algorithms [4-101 have been proposed. These algorithms reduce the computational complexity by first performing some simple tests before computing the distortion between the training vector and each codeword, and then rejecting those codewords which fail in the tests. In the next Section, we will briefly review some of these algorithms.

2

2.1 Partial distortion elimination algorithm

The partial distortion elimination (PDE) algorithm [4] allows early termination of the distortion calculation between a training vector and a codeword by introducing a premature exit condition in the search process. For each training vector x, the algorithm first calculates the distortion between x and an arbitrary codeword and takes this distortion as the current minimum distortion

dlin.

Then, for any other codeword y i = (yil, yi,,

.. .

, yik), if there exists q < k with the accumulated distortion for the first q samples in eqn. 2 larger than the current minimum distortion d:, , i.e.

S o m e existing fast closest codeword search algorithms

4

c

(x. - Y i J 2 2

dli"

" = l

this algorithm stops computing the distortion for code- word yi and begins trying the next codeword. This will reduce (k - q ) multiplications and 2(k - q) additions. Simulations [4] indicate that the PDE method can reduce a good number of multiplication operations and addition operations in the search process, and only increases some comparison operations.

2.2 Partial search partial distortion algorithm

The partial search partial distortion (PSPD) algorithm [ 5 ] builds up a partial codebook based on the mean value m, of a k-dimensional training vector x = (xl, x 2 ,

. . .

, xk), in which m, is defined as

1

m, = integer part of

I'

-

1

xj

+

0.5 k j = l

The algorithm then uses the PDE method to search the partial codebook for the closest codeword.

The PSPD algorithm first calculates the mean values of all codewords and sorts the codebook according to increasing order of the codeword means. For each training vector, it then finds the codeword y p with minimum mean difference to the training vector. The codewords with mean differences to y, less than a predetermined threshold T form the partial codebook. The PDE method is then employed to find the closest codeword in this partial codebook. Experimental results [6] show that the execution time of the PSPD algorithm is about 12% of that required by the LBG algorithm. However, some- times the closest codeword may not be located in the partial codebook, this will introduce more distortion than the LBG algorithm.

144

2.3 Fast nearest neighbour search algorithm

The fast nearest neighbour search (FNNS) algorithm [6] uses the triangle inequality to reject a great many unlikely codewords. For a vector x, it first finds a prob- ably nearby codeword y, with distortion d2(x, y,). This algorithm then eliminates those codewords which are impossible to be the closest codeword, based on the triangle inequality and a precomputed table which contains the distances of all pairs of codewords. That is, for each codeword y j , if

d ( y j , Y I ) > 24x,

Y O

through the triangle inequality, we have

d(x, ~ j

+

)d ( x ,

~3

2 d b j , Y,) > 24x, Y I )

The above inequality can be reduced to be 4x9 ~ j> )4x9 Y,)

Therefore, those codewords with distances to y, larger than 2d(x, y,) will be eliminated from consideration to be a candidate of the closest codeword. Simulations show that high saving rate over conventional full codebook search method can be achieved. However, this algorithm requires a table of size N2/2 to store the distances of all pairs of codewords. When N is large, the memory requirement is a serious problem.

2.4 Equal-average nearest neighbour search

The closest codeword search problem in vector quantisation is a nearest neighbour search (NNS) problem which can be stated as: given a set Y, of N prototypes in a k-dimensional space

Rk,

for a query point x E Rk, deter- mine which prototype is closest to x. Guan et al. [7] proposed an equal-average nearest neighbour search (ENNS) algorithm which uses hyperplanes orthogonal to the central line I to partition the search space. Each coor- dinate value of any point p = (pl, p 2 ,

. . .

, p 3

on I has the same value (i.e. pi = p j . i, j = 1, 2,

. ..

, k). Each point on a fixed hyperplane

H,

which is orthogonal to the central line I and intersects l at point

L,

= ( m H , m,,

. . .

, m,), will have the same mean value m,, such a hyperplane is called an equal average hyperplane. For an input vector x = (xl, x 2 ,

. . .

, xk), the algorithm first calculates its mean value m, with m, = (l/k) xj. The algorithm then finds the codeword y, which has the minimum mean difference to x and calculates the distance r , between x and y p . It is obvious that any other codeword which is closer to x than y, has to be located inside the hypersphere centred at x with radius r,. By projecting the hypersphere on I, two boundary projection points, L,, = (m,,, m,,,

. . .

, m,,) and Lmin = (mmin,

. .

.

, mmin,

. . .

, mmin) on 1 can be found, where

algorithm

and

(3)

(4)

The hypersphere can be bounded by two equal-average hyperplanes with mean values mW, and mmin. Hence, it is only necessary to search those codewords with mean values ranging from mmin to mWx. Fig. 1 shows the geometric interpretation of the method for a two-

(3)

dimensional case, the search area is bounded by two lines

I ,

and

I,,

which are perpendicular to the central line

I

at L,,, and L,, , respectively. 4 me I Fig. 1 0 input vector x ccdewords

Example of E N N S algorithm for 2-dimensional case

As discussed previously, the ENNS algorithm uses mean value as a feature to reject unlikely codewords and thus saves much of computation time. However, two vectors with the same mean value may have a large distance. For example, one vector represents an almost homogeneous block (i.e. entities in the vector are almost the same); the other represents an edge block (i.e. some entities will be larger than others). The distance between these two types of vectors will be large. To treat this phe- nomenon, we propose a new search method to reduce the search area of the ENNS algorithm. Since the variance of a vector is a simple measure to detect whether a vector is homogeneous, the proposed method uses both the mean value and the variance of a vector as two significant features to search for the closest codeword.

3 The proposed algorithm

In this Section, we will present the proposed algorithm, which uses the mean value and the variance of a vector as two significant features to speed up the closest codeword search process. Before describing the algorithm, we will first give some definitions and a theorem.

Definition 1 : Let x = ( x l , x 2 ,

..

., x k ) be a vector, define the mean value m, of x and the variance Vf as

1 ' k j = 1 m, = -

1

x j and k V: =

1

( x j - m,)' j = 1

Definition 2: Let 1 be a line on Rk, if any point p = (pl, p 2 ,

. . .

, p k ) on I satisfies the following condition

P k

p l = p z = " ' =

Let I be the central line of R' and L, be the projection By the above definitions, we have the following

I is called the central line of R'.

point of x on 1. It can be seen that L, = (m,, m,,

. . .

, m.). theorem.

ZEE Proc.-Vis. Image Signal Process., Vol. 141, N o . 3, June 1994

Theorem 1 : Let x = ( x l , x 2 ,

. . .

, x k ) be a vector and yi =

(yil, yiz ,

. . .

, yik) be a codeword. If the distortion between

x and y i is defined to be the squared Euclidean distance, i.e.

j = 1

then V, = d(x, L,),

_V,i

= d ( y i , Lyi) and

The proof is given in Appendix 7.1. By Theorem 1, we obtain the following corollary immediately.

Corollary 1 : Let x be a vector and dfin be a known current minimum distortion of x represented by a certain codeword. For any codeword y i , if ( V , - 3 d;, , y i will not be the closest codeword of x and it is unneces- sary to calculate d Z ( x , yi).

With the above theorem and corollary in hand, we now turn to describe the proposed algorithm. This algorithm consists of two steps. The first step is the same as the ENNS algorithm, the second is a new one. For a training vector x , the proposed algorithm first calculates the mean value m, and the squared root of the variance V, of x . The algorithm then finds the codeword y, with the minimum mean difference to x , calculates the distance r , between x and y, and sets the current minimum distortion d i i n as r i

.

For each codeword y i , the algorithm checks if myi is between mmin and mmx, where

If the answer is no, codeword yi is rejected without calcu- lating the distortion d2(x, y i ) . Otherwise, the second step is conducted. In the second step, if (V, - V y J 2 2 dfim, the codeword yi is rejected. If y i is not rejected, the distortion

d2(x, yi) is calculated. If d2(x, yi) < d i i n , the current minimum distortion df, is replaced by d2(x, y i ) and mmin and mmx are also updated.

Note that, in the ENNS algorithm, for any codeword y i with myi between mmin and mmxr the distortion d2(x, yi) is always evaluated. In contrast, the proposed algorithm will calculate d2(x, yJ only when (V, -

V,J2

<

This will avoid those codewords with mean values similar to m,, but with variances very different from V s being con- sidered to be the closest codeword of x . Hence, the proposed algorithm can reduce the search area and speed up the search process than the ENNS algorithm. Fig. 2 depicts the geometric interpretation of the proposed method for a two-dimensional case. Comparing Fig. 1 and Fig. 2, we can see that the search area, which is originally an area bounded by two lines I , and I, perpen- dicular to the central line l, has been reduced to be the two shaded squares.

A detailed description of how to apply the proposed algorithm to design a codebook is given below.

S t e p 0: Initialisation: Given N = codebook size, n = the number of training vectors, k = the vector dimension, Yo = initial codebook, E = distortion threshold. Set iteration counter c = 0, initial total distortion Step 1 ; Compute the mean value of each codeword in the codebook

x ,

and sort according to increasing order of the codeword means, i.e. the sorted codebook Y, D - , = CO.

(4)

is y

, = { y i

I

myi

<

my(i+

Step 2: Compute the squared root of the variance

V,;,

1

<

i

<

N - I} of each codeword y i

.

Y \".

/

'

2 v - central line 1 V

-

Fig. 2 0 input vector x 8 ccdewords

Example to illustrate proposed methodfor 2-dimensional case

Step 3 : For each training vector x,, find the closest codeword yict) in the codebook

Y,,

and assign x, to class i(t). The procedure includes the following substeps :

Step 3.1: Input a training vector x, = (xfl, x r Z , _ _ _ ,

x,J, compute its mean value m,, and its square root of variance V,,

.

S t e p 3.2: Find the codeword y, which has the minimum mean difference to x, (using binary search), i.e.

I

m,, - myp

I

<

I

m,, - myi

I

for all i # p Set

and

Step 3.3: Find the closest codeword yi(,) in

E,

and assign x, to class i(t). The search procedure is as follows Set d = 1 while(m,+, < m,,, or m p - d > m,,,)begin if(d2(x,, y p + d ) < d2,)begin if(m,+, < m,,,)begin if((Kt - < ( p + d ) ) 2 < dihlbegin

dk

= d2(xf, Y p + d ) dmin m m o r = mx,

+

- J(k) d m i n m . = m -- m'n xr J(k) i(t) = p

+

d end end

Table 1 : Comparison of execution time (in seconds) for codebook design. Values in parentheses denote the ratio of execution time of the current algorithm t o that of the LEG alaorithm

Codebook Method Design image

size

Lena Peppers Jet Baboon

128 LBG ENNS our method ENNS our method 256 LBG 512 LEG ENNS our method 1024 LBG ENNS our method 281 5 305 (01 08) 266 (0.094) 1 5608 509 (0.091) 383 (0.068) 865 (0.077) 576 (0.051) 1263 22732 1525 (0.067) 925 (0.041) 281 4 284 (0.101) 242 (0.086) 472 (0.084) 347 (0.062) 81 2 (0.072) 525 (0.047) 1464 (0.065) 866 (0.038) 5622 11 263 22589 2668 320 (0.1 20) 269 (0,101) 5334 535 (0.1 00) 392 (0.073) 921 (0.086) 602 (0.056) 1596 (0.074) 960 (0.044) 10709 21 679 2793 847 (0.303) 771 (0.276) 5583 1529 (0.274) 131 3 (0.235) 2798 (0.250) 2292 (0.204) 11212 2261 0 51 48 (0.228) 4001 (0.177)

Table 2: ComDarison of execution time fin seconds) for imaae encoding

Codebook Method Encoded image

size

Lena Peppers Jet Baboon

128 Full search ENNS our method ENNS out method ENNS our method ENNS our method 256 LBG 512 LBG 1024 LBG 140.9 14.1 (0.100) 11.5 (0.082) 23.6 (0.085) 17.2 (0.062) 40.8 (0.073) 26.7 (0.048) 278.6 557.6 11 08.5 71.4 (0.064) 40.7 (0.037) 140.1 15.0 (0.107) 11.9 (0.082) 27.4 (0.098) 18.8 (0.067) 47.9 (0.086) 30.2 (0.054) 89.3 (0.081) 51.1 ((0.046) 279.5 558.1 11 08.5 125.1 14.3 (0.114) 11.4 (0.091) 23.9 (0.096) 17.4 (0.070) 43.4 (0.087) 26.7 (0.053) 79.2 (0.080) 43.6 (0.044) 249.7 499.3 992.7 137.5 37.1 (0.270) 32.2 (0.233) 275.0 69.4 (0.252) 54.1 (0.197) 547.8 130.9 (0.239) 96.2 (0.176) 249.2 (0.229) 174.6 (0.1 60) 1087.9

(5)

end end end{of while}

end

S t e p 4 : Compute the total distortion for the cth iteration

D,

.

Here

D,

is defined to be

S t e p 5 : If (Dc- - D,)/D,

<

E, halt with final codebook

being

Y,,

.

Otherwise, go to Step 6.

S t e p 6 : Compute the centroid of each class. The cen- troids are regarded as the codewords of a new codebook. Set c = c

+

1 and to to Step 1 for next iteration. To speed up the codebook design procedure, the proposed algorithm needs two tables. One stores the mean values of all codewords, its size is N . The other stores the squared root of the variances, its size is also N . The total table size is 2N, which is smaller than the FNNS algo-

a

C

Fig. 3 Fout test images

a Lena b Peppers c let d Baboon

I E E Proc.-Vis. Image Signal Process., Vol. 141, N o . 3, June 1994

rithm. Note that the proposed algorithm does not produce any extra error than the LBG algorithm.

The encoder finds the closest codeword from a predesigned codebook for each input vector and then uses the codeword to represent the corresponding input vector. Therefore, the proposed algorithm can be used to find the closest codeword for each input vector to speed up the encoding process. The detail of the encoding procedure is similar to those in Step 3 of the codebook design algorithm described above.

4 Simulation results

To examine the efficiency of the proposed algorithm, we performed some experiments on a Sun SPARC-station- IPC using several 512 x 512 monochrome images with 256 grey levels. Each image is divided into 4 x 4 blocks, so that the training sequence contains 16384 16- dimensional vectors. The proposed algorithm was com- pared with the ENNS algorithm and the LBG algorithm in terms of the execution time required in codebook design and image encoding.

Table 1 shows the execution time required to design a codebook. The different images shown in Fig. 3 were used to design several different codebooks. Table 2 shows the time needed to encode an image given a predesigned codebook. In this simulation, the image Lena shown in Fig. 3a was used to design the codebook. The resulting codebook was then used to encode the four images (Lena, Peppers, Jet, and Baboon) shown in Fig. 3. From these two tables, we see that the proposed algorithm out-

b

d

(6)

performs the ENNS algorithm in both codebook design and image encoding.

5 Conclusions

A fast closest codeword search algorithm for vector quantisation has been proposed in this paper. This algorithm uses two significant features of a vector, mean value and variance, to reject a lot of unlikely codewords. It can speed up the search process in conventional VQ codebook design and encoding. The performance of the proposed algorithm has been evaluated in both codebook design and image encoding. The results obtained show that the proposed algorithm outperforms the ENNS algorithm and reduces a great deal of computa- tion time required by the LBG algorithm. Furthermore, it is worth mentioning that the proposed algorithm does not introduce any extra error other than the LBG algorithm.

6 References

1 GRAY, R.M.: ‘Vector quantisation’, IEEE ASSP Mag., 1984, pp. 4-29

2 LINDE, Y., BUZO, A., and GRAY, R.M.: ‘An algorithm for vector quantiser design’, IEEE Trans., 1980, C-28, pp. 84-95

3 EQUITZ, W.H.: ‘A new vector quantisation c!ustering algorithm’,

IEEE Trans., 1989, ASP-37, pp. 156g1575

4 BEI, C.D., and GRAY, R.M.: ‘An improvement of the minimum distortion encoding algorithm for vector quantisation’, I E E E Trans.,

1985,C-33, pp. 1132-1133

5 HSIEH, C.H., LU, P.C., and CHANG, J.C.: ‘Fast codebook gener- ation algorithm for vector quantisation of images’, Pattern Recogni- tion Lett., 1991, 12, pp. 605-609

6 ORCHARD, M.D.: ‘A fast nearest-neighbour search algorithm’,

IEEE ICASSP, 1991, pp. 2297-2300

7 GUAN, L., and KAMEL, M.: ‘Equal-average hyperplane partition- ing method for vector auantisation of image data’, Pattern Recogni- tion Lett., 1992, 13, pp. k93-699

8 CHENG, D.Y., and GERSHO, A.: ‘A fast codebook search algorithm for nearest-neighbour pattern matching’, IEEE ICASSP,

1986, pp. 6.14.1-6.14.4

9 SOLEYMANI, M.R., and MORGERA, S.D.: ‘A high-speed search algorithm for vector quanitisation’, IEEE I C A S S P , 1987, pp. 45.6.1- 45.6.3

10 PALIWAL, K.K., and RAMASUBRAMANIAN, V.: ‘Effect of ord- ering the codebook on the efficiency of the partial distance search algorithm for vector quantisation’, IEEE Trans., 1989, C-37, pp. 538-540

7 Appendix

7.1 Proof of Theorem 1

By definition 1, we have V’ = d(x, L,) and

By the triangle inequality, we can obtain = d(yi, Lyi). d ( x , ~ ik )d(x, L,i) - d b i , L,i) (6)

Since L, is the projection point of x on the central line I, thus

d(X, L,;) k d(x, LJ (7)

By inequality eqn. 7, inequality eqn. 6 can be reduced to be

Similarly, we can obtain d(x, ~ i3 )d b i , L,) - d(x, L,)

2 d(Yi, L,i) - d(x, LA

=

v,;

- V,

Combining inequalities eqn. 8 Theorem 1.

(9) and eqn. 9, we prove