ON MODELING GENETIC PATTERN SEARCH FOR BLOCK MOTION ESTIMATION
Jang-Jer Tsai
1and Hsueh-Ming Hang
1,21
Department of Electronics Engineering, National Chiao-Tung University, Hsinchu, Taiwan
2Dept. of Computer Sci. and Inform. Tech., National Taipei University of Tech., Taipei, Taiwan
[email protected], [email protected]
ABSTRACT
Pattern search algorithms, such as diamond search, hexagonal search and their variations, have been widely adopted by the block matching motion estimations in the modern video encoding systems. Recently we propose a weighting function (WF) to model the number of search points of a pattern search. Yet, WF fails to properly describe the behavior of the genetic pattern search algorithms due to some over-simplifications in their models. Therefore, we propose a refined weighting function (RWF) to more accurately describe both genetic and non-genetic pattern searches. In addition, we propose a new search algorithm, namely, the momentum directed genetic rhombus pattern search (MD-GRPS). It can accelerate the previous genetic rhombus pattern search by 8% on the average and this concept can be applied to the other genetic pattern searches.
Index Terms—Refined weighting function, Genetic
pattern search, Block motion estimation, Video coding.
1. INTRODUCTION
The modern video coding systems [1][2] adopt block motion estimation (BME) to remove inter-frame redundancy of image sequences. Despite the success of BME in compression, it requires a huge amount of computing power particularly for sophisticated coding algorithms that include multiple reference frames and variable size motion estimations. Therefore, researchers have been developing fast algorithms to reduce computation without sacrificing the coding efficiency. Among all BME algorithms, the pattern search algorithms, such as diamond search [3], hexagonal search [4] and their enhanced versions [5][6], are the most popular due to their simplicity and effectiveness. Moreover, because real image sequences may have different amount of motion activities in different parts of a long sequence the so-called adaptive schemes [7][8][9] that switch between two or more search patterns were also proposed.
In [8], we use the notion of weighting function (WF) to predict the efficiency of a pattern search algorithm. The WF is defined as the minimum number of search steps that a specific pattern search algorithm can achieve when the
matching error surface is monotonic. Therefore, its values depend on the search pattern used. Given the probability distribution of the motion vectors (MVs) of a video sequence, we can predict the performance (search points) of a BME algorithm by using its WF when it is applied to this particular video sequence.
In [8] and [10], we combine the genetic optimization method with pattern search to further reduce computation. The so-called genetic pattern search algorithm is thus devised. It matches the ideal minimum WF and indeed it shows substantial performance improvement. However, the afore-mentioned WF is not adequate in characterizing the behavior of the genetic pattern searches because WF does not represent the average performance (search points). This difference is due to the fact that, even when the matching error surface is monotonic, the genetic pattern search picks up the search direction randomly but the classical pattern searches move along the steepest descent path on the matching error surface towards the best matching point. The purpose of this paper is to construct a more accurate model (essentially, a better WF) for the genetic pattern search. In this process, we also devise a new type of genetic pattern search that further reduces the computation.
The remaining parts of this paper are organized as follows. Section 2 analyzes and models the behavior of a genetic pattern search. Based on this analysis, a new search algorithm is proposed in Section 3. Section 4 shows the experimental results of the proposed algorithm in comparison with some classical and genetic pattern searches. Finally, a brief summary is given in Section 5.
2. ANALYSIS ON GENETIC PATTERN SEARCH
Because WF does not well convey the randomness nature of the genetic pattern search, here we propose a refined weighting function (RWF). We assume the matching error (distortion) surface is unimodal and monotonic. In other
words, let the optimal point be the origin, and u=(x1, y1) and
v=(x2, y2) be two points in the domain. The distortion at u,
D(u), is smaller than D(v), if |x1|<|x2| and |y1||y2| or if
|x1||x2| and |y1|<|y2|. The RWF, RWF(x,y) is defined to be
the average number of search points (SPs) needed by a search algorithm, when the best matching point is located at (0,0) and the starting point is (x,y). As an example, we will
1980
construct the RWF for the genetic rhombus pattern search (GRPS) in [10]. The flow chart of GRPS is in Fig. 1 and its search patterns are shown in Fig. 2. In the search process, only one (black dot, for example) out of the four (grey and black) points in Fig. 2(a) is randomly chosen as the next check point. And the search ends when all four (black) points in Fig. 2(b) have been checked and all of them have larger matching errors than that of the center (white) point.
S1: Initial
Check the starting point and set it as the parent.
S2: Mutation
Randomly select one mutation from the un-checked points in the rhombus pattern
centered at the parent.
S4: End
Set the current survivor as the best motion vector. Parent Survive?
S3: Competition Compare the parent and the mutation to select one survivor according to a predefined
block matching cost criterion.
S3A: Set the surviving mutation as the next
parent. S3B:
Check if there is any other possible
mutations?
Y N
N Y
Fig. 1 GRPS flow chart
-4 -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4 (a) -4 -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4 (b)
Fig. 2 GRPS search pattern
Under the unimodal and monotonic matching error surface assumption, the number of mutations with matching error smaller than that of the current point is decided by the relative position of the current point to the best matched
point as shown in Fig. 3. Specifically, there are two types of
starting search point cases (S1 and S2) and two types of intermediate search point cases (M1 and M2) for GRPS, as
shown in Fig. 4. Herein, points A, B, C and D are the search
candidates (mutations), and point E denotes the best
matching point. In Fig. 4(a), only one out of the 4 points
centered at S1 may have a smaller matching error than S1 when point E has the same X or Y coordinate as S1.
Otherwise, it is the Fig. 4(b) case, in which 2 out of the 4
points centered at S2 may have smaller errors. Similarly, in
Fig. 4(c), only one out of the 3 points centered at M1 may have smaller matching error when point E has the same X or
Y coordinate as M1. Otherwise, in the case of Fig. 4(d), 2
out of the 3 points centered at M2 may have smaller errors.
Fig. 5 shows all possible search sequences starting from
S2. Eqn.(1) gives the average number of SPs (WS2) moving
from S2 one step towards the final destination, which is 5/3. We assume that the probability of selecting any candidate point is equal. Similarly, the average SPs moving from S1,
M1 and M2 (WS1, WM1 and WM2) are 5/2, 2 and 4/3,
respectively. 1 2 2 2 2 2 2 2 2 The feasible mutation number of GRPS
X-axis Y-ax is -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30
Fig. 3 The contour plot of the number of small-error mutations in the search area (GRPS), wherein the origin denotes the best matched point. S1 E A B C D S2 E A B C D M1 E A B C M2 E A B C (a) (b) (c) (d)
Fig. 4 Four typical cases of starting search points and intermediate search points.
Fig. 5 All possible search order for S2
Let the coordinate of the starting point be (x,y) and the best matching point be (0,0). The average SPs moving from
(x,y) to (0,0) is RWFGRPS(x,y). Fig. 6 shows the algorithm of
calculating RWF for GRPS according to the previous
analysis. The contour plot of RWFGRPS(x,y) is shown in Fig.
7(a). In comparison, Fig. 7(b) shows the real average search points when it is applied to the sequence ‘2X MD96’ in
Table I. The outer ring of Fig. 7(b) is empty because these points never become the best matching points. Because we have a good starting point predictor, a typical best matching point is fairly close to the starting point. Likewise, the RWF for the other genetic pattern searches can be constructed.
3. THE PROPOSED SEARCH ALGORITHM
We assume that a doable search method moves at most one unit distance horizontally or vertically per step, as
shown in Fig. 2(a). Then, the minimal number of search points for reaching point (x,y) is 'abs(x)+abs(y)+1'. At the ending stage, to decide the best motion vector generally requires examining at least the center point and its 4 neighboring points as shown in Fig. 2(b). Consequently, the smallest number of search points for motion vector (x,y) can be expressed by (2) and its contour plot is shown by Fig. 8(a). 3 5 2 1 4 1 2 2 2 3 1 4 1 2 3 2 1 3 1 4 1 2 » » » » » » » ¼ º « « « « « « « ¬ ª u u u » » » » ¼ º « « « « ¬ ª u u u u u u u S W (1) abs(y)) abs(x) 4 Max(5, ) , (x y RWF (2) A==0 B==0 0 0 N Y N Y A==0 B==0 1 1 Y N Y POS_DIR 0 0 5 WS2+4 WS1+4 Result 2 1 0 A>0 B>0 0 0 Y N Y N A>0 B>0 0 0 Y N Y N POS_DIR 1 1 0 WM2 WM1 Output 2 1 0 N + / + + + + + + Data Multiplexer Operation Weight(A,B) Procedure Legend: Weight(A,B-1) Weight(A-1,B) Y Weight(A-1,B) Weight(A,B-1) A = ABS(x) B = ABS(y) RWF(x,y) Fig. 6 Construction of RWF 7 10 15 20 25 50 50 75 75 75 75 RWFGRPS(x,y) X-axis Y-ax is -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30 7 7 7 10 10 1010 10 10 15 15 15 15 15 15 20
Average Search Points of GRPS on Sequence 2X MD96
X-axis Y-ax is -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30 (a) (b)
Fig. 7 Comparison between RWF of GRPS and its real average search points on the sequence ‘2X MD96’.
Comparing Fig. 7(a) to Fig. 8(a), it is obvious that the RWF of GRPS does not match the ideal RWF. How can we devise a search method that matches the ideal RWF?
Observing the ideal RWF carefully, we find a clue: the algorithm should proceed straightly towards the best matching point. Statistically, the successful direction of the previous search likely reappears at the current point. Therefore, instead of randomly select one mutation from the candidate child set, we select the mutation according to its likelihood to be a successful mutation. That is, it tends to move along the same direction of the prior successful searches. On the other hand, it can still change the search directions when the assumption of monotonic matching error surface is not totally valid. The flow chart of the proposed algorithm, namely, the momentum-directed GRPS
(MD-GRPS), is almost the same as Fig. 2, except that in step
2, the mutations are generated according to the search order
shown in Fig. 9. In comparison, when MD-GPRS is applied
on the sequence ‘2X MD96’ in Table I, its contour plot of
real average search points is shown in Fig. 8(b).
710 15 20 25 25 50 50 50 50
A doable smallest RWF(x,y)
X-axis Y-ax is -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30 7 7 10 10 10 10 10 15 15 15
Average Search Points of MD-GRPS on Sequence 2X MD96
X-axis Y-ax is -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30 (a) (b)
Fig. 8 Comparison between a doable smallest RWF and the real average search points when applying MD-GRPS on the
sequence ‘2X MD96’.
C P
PP
Fig. 9 The search order of mutations in MD-GRPS specified by the positions of the previous successful mutation (P) and the differential successful mutation before that (PP). And C denotes the current parent.
4. EXPERIMENTAL RESULTS
To test the proposed algorithm, four sequences with different MV variances (denoted as ‘1X’) are tested under
the setting given in Table I. Moreover, to test the extreme
cases, we generate four new test sequences consisting of the odd frames of these sequences (denoted as ‘2X’). They equal to the two times fast forward playback of the originals.
These 8 test sequences are coded by an MPEG-4 SP@L3
encoder. All the sequences are in the CIF (352X288) format. Only the first frame is coded as I frame, and all the
remaining frames are coded as P frame. The search range is 16, and the block size is 16x16.
Table I. Test Sequences and Their Settings
Abbreviation Sequence (K bps) Bitrate Frame rate (fps) of framesNumber MD96 mother and daughter 96 10 300
FM512 foreman 512 30 300
FB1024 football 1024 30 90
ST1024 steven 1024 30 300
Table II. ASP (Average Number of Search Points)
Type ASP MD-GRPS GRPS[8] PHS[6] DS[3] FS MD96 5.94 5.98 10.02 14.85 1024 FM512 6.80 7.13 10.57 16.17 1024 FB1024 10.35 11.89 14.18 22.36 1024 1X ST1024 7.24 7.65 11.40 16.96 1024 MD96 6.29 6.40 10.38 15.44 1024 FM512 8.25 9.07 12.02 18.72 1024 FB1024 13.27 15.75 17.15 27.39 1024 2X ST1024 8.61 9.28 13.00 19.49 1024 Average 8.34 9.14 12.34 18.92 1024
Table III. PSNR (Peak Signal to Noise Ratio)
Type PSNR MD-GRPS GRPS PHS DS FS MD96 40.05 40.08 39.85 39.99 39.80 FM512 34.04 34.05 33.92 34.06 34.06 FB1024 34.92 34.87 34.87 34.93 35.28 1X ST1024 29.16 29.39 29.33 29.44 29.48 MD96 38.66 38.66 38.44 38.60 38.41 FM512 32.36 32.34 32.23 32.38 32.42 FB1024 33.26 33.22 33.22 33.28 33.44 2X ST1024 27.86 27.99 27.88 27.97 28.10 Average 33.79 33.83 33.72 33.83 33.87
Table IV. Performance Comparison
Gain over GRPS MD-GRPS MD-GRPS over PHS MD-GRPS over DS MD-GRPS over FS
Type Sequence CG QG CG QG CG QG CG QG MD96 0.7% -0.02 68.7% 0.20 1.50 0.06 171.39 0.25 FM512 4.9% -0.01 55.4% 0.12 1.38 -0.02 149.59 -0.01 FB1024 14.9% 0.05 37.0% 0.05 1.16 -0.01 97.94 -0.36 1X ST1024 5.7% -0.23 57.5% -0.17 1.34 -0.29 140.44 -0.32 MD96 1.7% 0.00 65.0% 0.22 1.45 0.06 161.80 0.25 FM512 9.9% 0.02 45.7% 0.13 1.27 -0.02 123.12 -0.06 FB1024 18.7% 0.03 29.2% 0.03 1.06 -0.02 76.17 -0.18 2X ST1024 7.8% -0.13 51.0% -0.02 1.26 -0.11 117.93 -0.24 ʳ Average 8.0% -0.04 51.2% 0.07 1.30 -0.04 129.80 -0.08
The average number of search points (ASP) and the peak signal to noise ratio (PSNR) for various sequences and search algorithms are listed in Table II and Table III, respectively. FS denotes the full search. The optimal point is the one with the smallest block matching error in the search area. Because we do not use the rate-distortion optimization, the PSNR value of some fast algorithms may outperform that of FS. The initial MV predictor defined by the MPEG-4 standard is the only starting point in all cases. Table IV
shows pair-wise performance comparisons. In Table IV, the computing gain (CG) is defined as the ratio of ASP minus one, and the quality gain (QG) is the PSNR difference. For complicated sequences, MD-GRPS can be up to 19% faster than GRPS and its PSNR quality is about the same. On the average, the ASP of MD-GRPS is 8% faster than that of GRPS, 51% faster than PHS, 1.3 times faster than DS and 130 times faster than FS. And the PSNR of MD-GRPS is about the same as that of all the other search algorithms (+0.07dB~-0.08dB).
5. CONCLUSIONS
In this paper, we propose a refined weighting function (RWF) that models the SPs of a pattern search algorithm. With the help of RWF, we suggest a momentum-directed
genetic rhombus pattern search (MD-GRPS). Taking the
advantage of the correlation of successful search directions between two succeeding searches, the proposed design further improves the performance of the genetic rhombus pattern search (GRPS). This concept, momentum-directed genetic search, can be applied to all the other genetic pattern searches.
6. REFERENCES
[1] T. Wiegand, et al., “Overview of the H.264/AVC video coding standard”,” IEEE Trans. Circuits Syst.Video Technol., vol. 13, pp. 560–576, July 2003.
[2] A. H. Sadka, Compressed Video Communications, John Wiley and Sons Ltd, 2002.
[3] S. Zhu and K.-K. Ma, "A new diamond search algorithm for fast block-matching motion estimation," Proc. of Int. Conf.
Information, Communications and Signal Processing (ICICS’97),
vol. 1, p.p. 292-296, Sept. 9-12, 1997.
[4] C. Zhu, X. Lin and L.-P. Chau, "Hexagon-based search pattern for fast block motion estimation," IEEE Trans. Circuits Systems
for Video Technology, vol. 12, no. 5, p.p. 349-355, May 2002.
[5] Y. Nie and K.-K. Ma, "Adaptive irregular pattern search with matching prejudgment for fast block-matching motion estimation,"
IEEE Trans. Circuits and Systems for Video Technology, , vol.15,
no.6pp. 789- 794, June 2005
[6] L.-M. Po, et al., "Novel Point-Oriented Inner Searches for Fast Block Motion Estimation," IEEE Trans. Multimedia, vol.9, no.1pp.9-15, Jan. 2007
[7] S.-Y. Huang, et al., “Adaptive fast block-matching algorithm by switching search patterns for sequences with wide-range motion content”, IEEE Trans. Circuits and Systems for Video Technology , vol. 15, no. 11, p.p.1373-1384, Nov. 2005.
[8] J.-J. Tsai and H.-M. Hang, “On adaptive pattern selection for block motion estimation algorithms”, IEEE Proc. of Int. Conf. on
Acoustics, Speech, and Signal Processing (ICASSP’07), Hawaii,
U.S.A., Apr. 2007.
[9] I. Gonzalez-Diaz, and F. Diaz-de-Maria, “Improved Motion Classification Techniques for Adaptive Multi-Pattern Fast Block-Matching Algorithm”, IEEE Proc. of Int. Conf. Image Processing
(ICIP ’07), Texas, U.S.A., Oct. 2007
[10] J.-J. Tsai and H.-M. Hang, “A genetic rhombus pattern search for block motion estimation”, IEEE Proc. of Int. Symp. on Circuits
and Systems (ISCAS’07), New Orleans, U.S.A., May. 2007.
