ASED ON THE
ITE
WORD LENGTH
Pei-Chuan Liu, Wen-Thong Chang and Wen-Zen Shen Department of Electronic Engineering
Chiao-Tung University, Hsinchu 30050 Taiwan, R.O.C.
Abstract
Three parameter motion model can better describe the motion trajectory of picture element in video sequence. Combining the features of block process and recursive estimation, a block recursive algorithm is proposd to calculate these three motion parameters with compensating the variety of block position to get better stability. To get the simplification requirement for hardware implementation, the representation accuracy of displaced frame difference and gradient that are given is analyzed. From these analysis we propose a quantized block recursive algorithm with quantized grandient to reduce the computational complexity. With this simplification the quantized motion estimation algorithm could get sufficient computational speed to meet the requirement of real time application with the least accuracy loss.
1. Introduction
In the past 2 parameter motion model for video coding based on the assumption of pure 2-D translatory moving object is widely used. But for the actual image signal with 3- D motion, this assumption may be invalid. Recently, a three parameter motion model has been actively studied [l]. As Hotters[ 11 proposed, one parameter describes the change in focal length corresponding to the ratio Qf the focal lengthes before and after zooming, the other two parameters are used to describe the pan or translatory motion of camera. In order to represent the more complex motion of object, some other motion models with more motion parameters have been proposed [2], such as the 6 parameter affine transform, the
8
parameter bilinear transform and the 8 parameter perspective transform, etc. But in the real world such
a
special motion of object does not usually exist. So most of the motion situation in the real image sequence can be well described with thethree p a r a m e t e r motion model.
To calculate the motion parameters of moving object, most of the motion compensation methods belong to two categories : block matching algorithm[3] and the pel- recursive algorithm[4]. Because of the warping coordinate of pixels, the block matching algorithm can't be used to calculate the three motion parameters in three parameter motion model.
So
the pel-recursive algorithm is the only method that can be used for the motion estimation in three parameter motion model. But for motion estimation block process can get the more stable result than single pixelprocess. Using the block processing technique to estimate the coefficients of filter based on the LMS algorithm, Clark et al. proposed the block least mean square (BLMS) algorithm [5] and get better performance. It combines the well known sum square error and the mean square error to form the block mean square error(BMSE) performance measure. Combining the block processing operation of block matching method and the iterative searching procedure of pel-recursive algorithm, a block recursive algorithm is proposed to calculate the motion parameters of object. This method is proven to get the better performance than that of the pel recurive method. According to the equation derived in our proposed algorithm we find the block position is an important criterion to influence the stability of the proposed algorithm. Some big coordinate of pixels in a block will influence the estimation algorithm to be diverged. To solve the problem of "ability, some modifications are made for the block recursive algorithm. With the modification on the convergence factor the stability of the proposed algorithm is well improved.
In spite of the improvement on stability, the block recursive algorithm still meets the problem of large amount of calculations. This problem makes the proposed algorithm hard to be implemented and used for real time calculation. For this, some simplifications must be used.
Finite word length is a important criterion in solving the problem of digital signal processing. As compared to the infinite word length of analog signal, the finite word length of digital signal get the less representation range and the worse data accuracy. But on the concept of transmission or calculation, the finite word length of digital signal get the best quality of precision and the high speed of transmitting or calculating rate. Recently quantized state adaptive algorithms [6-71 have been used to reduce the numerical complexity and dynamic range of the least mean square adaption by replacing multiplifications with shifts, bit comparisons or table lookup. Using the concept of quantization, we investigate the influences caused by changing the word length on different calculation parameters and propose a simplified block recursive algorithm to improve the computational speed with the least performance loss. The trade for good performance on computational efficiency is the worse convergence speed. The rest of this paper is organized as follows. In section I1 we first briefly introduce the three parameter motion model and then propose a block recursive algorithm to calculate these parameters. For the better stability some modifications are made for the block recursive algorithm. For simplification, the influence of finite word length on different calculation
Contributed Paper
parameters is analyzed and a quantized block recursive algorithm is proposed in section 111. The multiplication reduction according to the coordinate relationship is also presented in this section. Finally, a conclusion is made in section IV.
I1
The block recursive algorithm
2.1 the three parameter motion model the transformation of the coordinate
The three parameter motion model can be described as
where (X,,Y,) and ( q , Y , ) are the coordinates of an image point before and after movement in the image space. Parameters a,,
a2
anda3
are the three motion parameters that are to be estimated. When parameter a, = 1, the three parameter model will degenerate to the traditional translational two parameter motion model with a2 =dx
and as = d y . To derive the estimation algorithm, the general relationship between the image point movement and the object motion in real world is first reviewd. For object motion, assuming that a particular point (xl,y,,z,)
has moved to a new position (x2,y2,z2) withx2=x1
+dx,
Y2'Yl +dYlz2
= 2, +dz,where (dx,dy,dz) is the 3-D motion vector in the object space. According to the camera projection rule, in the image space, we have
X, =F*xJzl Y l=Fy,/z,
and
X,=F*x/z2 Y,=Fy2% (3)
From Eqn.(2) and Eqn.(3), the relationship between (Xl, Y1) and (X,,
Y,)
is X, = F*x/z2 = F*(x,+dx)/(z,+dz) = F.(x,+dx)/(z,( l+k)) = F/( 1 +k) x,/z,+ F/( l+k) d d z , = a,X,+a2
Similarly, Y, = alY1 +a3 with k =d d z ,
a,= l/(l+k)a,=-
F x& (l+k) 21 F x- dY a3 =- ( l + k ) 21 (4)For camera zooming, the focal length will change from F, to F,. In this case
X
=F
,
*xl/zI Y ==F, *yl/zl andX2=F2*Xl/z, Y ,=F2*y1/zI
and the motion parameters will be, a,=F,IF, a2 == a3 = 0 2.2 The block recursive algorithm
Based on the Taylor series expansion, the block recursive motion estimation algorithm is derived as following. Assume that changes of the luminance signal in the image plane from frame k to frame k + l are only due to the zoom and pan of the camera and the translatory motion of moving object. Let the luminance at position (x,y) in frame k be defined as S,(x,y). If the motion model is assumed to be A(x, y ) = ( u p
+
a2. a,y+
a3), then the relationship between Sk(x,y) and S,+,(X, y) can be described asWhere a l , a2 and a.{ are the motion parameters of the moving object. The Objective of motion parameter estimation is to estimate these parameters based on the contents of S,+,(x,y) and Sk(x ,y). A common approach is to reduce the displaced frame difference of the two successive frames. The displaced frame difference(DFD) is defined as
A
DFD(x,y) = Sk+1(x7yl -s&A (x,y))
A A A A
= s,+,(x,y:l -
sdal
x+a2, a , y+a,) (7)A A A
where a, ,a2, as are the current estimates of the true motion parameters.
If a, ,a2 and as approach a,, a2 and a3 respectively, then DFD will approach zero. The objective of the proposed algorithm is to use a block of pixels to derive the U , ,a2 and as such that DFD is minimized. Since Sk+,(x,y) = Sk(A(x ,y)), the DFD can be expressed as S,(A(x ,y)) - S,(A (x,y)). Using the Taylor Series expansion to express the luminance function at point p o = A(x, y ) and motion level e= (a, ,a2, as) with respect to the motion parameter ai, we obtain
A A A
A A
A
A
A A A
where r(x,y) denote the higher order terms of the Taylor Series expasion.
where Gai(x,y) = Pose
aui
In order to describe Eqn.(9) directly from the image signal, the gradients with respect to the motion parameter can be replaced by the gradient in X and Y coordinates in the kth frame. By defining (x', y ' ) = A ( x , y )
,
we haveThus,
A
DFD(&y) = sk+,(x,y) sk(A (x>y)>
n
A = sk(A(x, y)) - sk(A (xiy>)
= (G,x
+
G, y ) X ( ~ lU , ) -+
6, X ( a 2 -U,) A A + G, x -a3> = (G,x
+
G, y ) U ,+
G, u2+
G, u3 a y h (12) a s k1
a s k1
where G,= and GI = are
X
and Ydirectional gradient functions on point A (x,y) and U , is the difference between
a,
anda,
.
According to the steepest descent algorithm [8], a linear estimator to minimize the square value of DFD can be written as
ax
A where G@) = [G, x+
G,,y G, GYlT withA @) is the estimated motion parameter vector at iteration p .
E' is the convergence factor of constant value.
Combining with the concept of block processing, a block recursive motion estimation algorithm is proposed by using N points of data to obtain a system of N linear equations to minimize the mean square value of DFD. This algorithm is based on the assumption that all pixels in a block have the same motion parameters. Writting in the matrix form we have
n
wht
D =
and
p is the iteration index of a block.
D is the displaced frame difference vector.
GXi@) = G,(xi@), yi@)) and GYi@) = G,,(xi@), yi@)) are the X and Y directional gradient values at 'the corresponding point (xi@), y i b ) ) in frame k+l.
2.3 the modification of block recursive algorithm If we apply the updating function of motion parameters derived in Eqn.( 14) into Eqn.( 1) to find the corresponding coordinate of pixels, we get
A
x,(P+l) = q @ + U xI+;2(P+I)
= {al +E'
c
[ ( G $ ) ~ ~ + G $ ) ~ ~ ) D F D ~ I }Xi&:P'+Et (G$)DFD~)IN i=l i=l ^ ( P ) N ..(PI A ( P ) N = a1 x i + ~ 2
+
E'? { [(GXlp)*~i.~i+G$'*yi.xi)+G~)]D~Di} r=l N i=l = xi@'+
E'C
[ ( G ~ ) . X ~ ~ C G ~ ) - X ~ - Y ~ + G ~ ) ) ~ ~ ~ ~ ] andFrom Eqn.(l7) we could find the update term of motion vector is propotional to the coordinates of pixel. The position of block is an important factor to influence the stability of the estimation algorithm. The estimated motion parameters of block with bigger coordinates may get a large change and
algorithm position a1 a2 a3 block recursive (5,5) 1.052 2.3 0.9 (-5, -5) 1.050 2.0 1.2 pel-recursive (5,5) 1.020 0.9 0.9 (-5, -5) 1.022 1.1 0.6
then diverge. If we decrease the convergence factor to increase the stability of algorithm, the convergence speed will be slowed down and so the recursive iteration must be increaed. With this, the convergence factor E ' in Eqn.( 14) could be modified to be DFD 6.56 5.41 4.1 1 4.10
&'=I
O OI
where and E;! are the convergence factors for zooming and
paaning motion parameters and (Xo, Yo) is the coordinates of the center point of block.
With the modification on convergence factor the block recursive algorithm will efficiently increase the stability. But there is a special case to cause the modified algorithm to be unstable. This situation happens in that the the x-directional coordinate and y-directional coordinate are one a small value and another a big value. In this case the unbalanced coordinate will cause xly or ylx to be a large number and so the update term of motion vector will be large and cause the algorithm to be unstable. To improve this, the convergence factor matrix will be changed as
where c is a constant value.
In Eqn.(l9) the constant c is used to make the value
x/(y+c) or yl(x+c) not to be so large to cause a unstable result. Another method for the improvement of the convergence factor is El 0 0 X02+Yo2
.;I
0 E2 0\
E2i
i o
0To see the performance of the block recursive algorithm two set of experiments are made for comparison. The image named as "Laboratory" is used for simulation and shown in Fig.1. The desired motion vector is (1.05, 2, 1). In these experiments the block with size of 8x8 is used to find the motion parameters. In the first set of experiments the output performances of the pel-recursive process and the block recursive process after 50 estimation iterations are compared. The blocks with center at
( 5 ,
5 ) and (-5, - 5 ) are used forsimulation. The simulation results are in Table.1. In Table.1 we can see the block process method can get the more accurate results than the single pixel process method. In the second set of experiments the output performances of the block recursive algorithm with different modifications are compared. There are two cases for simulation. In the first case blocks with large x-directional coordinate and large y- directional coordinate are used. The centers of block are at (51, 51) and (59, 59). The simulation results are shown in Table.2 and Fig.2. In TaMe.2 it shows the estimated motion parameters and DFD of different methods after 50 estimation iterations. In Fig.2 it shows the changing curve of the estimated zooming parameter to see the stability of algorithm. The modification methods derived in Eqn.( 18) to Eqn.(20) are labeled as modificationl
,
modification2 and modification3 respectively. According to the simulation results we could find the estimated motion parameters of the block recursive algorithm with no modification will get a divergent result. After using the modification on convergence factor, the stability of algorithm is clearly improved. The modification2 method gets the better stability than the modification1 method. The modification3 method gets the best stability but the slowest convergence speed. In the second case the blocks with large x-directional coordinate and small y-directional coordinate are used for simulation. The center positions of block are at (59, 3) and (83, 3). These results are shown in Table.3 and Fig.3. According to the results it could be found the modificationl method gets the unstable results. The modlification3 method also gets the best stability but the slowest convergence speed. The usage of the modification2 method or the modification3 method is a trade between the stability and the convergence speed. In the pel- recursive algorithm proposed in [4] it has made a modification on convergence factor for the large value of gradient. But in our analysis, most of the gradients in real image sequence is kept to be a small value. So under the consideration of calculation amount the modification for the large value of gradient is not made in our proposed.Table.1 The simulation results of the block recursive algorithm and the pel recursive algorithm. algorithm 15.76 1.2 11.55 0.9 11.31 (b)
Table.2 The simulation results for blocks with center at (a) (51, 51) (b) (59,
59) with different modifications.
(b)
3 ) with different modifications.
Table.3 The simulation results for blocks with center at (a) (59, 3) (b)
After making a modification on convergence factor for the stability, the block recursive algorithm still meets the problem of large amount of calculation. The complex computation makes the block recursive algorithm hard to be used for the real time application. So some simplifications must be made to dove this problem.
111
The block recursive algorithm with finite
word length
3.1 The quantized block recursive algorithm
As the constraint of the technique of image capturing or photoing, finite word length is the constraint on the quality of image of the digital system. To increase the quality of image picture the bit number of every pixel must be increased. On the other way, in order to increase the speed of processing or to decrease the size of memory for storing the image datas, the bit number of pixel must be decreased. So how to choose the word length for digital signal process system to get the best output performance is a important topic in disigning the signal processing system. Recently the quantized state adaptive algorithm has been sparked by the use of adaptive filters in high speed data computations and speech processing where the high data rate requires a computationally simple algorithm with large dynamic range. Using the concept of quantization, we simplify the block recursive algorithm by reducing the word length of some calculation parameters. The simplification strategy is to use the quantization on DFD or on gradient. The influences are introduced as following.
1. Quantization on DFD In the block recursive algorithm derived in section I1 the DFD is a important judgement of the output performance. The quantization on DFD will decrease the convergence accuracy of the estimation algorithm. On the other way the quantization on DFD will decrease the complexity of calculation DFD(xi@), yb))GXib) or DFD(xib),
yi@))G,.,@) and so the complexity of the block recursive algorithm will be reduced.
2. Quantization on gradient If we quantize the gradient to be less word length but keep the direction of gradient to be similar, the performance of the quantized algorithm will make the least influence. Under this situation the convergence speed will be influenced to be slower but the convergence accuracy will be kept similar. At the same time
the word length reduction of gradient will reduce the complexity of algorithm and increase the calculation speed.
Base on above analyses it could be found the quantization on gradient will reduce the complexity of algorithm with the least performance loss. So the quantization on gradient is used to simplify the block recursive estimation algorithm and Eqn.( 16) will be replaced by
(21) where QXl@),
e,,@)
are the quantized values of G,,@), GI,@).According to the analysis of the feature of image gradient, we find most of the gradients of pixel are small values. There are over 92 percent of gradients under the value of 32. According to the nonlinear distribution of gradient, a nonlinear quantization method is used for the quantiztion of gradient. The nonlinear quantization method to be used is to keep the first bit 1 of signal unchanged and change all the follwing bit I s to be bit Os. Taking the signal 0101 1010 as an example, the quantized signal will be 01000000. After using the nonlinear quantization on a 8-bit integer signal, the probability distribution of the quantization error eg will be
eg =0, I
2X(7-N) 511 eg= (2Ni1-2N)im1 W B G , W ~ 2 N - l , prob=-
According to the probability distribution in Eqn.(22), if we assume all the gradients are under value 32 and are normal distributed, the expective value of the error signal eg will be 4.8. If the nonlinear distribution of the gradient values is taking into account, the expective value of quantization error will be less. Under the consideration of implementation, because there is at most one bit 1 in the quantized signal Q, or Q,, the multiplication with Q, or Q, could be replaced by the shift bit operation. So the multiplication amount needed for the calculation QQ can be largely reduced.
There are some experimants made to see the output performance of block recursive algorithm with different word length of gradient. The image "Laboratory" is also used for simulation. The modification3 method in Eqn.(20) is used to get the better stability. The block with center at (25, 25) is used for simulation. The change curves of zooming motion parameter and mean square value of DFD are used to see the convergence speed and convergence accuracy. T h e simulation results are shown in Fig.4. In Fig.4 it shows the simulation results of the block recursive algorithm with the 5- bit to 8-bit linear quantization methods and the 3-bit
nonlinear quantization method. According to these simulation results we find that the reduction of word length of gradient only causes the convergence speed to be slower but convergence accuracy is kept similar. It also shows the convergence performance of the 3-bit nonlinear quantization method is similar to that of the 7-bit linear quantization method. This proves that the nonlinear quantization method used for the simplification of the block recursive algorithm can get good efficiency with the least performance loss. 3.2. The multiplication reduction according to the coordinate relationship
After using shift bit operation to replace the multiplication with the quantized gradient, the rest of multiplications needed for the calculation
QTD
are the multiplications with X and Y directional coordinates. If we consider the relationship between the coordinates in a block, these multiplications can also be largely reduced. It will be discussed in the following.Define the shift bit operation of the quantized gradient on DFD as
Dx,@) = DFD(x,@), ~,@))xQ,,~@) D,,@) = DFD(x,@), Y,@))XQ;~@)
The calculation
QTD
could be rewritten asV= [ vI , v z , v3IT= QTD
To complete the calculation of QQ, it remains the multiplications of coordinates xi and yi with Dxi@) and DYi@). If we consider the relationship between the coordinates of pixels in a block, these multiplications can be largely reduced. Taking a 4 x 4 block as an example, the coordinate relationships between pixels are shown in Fig.5. It can be seen that the X coordinates of the pixels in the same column and The
Y
coordinates of the pixels in the same row are the same. So there are only four different X indices and four different Y indices for the 16 coordinates of pixels. Denoting the left-upper corner coordinate as (x, y), the rest of the element can be denoted as ( x + h , y-i), h , i = 0,1,2,3. By definingthe ith column summation as
The vector v for a 4x4 block can be calculated as
According to Eqn.(26), for a 4 x 4 block only 2 multiplications and 39 additions are needed to complete the calculation of QTD.
After get the new motion parameters, the addresses of the new corresponding points with the new estimated motion parameter
^a,@+’)
must be calculated. To get the new address of ( a , x, +a2 , a,y;+a,), the multiplications with x, and y r are also needed. So the multiplication reduction according the coordinate relationship can also be used. According to above derivation, the computation requirement comparison of the original block recursive algorithm and the simplified method are shown in Table.3. According to the results we find the multiplications needed for the simplified algorithm will be over 30 times less than those needed for the original method. Such a large amount of calculation reduction makes the simplified block recursive algorithm easier to be implemented for high speed motion estimation.operation
coordinate relationshi
Table 4 The computational amount comparison
IV.
Conclusion
A block recursive algorithm to compute the three motion parameters for video coding is presented in this paper. With the modification of coordinate on the convergence factor the modified algorithm get the better stability and convergence accuracy. To solve the problem of computational complexity, the quantized gradient is is used and proven to get good result with a little performance loss. Combining with the computational complexity reduction according to the coordinate relationships, the amount of calculations can be 1,argely reduced and make the computational speed easier to meet the real time calculation requirement.
Reference
the ith row summation as[ 13 M.Hotter, “Differential Estimation of The Global Motion Parameters Zoom and Pan”, Signal Processing, vo1.16, no.3, March 1989, pp.249-265.
Y. Nakaya, H. Harashima, "Motion Compensation Based on Spatial Transformations", IEEE Trans. on CAS for Video Technology, vo1.4, no.3, Jun. 1994, pp- M. Bierling, "Displacement estimation by hierarchical b l o c k m a t c h i n g " , S P I E V o l . 1 0 0 1 , v i s u a l communications and Image Processing'88, pp.942-95 1,
1988.
Walker, D. R. and K. R. Rao, "Improved Pel- Recurssive Motion Compensation", IEEE Transactions on Commucations, vol. COM-32, no. 10, December G . A. Clark, S. K. Mitra, and S . R. Parker, "Block 339-356.
1984, pp.1128-1134.
Implementation of Adaptive Digital Filter", IEEE Trans. Circuits syst. , vol. CAS-28, June 1981, pp.584- 592.
[6] "Adaptive Equalization via Fast Quantized-State Methods", IEEE Trans. Commun., vol. COM-29, Oct. [7] A. Gersho, "Adaptive filtering with Binary reinforcement", IEEE Trans. Inform. Theory. vol. IT- 30, March 1984.
[8] W. A. Sethares and Richard Johnsen, JR. "A Comparison of T w o Quantized State Adaptive Algorithms",
IEEE
Trans. Acoust., Speech, signal Processing, ~01.37, No.1, Jan. 1989, pp.138-143. 198 1 , pp. 1492- 1501.0% I
Fig.2 The simulation result of zooming factor for blocks with center at (a) (51, 51) (b) (59, 59).
I
1
1 1 12-
a ' 3 Od 0.6 0.4 02 0 1.1,
1 (a) I@)Fig.3
The
simulation result of zooming factor for blocks with center at (a) (59. 3) (b) (83, 3).Fig.4 The simulation result of (a) zooming factor (b)
