Motion estimation-Digital visual effects. spring 2006

(1)

Motion estimation

Digital Visual Effects, Spring 2006

Yung-Yu Chuang

2005/4/12

(2)

Announcement

• The first part of project #2 (feature detection and matching) is due on Sunday, please send your source code and two images showing your results to TAs.

(3)

Outline

• Motion estimation

• Lucas-Kanade algorithm • Tracking

(4)

Motion estimation

• Parametric motion (image alignment) • Tracking

(5)

Parametric motion

(6)

(7)

(8)

Three assumptions

• Brightness consistency • Spatial coherence

(9)

(10)

(11)

(12)

Image registration

Goal: register a template image J(x) and an input image I(x), where x=(x,y)T.

Image alignment: I(x) and J(x) are two images

Tracking: I(x) is the image at time t. J(x) is a small patch around the point p in the image at t+1.

(13)

Simple approach

• Minimize brightness difference







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , (

(14)

Simple SSD algorithm

For each offset (u, v) compute E(u,v);

Choose (u, v) which minimizes E(u,v); Problems:

• Not efficient

(15)

(16)

Newton’s method

(17)

Newton’s method

• Root finding for f(x)=0 Taylor’s expansion:

(18)

Newton’s method

• Root finding for f(x)=0

x₀ x₁

(19)

Newton’s method

pick up x=x0 iterate compute update x by x+Δx until converge Minimize g(x) →_{find f(x)=g’(x)=0}

)

(

'

)

(

x

f

x

f

x







(20)

Lucas-Kanade algorithm







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , ( y x vI uI y x I v y u x I(  ,  )  ( , )  







    y x y x vI uI y x J y x I , 2 ) , ( ) , (







       y x y x x I x y J x y uI vI I u E , ) , ( ) , ( 2 0







       y x y x y I x y J x y uI vI I v E , ) , ( ) , ( 2 0

(21)

Lucas-Kanade algorithm







       y x y x x I x y J x y uI vI I u E , ) , ( ) , ( 2 0







       y x y x y I x y J x y uI vI I v E , ) , ( ) , ( 2 0









          



y x y y x y y x y x y x x y x y x y x x y x I y x J I v I u I I y x I y x J I v I I u I , , 2 , , , , 2 ) , ( ) , ( ) , ( ) , (









                            



y x y y x x y x y y x y x y x y x y x x y x I y x J I y x I y x J I v u I I I I I I , , , 2 , , , 2 ) , ( ) , ( ) , ( ) , (

(22)

Lucas-Kanade algorithm

iterate

shift I(x,y) with (u,v)

compute gradient image Ix, Iy

compute error image J(x,y)-I(x,y) compute Hessian matrix

solve the linear system (u,v)=(u,v)+(∆u,∆v) until converge









                            



y x y y x x y x y y x y x y x y x y x x y x I y x J I y x I y x J I v u I I I I I I , , , 2 , , , 2 ) , ( ) , ( ) , ( ) , (

(23)

Parametric model







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , (







  x x p) W(x; p) ( ) ( ) 2 ( I J E T y x y x d d p d y d x ) , ( ,           p) W(x; translation T y x yy yx xy xx y yy yx x xy xx d d d d d d p y x d d d d d d ) , , , , , ( , 1 1 1                       Ax d p) W(x; affine

Our goal is to find

(24)

Parametric model







  x x Δp) p W(x; ) ( ) 2 ( J I minimize with respect to Δp Δp p W p) W(x; Δp) p W(x;      ) ( ) ( Δp p W p) W(x; Δp) p W(x;      I I Δp p W x p) W(x;       I( ) I



           x x Δp p W p) W(x; 2 ) ( ) ( I J I minimize

(25)

Parametric model



           x x Δp p W p) W(x; 2 ) ( ) ( I J I image gradient

Jacobian of the warp

warped image                                            n y y y n x x x y x p W p W p W p W p W p W p W p W p W   2 1 2 1

(26)

Jacobian of the warp

For example, for affine

                                y yy yx x xy xx y yy yx x xy xx d y d x d d y d x d y x d d d d d d ) 1 ( ) 1 ( 1 1 1 p) W(x;                                            n y y y n x x x y x p W p W p W p W p W p W p W p W p W   2 1 2 1          1 0 0 0 0 1 0 0 y x y x p W

(27)

Parametric model



           x x Δp p W p) W(x; 2 ) ( ) ( I J I minimize



                     x x Δp p W p) W(x; p W ) ( ) ( 0 I I I J T







_            x p) W(x; x p W H Δp 1 I J( ) I( ) T



                   x p W p W H I I T Hessian

(28)

Lucas-Kanade algorithm

iterate

1) warp I with W(x;p)

2) compute error image J(x,y)-I(W(x,p)) 3) compute gradient image with W(x,p) 4) evaluate Jacobian at (x;p) 5) compute 6) compute Hessian 7) compute 8) solve 9) update p by p+ until converge p W   p W   I    _          x p) W(x; x p W ) ( ) ( I J I T Δp Δp







_            x p) W(x; x p W H Δp 1 I J( ) I( ) T I 

(29)

(30)

Coarse-to-fine strategy

J _warp Jw _refine I in a a  + J _warp Jw _refine I a a  + J pyramid construction J _warp Jw _refine I a  + I pyramid construction out

a

(31)

(32)

(33)

Tracking

(34)

Tracking

0 ) , , ( ) 1 , , (x  u y  v t   I x y t  I 0 ) , , ( ) , , ( ) , , ( ) , , ( ) , , (x y t  uI x y t  vI x y t  I x y t  I x y t  I _x _y _t 0 ) , , ( ) , , ( ) , , (x y t  vI x y t  I x y t  uI_x _y _t 0    _y _t xu I v I

I optical flow constraint equation brightness constancy

(35)

(36)

(37)

Area-based method

(38)

(39)

(40)

(41)

Demo for aperture problem

• http://www.sandlotscience.com/Distortions/Br eathing_Square.htm • http:// www.sandlotscience.com/Ambiguous/Barberpole _Illusion.htm

(42)

Aperture problem

• Larger window reduces ambiguity, but easily violates spatial smoothness assumption

(43)

Area-based method

• Assume spatial smoothness







   y x t y xu I v I I v u E , 2 ) , (

(44)

Area-based method

(45)

Area-based method

• The eigenvalues tell us about the local image st ructure.

• They also tell us how well we can estimate the flow in both directions

(46)

(47)

(48)

(49)

KLT tracking

• Select feature by

• Monitor features by measuring dissimilarity

 

 , ) 

( ₁ ₂

(50)

(51)

(52)

(53)

KLT tracking

(54)

KLT tracking

(55)

SIFT tracking (matching actually)



(56)

SIFT tracking



(57)

SIFT tracking



(58)

KLT vs SIFT tracking

• KLT has larger accumulating error; partly becau se our KLT implementation doesn’t have affine transformation?

• SIFT is surprisingly robust

• Combination of SIFT and KLT (example)

(59)

(60)

(61)

(62)

(63)

Single-motion assumption

Violated by • Motion discontinuity • Shadows • Transparency • Specular reflection • …

(64)

(65)

(66)

(67)

(68)

(69)

Robust statistics

• Recover the best fit for the majority of the data

(70)

(71)

(72)

(73)

(74)

(75)

(76)

(77)

(78)

(79)

(80)

(81)

(82)

(83)

(84)

(85)

(86)

(87)

(88)

(89)

(90)

(91)

(92)

(93)

(94)

Temporal artifacts

(95)

(96)

(97)

(98)

Reference

• B.D. Lucas and T. Kanade,

An Iterative Image Registration Technique with an Application to Stereo Vis ion

, Proceedings of the 1981 DARPA Image Understanding Workshop, 1981, pp1 21-130.

• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R.,

Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.

• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600.

• Michael Black and P. Anandan,

The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smoot h Flow Fields

, Computer Vision and Image Understanding 1996, pp75-104.

• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework

, International Journal of Computer Vision, 56(3), 2004, pp221 - 255.

• Peter Litwinowicz, Processing Images and Video for An Impressionist Effects

, SIGGRAPH 1997.

• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz,

Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004,