Motion estimation-Digital visual effects. spring 2005

Motion estimation

Digital Visual Effects, Spring 2005

Yung-Yu Chuang

2005/3/23

Announcements

• Project #1 is due on next Tuesday, submission mechanism will be announced later this week. • grading: report is important, results

(good/bad), discussions on implementation, interface, features, etc.

Outline

• Motion estimation

• Lucas-Kanade algorithm • Tracking

Motion estimation

• Parametric motion (image alignment) • Tracking

Three assumptions

• Brightness consistency • Spatial coherence

Image registration

Goal: register a template image J(x) and an input image I(x), where x=(x,y)T.

Image alignment: I(x) and J(x) are two images

Tracking: I(x) is the image at time t. J(x) is a small patch around the point p in the image at t+1.

Simple approach

• Minimize brightness difference







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , (

Simple SSD algorithm

For each offset (u, v) compute E(u,v);

Choose (u, v) which minimizes E(u,v); Problems:

• Not efficient

Newton’s method

• Root finding for f(x)=0 Taylor’s expansion:

Lucas-Kanade algorithm







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , ( y x vI uI y x I v y u x I(  ,  )  ( , )  







    y x y x vI uI y x J y x I , 2 ) , ( ) , (







       y x y x x I x y J x y uI vI I u E , ) , ( ) , ( 2 0







       y x y x y I x y J x y uI vI I v E , ) , ( ) , ( 2 0

Lucas-Kanade algorithm







       y x y x x I x y J x y uI vI I u E , ) , ( ) , ( 2 0







       y x y x y I x y J x y uI vI I v E , ) , ( ) , ( 2 0









          









y x x y y y y x y x x y x y x x y x I y x J I v I u I I y x I y x J I v I I u I , , 2 , , 2 ) , ( ) , ( ) , ( ) , (









                            













y x y y x x y x y y x y x y x y x y x x y x I y x J I y x I y x J I v u I I I I I I , , , 2 , , , 2 ) , ( ) , ( ) , ( ) , (

Lucas-Kanade algorithm

iterate

shift I(x,y) with (u,v)

compute gradient image Ix, Iy

compute error image J(x,y)-I(x,y) compute Hessian matrix

solve the linear system (u,v)=(u,v)+(∆u,∆v) until converge









                            













y x y y x x y x y y x y x y x y x y x x y x I y x J I y x I y x J I v u I I I I I I , , , 2 , , , 2 ) , ( ) , ( ) , ( ) , (

Parametric model







    y x y x J v y u x I v u E , 2 ) , ( ) , ( ) , (







  x x p) W(x; p) ( ) ( ) 2 ( I J E T y x y x d d p d y d x ) , ( ,           p) W(x; translation T y x yy yx xy xx y yy yx x xy xx d d d d d d p y x d d d d d d ) , , , , , ( , 1 1 1                       Ax d p) W(x; affine

Parametric model







  x x Δp) p W(x; ) ( ) 2 ( J I minimize with respect to Δp Δp p W p) W(x; Δp) p W(x;      ) ( ) ( Δp p W p) W(x; Δp) p W(x;      I I Δp p W x p) W(x;       I( ) I



           x x Δp p W p) W(x; 2 ) ( ) ( I J I minimize

Parametric model



           x x Δp p W p) W(x; 2 ) ( ) ( I J I image gradient

Jacobian of the warp

warped image                                            n y y y n x x x y x p W p W p W p W p W p W p W p W p W   2 1 2 1

Jacobian of the warp

For example, for affine

                                y yy yx x xy xx y yy yx x xy xx d y d x d d y d x d y x d d d d d d ) 1 ( ) 1 ( 1 1 1 p) W(x;                                            n y y y n x x x y x p W p W p W p W p W p W p W p W p W   2 1 2 1          1 0 0 0 0 1 0 0 y x y x p W

Parametric model



           x x Δp p W p) W(x; 2 ) ( ) ( I J I minimize



                     x x Δp p W p) W(x; p W ) ( ) ( 0 I I I J T







_            x p) W(x; x p W H Δp 1 I J( ) I( ) T



                   x p W p W H I I T Hessian

Lucas-Kanade algorithm

iterate

warp I with W(x;p)

compute error image J(x,y)-I(W(x,p)) compute gradient image

evaluate Jacobian at (x;p) compute compute Hessian compute solve update p by p+ until converge p W   p W   I    _          x p) W(x; x p W ) ( ) ( I J I T Δp Δp







_            x p) W(x; x p W H Δp 1 I J( ) I( ) T

Coarse-to-fine strategy

J _warp Jw _refine I in a a  + J _warp Jw _refine I a a  + J pyramid construction J _warp Jw _refine I a  + I pyramid construction out

a

Tracking

0 ) , , ( ) 1 , , (x  u y  v t   I x y t  I 0 ) , , ( ) , , ( ) , , ( ) , , ( ) , , (x y t  uI x y t  vI x y t  I x y t  I x y t  I _{x y t} 0 ) , , ( ) , , ( ) , , (x y t  vI x y t  I x y t  uI_{x y t} 0    _{y t} xu I v I

I optical flow constraint equation

Area-based method

Demo for aperture problem

• http://www.sandlotscience.com/Distortions/Br eathing_objects.htm • http:// www.sandlotscience.com/Ambiguous/barberpol e.htm

Aperture problem

• Larger window reduces ambiguity, but easily violates spatial smoothness assumption

Area-based method

• Assume spatial smoothness







   y x t y xu I v I I v u E , 2 ) , (

Area-based method

Area-based method

• The eigenvalues tell us about the local image st ructure.

• They also tell us how well we can estimate the flow in both directions

KLT tracking

• Select feature by

• Monitor features by measuring dissimilarity

 

 , )  ( _{1 2}

KLT tracking

KLT tracking

SIFT tracking (matching actually)



SIFT tracking



SIFT tracking



KLT vs SIFT tracking

• KLT has larger accumulating error; partly

because our KLT implementation doesn’t have affine transformation?

Single-motion assumption

Violated by • Motion discontinuity • Shadows • Transparency • Specular reflection • …

Robust statistics

• Recover the best fit for the majority of the data

Temporal artifacts

Reference

• B.D. Lucas and T. Kanade,

An Iterative Image Registration Technique with an Application to Stereo Vis ion

, Proceedings of the 1981 DARPA Image Understanding Workshop, 1981, pp1 21-130.

• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R.,

Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.

• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600.

• Michael Black and P. Anandan,

The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smoot h Flow Fields

, Computer Vision and Image Understanding 1996, pp75-104.

• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework

, International Journal of Computer Vision, 56(3), 2004, pp221 - 255.

• Peter Litwinowicz, Processing Images and Video for An Impressionist Effects

, SIGGRAPH 1997.

• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz,

Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004,