Motion estimation

112  Download (0)

Full text

(1)

Motion estimation

Digital Visual Effects Yung-Yu Chuang

with slides by Michael Black and P. Anandan

(2)

Motion estimation

• Parametric motion (image alignment)

• Tracking

• Optical flow

(3)

Parametric motion

direct method for image stitching

(4)

Tracking

(5)

Optical flow

(6)

Three assumptions

• Brightness consistency

• Spatial coherence

• Temporal persistence

(7)

Brightness consistency

Image measurement (e.g. brightness) in a small region remain the same although their location may change.

(8)

Spatial coherence

• Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.

• Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.

(9)

Temporal persistence

The image motion of a surface patch changes gradually over time.

(10)

Image registration

Goal: register a template image T(x) and an input image I(x), where x=(x,y)T. (warp I so that it

matches T)

Image alignment: I(x) and T(x) are two images

Tracking: T(x) is a small patch around a point p in the image at t. I(x) is the image at time t+1.

Optical flow: T(x) and I(x) are patches of images at t and t+1.

T fixed

I warp

(11)

Simple approach (for translation)

• Minimize brightness difference

 

y x

y x T v

y u x

I v

u E

,

) 2

, ( )

, (

) , (

(12)

Simple SSD algorithm

For each offset (u, v) compute E(u,v);

Choose (u, v) which minimizes E(u,v);

Problems:

• Not efficient

• No sub-pixel accuracy

(13)

Lucas-Kanade algorithm

(14)

Newton’s method

• Root finding for f(x)=0

• March x and test signs

• Determine Δx (small slow; large miss)→ →

(15)

Newton’s method

• Root finding for f(x)=0

(16)

Newton’s method

• Root finding for f(x)=0 Taylor’s expansion:

 

0 0 0 2

0

'' ( )

2 ) 1

( ' )

( )

( xf x f xf x

f

 ) ( ) ' ( ) ( x

0

f x

0

f x

0

f   

) (

'

) (

n n

n

f x

x

f

 

) (

'

) (

1

n n n

n

f x

x x f

x

 

(17)

Newton’s method

• Root finding for f(x)=0

x0 x1

x2

) (

'

) (

n n n

x f

x

f

 

(18)

Newton’s method

pick up x=x0 iterate

compute

update x by x+Δx until converge

Finding root is useful for optimization because Minimize g(x) find root for f(x)=g’(x)=0

) ( '

) (

x f

x x   f

(19)

Lucas-Kanade algorithm

 

y x

y x T v

y u x

I v

u E

,

) 2

, ( )

, (

) , (

y

x vI

uI y

x I v

y u x

I( , ) ( , )

 

y x

y

x vI

uI y

x T y

x I

,

) 2

, ( )

, (

 

y x

y x

x I x y T x y uI vI

u I E

,

) , ( )

, ( 2

0

 

y x

y x

y I x y T x y uI vI

v I E

,

) , ( )

, ( 2

0

(20)

Lucas-Kanade algorithm

 

y x

y x

x I x y T x y uI vI

u I E

,

) , ( )

, ( 2

0

 

y x

y x

y I x y T x y uI vI

v I E

,

) , ( )

, ( 2

0

 

 

 

y x

y y

x

y y

x

y x

y

x x y

x y

x y

x

x

y x I y

x T I v

I u

I I

y x I y

x T I v

I I u

I

, ,

2 ,

, ,

,

2

) , ( )

, (

) , ( )

, (

 

 

y x

y y x

x

y x

y y

x

y x

y x

y x y

x

x

y x I y

x T I

y x I y

x T I v

u I

I I

I I I

, ,

,

2 ,

, ,

2

) , ( )

, (

) , ( )

, (

(21)

Lucas-Kanade algorithm

iterate

shift I(x,y) with (u,v)

compute gradient image Ix, Iy

compute error image T(x,y)-I(x,y) compute Hessian matrix

solve the linear system (u,v)=(u,v)+(∆u,∆v)

until converge

 

 

y x

y y x

x

y x

y y

x

y x

y x

y x y

x

x

y x I y

x T I

y x I y

x T I v

u I

I I

I I I

, ,

,

2 ,

, ,

2

) , ( )

, (

) , ( )

, (

(22)

Parametric model

 

y x

y x T v

y u x

I v

u E

,

) 2

, ( )

, (

) , (

 

x

x p)

W(x;

p) ( ) ( ) 2

( I T

E

T y x

y

x p d d

d y

d

x , ( , )



p)

translation W(x;

T y x

yy yx

xy xx

y yy

yx

x xy

xx

d d

d d

d d

p

y x d

d d

d d

d

) ,

, ,

, ,

(

, 1 1

1





Ax d p)

affine W(x;

Our goal is to find p to minimize E(p) for all x in T’s domain

(23)

Parametric model

 

x

x Δp)

p

W(x; ) ( ) 2

( T

minimize I

with respect to Δp

p Δp p) W

W(x;

Δp) p

W(x;

) (

)

( Δp

p p) W

W(x;

Δp) p

W(x;

I

I

p Δp W p) x

W(x;

I

I( )





x

x p Δp

p) W W(x;

2

) ( )

( I T

minimize I

(24)

Parametric model





x

x p Δp

p) W W(x;

2

) ( )

( I T

I

image gradient

Jacobian of the warp warped image

n y y

y

n x x

x

y x

p W p

W p

W

p W p

W p

W

p W p W p

W

2 1

2 1

target image

(25)

Jacobian matrix

• The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function.

m

F : R

n

R

)) ,

, ( ),

, ,

( ),

, ,

( (

) ,

, (

2 1

2 1

2 2

1 1

2 1

n m

n n

n

x x

x f

x x

x f

x x

x f

x x

x F

) ,

, (

) ,

, (

or

) ,

, (

2 1

2 1

2 1

n m

n F

x x

x

f f

f

x x

x J

 

 

 

 

n m m

n

x f x

f

x f x

f

1

1 1

1

Δx x

x Δx

x ) ( ) ( )

( F J

F

F   

(26)

Jacobian matrix

   0 , 0 , 2

3

: R      R F

 cos

sin sin

cos sin

r v

r u

r t

 )

, , ( )

, ,

( r t u v

F   

















 

 

0 sin

cos

cos sin

sin cos

sin sin

sin sin

cos cos

cos sin

) , , (

r

r r

r r

v v

r v

u u

r u

t t

r t r

JF

(27)

Parametric model





x

x p Δp

p) W W(x;

2

) ( )

( I T

I

image gradient

Jacobian of the warp warped image

n y y

y

n x x

x

y x

p W p

W p

W

p W p

W p

W

p W p W p

W

2 1

2 1

target image

(28)

Jacobian of the warp

For example, for affine









y yy

yx

x xy

xx y

yy yx

x xy

xx

d y

d x

d

d y

d x

y d x d

d d

d d

d

) 1

( ) 1

( 1 1

p) 1 W(x;

n y y

y

n x x

x

y x

p W p

W p

W

p W p

W p

W

p W p W p

W

2 1

2 1





1 0

0 0

0 1

0 0

y x

y x

p W

dxx dyx dxy dyy dx dy

(29)

Parametric model





x

x p Δp

p) W W(x;

2

) ( )

( I T

I

x

x p Δp

p) W p W(x;

W ( ) ( )

0 I I I T

T

 

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

x p

W p

H I W I

T

(Approximated) Hessian

min

Δp

arg

(30)

Lucas-Kanade algorithm

iterate

1) warp I with W(x;p)

2) compute error image T(x,y)-I(W(x,p)) 3) compute gradient image with W(x,p) 4) evaluate Jacobian at (x;p)

5) compute

6) compute Hessian 7) compute

8) solve

9) update p by p+

until converge

p W

p

W

I

x

p) W(x;

p x

W T( ) I( ) I

T

Δp

Δp

 

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

I

(31)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(32)

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

(33)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(34)

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

(35)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(36)

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

(37)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(38)

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

(39)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(40)

x

p) W(x;

p x H W

Δp 1 I T( ) I( )

T

(41)

x W(x;p)

p H W

Δp 1 I T( ) I( )

T

(42)

Coarse-to-fine strategy

J warp Jw refine I

ain

a

+

J warp Jw refine I

a

a

+ J

pyramid construction

J warp Jw refine I

a

+

I

pyramid construction

a

out

(43)

Application of image alignment

(44)

Direct vs feature-based

• Direct methods use all information and can be very accurate, but they depend on the fragile

“brightness constancy” assumption.

• Iterative approaches require initialization.

• Not robust to illumination change and noise images.

• In early days, direct method is better.

• Feature based methods are now more robust and potentially faster.

• Even better, it can recognize panorama without initialization.

(45)

Tracking

(46)

Tracking

I(x,y,t) (u, v) I(x+u,y+v,t+1)

(47)

Tracking

0 )

, , ( )

1 ,

,

(x u y v t I x y t I

0 )

, , ( )

, , ( )

, , ( )

, , ( )

, ,

(x y t uI x y t vI x y t I x y t I x y t

I x y t

0 )

, , ( )

, , ( )

, ,

(x y t vI x y t I x y t

uIx y t

0

y t

xu I v I

I optical flow constraint equation brightness constancy

(48)

Optical flow constraint equation

(49)

Multiple constraints

(50)

Area-based method

• Assume spatial smoothness

(51)

Area-based method

• Assume spatial smoothness

 

y x

t y

xu I v I

I v

u E

,

) 2

, (

(52)

Area-based method

must be invertible

(53)

Area-based method

• The eigenvalues tell us about the local image structure.

• They also tell us how well we can estimate the flow in both directions.

• Link to Harris corner detector.

(54)

Textured area

(55)

Edge

(56)

Homogenous area

(57)

KLT tracking

• Select features by

• Monitor features by measuring dissimilarity

, ) ( 1 2 min

(58)

Aperture problem

(59)

Aperture problem

(60)

Aperture problem

(61)

Demo for aperture problem

• http://www.sandlotscience.com/Distortions/Br eathing_Square.htm

• http://www.sandlotscience.com/Ambiguous/Ba rberpole_Illusion.htm

(62)

Aperture problem

• Larger window reduces ambiguity, but easily violates spatial smoothness assumption

(63)
(64)
(65)

KLT tracking

http://www.ces.clemson.edu/~stb/klt/

(66)

KLT tracking

http://www.ces.clemson.edu/~stb/klt/

(67)

SIFT tracking (matching actually)

Frame 0  Frame 10

(68)

SIFT tracking

Frame 0  Frame 100

(69)

SIFT tracking

Frame 0  Frame 200

(70)

KLT vs SIFT tracking

• KLT has larger accumulating error; partly

because our KLT implementation doesn’t have affine transformation?

• SIFT is surprisingly robust

• Combination of SIFT and KLT (example)

http://www.frc.ri.cmu.edu/projects/buzzard/smalls/

(71)

Rotoscoping (Max Fleischer 1914)

1937

(72)

Tracking for rotoscoping

(73)

Tracking for rotoscoping

(74)

Waking life (2001)

(75)

A Scanner Darkly (2006)

• Rotoshop, a proprietary software. Each minute of animation required 500 hours of work.

(76)

Optical flow

(77)

Single-motion assumption

Violated by

• Motion discontinuity

• Shadows

• Transparency

• Specular reflection

• …

(78)

Multiple motion

(79)

Multiple motion

(80)

Simple problem: fit a line

(81)

Least-square fit

(82)

Least-square fit

(83)

Robust statistics

• Recover the best fit for the majority of the data

• Detect and reject outliers

(84)

Approach

(85)

Robust weighting

Truncated quadratic

(86)

Robust weighting

Geman & McClure

(87)

Robust estimation

(88)

Fragmented occlusion

(89)
(90)
(91)

Regularization and dense optical flow

• Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.

• Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.

(92)
(93)
(94)
(95)
(96)
(97)
(98)

Input image Horizontal motion

Vertical motion

(99)
(100)
(101)

Application of optical flow

video

matching

(102)
(103)

Input for the NPR algorithm

(104)

Brushes

(105)

Edge clipping

(106)

Gradient

(107)

Smooth gradient

(108)

Textured brush

(109)

Edge clipping

(110)

Temporal artifacts

Frame-by-frame application of the NPR algorithm

(111)

Temporal coherence

(112)

References

• B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, Proceedings of the 1981 DARPA Image

Understanding Workshop, 1981, pp121-130.

• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R., Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.

• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600.

• Michael Black and P. Anandan, The Robust Estimation of Multiple Motions:

Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image Understanding 1996, pp75-104.

• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying

Framework, International Journal of Computer Vision, 56(3), 2004, pp221 - 255.

• Peter Litwinowicz, Processing Images and Video for An Impressionist Effects, SIGGRAPH 1997.

• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz,

Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004, pp584-591.

Figure

Updating...

References

Related subjects :