Structure from motion II-Digital visual effects. spring 2005

Structure from motion II

Digital Visual Effects, Spring 2005

Yung-Yu Chuang

2005/5/4

with slides by Richard Szeliski, Steve Seitz, Marc Pollefyes and D aniel Martinec

Announcements

• Project #2 artifacts voting.

• Project #3 will be online tomorrow, hopefully. • Scribe schedule.

Outline

• Factorization methods

– Orthogonal – Missing data – Projective

– Projective with missing data

Recap: epipolar geometry

0

Fx

Structure from motion

2D feature

tracking 3D estimation _{(bundle adjust)}optimization

geometry fitting

Notations

• n 3D points are seen in m views • q=(u,v,1): 2D image point

• p=(x,y,z,1): 3D scene point  : projection matrix

 : projection function

• qij is the projection of the i-th point on image j

 ij projective depth of qij

)

(

_{j i} ij

p

q









(x, y, z)  (x / z, y / z) z ij 



SFM under orthographic projection

2D image point orthographic projection matrix 3D scene point image offset

t

Πp

q





1 2 23 31 21 • Trick

– Choose scene origin to be centroid of 3D points – Choose image origins to be centroid of 2D points – Allows us to drop the camera translation:

Πp

factorization (Tomasi & Kanade)









n 3 3 2 n 2 _    n



1 2 n 2 1 q q p p p q _{ }

projection of n features in one image:





n 3 3 2m n 2m 2 1 2 1 2 1 2 22 21 1 12 11                               n m mn m m n n p p p Π Π Π q q q q q q q q q         

projection of n features in m images

W

measurement

M

motion

S

shape

n 3 3 m 2 n 2m

W





M



'

S



'

• Factorization Technique

– W is at most rank 3 (assuming no noise)

– We can use singular value decomposition to factor W:

Factorization

– S’ differs from S by a linear transformation A:

– Solve for A by enforcing metric constraints on M

)

)(

(

'

'

S

MA

AS

M

W





1 n 3 3 m 2 n 2m

W





M



S



Metric constraints

• Orthographic Camera

– Rows of  are orthonormal:

• Enforcing “Metric” Constraints

– Compute A such that rows of M have these propertie s

M

A

M

'



        T ₀1 ₁0

Trick (not in original Tomasi/Kanade paper, but in followup work) • Constraints are linear in AAT _:

• Solve for G first by writing equations for every _i in M • Then G = AAT _{by SVD (since U = V)}  T T T T _{  A A   G where G  AA}          ' ' ' ' 1 0 0 1

n m 2 n 3 3 m 2 n 2m

W





M



S





E



Factorization with noisy data

• SVD gives this solution

– Provides optimal rank 3 approximation W’ of W

n m 2 n 2m n 2m

W





W



'



E

 • Approach

– Estimate W’, then use noise-free factorization of W’ as before

– Result minimizes the SSD between positions of image features and projection of the reconstruction

Factorization method

with missing data

Why missing data?

• occlusions

• tracking failure

W is only partially filled, factorization doesn’t work

Tomasi & Kanade

• Hallucination/propagation

4 points in 3 views determine structure and motion

Tomasi & Kanade

Tomasi & Kanade

Tomasi & Kanade

• Disadvantages

– Finding the largest full submatrix of a matrix with mi ssing elements is NP-hard.

– The data is not used symmetrically, these inaccuraci es will propagate in the computation of additional m issing elements.

Shum, Ikeuchi & Reddy

• Treat SVD as a PCA with missing data problem w hich is a weighted least square problem.

• Assume that W consists of n m-d points with me an t and covariance . If the rank of W is r, the problem of PCA is to find U,S,V such that is minimal. • If W is incomplete, it becomes T T _USV et W  





2 visible is T 2 1 min 



  ij q j i j ij t u v q 

Shum, Ikeuchi & Reddy

• To be solvable, the number of observable elem ents c in W must be larger than r(m+n-r)

• If we arrange W as an c-d vector w, we can rew rite it as

• To reach minimum, u and v satisfies: f f T 2 1 min  

Gv

w

Bu

t

w

f











0

T T T T





















w

G

Gv

G

t)

(w

B

Bu

B

Shum, Ikeuchi & Reddy

• Nonlinear, solved by iterating between “fixing v and solving u” and “fixing u and solving v”

1) initialize v 2) update

3) update

4) stop if convergence, or go back to step 2

• The above procedure can be further simplified by taking advantage of the sparse structure.

t)

(w

B

u







w

G

v





Shum, Ikeuchi & Reddy

Linear fitting

• Try to find a rank-r matrix so that i s minimal.

• Each column of W is an m-d vector. SVD tries to find an r-d linear space L that is closest to thes e n m-d vectors and projects these vectors to L. • A matrix describes a vector space.

Linear fitting

• Without noise, each triplet of columns of M

exactly specifies L. When there is missing data, each triplet only forms a constraint.

• For SFM, r=3, We can combine constraints to find L ) , , ( L ) , , (ij k span Ai Aj Ak 

Linear fitting

t t S   L _t t S   L

Let Nt denote a matrix representation of , that is, each column of Nt is a vector orthogonal to the space .

If N=[N1,N2,…Nl], then L is the null space of N.

Because of noise, the matrix N will typically have full rank. Taking the SVD of N, and find its three least significant components. If fourth smallest singular value of this matrix is less than 0.001, the result is unreliable.

This method can be used as the initialization for Shum’s method.

t

S

t

Factorization method

Factorization for projective projection





n 4 4 3m n 3m 2 1 2 1 2 2 1 1 2 2 22 22 21 21 1 11 12 11 11 11                               n m mn mn m m m m n n n p p p Π Π Π q q q q q q q q q                   i j ij ij

q





p



projective depth

W has rank at most 4. The problem is that we don’t know .

Sturm & Triggs

For the p-th point, its projective depths for the i-th and j-th images are related by

Factorization method with

projective projection and

Mahamud et. al.

2 2

_|

₍

₎

_|

|

|













_{ij ij ij i j ij ij i j} ij

z

q

M

P

q

M

P

E

2 1 2 ) (

₍

₎

i i n j j i ij P i

q

M

P

C

m

E











Project #3 Matchmove

• Assigned: 5/4

• Due: 11:59pm 5/24 • Work in pairs

• Implement Tomasi/Kanade factorization metho d.

• Some matlab implementations are provided as r eference for implementation details.

Bells & whistles

• Tracking

• Extensions of factorization methods (Jacobs, Ma hamud are recommended)

• Bundle adjustment

Artifacts

• Take your own movie and insert some objects into it.

• Sony TRV900, progressive mode, 15fps • Capturing machine in 219

Submission

• You have to turn in your complete source, the executable, a html report and an artifact.

• Report page contains:

description of the project, what do you learn,

algorithm, implementation details, results, bells and whistles…

• Artifacts must be made using your own program.

Reference software

• Famous matchmove software include 3D-Equaliz er, boujou, REALVIS MatchMover, PixelFarm PFT rack... Most are very expensive

• We will use Icarus, predecessor of PFTrack. It w ill be available at project’s page (id/password).

ICARUS

• Three main components:

– Distortion – Calibration

– Reconstruction

• Capturing video

– Enough depth variance – Fixed zoom if possible – Static scene if possible

Reference

• Heung-Yeung Shum, Katsushi Ikeuchi and Raj Reddy,

Principal Component Analysis with Missing Data and Its Application to Polyhedral Object Modeling

, PAMI 17(9), 1995. • David Jacobs,

Linear Fitting with Missing Data for Structure from Motion, Comput

er Vision and Image Understanding, 2001 • Peter Sturm and Bill Triggs,

A factorization Based Algorithm for Multi-Image Projective Structu re and Motion

, ECCV 1996.

• Shyjan Mahamud, Martial Hebert, Yasuhiro Omori and Jean Ponce,

Provably-Convergent Iterative Methods for Projective Structure fr om Motion