• 沒有找到結果。

# fundamental matrix

N/A
N/A
Protected

Share "fundamental matrix"

Copied!
80
0
0

(1)

### Structure from motion

Digital Visual Effectsg Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz, Zhengyou Zhang and Marc Pollefyes

(2)

### Outline

• Epipolar geometry and fundamental matrix

S f i

• Structure from motion

• Factorization method

• Applications

• Applications

(3)

(4)

### The epipolar geometry

epipolar geometry demo

C C’ x x’ and X are coplanar C,C ,x,x and X are coplanar

(5)

### The epipolar geometry

What if only C C’ x are known?

What if only C,C ,x are known?

(6)

### The epipolar geometry

All points on  project on l and l’

All points on  project on l and l

(7)

### The epipolar geometry

Family of planes  and lines l and l’ intersect at e Family of planes  and lines l and l intersect at e and e’

(8)

### The epipolar geometry

epipolar pole

= intersection of baseline with image plane

epipolar geometry demo

= intersection of baseline with image plane

= projection of projection center in other image

epipolar plane = plane containing baseline epipolar plane plane containing baseline

epipolar line = intersection of epipolar plane with image

(9)

C C’

p R p’

C

T=C’-C

### p  

Two reference frames are related via the extrinsic parameters

(10)

###  

T T0 0Tz TTy

Multiply both sides by

T



 



0 0

x y

x z

T T

T T T

T

T

T

essential matrix

(11)

### Epp pp

Let M and M’ be the intrinsic matrices, then

1

1

1

1

1

### x

fundamental matrix

(12)

### The fundamental matrix F

• The fundamental matrix is the algebraic representation of epipolar geometry

representation of epipolar geometry

Th f d t l t i ti fi th diti

• The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’

i th t i

in the two images

T

T

(13)

### The fundamental matrix F

F is the unique 3x3 rank 2 matrix that satisfies xTFx’=0 for all x↔x’

for all x↔x

1. Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

2 Epipolar lines: l F ’ & l’ FT 2. Epipolar lines: l=Fx’ & l’=FTx

3. Epipoles: on all epipolar lines, thus eTFx’=0, x’

eTF=0, similarly Fe’=0

e F 0, similarly Fe 0

4. F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)

5. F is a correlation, projective mapping from a point x to a line l=Fx’ (not a proper correlation, i.e. not invertible)

(14)

### The fundamental matrix F

• It can be used for

– Simplifies matching

– Allows to detect wrong matchesAllows to detect wrong matches

(15)

### Estimation of F — 8-point algorithm

• The fundamental matrix F is defined by

### x

for any pair of matches x and x’ in two images.

f11 f12 f13

• Let x=(u,v,1)T and x’=(u’,v’,1)T,

21 22 23

13 12

11

f f

f

f f

f

f f

f F



 f31 f32 f33

each match gives a linear equation

0 '

' '

' '

' f11 uv f12 uf13 vu f21 vv f22 vf23 u f31 v f32 f33 uu

(16)

### 8-point algorithm

11

f f

1

´

´

´

´

´

´ 13

12

1 1

1 1

1 1

1 1

1 1 1

1

f f f v

u v

v v u

v u

v u u

u

1 0

´

´

´

´

´

´

22 21 2

2 2

2 2 2

2 2

2 2 2

2

1 1

1 1

1 1

1 1

1 1 1

1

f v f

u v

v v u

v u

v u u

u

1

´

´

´

´

´

´

31 23

f

v f u

v v

v u

v u

v u u

un n n n n n n n n n n n

33 32

 f f

33

 f

• In reality, instead of solving , we seek f to minimize subj. . Find the vector

### Af  0

Af j f  1

corresponding to the least singular value.

(17)

### 8-point algorithm

• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to

### FF ' det F ' 0

F that minimizes subject to .

### F  F ' det F '  0

• It is achieved by SVD. Let , where y

### F  U Σ V

,

let

1

0 0

0 0

Σ

0 0

0 0

Σ'

1

 , let

3 2

0 0

0 0

Σ

0 0

0

0 0

Σ'2

then is the solution.

(18)

### 8-point algorithm

% Build the constraint matrix

A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...[ ( , ) ( , ) ( , ) ( , ) ( , ) x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...

x1(1,:)' x1(2,:)' ones(npts,1) ];

[U,D,V] = svd(A);

% Extract fundamental matrix from the column of V

% corresponding to the smallest singular value.p g g F = reshape(V(:,9),3,3)';

% E f k2 t i t

% Enforce rank2 constraint [U,D,V] = svd(F);

F = U*diag([D(1 1) D(2 2) 0])*V';

F = U diag([D(1,1) D(2,2) 0]) V ;

(19)

### 8-point algorithm

• Pros: it is linear, easy to implement and fast

C ibl i

• Cons: susceptible to noise

(20)

### Problem with 8-point algorithm

11

f f

1

´

´

´

´

´

´ 13

12

1 1

1 1

1 1

1 1

1 1 1

1

f f f v

u v

v v u

v u

v u u

u

1 0

´

´

´

´

´

´

22 21 2

2 2

2 2 2

2 2

2 2 2

2

1 1

1 1

1 1

1 1

1 1 1

1

f v f

u v

v v u

v u

v u u

u

10000 10000 100 10000 10000 100 100 ~100 1

1

´

´

´

´

´

´

31 23

f

v f u

v v

v u

v u

v u u

un n n n n n n n n n n n

~10000 ~10000 ~100 ~10000 ~10000 ~100 ~100 ~100 1

!

Orders of magnitude difference

between column of data matrix 33

32

 f f

! between column of data matrix

 least-squares yields poor results

33

 f

(21)

### Normalized 8-point algorithm

1. Transform input by , 2 C ll 8 i b i

i

i

'i

'i

'

### ˆ ˆ

2. Call 8-point on to obtain 3.

i i

Τ

1

(22)

### Normalized 8-point algorithm

normalized least squares yields good results

T f i [ 1 1] [ 1 1]

(700,500)

(0,500) (-1,1) (1,1)

 2

Transform image to ~[-1,1]x[-1,1]

( , )

( , ) ( , ) ( , )

500 1

2

1 700 0

2

(0,0)

1

(0,0) (700,0) (-1,-1) (1,-1)

(23)

### Normalized 8-point algorithm

[x1, T1] = normalise2dpts(x1);

[x2, T2] = normalise2dpts(x2);

A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...

x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...

[ , ] p ( );

x1(1,:)' x1(2,:)' ones(npts,1) ];

[U D V] svd(A);

[U,D,V] = svd(A);

F = reshape(V(:,9),3,3)';

F reshape(V(:,9),3,3) ; [U,D,V] = svd(F);

F = U*diag([D(1,1) D(2,2) 0])*V';

% Denormalise F = T2'*F*T1;

(24)

### Normalization

function [newpts, T] = normalise2dpts(pts) c = mean(pts(1:2,:)')'; % Centroid

newp(1,:) = pts(1,:)-c(1); % Shift origin to centroid.p( , ) p ( , ) ( ); g newp(2,:) = pts(2,:)-c(2);

meandist = mean(sqrt(newp(1,:).^2 + newp(2,:).^2));

scale = sqrt(2)/meandist;

T = [scale 0 -scale*c(1) 0 scale -scale*c(2) 0 0 1 ];

t T* t newpts = T*pts;

(25)

### RANSAC

repeat

select minimal sample (8 matches) compute solution(s) for F

determine inliers

until (#inliers,#samples)>95% or too many times ( , p ) y compute F based on all inliers

(26)

(27)

(28)

(29)

(30)

### Structure from motion

Unknown Unknown Unknown Unknown camera camera viewpoints viewpoints

structure for motion: automatic recovery of camera motion and scene structure from two or more images. It is a self calibration technique and called automatic camera tracking calibration technique and called automatic camera tracking or matchmoving.

(31)

### Applications

• For computer vision, multiple-view shape reconstruction novel view synthesis and reconstruction, novel view synthesis and autonomous vehicle navigation.

F fil d ti l i ti f CGI

• For film production, seamless insertion of CGI into live-action backgrounds

(32)

### Matchmove

example #1 example #2 example #3 example #4

(33)

### Structure from motion

2D feature

tracking 3D estimation optimization (bundle adjust)

geometry fitting

SFM pipeline SFM pipeline

(34)

### Structure from motion

• Step 1: Track Features

Detect good features Shi & Tomasi SIFT – Detect good features, Shi & Tomasi, SIFT – Find correspondences between frames

• Lucas & Kanade-style motion estimationy

• window-based correlation

• SIFT matching

(35)

### KLT tracking

http://www ces clemson edu/~stb/klt/

http://www.ces.clemson.edu/ stb/klt/

(36)

### Structure from Motion

• Step 2: Estimate Motion and Structure

Si lifi d j ti d l [T i 92]

– Simplified projection model, e.g., [Tomasi 92]

– 2 or 3 views at a time [Hartley 00]

(37)

### Structure from Motion

• Step 3: Refine estimates

“B dl dj t t” i h t t – “Bundle adjustment” in photogrammetry – Other iterative methods

(38)

### Structure from Motion

• Step 4: Recover surfaces (image-based triangulation silhouettes stereo )

triangulation, silhouettes, stereo…)

Good mesh Good mesh

(39)

(40)

(41)

### Notations

• n 3D points are seen in m views

( 1) 2D i i

• q=(u,v,1): 2D image point

• p=(x,y,z,1): 3D scene point

• : projection matrix

• : projection function

• : projection function

• qij is the projection of the i-th point on image j

 j ti d th f

• ij projective depth of qij

ij

j

i

### ) 

(x y z) (x / z y / z)

### q    

(x, y, z) (x / z, y / z)

ijz

(42)

### Structure from motion

• Estimate and to minimize

j

i

) );

( ( log )

, ,

, ,

, (

1 1 1

1 j i ij

m

j

n

i

ij n

m p p w P Π p q

Π

Π

### 

j

otherwise

j in view visible

is if

0

1 i

ij

w p

otherwise

j 0

• Assume isotropic Gaussian noise, it is reduced top ,

2 1

1, , , , , ) ( )

(Π Π p p

### 

m n wij Π jpi qij

1 1 1

1, , , , , ) ( )

( j i ij

j i

ij n

m p p w Π p q

Π

Π

### 

• Start from a simpler projection model

• Start from a simpler projection model

(43)

### Orthographic projection

• Special case of perspective projection

Di t f th COP t th PP i i fi it – Distance from the COP to the PP is infinite

Image World

– Also called “parallel projection”: (x, y, z) → (x, y)

(44)

### SFM under orthographic projection

2D image Orthographic projection

incorporating 3D rotation 3D scene

image offset point incorporating 3D rotation 3D scene

point

offset

### q  

1

21 23 31 21 2 23 31 21

• Trick

– Choose scene origin to be centroid of 3D pointsg p – Choose image origins to be centroid of 2D points – Allows us to drop the camera translation:Allows us to drop the camera translation:

(45)

###   

projection of n features in one image:

n 3 3

n 2

2

n

### 

1 2 n

2

1 q q p p p

q

1 1 12

11

q q q n Π

projection of n features in m images

32

1 2 2

22 21

1 12

11

n n

n

p p

Π p q

q

q

n 3 3

n 2m 2m

2 1

qm qm qmn Πm

3 n 2m

2m

measurement

motion

### S

shape

Key Observation: rank(W) <= 3

(46)

3 3 2

2

### MS

known solve for

• Factorization Technique

n 3 3

m n 2

2m

Factorization Technique

– W is at most rank 3 (assuming no noise)

– We can use singular value decomposition to factor W:

3 3

2

2

### M ' S '

We can use singular value decomposition to factor W:

n 3 3

m n 2

2m

– S’ differs from S by a linear transformation A:

### W  

1

– Solve for A by enforcing metric constraints on M

(47)

### Metric constraints

• Orthographic Camera

R f  th l

  T 01 10

– Rows of  are orthonormal:

• Enforcing “Metric” Constraints

  0 1

– Compute A such that rows of M have these properties

### M 'AMM ' 

Trick (not in original Tomasi/Kanade paper, but in followup work)

• Constraints are linear in AAT :

T T T

T A A G where G AA

' ' ' '

1 0

0 1

• Solve for G first by writing equations for every i in M

• Then G = AAT by SVD (since U = V)

0 1

Then G AA by SVD (since U V)

(48)

### W

n 2m 3 3 n 2m n

2m

• SVD gives this solutiong

– Provides optimal rank 3 approximation W’ of W

n 2m n 2m n 2m

###  E

• Approach

– Estimate W’, then use noise-free factorization of W’

as before

– Result minimizes the SSD between positions of image features and projection of the reconstruction

(49)

(50)

### Extensions to factorization methods

• Projective projection Wi h i i d

• With missing data

• Projective projection with missing data

(51)

(52)

### Levenberg-Marquardt method

• LM can be thought of as a combination of steepest descent and the Newton method steepest descent and the Newton method.

When the current solution is far from the correct one the algorithm behaves like a correct one, the algorithm behaves like a

steepest descent method: slow, but guaranteed to converge When the current solution is close to converge. When the current solution is close to the correct solution, it becomes a Newton’s method

method.

(53)

T

(54)

(55)

### Levenberg-Marquardt method

• μ=0 → Newton’s method

d h d

• μ→∞ → steepest descent method

• Strategy for choosing μ

– If error is not reduced, keep trying larger μ until it does

– If error is reduced, accept it and reduce μ for the next iteration

(56)

• Bundle adjustment (BA) is a technique for simultaneously refining the 3D structure and simultaneously refining the 3D structure and camera parameters

It i bl f bt i i ti l

• It is capable of obtaining an optimal

reconstruction under certain assumptions on

i d l F G i

image error models. For zero-mean Gaussian image errors, BA is the maximum likelihood

ti t

estimator.

(57)

• n 3D points are seen in m views

i h j i f h i h i i j

• xij is the projection of the i-th point on image j

• ajj is the parameters for the j-th camera

• bi is the parameters for the i-th point

• BA attempts to minimize the projection error

• BA attempts to minimize the projection error

Euclidean distance

predicted projection Euclidean distance

(58)

(59)

3 views and 4 points

(60)

(61)

(62)

(63)

Multiplied by

(64)

### Issues in SFM

N li l di i

• Nonlinear lens distortion

• Degeneracy and critical surfaces

• Prior knowledge and scene constraints

• Multiple motions

• Multiple motions

(65)

every 50th frame of a 800-frame sequencey q

(66)

lifetime of 3192 tracks from the previous sequencep q

(67)

track length histogramg g

(68)

(69)

### Nonlinear lens distortion

effect of lens distortion

(70)

### Prior knowledge and scene constraints

add a constraint that several lines are parallelp

(71)

### Prior knowledge and scene constraints

add a constraint that it is a turntable sequenceq

(72)

(73)

(74)

### 2d3 boujou

Enemy at the Gate, Double Negative

(75)

### 2d3 boujou

Enemy at the Gate, Double Negative

(76)

(77)

### VideoTrace

http://www.acvt.com.au/research/videotrace/

(78)

(79)

### Project #3 MatchMove

• It is more about using tools in this project

Y h i h lib i

• You can choose either calibration or structure from motion to achieve the goal

• Calibration

• Voodoo/Icarus

• Examples from previous classes #1 #2

• Examples from previous classes, #1, #2

(80)

### References

• Richard Hartley, In Defense of the 8-point Algorithm, ICCV, 1995.

• Carlo Tomasi and Takeo Kanade,, Shape and Motion from Image p g Streams: A Factorization Method, Proceedings of Natl. Acad. Sci., 1993.

Manolis Lourakis and Antonis Argyros The Design and

• Manolis Lourakis and Antonis Argyros, The Design and

Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm, FORTH- ICS/TR 320 2004

ICS/TR-320 2004.

• N. Snavely, S. Seitz, R. Szeliski, Photo Tourism: Exploring Photo Collections in 3D, SIGGRAPH 2006.,

• A. Hengel et. al., VideoTrace: Rapid Interactive Scene Modelling from Video, SIGGRAPH 2007.

• Michael Grossberg, Shree Nayar, Determining the Camera Response from Images: What Is Knowable, PAMI 2003. • Michael Grossberg Shree Nayar Modeling the Space

• Why blending: parallax, lens distortion, scene motion exposure difference. motion,

steepest descent method: slow, but guaranteed to converge When the current solution is close to converge. When the current solution is close to the correct solution, it becomes

• external or extrinsic (pose) parameters including rotation and translation:.. where is

When the current solution is far from the correct one the algorithm behaves like a correct one, the algorithm behaves like a. steepest descent method: slow, but guaranteed

structure for motion: automatic recovery of camera motion and scene structure from two or more images.. It is a self calibration technique and called automatic camera tracking

It is a self calibration technique and called automatic camera tracking calibration technique and called automatic camera tracking or

Removing Camera Shake From a Single Photograph Coded Exposure Photography: Motion Deblurring Photo Tourism: Exploring Photo Collections in 3D Photo Tourism: Exploring Photo

Removing Camera Shake From a Single Photograph Coded Exposure Photography: Motion Deblurring Photo Tourism: Exploring Photo Collections in 3D AutoCollage. Photographing Long

Natural Video Matting using Camera Arrays Removing Camera Shake From a Single Photograph Coded Exposure Photography: Motion Deblurring Photo Tourism: Exploring Photo Collections in

A model of shape and motion model was learned from training sequences of hand motion against a plain background, tracked by Kalman filter (using signed.. The Condensation

• Michael Grossberg, Shree Nayar, Determining the Camera Response from Images: What Is Knowable, PAMI 2003. • Michael Grossberg, Shree Nayar, Modeling the Space of Camera

• Michael Grossberg, Shree Nayar, Determining the Camera Response from Images: What Is Knowable, PAMI 2003. • Michael Grossberg, Shree Nayar, Modeling the Space of Camera

• The scene with depth variations and the camera has movement... Planar scene (or a

• The scene with depth variations and the camera has movement... Planar scene (or a

Define instead the imaginary.. potential, magnetic field, lattice…) Dirac-BdG Hamiltonian:. with small, and matrix

Because simultaneous localization, mapping and moving object tracking is a more general process based on the integration of SLAM and moving object tracking, it inherits the

structure for motion: automatic recovery of camera motion and scene structure from two or more images.. It is a self calibration technique and called automatic camera tracking

a Called the risk-neutral probability measure... 560 uses the money market account as numeraire... – It expresses the price of any asset relative to the money market account.. 299)

Zhang, “A ﬂexible new technique for camera calibration,” IEEE Tran- scations on Pattern Analysis and Machine Intelligence,

(6) Matlab vehicle equation of motion program development (7) Pitch and yaw channels’ tracking-loop bandwidth

The major task of the research is to provide a sort of automatic system solution for producing the RF Power module in fast and accuracy by several core module from

Input data is the window data and the fall motion technique is the machine learning model with the extension of the checkpoint ensemble method...