• 沒有找到結果。

Multi‐view 3D Reconstruction  for Dummies

N/A
N/A
Info
下載
Protected

Academic year: 2022

Share "Multi‐view 3D Reconstruction  for Dummies"

Copied!
46
0
0

加載中.... (立即查看全文)

立即下載 ( 46 頁 )

全文

(1)

=

+ +

Multi‐view 3D Reconstruction  for Dummies

Jianxiong Xiao

(2)

SFMedu Program with Code

=

+ +

Download from: 

http://mit.edu/jxiao/Public/software/SFMedu/

(3)

Camera projection

• When people take a picture of a point:

x = K R t  X

(4)

Camera projection

• When people take two pictures with same  camera setting:

x

1

= K R 

1

t

1

X

x

2

= K R 

2

t

2

X

(5)

Camera projection

• When people take three pictures with same  camera setting:

x

1

= K R 

1

t

1

X

x

2

= K R 

2

t

2

X

x

3

= K R 

3

t

3

X

(6)

Image 1

Image 2

Image 3

R

1

,t

1

R

2

,t

2

R

3

,t

3

X

Camera projection

x

1

x

2

x

3

(7)

Image 1

Image 2

Image 3

R

1

,t

1

R

2

,t

2

R

3

,t

3

X

1

Camera projection

X

2

X

3

X

4

X

5

X

6

X

7

x

11

x

12

x

13

(8)

Camera projection

x

11

= K R 

1

t

1

X

1

x

12

= K R 

2

t

2

X

1

x

13

= K R 

3

t

3

X

1

x

12

= K R 

1

t

1

X

2

x

22

= K R 

2

t

2

X

2

x

23

= K R 

2

t

2

X

3

x

33

= K R 

3

t

3

X

3

Point 1 Point 2 Point 3

Image 1 Image 2 Image 3

Same Camera Same Setting = Same  K

(9)

Image 1

Image 2

Image 3

R

1

,t

1

R

2

,t

2

R

3

,t

3

Triangulation

x

1

x

2

x

3

X

(10)

Structure From Motion

• Structure = 3D Point Cloud of the Scene

• Motion = Camera Location and Orientation

• SFM = Get the Point Cloud from Moving Cameras

• Structure and Motion: Joint Problems to Solve

=

+ +

(11)

Multi‐view Stereo (MVS) Structure from Motion (SFM)

Pipeline

(12)

Multi‐view Stereo (MVS) Structure from Motion (SFM)

Pipeline

(13)

Two‐view Reconstruction

(14)

Two‐view Reconstruction

(15)

Two‐view Reconstruction

keypoints

keypoints

match fundamental matrix

essential

matrix [R|t] triangulation

(16)

Keypoints Detection

keypoints keypoints

match fundamental matrix

essential

matrix [R|t] triangulation

(17)

Descriptor for each point

keypoints keypoints

match fundamental matrix

essential

matrix [R|t] triangulation SIFT

descriptor SIFT descriptor

(18)

Same for the other images

keypoints keypoints

match fundamental matrix

essential

matrix [R|t] triangulation SIFT

descriptor

SIFT descriptor

SIFT descriptor SIFT

descriptor

(19)

Point Match for correspondences

keypoints keypoints

match fundamental matrix

essential

matrix [R|t] triangulation SIFT

descriptor

SIFT descriptor

SIFT descriptor SIFT

descriptor

(20)

Point Match for correspondences

keypoints keypoints

match fundamental matrix

essential

matrix [R|t] triangulation SIFT

descriptor

SIFT descriptor

SIFT descriptor SIFT

descriptor

(21)

Image 1

Image 2

R

1

,t

1

R

2

,t

2

X

Fundamental Matrix

x

1

x

2

x 1 T Fx 2  0

x

1

 x

2

(22)

Estimating Fundamental Matrix

• Given a correspondence

• The basic incidence relation is

x

1T

Fx

2

 0

x

1

 x

2

x

1

x

2

, x

1

y

2

, x

1

, y

1

x

2

, y

1

y

2

, y

1

, x

2

, y

2

,1

 

f

11

f

12

f

13

f

21

f

22

f

23

f

31

f

32

f

33

 

 

 

 

 

 

 

 

 

 

 

 

 0

Need 8 points

(23)

Estimating Fundamental Matrix

x

1T

Fx

2

 0

x

11

 x

12

, x

12

 x

22

, x

13

 x

32

, x

14

 x

42

, x

15

 x

52

, x

16

 x

62

, x

17

 x

72

, x

18

 x

82

x

11

x

12

x

11

y

12

x

11

y

11

x

12

y

11

y

12

y

11

x

12

y

12

1 x

12

x

22

x

12

y

22

x

12

y

12

x

22

y

12

y

22

y

12

x

22

y

22

1 x

13

x

23

x

13

y

32

x

13

y

13

x

32

y

13

y

32

y

13

x

32

y

32

1 x

14

x

24

x

14

y

24

x

14

y

14

x

42

y

14

y

42

y

14

x

42

y

42

1 x

15

x

25

x

15

y

52

x

15

y

15

x

52

y

15

y

52

y

15

x

52

y

52

1 x

16

x

62

x

16

y

62

x

16

y

16

x

62

y

16

y

62

y

16

x

62

y

62

1 x

17

x

72

x

17

y

27

x

17

y

17

x

72

y

17

y

72

y

17

x

72

y

72

1 x

18

x

82

x

18

y

82

x

18

y

18

x

82

y

18

y

82

y

18

x

82

y

82

1

 

 

 

 

 

 

 

 

 

 

 

 

f

11

f

12

f

13

f

21

f

22

f

23

f

31

f

32

f

33

 

 

 

 

 

 

 

 

 

 

 

 

 0

for 8 point correspondences:

Direct Linear Transformation (DLT)

Ax  b

Af  0

(24)

RANSAC to Estimate Fundamental Matrix

• For many times

– Pick 8 points

– Compute a solution for      using these 8 points – Count number of inliers

• Pick the one with the largest number of inliers

F

(25)

Image 1

Image 2

R

1

,t

1

R

2

,t

2

X

Fundamental Matrix  Essential Matrix

x

1

x

2

x 1 T Fx 2  0

E  K 1 T FK 2

(26)

Image 1

Image 2

R

1

,t

1

R

2

,t

2

X

Essential Matrix 

x

1

x

2

x 1 T Fx 2  0

E  K 1 T FK 2

  R t

(27)

Result 9.19. For a given essential matrix

and the first camera matrix       , there are four  possible choices for the second camera matrix     :

Page 259 of the bible (Multiple View Geometry, 2ndEd)

Essential Matrix    R t

E  Udiag 1,1, 0   V

T

,

P

1

 I 0  

P

2

P

2

 UWV 

T

u

3



P

2

 UWV 

T

u

3



P

2

 UW 

T

V

T

u

3



P

2

 UW 

T

V

T

u

3



W

0 1 0

1 0 0

0 0 1

 

 

(28)

Four Possible Solutions

Page 260 of the bible (Multiple View Geometry, 2ndEd)

(29)

In front of the camera?

• Camera Extrinsic

• Camera Center

• View Direction

  R t

Xcam Ycam

Zcam





 R

Xworld Yworld

Zworld





 t

Xworld Yworld

Zworld





 R1

Xcam Ycam

Zcam





 t





 RT

Xcam Ycam

Zcam





 RTt

Xcam Ycam

Zcam





 0 0 0





C

Xworld Yworld

Zworld





 RT 0 0 0





 RTt  RTt

0 0 1





 0 0 0





RT 0 0 1





 RTt





 C

 

 R(3,:)

T  RTt

 R

Tt

 R(3,:)T

Camera Coordinate System World Coordinate System

(30)

In front of the camera?

• A point

• Direction from camera center to point

• Angle Between Two Vectors

• Angle Between      and View Direction

• Just need to test 

X

X  C

A  B  A B cos

X  C

   R(3,:)

T

 0?

X  C

(31)

Pick the Solution

Page 260 of the bible (Multiple View Geometry, 2ndEd)

With maximal number of points in front of both cameras. 

(32)

Two‐view Reconstruction

keypoints

keypoints

match fundamental matrix

essential

matrix [R|t] triangulation

(33)

Multi‐view Stereo (MVS) Structure from Motion (SFM)

Pipeline

(34)

Taught Next

Pipeline

(35)

Merge Two Point Cloud

(36)

Merge Two Point Cloud

There can be only one   R

2

t

2



(37)

Merge Two Point Cloud

• From the 1 st and 2 nd images, we have and

• From the 2 nd and 3 rd images, we have and

• Exercise:  How to transform the coordinate 

system of the second point cloud to align with  the first point cloud so that there is only one

? 

R

1

t

1

   R

2

t

2



R

2

t

2

  

  R

3

t

3



R

2

t

2

  

(38)

Merge Two Point Cloud

(39)

Oops

See From a Different Angle

(40)

Bundle Adjustment

(41)

Image 1

Image 2

Image 3

R

1

,t

1

R

2

,t

2

R

3

,t

3

X

1

Camera projection

X

2

X

3

X

4

X

5

X

6

X

7

x

11

x

12

x

13

(42)

Camera projection

x

11

= K R 

1

t

1

X

1

x

12

= K R 

2

t

2

X

1

x

13

= K R 

3

t

3

X

1

x

12

= K R 

1

t

1

X

2

x

22

= K R 

2

t

2

X

2

x

23

= K R 

2

t

2

X

3

x

33

= K R 

3

t

3

X

3

Point 1 Point 2 Point 3

Image 1 Image 2 Image 3

Same Camera Same Setting = Same  K

(43)

Rethinking the SFM problem

• Input: Observed 2D image position

• Output:

Unknown Camera Parameters (with some guess) Unknown Point 3D coordinate (with some guess)

R

1

t

1

 , R 

2

t

2

, R 

3

t

3



x

1 1

x

12

x

13

x

1 2

x

222

x

32

x

33

X

1

, X

2

, X

3

,

(44)

Bundle Adjustment

A valid solution      and must let

x

11

x

2 1

x

3 1

x

12

x

2

22

x

2 3

x

3 3

R

1

t

1

 , R 

2

t

2

, R 

3

t

3

 X

1

, X

2

, X

3

,

Observation Re‐projection

x

11

= K R 

1

t

1

X

1

x

12

= K R 

2

t

2

X

1

x

13

= K R 

3

t

3

X

1

x

12

= K R 

1

t

1

X

2

x

22

= K R 

2

t

2

X

2

x

23

= K R 

2

t

2

X

3

x

33

= K R 

3

t

3

X

3

=

(45)

Bundle Adjustment

A valid solution      and must let the Re‐projection close to the 

Observation, i.e. to minimize the reprojection error

R

1

t

1

 , R 

2

t

2

, R 

3

t

3

 X

1

, X

2

, X

3

,

min  x i j  K R  i t i X j2

j

i

(46)

Solving This Optimization Problem

• Theory:

The Levenberg–Marquardt algorithm

• Practice:

The Ceres‐Solver from Google

http://en.wikipedia.org/wiki/Levenberg‐Marquardt_algorithm

http://code.google.com/p/ceres‐solver/

參考文獻

立即下載 ( PDF - 46 頁 - 1.02 MB )

相關文件

Label Space Coding for Multi-label Classification

HAMR =⇒ lower 0/1 loss, similar Hamming loss BCH =⇒ even lower 0/1 loss, but higher Hamming loss to improve Binary Relevance, use. HAMR or BCH =⇒ lower 0/1 loss, lower

Label Space Coding for Multi-label Classification

• label embedding: PLST, CPLST, FaIE, RAk EL, ECC-based [Tai et al., 2012; Chen et al., 2012; Lin et al., 2014; Tsoumakas et al., 2011; Ferng et al., 2013]. • cost-sensitivity: CFT,

Label Space Coding for Multi-label Classification

classify input to multiple (or no) categories.. Multi-label Classification.

Matrix Factorization and Factorization Machines for Recommender Systems

For categorical features like IDs, we have ID: field ID index: feature Each field has many 0/1 features. But how about

Matrix Factorization and Factorization Machines for Recommender Systems

Matrix Factorization is an effective method for recommender systems (e.g., Netflix Prize and KDD Cup 2011).. But training

Matrix Factorization and Factorization Machines for Recommender Systems

Matrix factorization (MF) and its extensions are now widely used in recommender systems.. In this talk I will briefly discuss three research works related to

ROC Representation for the Discriminability of Multi-Classification Markers

Receiver operating characteristic (ROC) curves are a popular measure to assess performance of binary classification procedure and have extended to ROC surfaces for ternary or

Stokes Phenomena and Non‐perturbative Completion in the multi‐cut matrix models

Normalizable moduli (sets of on‐shell vacua in string theory) Scale