A C R S 2 0 1 7

Student：Pin-Yun Chen, Kuan-Ying Lin Adviser：Yi-Hsing Tseng

ACRS 2017

Automatic Image Matching for Space Intersection

of Spherical Panorama Images

• Introduction

• Spherical Panorama Image (SPI)

• Portable Panoramic Image Mapping System (PPIMS)

• SPI Matching Strategy

• Matching method

• Matching procedure

• Experiment Analysis

• Test field

• Image matching

• Conclusions

Outline

Introduction

SPI

• Spherical Panorama Image (SPI)

180°

360°

(X,Y,Z)?

SPI

• Spherical Panorama Image (SPI)

5

PPIMS

• Portable Panoramic Image Mapping System (PPIMS)

• Combining GNSS receiver

Camera Specs

Model Sony NEX-5N

Resolution 16 megapixels Focal length 16 mm

Pixel size 4.8 𝜇𝑚

Field of view 80° × 53°

PPIMS

• PPIMS Coordinate System

7

Z

^M

Mapping frame

Y

^M

X

^M

X

^B

Body frame

Z

^B

Y

^B

𝑥

^𝐶

𝑧

^𝐶

𝑦

^𝐶

Camera frame

𝑟 _𝑐 ^𝐵 _𝑖

𝑅 _𝑐 ^𝐵 _𝑖

PPIMS

• Calibration Result of PPIMS Relative Orientation

Camera 𝒙_𝑪^𝑩_𝒊(𝐦) 𝒚_𝑪^𝑩_𝒊(𝐦) 𝒛_𝑪^𝑩_𝒊(𝐦) 𝛚_𝑪^𝑩_𝒊(°) 𝛗_𝑪^𝑩_𝒊(°) Ҡ_𝑪^𝑩_𝒊(°) No.1 -0.0965 0.1017 -0.0896 359.1348 109.4978 226.0600 No.2 -0.1453 0.0021 -0.0905 359.7416 109.8292 180.1320 No.3 -0.1050 -0.1021 -0.0937 0.0605 109.0724 135.5507 No.4 0.0019 -0.1517 -0.0958 357.1614 108.9608 92.6564 No.5 0.0982 -0.1041 -0.0896 359.4705 109.8352 46.0271 No.6 0.1462 -0.0023 -0.0907 359.4214 109.9843 0.1291 No.7 0.1078 0.1071 -0.0926 0.0300 110.0384 314.9899

No.8 0.0051 0.1425 -0.0886 0 110 270

Distribution of 8 cameras

Lever-arm vector & Boresight angle

PPIMS

• Formation of PPIMS SPI

•

9

PPIMS

• Bundle adjustment of multi-station SPIs (Lin, 2014)

• SPI center

• Ground control point

• SPI point

P (Object point) o

𝑟_𝐶^𝐵_𝑖

𝑟_𝑃^𝐵

𝑝^′(SPI Point) 𝑐_𝑖

𝑟

_𝑝′^𝐵

𝑥_𝑆1

𝑦_𝑆1 𝑧_𝑆1

𝑦_𝑆3 𝑥_𝑆3

𝑧_𝑆3

𝑥_𝑆2 𝑦_𝑆2 𝑧_𝑆2

P

𝑺𝑷𝑰 1

𝑺𝑷𝑰 2

𝑺𝑷𝑰 3

𝑟

_𝑝′^𝐵

+ 𝑣

_𝑟

𝑝′𝐵

= 1

𝜆′

_𝑝′^𝐵

∙ 𝑅

_𝑀^𝐵

∙ (𝑟

_𝑃^𝑀

− 𝑟

_𝐵^𝑀

) − 𝑟

_𝑐^𝐵_𝑖

SPI Matching Strategy

Matching Method

• Matching index

• SNCC (Sum of Normalized Cross Correlation) • YARD (Yet Another Reconstruction Dataprogram)

𝑁𝐶𝐶

_𝑘

= σ σ(𝑓

_𝑖𝑗𝑘

− 𝑓

_𝑘

)(𝑔

_𝑖𝑗

− 𝑔) σ σ(𝑓

_𝑖𝑗𝑘

− 𝑓

_𝑘

)

²

σ σ(𝑔

_𝑖𝑗

− 𝑔)

²

𝑆𝑁𝐶𝐶 = 1

𝑁 ෍ 𝑁𝐶𝐶

_𝑘

𝑌𝐴𝑅𝐷 = σ σ 𝑓

_𝑖𝑗

− 𝑓)(𝑔

_𝑖𝑗

− 𝑔

σ σ 𝑓

_𝑖𝑗

− 𝑓

²

σ σ 𝑔

_𝑖𝑗

− 𝑔

²

𝑓

_𝑖𝑗𝑘

: The gray value of pixel i, j in match image k 𝑓

_𝑘

: The mean value of match image k

𝑔

_𝑖𝑗

: The gray value of pixel (i, j) in base image 𝑔: The mean value of base image

N: Number of the overlapped SPIs

𝑓

_𝑖𝑗

: The gray value of pixel (i, j) in average image

▪ Based on Normalized Cross Correlation ▪ Use average image for matching

Average image ( f ) = 1

𝑁 ෍ 𝑓

_𝑖𝑗k

Matching Method

• Search conjugate points along epipolar line.

13

Matching Procedure

Matching Procedure

• Back projection

• Find the possible position of interest point on other SPIs based on the collinearity condition.

15

Matching Procedure

• Matching in the object space

• Generate the virtual surface in the object space for each assumed point.

• Bilinear image resampling.

Matching Procedure

• Similarity check

• Generate similarity profile along the searching direction

• The maximum similarity locates is regarded as the object point position.

17

Base image

Match image

Experiment Analysis

Test Field

• First floor of Dept. of Geomatics in NCKU

• 5 image stations

• 6 control points

• 24 tie points

19

Test Field

• Tie points on SPIs

SPI 2

SPI 3

SPI 4

Test Field

• Tie points on SPIs

21

SPI 5

SPI 6

Test Field

• E.O. of 5 SPIs

ෝ𝝈_𝐗(m) 𝝈ෝ_𝐘(m) 𝝈ෝ_𝐙(m) ෝ𝝈_𝛚(˚) 𝝈ෝ_𝛗(˚) ෝ𝝈_𝛋(˚) SPI 2 ±0.006 ±0.005 ±0.004 ±0.043 ±0.030 ±0.029 SPI 3 ±0.005 ±0.005 ±0.004 ±0.042 ±0.032 ±0.028 SPI 4 ±0.004 ±0.004 ±0.003 ±0.042 ±0.032 ±0.027 SPI 5 ±0.005 ±0.004 ±0.003 ±0.042 ±0.029 ±0.025 SPI 6 ±0.005 ±0.004 ±0.003 ±0.050 ±0.036 ±0.032

Point ID ∆X (m) ∆Y (m) ∆Z (m) ∆𝐝 (m)

1 0.003 -0.007 0.004 0.008

4 0.000 -0.003 0.001 0.003

8 -0.012 0.000 0.007 0.014

27 0.001 0.000 0.000 0.001

36 0.000 0.000 -0.001 0.001

Mean. -0.002 -0.002 0.002

RMSD ±0.006 ±0.003 ±0.004

Differences of check points Distribution of image stations and check points

Image Matching

• 5 test targets

• Windows size：35×35 pixel

• Searching range: Initial position ± 0.25m

• Searching distance: 0.01m

• Different test cases

23

Case 1 Case 2 Case 3 Case 4

Match Image Type Image Space Image Space Object Space Object Space

Matching Index SNCC YARD SNCC YARD

Image Matching

Max. Similarity ^{SPI 2 SPI 3 SPI 5 SPI 6}

Case 1 0.248 Case 2 0.472 Case 3 0.232

Max. Similarity ^{SPI 2 SPI 4 SPI 5}

Case 1 0.193 Case 2 0.523 Case 3 0.563

Image Matching

25 Max. Similarity ^{SPI 4 SPI 5}

Case 1 0.242 Case 2 0.333 Case 3 0.465 Case 4 0.624

Max. Similarity ^{SPI2 SPI4 SPI5}

Case 1 0.533 Case 2 0.757 Case 3 0.706 Case 4 0.777

Image Matching

Point ID Case 1 Case 2 Case 3 Case 4

1 X O X X

4 X O O O

8 X X O O

27 O O O O

36 O O O O

O: Yes X: No

Max. Similarity ^{SPI 2 SPI 3 SPI 5} Case 1 0.684

Case 2 0.787 Case 3 0.897

Show high similarity at central position?

Image Matching

• Maximum similarity value

27

Case 1 Case 2 Case 3 Case 4

Matching Index SNCC YARD SNCC YARD

Match Image Type Image Space Image Space Object Space Object Space Point ID

1 0.248 0.472 0.232 0.411

4 0.193 0.523 0.563 0.747

8 0.242 0.333 0.465 0.624

27 0.533 0.757 0.706 0.777

36 0.684 0.787 0.897 0.922

Average 0.380 0.574 0.573 0.696

YARD index > SNCC index

Image Matching

• Point searching results

^{Point ID} _{∆X (m)} ^{Case 1}_{∆Y (m) ∆Z (m) ∆X (m)} ^{Case 2}_{∆Y (m) ∆Z (m)}

1 -0.007 0.002 0.006 -0.007 0.002 0.006

4 -0.026 0.043 0.007 -0.005 -0.002 -0.003

8 -0.012 0.060 -0.006 -0.012 0.060 -0.006

27 0.000 0.000 0.001 0.000 0.000 0.001

36 0.011 0.003 -0.004 0.011 0.003 -0.004

Mean -0.007 0.022 0.001 -0.003 0.013 -0.001

RMSD ±0.014 ±0.033 ±0.005 ±0.008 ±0.027 ±0.005

Point ID Case 3 Case 4

∆X (m) ∆Y (m) ∆Z (m) ∆X (m) ∆Y (m) ∆Z (m)

1 -0.007 0.002 0.006 -0.015 -0.017 0.010

4 -0.005 -0.002 -0.003 -0.005 -0.002 -0.003

8 -0.013 0.001 0.004 -0.013 0.001 0.004

27 0.000 0.000 0.001 0.000 0.000 0.001

36 0.011 0.003 -0.004 0.011 0.003 -0.004

Mean -0.003 0.001 0.001 -0.004 -0.003 0.002

Object Space > Image Space

Image Matching

• Point searching results

29 Point ID

Case 1 Case 2

∆X (m) ∆Y (m) ∆Z (m) ∆X (m) ∆Y (m) ∆Z (m)

1 -0.007 0.002 0.006 -0.007 0.002 0.006

4 -0.026 0.043 0.007 -0.005 -0.002 -0.003

8 -0.012 0.060 -0.006 -0.012 0.060 -0.006

27 0.000 0.000 0.001 0.000 0.000 0.001

36 0.011 0.003 -0.004 0.011 0.003 -0.004

Mean -0.007 0.022 0.001 -0.003 0.013 -0.001

RMSD ±0.014 ±0.033 ±0.005 ±0.008 ±0.027 ±0.005

Point ID Case 3 Case 4

∆X (m) ∆Y (m) ∆Z (m) ∆X (m) ∆Y (m) ∆Z (m)

1 -0.007 0.002 0.006 -0.015 -0.017 0.010

4 -0.005 -0.002 -0.003 -0.005 -0.002 -0.003

8 -0.013 0.001 0.004 -0.013 0.001 0.004

27 0.000 0.000 0.001 0.000 0.000 0.001

36 0.011 0.003 -0.004 0.011 0.003 -0.004

Mean -0.003 0.001 0.001 -0.004 -0.003 0.002

RMSD ±0.009 ±0.002 ±0.004 ±0.010 ±0.008 ±0.005

YARD index > SNCC index

YARD index ≅ SNCC index

Conclusions

Conclusions

• PPIMS SPIs is applied to space intersection in this study, and the precision can reach cm level with reliable quality of image station EOPs.

• The feasibility of matching strategy proposed are validated.

• Matching in the object space shows better performance for searching conjugate point.

• The similarity value using YARD index is higher than SNCC index in most cases. However, searching the object point position by YARD index is equal to SNCC index while matching in the object space.

31

Thanks for your listening!

Q & A