A PRELIMINARY STUDY OF IMAGE-BASED INDOOR NAVIGATION WITH PANORAMIC CAMERA

(1)

A Preliminary Study of

Image-base Indoor

Navigation with

Panoramic Camera

(2)

Motivation

Department of Geomatics, National Cheng Kung University

2

(3)

Motivation

(4)

Introduction

4

Image-based method perform well in indoor environments

Input image

Output result

www.ctimes.com.tw www.apk3.com

(5)

Flow chart

Establish the control images database Find the corresponding images Measure the coordinate of corresponding points Calculate the position and orientation Control points Back-resection Bundle adjustment SURF RANSAC Experiment

(6)

6

Limitation of FOV Few conjugate points

1.Maybe get the bad intersection geometry

2.Need lots of images to establish the control image database

Frame image

(7)

!?

What will happened if we replace the frame image by panoramic image?

(8)

Panoramic image

• Ladybug 5 • 8000 x 4000

• Horizontal field of view: 360 degrees • Vertical field of view: about 150 degrees • Contain lot of image feature information

(9)

• Calculate the exterior orientation – Single image resection

– SPI bundle adjustment

• Collection of a set control images

Images

Position information

(10)

Single image resection

(11)

Bundle adjustment

SPI E(m) N(m) h(m) _ω(˚) _φ(˚) _κ(˚) S1 109.06 239.63 -3.77 _4.09 _1.59 _-97.25 S5 106.18 258.76 -2.29 _5.93 _3.52 _90.31 S6 103.45 236.77 -4.31 _5.06 _0.54 _73.98 SPI E(m) N(m) h(m) _ω(˚) _φ(˚) _κ(˚) S2 _109.68 _267.22 _-3.75 _-1.57 _-0.05 _-99.34 S3 _110.37 _282.46 _-3.66 _-0.22 _-3.26 _-92.13 S4 _107.57 _290.49 _-3.77 _2.72 _2.28 _87.17 Check point Control image Control point

(12)

Image matching methods

• Manual matching

– Search conjugate points

– Compute the vectors of conjugate points • Auto-matching with panoramic image

– Detect feature points and matching (SURF)

– Eliminate error matching conjugate points (RANSAC)

(13)

Experiment introduction

• Case I

– Movement test

– Distance between 2 station is 0.95 meters – A1, A2, A3, A4, A5

A1

A5

A2 A3

(14)

Trajectory Trajectory

Movement result

14

(15)

Experiment introduction

• Case II

– Rotational test

– Rotation angle is 90 degrees clockwise – B1, B2, B3, B4 B1

B2 B4

(16)

Rotation result

Rotational direction

16

(17)

Experiment introduction

• Case III

– Oblique test

– Tilt angle is about 5 degrees – C1, C2, C3

(18)

Oblique result

downward upward

Before: red image After: blue image

(19)

(20)

Epipolar geometry

20 epipolar plane

Epipolar plane: plane defined by the camera centers and word point

Epipolar line: set of points that project to the same point in left image, when projected to right image forms a line

(21)

• The Essential matrix can be computed directly from the camera coordinates with singular value decomposition

• 𝑥𝑥_𝑐𝑐 𝑦𝑦𝑐𝑐 𝑧𝑧𝑐𝑐 𝑒𝑒𝑒𝑒11₂₁ 𝑒𝑒𝑒𝑒12₂₂ 𝑒𝑒𝑒𝑒13₂₃ 𝑒𝑒₃₁ 𝑒𝑒₃₂ 𝑒𝑒₃₃ 𝑥𝑥_𝑞𝑞 𝑦𝑦_𝑞𝑞 𝑧𝑧_𝑞𝑞 = 0 • 𝐴𝐴 = 𝑥𝑥_𝑠𝑠𝑥𝑥_𝑞𝑞 𝑥𝑥𝑠𝑠𝑦𝑦𝑞𝑞 𝑥𝑥_𝑠𝑠𝑧𝑧_𝑞𝑞 𝑦𝑦𝑠𝑠𝑥𝑥𝑞𝑞 𝑦𝑦𝑠𝑠𝑦𝑦𝑞𝑞 𝑦𝑦_𝑠𝑠𝑧𝑧_𝑞𝑞 𝑧𝑧_𝑠𝑠𝑥𝑥_𝑞𝑞 𝑧𝑧_𝑠𝑠𝑦𝑦_𝑞𝑞 𝑧𝑧_𝑠𝑠𝑧𝑧_𝑞𝑞 • 𝑥𝑥 = 𝑒𝑒𝑒𝑒11₂₁ 𝑒𝑒𝑒𝑒12₂₂ 𝑒𝑒𝑒𝑒13₂₃ 𝑒𝑒₃₁ 𝑒𝑒₃₂ 𝑒𝑒₃₃

• One conjugate point can provide one equation

(22)

Movement result

(23)

(24)

Oblique result

(25)

Movement Rotation Oblique

Affine(Avg.) 31.8% 38.3% 35%

Essential(Avg.) 77.8% 96% 93%

Total umber of pairs(Avg.) 445 1871 1252

Search ability of RANSAC

(26)

• The Essential matrix can be decomposed into R and t with SVD

• Assume projection matrix of control image

– P = [I | 0]

• There are 4 possible solutions of query image

– P_A=[R_a |t] – P_B=[R_a |-t] – P_C=[R_b |t] – P_D=[R_b |-t]

26

(27)

θ P p’ 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑃𝑃𝐵𝐵 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑝𝑝𝑝𝐵𝐵 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑃𝑃𝐵𝐵 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑝𝑝𝑝𝐵𝐵 𝜃𝜃 = cos−1( 𝑅𝑅𝐵𝐵𝑀𝑀𝑟𝑟𝑃𝑃𝐵𝐵 ∙ 𝑅𝑅𝐵𝐵𝑀𝑀𝑟𝑟𝑝𝑝𝑝𝐵𝐵 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑃𝑃𝐵𝐵 𝑅𝑅_𝐵𝐵𝑀𝑀𝑟𝑟_𝑝𝑝𝑝𝐵𝐵 ) θ ≈ 0˚

Possible solution

𝑧𝑧𝐵𝐵 𝑦𝑦𝐵𝐵 𝑥𝑥𝐵𝐵 𝑌𝑌𝑀𝑀 𝑋𝑋𝑀𝑀 𝑍𝑍𝑀𝑀

(28)

28

(29)

(30)

Conclusion

• RANSAC is an algorithm which we can change model for the computation case by case

• Coplanarity constraint is useful for spherical panoramic image • The vectors in the panoramic image is more complicated than

frame images

– Movement: Radial transpire and gather – Rotation: straight

– Oblique: swirl

• Matching ability of SURF:

rotation(~1800)>oblique(~1200)>movement (~450)

(31)

Conclusion

• Essential matrix is not only suitable for the RANSAC model but is good at calculating the relative orientation between two panorama images.

• The way which is used for judging the ambiguity in frame images is not suitable for panoramic image.

• With the angle between vectors, we can get the correct orientation in 4 possible solutions.

(32)

• Try another feature detection and matching method such as

SIFT.(Maybe it can provide more robust and reliable conjugate points.)

• Develop the method to find the corresponding control images with query image. (A possible solution is do the image

matching and record the number of conjugate points)

• Use other panoramic cameras to test the robustness of our method.

• Calculate the scale of control and query image.( If two

corresponding control images are found, this mission can be solved.)

32

(33)

• Berthold K.P. Horn, Recovering Baseline and Oreintation from ‘Essential’ Matrix

• Cansm Yildiz, An Implementation on Recognizing Panoramas

• David Nister, An Efficient Solution to the Five-Point Relative Pose Problem

• Daesik Kim, 3D Reconstruction

• Henrik Stewenius, Christopher Engels, David Nister, Recent Developments on Direct Relative Orientation

• Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool, Speed-Up Robust Features

• Richard Hartley and Andrew Zisserman, Multiple View Geometry in computer vision

• Richard I. Hartley, In Defence of the 8-point Algorithm

• Van Vinh Nguyen, Jin Guk Kim, and Jong Weon Lee, Panoramic

Image-References

(34)

A PRELIMINARY STUDY OF IMAGE-BASED INDOOR NAVIGATION WITH PANORAMIC CAMERA