IMPLEMENTATION OF BUNDLE ADJUSTMENT AND 3D MODELING

(1)

IMPLEMENTATION OF BUNDLE ADJUSTMENT AND 3D MODELING

1 Paul Wu (吳康寶),²Chiou-Shann Fuh (傅楸善), ²

1

Department of Civil Engineering,

²

Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan,

E-mail: r06521808@ntu.edu.tw fuh@csie.ntu.edu.tw

1. INTRODUCTION

There are mainly two ways to get the points’ 3D location from photos. One way is using linearized versions of Equations to acquire an approximated points’

3D location. The other way is using non-linear minimization to get much more accurate points’ 3D locations.

This project uses both methods to obtain points’ 3D location from photos and compare the results. By designing experiment process and taking the photos to obtain the required materials. Then using the existing software and own writing program to fulfill what the topic question asked.

2. METHOD 3.1. Experiment using object

Fig. 1: Object image.

3.2. Using equipment

3.3. Image acquire

Table 1: Image parameters.

3.3.1. Coordinate system

Sensor CMOS

Sensor size 22.5 x 15.0 mm Effective pixels 20.2-

megapixel Highest

resolution

6000×4000 Shutter speed 1/4000 seconds

to 30 seconds

Image size 1920×1080 pixel

Pixel size 0.0117mm

DOF 1/3 pixel

(Wenzel et al., 2013a) Object direction accuracy 2.2605mm

Image overlapping rate 80%

Numbers of images 20 photos

GSD 0.6103mm

Focal length 50mm

Image base distance 321.4409mm

Object distance 2500.0mm

(2)

Fig. 2: Coordinate system.

3.3.2. Image parameters

Resolution: 1920 ∗ 1280 pixels 𝑓 = 𝑝 ∗𝑃𝑠𝑖𝑧𝑒

𝐺𝑆𝐷 (1) 𝑝: Object distance

𝑓: Focal length 𝑃𝑠𝑖𝑧𝑒: Pixel size

𝐺𝑆𝐷: Ground sample distance

Table 2: Expected accuracy.

3.3.4. Expected accuracy

Control points’ accuracy: Instrument distance measurement error is ±2.0𝑚𝑚. The angle measurement error is close to 0. Hence only the distance measurement error is considered.

Tie points’ accuracy and bundle adjustment standard deviation: Tie points’ accuracy is expected±1.0𝑝𝑖𝑥𝑒𝑙 by using manual matching. But considering other unstable reason, setting the Tie points’

accuracy and bundle adjustment standard deviation to

±3.0𝑝𝑖𝑥𝑒𝑙.

Dense point cloud: Based on bundle adjustment standard deviation, the Dense point cloud accuracy is

±4.0𝑝𝑖𝑥𝑒𝑙 and multiply with GSD. So, Dense point cloud is ±2.4𝑚𝑚. Check points’ accuracy is ±2.4𝑚𝑚.

So the dense point cloud accuracy can be checked by check points.

3.3.4. Instrument setting

Based on object size, single photo coverage scale, image overlapping rate, the numbers of photo and stations can be estimated.

Numbers of photo at one station: 4 photos.

object height

image height ∗ (1 − image overlapping rate)

= 1.62 − 0.78

0.78 ∗ (1 − 0.8)= 5.37 Numbers of station: 5 stations.

object width

image width ∗ (1 − image overlapping rate)

= 0.81

1.17 ∗ (1 − 0.8)= 3.46 3.3.5. 20 image demonstration

Fig. 3: 20 image demonstration.

3.4. PhotoScan modeling result

3.4.1 Image exterior orientation

Table 3: Exterior orientation.

Imag e ID

𝑿(𝒎) 𝒀(𝒎) 𝒁(m) 𝝎(。) 𝝋(。) 𝜿(。) _MG

_034 7.JP

G - 1.54

336 2.06 8919

- 0.33

283 117.

7268 - 41.6

245 17.0 0755

_MG _034 9.JP

G - 1.16

903 1.88 0028

- 0.34

32

115.

7802 - 32.5

964 12.0 8462

_MG _035 1.JP

G - 0.85

213 1.71 5935

- 0.34

202 114.

7653 - 28.8

108 9.53 2835

_MG _035 3.JP

G - 0.50

431 1.52 3421

- 0.34

806 114.

0287 - 22.6

666 8.17 4321

_MG _035 5.JP

G - 0.18

185 1.35

745 - 0.36

057 113.

4996 - 15.1

185 5.48 2461

Target Accuracy

Control/Check point ±𝟐. 𝟎𝐦𝐦 Image measurement ±𝟑. 𝟎𝐩𝐢𝐱𝐞𝐥

Tie point ±𝟑. 𝟎𝐩𝐢𝐱𝐞𝐥 Bundle adjustment

standard deviation

±𝟑. 𝟎𝐩𝐢𝐱𝐞𝐥 Dense point cloud ±𝟐. 𝟒𝐦𝐦

(3)

_MG _035 7.JP G

- 1.57

68

2.03 0452

- 0.33

045 111.

8529 - 42.4

842 12.6 1869

_MG _035 9.JP G

- 1.18

302 1.82 6311

- 0.34

223 108.

1277 - 31.8

156 8.24 8543

_MG _036 1.JP G

- 0.83

96

1.69 0538

- 0.34

316 107.

0658 - 27.2

377 5.60 5208

_MG _036 3.JP G

- 0.54

132 1.54 9592

- 0.35

626 106.

9674 - 21.5

849 4.09 0894

_MG _036 5.JP G

- 0.17

024 1.34 1238

- 0.34

14

105.

2016 - 11.0

34

1.84 6011

_MG _036 7.JP G

- 1.55

723 1.98 6771

- 0.32

066 97.9 3882

- 41.1

716 2.60

47

_MG _036 9.JP G

- 1.18

974 1.83 8607

- 0.32

292 96.7 6505

- 31.7

757 1.05 3093

_MG _037 1.JP G

- 0.86

043 1.70 2848

- 0.32

81

96.6 5528

- 28.2

164 0.78 9317

_MG _037 3.JP G

- 0.52

83

1.51 0712

- 0.33

655 96.3 9484

- 21.0

987 0.41 5667

_MG _037 5.JP G

- 0.19

202 1.33 6401

- 0.33

315 95.5

452 - 14.6

579 - 0.04

392 _MG

_037 7.JP G

- 1.54

454 1.99 0102

- 0.30

752 94.4 6502

- 40.4

076 0.86 1656

_MG _037 9.JP G

- 1.19

234 1.83 9949

- 0.31

559 95.6 2281

- 30.5

368 4.25 7203

_MG _038 1.JP G

- 0.85

906 1.71 7718

- 0.31

024 94.7 9497

- 26.8

495 2.72 5674

_MG _038 3.JP G

- 0.53

886 1.59 9558

- 0.31

976 95.4 3238

- 22.9

422 2.11 3581

_MG _038

- 0.16

1.39 3237

- 0.31

93.4 4242

- 10.5

- 0.42

5.JP G

92 358 031 447

_MG _034 7.JP

G - 1.54

336 2.06 8919

- 0.33

283 117.

7268 - 41.6

245 17.0 0755

3.4.2 Camera calibration

Fig. 4: Camera calibration.

Fig. 5: Radial distortion.

(4)

Fig. 6: Decentering distortion.

3.4.3 Result Demonstration

Fig. 7: Camera position and object.

\ Fig. 8: Build dense cloud.

Fig. 8: 3D model demonstration.

The sides of the cylinder are connected to the plane, and because of the viewing angle, there is a block defect.

The middle of the two columns is obscured by the left and right lines of sight, resulting in a model that is not completely complete. For these missing parts, consider replenishing the details in the form of a reshoot. In addition to the above problems, the rest of the modeling is complete and without missing details. The texture, color tone, unevenness and mottled on the top and left and right sides, and the missing blocks on the cylinder can be clearly identified by the human eye; in the data, the subsequent inspection points are checked. Provide more objective accuracy indicators.

3.5 Control point and check point accuracy

Table 4: PhotoScan evaluating control point accuracy.

Control point ID

Accuracy(

m)

Control point ID

Accuracy (m)

P01 0.005 P13 0.006

P02 0.004 P14 0.006

P03 0.002 P15 0.001

P04 0.006 P18 0.005

P05 0.002 P21 0.002

P08 0.002 P23 0.003

P09 0.002 P26 0.005

P10 0.002 P28 0.003

P11 0.004 P29 0.007

P12 0.006 P30 0.0001

Table 5: PhotoScan evaluating control point accuracy.

Check point ID

Accuracy(

m)

Check point ID

Accuracy(

m)

P06 0.002 P20 0.004

P07 0.001 P22 0.001

P16 0.001 P24 0.007

P17 0.003 P25 0.004

(5)

P19 0.002 P27 0.001 Checkpoint accuracy and planning accuracy:

- Planning accuracy: ±2.4mm = 4 pixels

-

Average accuracy of checkpoints calculated by PhotoScan: ±3.6mm = 6 pixels

This project uses both methods to obtain points’ 3D location from analysis of error sources:

GSD=0.6103mm is known from photography planning, so the accuracy of the checkpoint of the result is 2 pixels worse than the planning accuracy. The possible reasons are as follows:

Photo rendezvous point of view: As described above, the small angle of camera rendezvous causes large errors in the solution parallel to the baseline. This does not rule out the negative impact on accuracy.

Miscalculation of the error of the artificial point control point: The pixel measurement error is given only 1 pixel in the planning, but it is very difficult to implement.

Because the center of the standard is not sharp enough, there are multiple suitable points. Can be used as the center of the standard, and the appropriate range of points is also greater than 1 pixel, resulting in artificial errors in the wrong point, resulting in errors.

3.6 Programing steps

3.6.1. Guess the execution flow of the PhotoScan program

First of all, it is necessary to obtain the initial value of the outer bearing. It is inferred that the PhotoScan uses SIFT or ASIFT to use the image matching method to generate the possible initial values of the outer orientation of the photo.

Set the initial value of the outer orientation of each photo to adjust the beam method.

Bring the better orientation to the front, or transfer the photo to the same surface conduct dense point matching on the same surface.

3.6.2. Simulate the PhotoScan process

Use a single piece of resection to obtain the outer bearing initial value, taking the outer orientation of the PhotoScan program output as the initial value. Since this item was not taken into account during the measurement, the instrument can be placed on the shooting point, and the initial value of the outer orientation can be brought into the program.

The beam orientation is adjusted by the external orientation after the single piece is post-crossed to obtain a more accurate external orientation parameter.

Perform a forward intersection using the additional orientation parameter to simulate the object space position of the point matching point.

3.6.3. Resection

Since there is six unknowns in the outer orientation of an image, at least three control points are needed, that is, six equations can solve the external orientation.

However, since it is a nonlinear equation, an initial value is required to solve the problem. And the initial value will affect the result of regional convergence, so you need to give the appropriate initial value.

3.6.3. Bundle adjustment

On an image, the control point, the photographing center, the object point, and three can list two collinear equations, and the collinear equations of all the control points of the entire image are listed as observations.

Equations, then all the observation equations of all images are listed, and the indirect flat interpolation mode is used to iteratively calculate the accuracy of the external and external bearings after all the images are corrected.

The control points of each image are measured, the coordinates of the image point are measured, and the external orientation is unknown. The indirect adjustment model is used to find the unknown value. The experimental data of the program is based on two photographs, and iteratively calculates the accuracy of the outer azimuth and the outer azimuth. The program experiment data was tested with 2 images.

3.6.3. Intersection

The program uses Bashar Alsadik's (2013) forward communication function. After reading in the matching point, first convert the pixel unit of the photo to the unit point coordinate of mm (the forward rendezvous program requirement, that is, the length and width of the image pixel, and the actual length. Wide information in the coordinates of the incoming point), incoming outer orientation parameters, pp system point coordinates.

The program experiment uses two images for matching and pre-delivery. The parameters are as follows.

Match point method ASIFT: Based on the improvement of the SIFT algorithm, considering the change of the angle of view produced by the latitude and longitude of the camera's optical axis, the true affine is unchanged. Feature extraction is similar to the SIFT algorithm. There are mainly the following steps:

Simulate changes in the direction of the optical axis, causing the deformation of the picture, and bringing the longitude and latitude into the simulation at a fixed insertion angle. According to personal understanding, the main difference lies in this. ASIFT considers the change of the optical axis angle, SIFT only considers the image enlargement and reduction of the Gaussian

(6)

difference pyramid, and does not consider the tilt of the angle of view, resulting in a rapid decline in the matching accuracy when the angle of view changes.

Table 6: Resection result.

Fig. 9: Resection result.

Fig. 10: Enlarge resection result.

Perform SIFT algorithm feature extraction on the simulation map; find the extreme points in the image, and the particle centered, calculate the gradient direction of its neighboring points. This gradient direction is made into a histogram according to 360 degrees, and the segmentation points are divided into eight directions,

and the intensity of each direction is marked to indicate the characteristics of the key points.

Feature Point Matching: Calculate the straight line distance of the features of the key points of the two feature points. The smaller the similarity, the higher the similarity, and the less than a threshold value.

ASIFT Matching Program Description: Using the ASIFT suite developed by Guoshen Yu and Jean- Michel Morel (2011), it packages the ASIFT program into an exe file and provides it to matlab as a function call. Call this function in the program to complete ASIFT point matching, store feature points in keys, matching points in the matching file, and produce horizontal and vertical matching point connection diagram. Then write the program to read its asift point matching results for subsequent forward rendezvous.

Fig. 11: ASIFT result.

3.8 Software SURE using

SURE matches the SGM algorithm used. Image capture position: Output the external orientation generated by PhotoScan as the external orientation input of SURE.

X(m) Y(m) Z(m) ω(rad) φ(rad) κ(rad) σ(m) _M

G_0 347.

JPG - 0.17 543

1.34 0248 4

- 0.34 133

1.82 514

- 0.19 215

0.03 2317

0.00 3854 _M

G_0 349.

JPG - 1.16 8954

1.89 0543

- 0.34 314

2.02 0858

- 0.56 868

0.21 1133

0.00 3327 光束

法平差

X(m) Y(m) Z(m) ω(rad) φ(rad) κ(rad) σ(m)

_M G_0 347.

JPG - 0.17 6472

1.34 0045 7

- 0.33 912

1.82 2764

- 0.19 239

0.03 2257

0.00 473

_M G_0 349.

JPG - 1.17 1479 5

1.89 9741

- 0.34 513

2.02 1791

- 0.56 868

0.21 1133

0.00 6321

(7)

Fig. 14: Camera position

Fig. 15: SURE modeling result.

Fig. 16: SURE modeling result.

Can be found on the side, the dense point cloud on the side of the column has improved significantly, the texture is still preserved, and there is no big break or hole; the middle of the two columns because the angle is limited, is within the expected error; but in There was a large break in the head, which was presumed to be a too-large station, which affected the overturning angle at the top. The result of the outer position calculation was poor, while the poor outer position resulted in the inability to convert the paired nuclear line image. There is no point. The reason why the right column produces two colors is that the size of the image light source of the supplementary image is different, and thus the color difference occurs. Although the quality of the point cloud is very dense, but the control point mark, only two

can be seen clearly, the rest cannot be identified, so there is no subsequent checkpoint check.

4. RESULT

Change to the correct shooting mode and increase the image coverage on both sides. However, in the results of PhotoScan, the left side and the top of the column have not been well improved. Therefore, it is presumed that the texture is less obvious, resulting in a poor match.

And this idea is also verified when using SURE, because SURE point cloud column has a complete point cloud on both sides, and SURE production point method is to change the image to the nuclear line, and then you can make a point cloud before the point cloud .

5. CONCLUSION

The outward orientation of the two solutions differs by about 2 to 3 mm. It can be seen that after the adjustment of the beam method, the power of the later examination unit is larger than that of the rear. There is no reasonable explanation about this part, but the current idea is that when the beam method is adjusted, a control point has two observation equations, and these two equations will pull and solve each other, and there are more equations to go to the end of the image. The outer position solves and the answer deviates from the initial approximation, so the posterior unit power is larger than the posterior meeting.

6. DISSCUSSION

Differences in outer orientation values obtained by resection and beam method adjustment: In the rear rendezvous mode, the coordinates of the control point are regarded as constants, and in the beam mode mode, the control point is regarded as an observational measurement. The answer calculated by the interpreter is not the same, but similar. Theoretically, if the beam method is adjusted, the coordinates of the control point are still regarded as constants, because the outer bearing value is the same as that of the resection.

point cloud accuracy: It is somewhat impractical to compare the accuracy of the two sides of the check point because they use different algorithms. PhotoScan uses feature point matching; SURE uses SGM matching.

And SURE may not be matched because the orientation is not accurate enough. Thus no further, to refine the SURE results in order to measure the checkpoints, and to compare with PhotoScan accuracy.

REFERENCES

[1] Wenzel, K., M. Abdel-Wahab, A. Cefalu, and D. Fritsch,

“High-resolution surface reconstruction from imagery for close range cultural Heritage applications” , International Archives of Photogrammetry and Remote Sensing, ISPRS 2012 Congress, Vol. 39, Part B5, 2012.

(8)

[2] Bashar Alsadik “STEREO IMAGE TRIANGULATION – LINEARCASE” URL:

https://www.mathworks.com/matlabcentral/fileexchange/4 1739-stereo-image-triangulation-linear-case