Chapter 3 Parameter Estimation in Multiple View Geometry
3.3 Experiments
3.3.2 The analysis in the parameter estimation methods
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
53
Figure 3-5 Corresponding points which are used in simulate and real images: (a) Simulated model house; (b) Dinosaur; (c) Fountain-P11; (d) Herz-Jesu-P8; (e) Model house.
3.3.2 The analysis in the parameter estimation methods
As shown in Table 3-2, we compare the parameter estimations with public codes [3]. The ROPSO is the method we proposed and LMedS implement by ourselves. We also find that the errors of the ROPSO are less than other methods for all the different data set. We think that because the corresponding points with some outliers and M-Estimator evaluation also use low weight outlier data, the average errors will increase. Likewise the errors got from other
(e) (d) (c)
‧
methods which Armangué and Salvi implemented are also larger than the ROPSO, because they consider low weight outlier data too. We also find that if there are plane objects (e.g.
buildings) in the captured images, it will cause the error of the RANSAC to be larger. It is because that it is easy to sample the corresponding points on the same plane and make the estimation degenerate. We think it is not suitable to using RANSAC to estimate these images directly. We also combine the ROPSO with GA, the GA-PSO, and the experiments shows that most results are improved.
Table 3-2 The average errors of the parameter estimations for the fundamental matrix.
Data set Method
Simulated
model house Dinosaur Fountain-P11 Herz-Jesu-P8 Model house Seven-point 2.184 46.184 67.161 32.352 21.550 Least square 15.790 364.453 144.848 18.145 291.222 SVD 6.236 3.469 10.867 9.692 23.827 Gradient 6.424 4.894 25.541 4.673 26.447 M-Estimators 0.0068 9.463 0.180 0.110 1.862 LMedS 0.0063 0.093 0.102 0.112 0.185 RANSAC 6.236 3.469 10.867 9.692 23.827
MLESAC 0.807 0.891 0.925 3.942 1.346
MAPSAC 0.829 0.475 0.847 3.941 1.403
GA 0.018 0.132 0.163 0.181 0.302
ROPSO 0.0055 0.037 0.085 0.050 0.150
GA-PSO 0.0034 0.033 0.080 0.048 0.118
3.4 Summary
Many computer vision problems are related to the problem of parameter estimation in noisy data. This chapter discuss about several kinds of parameter estimation methods, including linear methods, robust methods, and evolution methods. We discuss and analyze
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
55
them in the parameter estimation for an example, fundamental matrix estimation, in multiple view geometry. We find that, if the method with weighted outlier data, it will cause the results not robust enough. The combination with different method may be more robust for the estimation. The proposed ROPSO combined with the PSO and the LMedS is more robust in the experiment. The GA is also helpful for combining with our method. Besides, the plane objects (e.g. the building) captured will easily make the estimation be with degeneracy. It needs to combine with the processing for the degeneracy.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
56
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
57
Chapter 4
Robust Methods for Critical Configuration
4.1 Introduction
In the section 2.4.1, we discuss about that when the number of the corresponding points are enough, we can use robust methods, robust parameter estimations, to process most critical configuration. Many computer vision problems are related to the problem of parameter estimation in noisy data. Such as line fitting, homography estimation, fundamental matrix estimation, image matching, camera calibration, and pose estimation [28][29][38][115].
Taking the homography estimation as an example, the homography is a 3×3 matrix which is a plane projective transformation in two view images as shown in Figure 2-1. The homography has eight variables. It needs at least four corresponding points in the same plane in two view images to estimate the matrix. The corresponding points mean the image matched points in a pair of stereo images. We can automatically detect corresponding points with noise in the pair of stereo images. If we use brute-force search, it needs C4n times to try all possible solution to estimate the best parameters, where n is the number of the corresponding points.
For another popular example of the parameter estimation, the fundamental matrix is a 3×
3 matrix which is the relationship between any two views which capture the same scene. As
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
58
shown in Figure 2-2, the projection points from the scene occur in both view images. It can be used to help us estimate the depth from a pair of the stereo images. The fundamental matrix also has eight variables. It needs at least eight corresponding points to estimate the matrix.
However, if we use brute-force search, it needs C8n times to try all possible solution to estimate the best and this is impracticable with the number n of the corresponding points.
Therefore, there are several methods sampling the corresponding points to estimate the parameters.
Armangué and Salvi [3] and Huang et al. [51] surveyed the most used methods for estimating the fundamental matrix. They showed that the robust method can cope with potential outliers, such as LMedS [75], RANSAC [99], MLESAC [96] and MAPSAC [100]
which are random sampling methods. Although Brahmachari and Sarkar [9] and Ni et al. [65]
proposed some algorithms which are able to tolerate up to 90% outliers, several experiments showed that the errors are larger than 50 pixels. In this research, we focus on robust and accurate results in order to apply to high accurate requirement applications. Therefore, we do not consider the data with high outlier rates.
We also propose an idea of using the Particle Swarm Optimization (PSO) combined with the LMedS and the orthogonal array to improve parameter estimation [17]. We think that the PSO is better than the gradient method and the random sampling methods in the parameter estimation described in section 3.2.3. There are also many researches proposing orthogonal PSO methods which combine the orthogonal array with the PSO to solve the problems in other different fields [44][105][112]. The orthogonal array is one of the major tools used in Taguchi method. It is an important technique to conduct a robust experimental design. In this research, we also use the PSO with the orthogonal array to solve the parameter estimation
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
59
problem in computer vision. The difference is that we use the orthogonal array to make the representative initial particles in the PSO for more stable, and use the LMedS produce an initial global best particle for accurately parameter estimation.
Therefore, this chapter focuses on the discussion about how to use some of the combined methods to estimate more robust and accurate parameters. The main goals of this research are as follows. First, we formulate the parameter estimation problem to the traditional PSO problem. The orthogonal array is used to make the representative initial particles for more stable. The LMedS is also used to produce an initial global best particle for accurately estimating the parameters. Several robust methods are used to be combined with the proposed methods. Moreover, we design some experiments to discuss the optimization of the parameter which the methods used. Second, we use more real ground truth data from Visual Geometry Group [103] and Strecha [90] for experiments and test for accuracy on different data set with a variety of characteristics. Third, we also analyze that if the combined methods are better than traditional methods by statistical method.
4.2 Parameter Estimation by ROPSO
In this section, we will introduce the proposed ROPSO algorithm. We also apply the algorithm to the three instances: homography estimation, fundamental matrix estimation, and trifocal tensor estimation.
4.2.1 ROPSO algorithm
The PSO is a useful evolutionary algorithm for solving high dimensions problems. It uses nature-inspired search to optimize the solution and costs less sampling times. The details
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
60
are described in the section 3.2.3. In this research, we first need to formulate the parameter estimation problem into the PSO problem. It is assumed that there are data points xi (e.g. for line fitting) or corresponding points (xi, xi') (e.g. for homography and fundamental matrix estimation) for i = 1, 2, 3, …, n, where n is the number of the points. If at least np data points or corresponding points are required for estimating the parameters, we use np-dimensional space with each axis being the integer intervals [1, n]. The particles are the coordinates in the np-dimensional space, and any two of the axis values are not equal.
The initial particles are important for PSO algorithm. If the particles concentrate on somewhere, they would be led to local optimum. Thus, we use the orthogonal array to help us select the representative points to be the initial particles, and avoid the initial particles concentrating somewhere in the np-dimensional space. Then, the particles' positions will update by (3-2) and (3-3). Besides, we must define the fitness function and it is used to estimate the particles' quality. The less the fitness value is, the best the position of the particle is. We call the method Robust Orthogonal Particle Swarm Optimization (ROPSO). The procedure of the proposed ROPSO is now described as follows. Figure 4-1 shows the flowchart of the proposed method with application to the fundamental matrix estimation.
Step 0: Preprocessing. Detect data points or the corresponding points from image pair.
Step 1: Initialization. The initial particles are selected according to the orthogonal array.
Each particle has np axis values matched to the np factors in the orthogonal array. If the element of the factor is 1, the particle's axis value is set to the first data points. Likewise, if the element of the factor is 2, the particle's axis value is set to the last points. Test two kind of initial global best particle which are making from the rough LMedS or only the orthogonal array. The rough LMedS means that the probability Pr in equation (3-1) is set to a small value,
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
61
and the LMedS uses fewer samples to estimate a rough solution. Then, select the best one which has the best fitness value. The particles' initial velocities are set randomly.
Step 2: Fitness. Use the fitness function to estimate each particle's quality, and decide
their past best positions and whole global best position. The fitness function is set to the residual rQ as the equation (4-1). Q is the τth percentile residual. If we set the fitness to the median residual, the τ can be set as 50. This concept is similar to LQS described in section 3.2.2.
Q
fitness r (4-1)
Step 3: Iterations. Decide the parameters w, c1, and c2. We can use (3-2) and (3-3) recursively to update the particles' positions.
Step 4: Constraints. Set the Vmax to about 10-20% of the problem range in each dimension. It follows the suggestion of the reference [22]. In our experiments, we set it to 20% for convergence quickly. If the particle's any two of the axis values are the same, we repeat update the particle by (3-2) and (3-3) until it satisfies this constraint which the matched image points are different.
Step 5: Stopping condition. Repeat steps 3 to 4 until the stopping condition which all particles and the global best particle are within about 10-20% of the whole np-dimensional space apart or at maximum iterations. It is because that the Vmax is set to about 10-20% of the problem range in each dimension, the solution will converge. Therefore, we can stop the process early.
The computational complexity is the same as the LMedS. It is dependent on the number
‧
of the data points or the corresponding points and the number of the sampling times.
Therefore, the computational complexity is O(nm) where n is the number of the points and m is the number of the sampling times.
Image pair
Replacing the worst particle by crossover operation between the best and second best ones
Figure 4-1 Flowchart of the proposed method with application to the fundamental matrix estimation.
For example, the line fitting is the simplest case in the parameter estimations. We will give a brief introduction about the line fitting problem. The orthogonal array and the fitness function used in the ROPSO are also described as follows.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
63
4.2.1.1 The line fitting problem
Some estimated parameters can be represented as an implicit nth order polynomial equation in homogenous coordinates of the form
0
X A Xn n n ,
where the homogenous coordinates of the points Xnis in the n-dimensional space, and Anis a nn parameter matrix. For the simplest case line fitting, given a distribution of the data points in the 2D Euclidean plane, use the known mathematical model of a straight line, dx+ey+1=0, to minimize the sum of the distances between the given points and the line. It is a second order polynomial equation and it can be expanded to the following equation
2+ + 2+ + + = 0
ax bxy cy dx ey f ,
where a = b = c = 0, and f = 1. It is easy to estimate the line by the method of least squares.
However, it is very sensitive to outliers. In order to deal with this problem, robust parameter estimation is required.
The ROPSO described in section 4.2.1 can estimate the line robustly. The orthogonal array used in step 1 and the fitness function used in step 2 are as follows.
4.2.1.2 The orthogonal array for line fitting
Because any two points can be used to estimate a line, it is a 2-dimensional PSO problem described in section 3.2.3. The orthogonal array only has two factors as shown in Table 4-1.
We can use four particles selected by the orthogonal array to be the representative particles. If the element of the factor is 1, the axis value of the particle is set to the first data point. Each
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
64
data point is assigned a serial number ID. Otherwise if the element of the factor is 2, the axis value of the particle is set to the last data point.
Table 4-1 A L4(22)Orthogonal array Factors Experiment number
A B
1 1 1
2 1 2
3 2 1
4 2 2
4.2.1.3 Fitness function for line fitting
The residual of the fitness function described in the step 2 in the section 4.2.1 is shown in the equation (4-2). It is set to be the difference between the measured y coordinate of the data point and its estimated y coordinate. A better fitting line will minimize the residual rQ calculated from the inlier data points.
( 1)
i i i
r y dx e (4-2)
4.2.2 Homography estimation
This section will give a brief introduction about the orthogonal array and the fitness function used in the RPOSO. The homography estimation problem has described in the section 2.2.1.1.
‧
4.2.2.1 The orthogonal array for the homography estimation
In the homography estimation, the number of factors is set to be 4 with strength 3, as shown in Table 4-2. It is because that the homography use at least four corresponding points to estimate the parameters in the equation (2-1). Eight particles are selected by the orthogonal array to be the representative particles. If the element of the factor is 1, the axis value the particle is set to the first corresponding points as the line fitting. Likewise, if the element of the factor is 2, the axis value of the particle is set to the last corresponding points.
Table 4-2 A L8(24) Orthogonal array
4.2.2.2 Fitness function for the homography estimation The residual
Q
r in the fitness function for the homography estimation is shown in the equation (4-3). It is set to be the difference between the measured coordinate x of the i image point and its transformed coordinateHxi.
i i i
r d x Hx, (4-3)
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
66
4.2.3 Fundamental matrix estimation
In two views, the epipolar geometry [28] is an important constraint in computer vision.
The epipolar geometry is the intrinsic geometry between two views that only depend on the internal parameters and relative poses of cameras. We can use the fundamental matrix to represent algebraically the epipolar geometry. The fundamental matrix estimation is one of the representative problems in computer vision
Besides, we must avoid the selected points coming from the same plane which lead to the wrong estimation of the fundamental matrix. Therefore, we use the bucketing techniques [114] to reduce the effect. In the experiment, the image plane is partitioned into nb
2 square windows (buckets), e.g. nb=8. The fundamental matrix estimation problem, its orthogonal array, and the fitness function are described as follows. The fundamental matrix estimation problem has described in the section 2.2.1.2.
4.2.3.1 The orthogonal array for the fundamental matrix estimation
In our experiments, we set the orthogonal array of strength to be 3 and the number of factors to be 8, as shown in Table 4-3. In order to avoid the selected points coming from the same plane, we partition the image plane into nb
2 square buckets and to sort the bucket from left to right and top to bottom. Then, the initial sixteen particles are selected according to the orthogonal array. Each particle has eight axis values matched to the eight factors. If the element of the factor is 1, the axis value of the particle is set to the number from 1 to n of the corresponding points randomly selected within first bucket which contains points. Likewise, if the element of the factor is 2, the axis value of the particle is set to that within last bucket which contains points. We also can replace one of initial global best particle which are making from the rough LMedS described in the step 1 in the section 4.2.1. Then, select the best one
‧
which has the best fitness value.
Table 4-3 A L16(28) Orthogonal array
4.2.3.2 Fitness function for the fundamental matrix estimation
Image 1 Image 2
FT
Image 1 Image 2
F
Figure 4-2 The geometric distances of the epipolar geometry.
‧
The geometric distance is used as the measurement for fundamental matrix estimation.
As shown in Figure 4-2, the residual ri is defined by the geometric distance as the equation (4-4) and (4-5). In (4-5), the Euclidean distance of point xi to its epipolar line F xT i is
4.2.4 Trifocal tensor estimation
We must avoid the selected points falling in the same plane which lead to the wrong estimation of the trifocal tensor. Therefore, we use the bucketing techniques [114] to reduce the effect. In the experiment, the image plane is partitioned into nb
2 square windows (buckets), e.g. nb=8.
Besides, we must define the fitness function in equation (4-6) and it is used to estimate the quality of the particles. The less the fitness value is, the best the position of the particle is.
Then, the proposed ROPSO method uses the PSO combining the LMedS and the orthogonal array to improve the trifocal tensor estimation.
4.2.4.1 The orthogonal array for the trifocal tensor estimation
In the PSO, the initial positions of the particles may influence on the results. Thus, we
‧
use the orthogonal array to help us select the representative points to be the initial particles, and avoid the initial particles concentrating somewhere in the 7-dimensional space. In our experiments, we set the orthogonal array of strength to be 3 and the number of factors to be 7 as shown in Table 4-4. Then, the positions of the particles will update by (3-2) and (3-3).
Table 4-4 A L8(27) Orthogonal array
4.2.4.2 Fitness function for the trifocal tensor estimation
For estimating the trifocal tensor by the proposed method, we use the geometric distance as the measurement. The geometric distance r is defined as (4-6), the Euclidean distance between the matched point x in third view and the transfer point xx lij iTj calculated from x and l which is passing through the pointx .
,
x ijTij
r d x l (4-6)
4.3 The Combination of the ROPSO and Genetic Algorithm
The advantage of the genetic algorithm is that it will be evolved toward better solutions.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
70
After the natural evolution such as crossover operation, it may combine the better corresponding point selections and get better solution. Therefore, we think that the best two particles in PSO with the crossover operation may increase the probability of getting global best in PSO. We refine the algorithm described in section 4.2.1 to improve the ROPSO, we call it as GA-PSO.
Step 0: Preprocessing. It is the same as the Step 0 in the section 4.2.1. Detect data points or the corresponding points.
Step 1: Initialization. It is the same as the Step 1 in the section 4.2.1. Select the initial particles according to the orthogonal array.
Step 2: Fitness. It is also the same as the Step 2 in the section 4.2.1.
Step 3: Iterations. It’s the same as the Step 3 in the section 4.2.1. Decide the fitness
Step 3: Iterations. It’s the same as the Step 3 in the section 4.2.1. Decide the fitness