When a sensed image is available, we can start to match the reference feature points in the matching database with the feature points extracted from the sensed image. We use SPP = {SPPi,i =1, 2, .., 6} obtained above to detect the overlapping area between the reference and sensed images. During the view registration process a reference point pair( , , , )
s t
s i t i
p p is fetched in order from SPP where ,
s is
p and ,
t it
p are the is-th and it-th reference feature points from sub-regions Rs and Rt, respectively. If either point fails to find any matched point in the sensed image, then delete all the reference point pairs in SPP involving the unmatched reference point, ,
s is
p or ,
t it
p . We define the size ratio of the updated SPPi to its initial set as the overlap index. When the overlap index is low for a particular SPPi, it implies the chance that the two reference sub-regions of SPPi overlap with the sensed image is also low. An algorithm for the on-line registration process is given below:
Algorithm OLRP (On-Line Registration Process) Input:
1. The sets of reference and sensed feature points: P={ , , , }p p1 2 pn and
1 2
{ , , , m} Q= q q q .
2. The lists of sorted reference point pairs in the six region pairs: SPP={SPPi, i =1, 2, .., 6}.
3. The size of initial SPPi,: Si= |SPPi| for i = 1, 2,…, 6.
Output:
1. The final corresponding point set:CPS( )f 2. The final transformation matrix: T( f ) Initialization:
Initialize the overlap index OIi = 1 for i = 1, 2,…, 6.
For c = 1, 2 … , cmax (cmax = 5)
1. Fetch the first element ( , , , )
s t
s i t i
p p from the sorted point pair list SPPi whose overlap index OIi is maximum (if there is a tie, break the tie arbitrarily).
2. Find the matched points in the sensed image for each of ,
s is
p and ,
t it
p based on the normalized cross correlation measure. Assume the resulting matched point sets are CMs = {qk, k =1, 2, .., ns} and CMt ={ql, l =1, 2, .., nt} for ,
s is
p and ,
t it
p , respectively. If CMs (or CMt) is empty, then delete all the reference point pairs involving the unmatched reference point, ,
(1) Compute the affine transformation matrix T(0) using the two point pairs
, ,
( , )
s t
s i t i
p p and ( , )q qk l (refer to Section 5.1).
(2) Invoke the IMU algorithm to determine the homography matrix T(r) using T(0) as the initial solution and to find CPS(r) (refer to Section 5.2).
(3) Check the stopping criteria: if the size of the corresponding point set CPS(r) is greater than a pre-defined threshold, then return T( f )= T(r) and terminate the process with “success”; otherwise, continue.
End For
The iteration number of the above cycle is bounded by a fixed number (5 in our case), as shall be explained at the end of the next section.
5.6 Experimental Results
A) The Iterative View Registration under the Homography Transformation
We apply our method to register two aerial images. Fig. 5.4(a) shows the reference image of size 500 by 500. A synthetic sensed image with severe perspective deformation is generated and shown in Fig. 5.5(b).
(a) (b)
Fig. 5.4: (a) The reference image. (b) The synthetic sensed image.
The reference features points are extracted using the Gabor filtering technique. A reference matching database is constructed offline using the five planning strategies. Given the sensed image the feature points are extracted first. Then the online registration process is invoked to register the two images. The first starting reference point pair is fetched from the reference matching database and the corresponding sensed point pair is found right away in the case, since this feature point pair is in the overlapping area. Both pairs are shown in the images as the two superimposed triangles. They lead to an affine transformation T(0). Then, the transformation model is updated by the iterative algorithm IMU and converges in two iterations.
Table 5.2 lists three estimated transformation matrices T(c) for c = 0, 1, 2. To demonstrate how the transformation matrix converges, Fig. 5.5(a) shows the feature points
using T(c), c = 0, 1, 2. Furthermore, Figs. 5.5(b)–5.5(d) show the registration results under the three transformation models. The RMSE of distances between the sixteen matched point pairs is 0.75 pixels, so it implies the final homography model is rather accurate.
(a) (b)
(c) (d) Fig. 5.5: (a) The partial overlapping between the image boundaries of the reference and three
transformed sensed images. (b)-(d) The view registration results under T(0), T(1), and T(2), respectively.
TABLE 5.2THE TRANSFORMATION PARAMETERS PRODUCED IN THE THREE ITERATIONS
M3×3
m11 m12 m13 m21 m22 m23 m31 m32 m33
T(0) -0.6110 -1.4876 664.0500 1.0264 -0.8557 161.9824 0 0 1 T(1) -0.7977 -0.9341 674.8476 1.0080 -0.7286 173.6283 -0.0005 0.0018 1 T(2) -0.7995 -0.9120 682.9316 1.0400 -0.7318 175.3778 -0.0005 0.0020 1
B) Image Noise Resistance
To demonstrate the usefulness of strategy 1 of the offline planning in combating with image noise, we generate 100 noisy reference image copies by adding Gaussian noise with signal-to-noise ratio 6.2 dB to the original reference image shown in Fig. 5.4(a). Fig. 5.6 shows the effect of image noise on the ranking of the reference feature points according to the descending order of the products of normalized energy and orientation factors E(pk)O(pk), k∈{1, 2, .., n}. The horizontal axis indicates the ranking sequence of the reference feature points before the introduction of image noise. For each of the 100 noisy reference image copies, the feature point ranking process is applied. The vertical axis indicates the new ranking number for each feature point in the horizontal ranking sequence. The mean of the new ranking number is indicated by the marker “*”, and the corresponding standard deviation of the new ranking number is indicated by the blue vertical bars centered at the mean rank at each horizontal ranking place. We add a dashed line of slope 45o to serve as the reference line for the ranking change evaluation. Any rank marker located above the reference line indicates a ranking setback under the influence of the image noise, any rank marker located below the line indicates a ranking improvement, and any rank marker located on the line indicates no ranking change. The experimental result shows that the new ranking numbers are fairly close to the old ones. Therefore, the ranking based on the product of E(pk)O(pk) is fairly stable in
Fig. 5.6: The effect of image noise on the ranking of reference feature points according to the product of normalized energy and orientation factor E(pk)O(pk), k∈{1, 2, .., n}. (See the text).
C) The Efficiency of Online Registration between Two Partially Overlapped Images
In this experiment we use two types of images, building, and landscape painting, to demonstrate the capability of our method in handling the registration of two partially overlapped images. Figs. 5.7(a) and 5.7(b) show the two building images superimposed with the two partitioned lines of the entire image and the labels of extracted feature points. The left image serves as the reference image. It can be seen that the overlapping area contains R2 and R4. From Table 5.3, we can explain the importance of using {OIi}i=1,2,…,6 to guide the registration process. Initially, we select SPP6 based on the maximum OIi value and fetches the first sorted point pair (#40, #57) from it. The processing results show |CPS| = 0 and no matched point pairs are found in the sensed image for either reference point of the pair (#40,
#57). This indicates that the region pair (R3, R4) of SPP6 is not totally in the overlapping area.
Then all entries in the six lists {SPPi}i=1,2,…,6 involving one of the reference points #40 and
#57 will be removed, and the corresponding overlap indices {OIi}i=1,2,…,6 are updated accordingly. Next, we select SPP1 whose updated OI index is the largest and the leading point pair (#20, #7) of SPP1 is fetched. The online registration process fails again with a final size
|CPS| = 0. It indicates the overlapping area is not found yet. The third attempt chooses SPP5
whose updated OI index is the largest and the reference point pair (#18, #38) is fetched from SPP5. These two reference points immediately lead to a successful registration with a final size |CPS| being 23. Thus, after three attempts (< 5) we find the point pair (#18, #38) that is totally in the overlapping area (R2, R4). The execution time for this on-line registration process is 0.312 seconds. Fig. 5.7(c) shows the final registration result of the reference and sensed images.
(a) (b)
(c)
Fig. 5.7: (a)-(b) Two real building images that are partially overlapped. (c) The final registration result.
TABLE 5.3THE STATISTIC OF THE ONLINE REGISTRATION PROCESS FOR REGISTERING TWO BUILDING
We apply the online registration process to another set of three synthetic landscape images shown in Figs. 5.8(a)-5.8(c). The reference image is given in Fig. 5.8(b). The final registration result is given in Fig. 5.8(d).
(a) (b) (c)
(d)
Fig. 5.8: (a) -(c) Three synthetic landscape images used for view registration. (d) The final registration result.
Table 5.4 gives the respective registration efficiencies with and without the five off-line planning strategies. We list the total number of attempts to fetch a reference point pair (PP) from the reference matching database SPP to complete a successful view registration. We also
record the total computational time (T) taken to complete a successful view registration.
Without the use of planning strategies, only the IMU algorithm will be employed to find a solution model T(r) using a random drawing of two starting reference points from the set of all possible combinations of the reference point pairs. The model T(r) is correct, if |CPS(r)| is greater than a specified threshold. The IMU process is repeated until a successful view registration is completed (note the reference and sensed images are well overlapped in both cases). The registration statistics are collected for 100 successful runs. Let PPavg denote the average numbers of attempts to randomly draw a reference point pair until a successful view registration is completed and let Tavg be the average of the computer execution time taken for completing a successful view registration. The results indicate our method can cut down the computer execution time by using the offline planning strategies. The time reduction benefitted from offline planning strategies is larger, when there are more feature points in the given pair of images.
TABLE 5.4REGISTRATION EFFICIENCY COMPARISON WITH AND WITHOUT THE OFF-LINE PLANNING STRATEGIES
Registration with
From our experience the online registration process generally obtain a correct solution within 5 iterations. To put into a more formal statement, under the assumptions that the invariant feature points can be reliably extracted by the feature extractor and that the overlapping area covers at least two sub-regions (a 50% overlapping area ratio), the online registration process will find the overlapping area between the reference and sensed images
using the OI index within a finite number of attempts (4, most of the time in our case). The ensuing view registration will succeed, since the first sorted point pair fetched from the database SPP is totally in the overlapping area and will find a correct matched pair in the sensed image. If the database access number, denoted by NOLRP, exceeds a specified bound (5 in our case), it is likely that the two images are not overlapped at all or only slightly overlapped. So the registration process should be terminated. Of course, we can increase the bound on NOLRP when considering those cases with an overlapping percentage less than 50%.