Chapter 3 Construction of 3D Images from KINECT Images
3.3 Review of a method for Geometric Correction of 3D Images
3.3.2 Correction Algorithm and Experimental Results
In this section, the method for finding the approximating paraboloid is described and some experimental results are shown. The criterion of minimum sum of squared errors (MSSE) is used to decide the parameters of the approximating shape which is a paraboloid. The following is the detail of this process.
First, let the equation of the paraboloid be described by:
where C is the distance between the KINECT device and the apex of the paraboloid, as shown in Figure 3.6. The equation for computing the value SSE of the SSE is:
where (xi, yi, zi) are the coordinates of a sample point. To find the coefficients A, B and (3.16), (3.17) and (3.18) become the three-variable linear equations after substituting all the values of xi, yi and zi into the simultaneous equations. By solving these three
23
demanded paraboloid is so obtained. Finally, we show some examples of the experimental results in Figures 3.7 and 3.8.
(a) (b)
Figure 3.7 A correction result of Figure 3.5 (b). (a) Before correction. (b) After correction.
(a) (b)
Figure 3.8 A 3D image seen from above. (a) Before correction. (b) After correction.
24
Chapter 4
Modeling of Around-car Events
4.1 Construction of Around-car 3D Images
In this chapter, we describe the method we propose for modeling of around-car events. In Section 4.1.1, the iterative closest point (ICP) algorithm and the K-D tree algorithm [10] used in this study are reviewed. In Section 4.1.2, how the calibration work is done by using the ICP algorithm to find the geometric relationship between every two KINECT devices is presented. With the calibration information, the idea and algorithm for constructing the around-car model are proposed in Sections 4.1.3 and 4.1.4, respectively. Finally, we show some experimental results in Section 4.1.5.
4.1.1 Review of Iterative Closest Point (ICP) Algorithm and K-D Tree Algorithm
The iterative closest point (ICP) algorithm can be employed to minimize the difference between two groups of points. The concept of the algorithm is simple. It revises the transformation, including rotation and translation, from one object to the other, iteratively in order to minimize the total distance measure between the two groups of points.
The ICP is often used to match objects to compute their similarity. It is found useful for constructing 2D or 3D images from different views because object
25
registration or stitching, which is needed in this study, requires shape matching before the work is conducted.
A K-D tree algorithm is also employed in this study to reduce the amount of computation for searching the closest point using the ICP algorithm. The method for constructing a K-D tree, simply speaking, is to separate the group of points in concern into two parts by the median of the X, Y and Z coordinates sequentially. When searching the closest point, it takes less time to search a K-D tree rather than to search all the points. The complexity of searching becomes O(N2/3) from O(N).
4.1.2 Calibration between Neighboring KINECT Devices Using ICP
In Section 4.1.1, it was mentioned that the ICP algorithm can be employed to minimize the difference between two groups of points. This inspired us to get the idea that we might use the ICP algorithm to “register” two objects if the two groups of points composing the objects are similar in their features. So, a method of calibration of the geometric relationship between two neighboring KINECT devices using the ICP algorithm is proposed in this study, and is described in this section.
First, a carton is put between every two neighboring KINECT devices as the calibration target. Then, depth images are acquired with these two KINECT devices, respectively. At this moment, we have two similar groups of points which come from an identical object, the carton, but appear in two different views. After the ICP algorithm is applied to register them, geometric relationship parameters, including the horizontal included angle and the displacements along the x-, y- and z-axes can be obtained. This completes the calibration of the geometric relationship between two neighboring KINECT devices. A detailed description of this process as an algorithm is
26
as follows.
Algorithm 4.1: calibration between two neighboring KINECT devices using the ICP algorithm.
Input: depth images I1 and I2 of a calibration target (a carton) acquired respectively with two neighboring KINECT devices D1 and D2.
Output: the geometric relationship parameters, a horizontal included angle i and a 3D displacement di, of D1 and D2.
Steps:
Step 1 Convert the coordinates of the depth images I1 and I2 into two groups G1 and G2 of 3D points, respectively, by the coordinate conversion scheme described in Section 3.2.1.
Step 2 Move group G1 of 3D points iteratively through a series of transformations Ti, each including a horizontal included angle i and a 3D displacement di = (dix, diy, diz) along the x-, y- and z-axes, respectively.
Step 3 For each transformation Ti = (i, di), compute the Euclidean distances Djk
from each point Pj of G1 to every point Pk of G2.
Step 4 For each point Pj, choose the minimum distance Djm from the distances Djk. Step 5 Sum up all the Djm as the distance Di for Ti between G1 and G2.
Step 6 Find the transformation Ti = (i, di) with the minimum Di as the desired relationship parameters.
4.1.3 Idea of Proposed Method for Around-car Scene Model Construction
According to the discussions in Section 3.2, we can construct a 3D partial scene model by the use of a single KINECT device. To construct the around-car 3D scene
27
model, a technique to use of multiple KINECT devices simultaneously is needed. And the relationship information for every two KINECT devices, which is obtained during calibration as described previously, is also needed. Under these conditions, we can start to construct the around-car scene model.
4.1.4 Construction Algorithm
In Section 2.1, we discuss how to use a desktop computer to control multiple KINECT devices to work simultaneously via a USB expansion card. As for calibration, the proposed method for acquiring the geometric relationship parameters between neighboring KINECT devices has been described in Section 4.1.2. A flowchart of the proposed around-car model construction algorithm is shown in Figure 4.1. And a detailed description of the algorithm is as follows.
Algorithm 4.2: around-car scene model construction.
Input: images I1 through I14 acquired with the 14 around-car KINECT devices, respectively.
Output: an around-car model Mcar. Steps:
Step 1 Construct 3D models M1 through M14 using images I1 through I14, respectively, by the 3D image construction scheme described in Section 3.2.3.
Step 2 For i = 1~13, compute the geometric relationship parameters Ri between the 3D models Mi and Mi+1 iteratively by the calibration algorithm (Algorithm 4.1) described in Section 4.1.2, with each Ri including a horizontal angle i
and a 3D displacement di = (dix, diy, diz) along the x-, y- and z-axes, respectively.
Step 3 For i = 1~13, perform the following steps.
28
Figure 4.1 A flowchart of the around-car model construction algorithm.
4.1.5 Experimental Results
In this section, some experimental results of conducting Algorithm 4.2 for
29
around-car scene model construction are shown. At first, the comparative illustrations from Section 4.1.2 are shown in Figure 4.2. From Figure 4.2(b), it can be seen that after calibration, the 3D models of the identical cartons constructed from images acquired with two neighboring KINECT devices overlap each other well. Then, the constructed around-car model is shown from each of the three sides of the car respectively, as shown in Figure 4.3.
(a) (b)
Figure 4.2 Comparison of constructed 3D scene models before and after calibration. (a) Before calibration. (b) After calibration.
(a)
(b) (c)
Figure 4.3 The constructed around-car model. (a) Front view. (b) Left side view. (c) Right side view.
30
4.2 Modeling of a By-passing Car Using 3D Distance-weighted Correlation (DWC)
4.2.1 Review of DWC
The measure of distance-weighted correlation (DWC) was proposed by Fan and Tsai [11] originally for automatic Chinese seal identification. The measure is defined as the minimum distance between two groups S and T of pixels of seal imprint images after the seal imprint images are overlapped. For each pixel p in S, a pixel in T with used to decide an effective distance; distances larger than this threshold is discarded.
Finally, the DWC defined for the two groups of pixels, S and T, is defined as follows:
where the coefficient 1/2 is included to treat S and T symmetrically; and NS and NT are the total numbers of pixels in S and T, respectively. It can be verified that 0CK 1
31
identification, is a general measure for point-type object shape matching.
4.2.2 Proposed 3D DWC
An extension of the above-reviewed “2D” DWC measure is proposed in this study for the purpose of 3D scene model construction. However, the 2D DWC was used to decide whether the images of two object shapes are similar or not, while the proposed 3D DWC is used to compute the displacement of two similar objects in this study.
4.2.3 Idea of Modeling of a By-passing Car Using 3D DWC
The depth range of the KINECT device is limited and the horizontal angle is only 57 degrees, so we cannot acquire the 3D information of the entire scene from a single image frame. Therefore, it is difficult to construct a complete model of a scene with only one KINECT device, or equivalently, with only a pair of color and depth images acquired with a single KINECT device. But we discovered that the models in any two sequential image frames acquired with two neighboring KINECT devices, when acquired with a high frame rate, are very similar. Since it is possible to overlap the two similar objects by the 3D DWC as mentioned previously, the 3D DWC can thus help us construct the model of a scene from different frames acquired with distinct KINECT devices. As the 3D information increases, it is possible to construct a more complete model of a scene.
4.2.4 Modeling Algorithm
In the proposed scene modeling algorithm, it is assumed first that a 3D model is
32
already constructed by one KINECT device as described in Section 3.2. Then, by the use of the 3D DWC, we compute the displacement between the 3D models which are constructed from the images acquired with every two KINECT devices. By using all the resulting displacements, it is feasible to construct a more complete model of a scene. In this study, the mainly concerned object in the scene model is a by-passing car. A flowchart of the modeling algorithm is shown in Figure 4.4. A detailed description of the algorithm is as follows.
Algorithm 4.3: modeling of a by-passing car using 3D DWC.
Input: a series I of image pairs I1, I2, …, In acquired of a by-passing car Mcar with n background image parts beyond the lane in the next step.
Step 2 Perform the following steps to filter out unrelated scene parts other than the car in each image pair Ij, respectively, by the thresholds T learned in Step. 1:
2.1 for each pixel p in the depth image Idj, if the depth value of p is larger than T, then remove p from Idj;
2.2 for each pixel p' in the color image Icj, if the corresponding pixel p in Idj
is removed, then also remove p' from Icj.
Step 3 For each image pair Ij,construct a 3D model of the by-passing car Mj by the 3D image construction scheme described in Section 3.2.3.
Step 4 For j = 1 through n 1 (i.e., for every two neighboring car models), perform the following steps to construct a complete car model Mcar:
33
4.1 compute the 3D displacement dj = (djx, djy, djz) along the x-, y- and z-axes, respectively, by matching the two partial car models Mj and Mj+1
by the use of the 3D DWC measure described previously.
4.2 overlap the two partial car models Mj and Mj+1 in accordance with the computed 3D displacements dj to form a new model to replace the original partial model Mj+1.
Step 5 Display the resulting car model Mcar of the last step by drawing all the overlapped points of Mcar in the 3D space with the OpenGL.
A series of
Figure 4.4 A flowchart of the car modeling algorithm.
34
4.2.5 Experimental Results
In this section, an example of the 3D models of the by-passing car resulting from executing Algorithm 4.3 described previously is shown in Figure 4.5. It is obvious that with more image frames, the constructed model of the by-passing car becomes more complete. Furthermore, from Figure 4.5(a) we can see only the middle part of the car, but from Figure 4.5(d) we can see its rear part. This experimental result shows the effectiveness of the use of the 3D DWC measure for 3D object modeling.
(a) (b)
(c) (d)
Figure 4.5 The models of a by-passing car constructed by overlapping different numbers of frames. (a) By one frame. (b) By five frames.(c) By ten frames. (d) By fifteen frames.
35
4.3 Restoration of Car-accident Scenes
4.3.1 Idea of Restoring Car-accident Scenes Using 3D DWC
Since the depth range yielded by the KINECT device is limited, as described previously, once the distance between the KINECT device and the object is out of the range, it becomes impossible to acquire the depth information. In addition, when a car accident occurs, we need to construct the models of the involved cars in order to clarify the responsibility of the car accident by inspecting the relative geometric conditions of the constructed car models. But the distance between the cars in a car accident is usually smaller than the minimum value of the depth range. Nevertheless, from the Section 4.2, we know that it is possible to increase 3D information by overlapping the models in continuous frames. That is, we can construct a more complete model of the involved car in the car accident by overlapping the models in the previous frames. This means that we can use the 3D DWC measure to restore a car accident scene. More details are presented in the following.
4.3.2 Restoring Algorithm
First, the depth and color images of the involved cars which appear just before the car collision happens are needed. Then, the backgrounds and the depth information, which do not belong to the car, are filtered out to construct the car models more easily. Next, the displacement for overlapping the car models is computed by the use of the 3D DWC measure, as done previously. Once the models of the involved cars are overlapped accurately with the displacement, we can restore the car accident scene by constructing more complete models of the involved cars. A
36
flowchart of the restoration algorithm is shown in Figure 4.6. A detailed description of the algorithm is as follows.
Algorithm 4.4: restoring of a car-accident scene using 3D DWC.
Input: an image sequence I1 through In of an involved car (other than the one on which the proposed system is installed) in a car-accident.
Output: a complete model Mcar of the involved car.
Steps:
Step 1 Pick out the image Ij of the involved car which appears just before the car collision occurs by inspecting the image sequence I1 through In acquired of the involved car.
Step 2 Construct 3D models M1 through Mj from images I1 through Ij by the 3D image construction scheme described in Section 3.2.3.
Step 3 For k = 1, 2, …, j1, compute the 3D displacement dk = (dkx, dky, dkz) along the x-, y- and z-axes, respectively by matching the car models Mk and Mk+1 by the use of the 3D DWC scheme mentioned in Section 4.2.2.
Step 4 For k = 1, 2, …, j1, overlap the car models Mk and Mk+1 in accordance with the computed 3D displacements dk to form a more complete model of the involved car Mcar.
Step 5 Display Mcar by drawing all the points in Mcar in the 3D space with the OpenGL.
4.4 Experimental Results
Due to security concerns, a toy car was used to simulate car accidents in this experiment. Figure 4.7 shows the models of the car before the instant of collision in a simulated car accident in different frames, respectively. The green lines simulate the
37
range of our car on which the proposed system is installed. In Figure 4.7(d), we can see that part of the toy car passes through the area surrounded by the green lines. It means that the toy car collides with our car. By the constructed model, it is feasible to restore a car-accident scene and check the responsibility in law.
An image sequence of the involved
car
3D image construction
3D car
models 3D DWC
Displacement of matching OpenGL
a complete model of the involved car
Pick out the specific image
sequence
Figure 4.6 A flowchart of the restoring algorithm.
38
(a) (b)
(c) (d)
Figure 4.7 Models of an involved car before the instant of collision in different frames. (a) 60 frames before. (b) 40 frames before. (c) 20 frames before. (d) At instant of collision.
39
Chapter 5
Real-time Response to Around-Car Dangerous Events
5.1 Monitoring of Ramps on Roads
5.1.1 Idea of Monitoring of Ramps on Roads
The slope of the ramp, especially that of a downhill one, is important information for a driver while driving on the road. The speed of the car will rise because of the gravity when driving on a downhill road. As the speed of the car goes up, it reduces the driver’s reaction time. That will be a potential danger to driving safety. In order to reduce the danger, we propose a method of using KINECT images and trigonometric functions and mathematical geometry to measure the slope of the ramp. In this way, the driver can know the slope of the ramp before driving downhill. If the driver is able to slow down earlier on a steep downhill road, the driving safety is improved.
5.1.2 Monitoring Algorithm and Experimental Results
In the proposed method, the KINECT device we use is mainly the one affixed in the front of the car, which is tilted down thirty degrees in order to increase the acquired depth information of the road, as shown in Figure 5.1. As seen in Figure 5.2, let θ3 represent the slope of the ramp. Then, it can be easily figured out from
40
trigonometry that
where θ1 is 30o as mentioned previously, and according to trigonometry again, we have:
where ΔY and ΔZ can be computed by points P(Y1, Z1) and Q(Y2, Z2) which are selected from the centerline of the KINECT device on the ramp. By Equation 5.1, the slope of the ramp θ3 can be computed. A flowchart of the algorithm is shown in Figure 5.3. A detailed description of the algorithm for this process is as follows.
Algorithm 5.1: computing the slope of a ramp.
Input: a depth image I acquired with the KINECT device affixed in the front of the car.
Output: a slope of a ramp.
Steps:
Step 1 Select the lowest pixel point P on the image boundary from the centerline of the depth image I.
Step 2 Select the center point Q of the depth image I.
Step 3 Convert the coordinates and the depth values of pixel points P and Q into 3D points P(X1, Y1, Z1) and Q(X2, Y2, Z2), respectively, by the coordinate conversion scheme described in Section 3.2.1.
Step 4 Compute ΔY and ΔZ by subtracting the values of Y1 and Z1 of point P from
41
Step 6 Compute as output the slope of the ramp θ3 by Equation 5.1.
Figure 5.1 The KINECT device affixed in the front of the car and it is tilted down thirty degrees.
Figure 5.2 Geometry of slope computation where the orange-colored line represents a slope.
42
Figure 5.3 A flowchart of computing slopes algorithm.
Once the slope of the ramp θ3 is computed, it is feasible to compute the magnitude of gravity g, as shown in Figure 5.4. From the figure, we have:
where a represents the acceleration of the car along the ramp. Some results of computations of the value a in different slopes are shown in Table 5.1. Assume that the ramp is 100m long and the car speed is 40 km/hr. It takes 9 seconds to pass the ramp without the acceleration a. Once we have the acceleration, the time is shorter than 9 seconds. By the constant acceleration equation, we have:
where t denotes the passing time, s is the car speed, and l is the length the car goes. By
a = gcos(90o – θ3) (5.3)
1 2
l s t 2 at (5.4)
43
substituting the acceleration a = 17.64 km/hr, the car speed s = 40 km/hr, and the slope length l = 100m into Equation (5.4), we can get the solution, the passing time, to be about 4.5 seconds which is a half of 9 seconds. It means that if the slope of the ramp is more than 30 degrees, the time will be less than a half of the time required for normal driving on flat road surfaces. Therefore, the driver should slow down the car speed to avoid danger.
Figure 5.4 The components of gravity g.
Table 5.1 The results of computations in different slopes.
Slope θ3 10o 20o 30o 40o 50o 60o
Acceleration a 6.12 12.06 17.64 22.68 27.03 30.55 km/hr
We now show some experimental results of applying the above algorithm to real situations. Figures 5.5 and 5.6 show two different results on the ground and on a ramp, respectively. Figures 5.5(a) and 5.6(a) show respective images of the ground and the
44
ramp in front of the car while Figures 5.5(b) and 5.6(b) respectively show the slopes of the ground and the ramp, respectively. The repeated values in Figures 5.5(b) and 5.6(b) denote keep monitoring the ramp.
(a)
(b)
Figure 5.5 The experimental result on the ground. (a) Front view from the car. (b) The values of the slope.
(a)
(b)
Figure 5.6 The experimental result on the ramp. (a) Front view from the car. (b) The values of the slope.
45
5.2 Monitoring of Limitations of Heights on Roads
5.2.1 Idea of Monitoring of Limitations of Heights on Roads
When driving on the road, there are some places where limitations on the car
When driving on the road, there are some places where limitations on the car