The ”grand challenge” problem for field robotics is: Create smart, reliable, mo-bile machines, capable of moving more capably than equivalent manned ma-chines in a wide variety of unstructured environments.
...
There are immediate applications, both in terms of complete systems and in terms of robotic components used in other applications. But there is also a rich set of open problems, in fundamental research as well as in applications, that will keep us all busy well into the future.
(Thorpe and Durrant-Whyte, 2001)
A
new discipline has been established at the intersection of SLAM and moving object tracking in this work. Simultaneous localization, mapping and moving object tracking can be treated as an innovation to seamlessly integrate SLAM and moving object tracking, or an improvement of SLAM and moving object tracking re-spectively.In the localization and mapping problem, information associated with stationary ob-jects are positive; moving obob-jects are negative, which degrades the results. Conversely, measurements belonging to moving objects are positive in the moving object tacking prob-lem; stationary objects are negative information, and are filtered out. The central thesis of this work is that both stationary objects and moving objects are positive to the whole problem and they are mutually beneficial.
7.1. Summary
In this dissertation, we established a probabilistic framework for integrating SLAM and moving object tracking. The first solution, SLAM with generic objects, is a general ap-proach which is similar to existing SLAM algorithms but with motion modelling of generic objects. Unfortunately. it has a very high dimensionality and is computationally demand-ing. In practice, its performance is often degraded because of highly maneuvering objects.
Consequently, we provided the second solution, SLAM with DATMO, in which the estimation problem is decomposed into two separate estimators. By maintaining separate posteriors for stationary objects and moving objects, the resulting estimation problems are much lower dimensional than SLAM with generic objects. This makes it possible to up-date both filters in real-time. The critical requirement for successful implementation of SLAM with DATMO is correct moving object detection. In addition to move-stop hypoth-esis tracking, we provided a consistency based approach and a moving object map based approach for detecting moving objects reliably.
Assuming that the static environment assumption is valid, SLAM is still limited to indoor environments, or outdoor environments with specific characteristics. For accom-plishing simultaneous localization, mapping and moving object tracking from ground ve-hicles at high speeds in crowded urban environments, we provided several algorithms and guidelines to eliminate the gaps between theory and implementation. These gaps are cate-gorized into three classes: perception modelling, motion modelling and data association.
We used the hierarchical object based representation to tackle the perception mod-elling issues of SLAM and moving object tracking. The hierarchical object based represen-tation integrates direct methods, grid-based approaches and feature-based approaches. In addition, we used the sampling and correlation based range image matching algorithm to tackle the uncertain and sparse data issues. Our experimental results have demonstrated that the hierarchical object based representation is an efficient and feasible way to accom-plish city-sized SLAM.
Theoretically, motion modelling is as important as perception modelling in Bayesian approaches. Practically, reliable pose predictions from the learned motion models of the robot and moving objects are essential to tasks such as collision warning, dynamic obstacle avoidance and planning. We began with a description of the model selection and model complexity issues. We explained why it is not correct to use the IMM algorithm with the stop model simplified from the constant velocity model for tackling move-stop-move target tracking. The corresponding solutions, the stationary process model and the move-stop hypothesis tracking, are described.
The data association problem is unavoidable because of uncertainty in the real world.
We addressed three data association problems in practice: in the small, in the cluttered and in the large. We described three general principles to solve data association: information exploiting, ambiguity modelling and covariance increasing. Geometric information from perception modelling as well as kinematic information from motion modelling are used to
7.2 FUTURE EXTENSIONS
remove the ambiguity. We used the correlation based image registration algorithm along with multi-scale pyramids to solve the revisiting problem robustly and efficiently.
After these theoretical and practical developments, the described formulas and algo-rithms were carried out with the Navlab8 and Navlab11 vehicles at high speeds in crowded urban and suburban areas. The copious results indicated that simultaneous localization, mapping and moving object is indeed feasible.
7.2. Future Extensions
This dissertation raises several interesting topics and there are a number of possible extensions for improving the performances of the system and the algorithms in both theo-retical and practical ways.
Between SLAM with GO and SLAM with DATMO
Since the full solution of simultaneous localization, mapping and moving object, SLAM with GO, is computationally demanding and infeasible in practice, we have presented and implemented the second solution, SLAM with DATMO. The experimental results using laser scanners and odometry have demonstrated the feasibility of SLAM with DATMO.
Recall that correct moving object detection is critical for successfully implementing SLAM with DATMO. Nevertheless, in the cases of using sonar and cameras, classifying moving objects and stationary objects may be difficult where a more robust but tractable solution is needed.
Fortunately, it is possible to find an intermediate solution between SLAM with DATMO and SLAM with GO as illustrated in Figure 7.1. In this dissertation, we have pointed out some potential extensions such as detection without thresholding in Section 6.2 and simul-taneous multiple moving object tracking in Section 4.7.
Figure 7.1. Between SLAM with GO and SLAM with DATMO.
Heterogeneous Sensor Fusion
For understanding complex scenes and increasing reliability and integrity of the robot, heterogeneous sensor fusion is the key. In this work, the Bayesian framework of simulta-neous localization, mapping and moving object tracking provides the guidance for fusing
measurements from perception and motion sensors. The experimental results using laser scanners and odometry/IMU are shown to be promising.
Nevertheless, laser scanners may not be sufficient to fully understand a complex scene.
For instance, traffic signs, lights and lanes can not be recognized. Besides, laser scanners may fail to produce reliable measurements in the situations addressed in Section 6.7. There-fore, other heterogeneous information should be included and fused to boost reliability and integrity.
Visual images from cameras contain rich information for scene understanding and compensate for some of the disadvantages of laser scanners. There are a number of ways to improve system performance using state-of-the-art algorithms from the computer vision literature. For example, pedestrian detection using laser scanners is difficult because the number of measurement points associated with a pedestrian is often small in our applica-tions. Recognition algorithms can be used to confirm the results of ladar-based detection.
Because only portions of the image with high likelihood have to be processed and range measurements from laser scanners can be used to solve the scale issue, the recognition process can be speeded up and run in real-time.
4-D Environments
The real world is indeed four-dimensional, three dimensions for space and one di-mension for time. Figure 7.2 shows two examples of 4-D environments. Accomplishing simultaneous localization, mapping and moving objects using 3-D perception and motion sensors is essential to successfully deploy a robot in such environments.
Figure 7.2. 4-D environments.
From a theoretical point of view, the formulation of simultaneous localization, map-ping and moving objects in 4-D environments is the same as the described formulas in this
7.3 CONCLUSION
dissertation. However, because of the higher dimensionality in 4-D environments, uncer-tainty estimation and analysis would be more difficult.
From a practical point of view, perception and motion modelling should be modified according to sensor capability. Because of the richness of 3-D spatial information, data association should be easier and more robust. However, more computational power is required to process large amount of perception and motion data.
Toward Scene Understanding
Estimating the states and motion patterns of the robot and moving objects can be treated as the lowest level of scene understanding. The described algorithms should be sufficient for safe driving in which the robot, or agent, provides proper warnings to as-sist human drivers. For autonomous driving among human drivers, higher level scene understanding such as event or scenario recognition is critical.
In the AI literature, there are a number of studies about activity, behavior and interac-tion modelling. Most related studies are based on simulainterac-tions or experiments conducted with the use of stationary sensors in indoor or controlled outdoor environments. Our work would make it feasible to conduct experiments in outdoor, dynamic, uncontrolled and very large scale environments. Integrating activity, behavior and interaction modelling into the current framework would lead to a higher level scene understanding.
7.3. Conclusion
It is our hope that this dissertation demonstrates that performing SLAM and moving object tracking concurrently is superior to doing just one or the other. We have answered some important and fundamental questions about formulation, perception modelling, mo-tion modelling and data associamo-tion. Addimo-tionally, we have demonstrated that simulta-neous localization, mapping and moving object tracking is indeed feasible from ground vehicles at high speeds in urban environments. We hope that this thesis will serve as a basis for pursuing the questions in fundamental research as well as in applications related to scene understanding or other domains.