Objects Matching Algorithm

The object matching algorithm in the sub-module in Fig. 4 tries to find matches for the objects using the matching function. Because our algorithm tries to handle various conditions, simply matching the objects detected in current frame to the objects detected in the previous frame is not enough. In our matching algorithm, the process is divided into several stages. For short, the objects detected in the current frame and the objects detected in the previous frame are denoted as CurrObj and PrevObj respectively, and the diagram is shown in Fig. 9.

In the matching algorithm, the objects detected in current frame and the objects detected in previous frame are taken into the matching function to find a best match. In our algorithm, we use the object trajectory to stores the tracked objects in each frame for each object entry.

C u r r e n t

Fig. 9. The process of the whole matching algorithm

25

So, the current objects that found matches here are appended to their object trajectories respectively. Because the matching function terminates when no more good matches could be found, there could be some current objects and previous objects left unmatched and they are stored in the CurrObjRest and PrevObjRest respectively. The current objects that left unmatched may be resulted from the appearance of new objects, the split of occluded objects or temporally disappeared objects revealed. Similarly, the previous objects that left unmatched may be resulted from the occlusion or the disappearance of the objects. Sub-modules designed to handled the events and condition will be presented in detail. The whole process of the matching algorithm will be presented after these sub-modules are presented.

3.3.3.1 Occluded Objects Matching

For those previous objects that left unmatched, our matching algorithm first tries to find if there is any objects occlusion events. As mentioned earlier, since both the conditions of occlusion and disappearance of objects will leave the previous objects unmatched, the COLLISION state must be used to judge if there is an occlusion event. Fig. 10 illustrates the relationship of occluded objects and Fig. 11 shows the diagram of occlusion objects matching process. Assume that the video objects VOt-11 and VOt-12 in time t-1 may collide with each other in the future, so both the objects are in the COLLISION state and we define that these two objects are in the same ‘collision group’. In addition, we also assume that the two objects occlude at time t and thus only one isolated object is detected. As described earlier, the current

VO_t-1¹ COLLISION

VO_t-1²

COLLISION VO_t¹

(a) (b)

VO_t-1¹ COLLISION

VO_t-1²

COLLISION VO_t¹

(a) (b)

Fig. 10. The relationship of occlusion objects;

(a) Before occlusion; (b) after occlusion

26

objects and the previous objects are first taken into the matching function and thus one of the objects in t-1, says VOt-12, is matched to the VOt1 at time t. After the 1-to-1 matching function, the video object VOt-11 is left unmatched. To handle the occlusion events, our algorithm first checks the possibility of objects occlusion by examining the COLLISION state of the unmatched previous object, for example the VOt-11 in Fig. 10, and the previous object in COLLISION state is taken into the matching function. Then, the current object that has being matched to the previous object which is also in the ‘collision group’ is also taken into the matching function. If a match is successfully found, it implies that there is indeed an occlusion since the current object can match to the previous objects in COLLISION state and it satisfies the real situation of the objects occlusion.

Once the event of the objects occlusion is detected, both the occluded objects go to the OCCLUSION state and they share the same object trajectory until they split into two. We define that these occluded objects are in the same ‘occlusion group’. Note that because the individual motion is required to track the each object when the occluded objects split, our algorithm keeps estimating the individual trajectory when the objects are occluded. The tracked objects are appended to respective object trajectories.

3.3.3.2 Split Objects Matching

Unmatched

Fig. 11. The occluded objects matching process

27

The split objects matching process handles the event of object split and matches the split objects. The relationship of the split objects and the diagram of the process are shown in Fig.

12 and Fig. 13. Assume the object VOt-11 in time t-1 is merged from two objects and they split

into two objects, VOt1 and VOt2, at time t. According to the matching function performed on current and previous objects, the previous object VOt-11 is matched to one of the current objects, for example VOt2. Therefore, the previous object VOt1 is left unmatched. Remember that there are explicit occlusion and implicit occlusion. Therefore, the conditions of split event become more complex. Because we cannot judge the possibility of split event simply with the OCCLUSION state, we need to divide the split object matching process into two steps.

In the first step, we try to detect the split events from explicitly occluded objects. First, VO_t¹

Fig. 12. The relationship of split objects;

(a) Before splitting; (b) after splitting

Occlusion

Fig. 13. The split object matching process

28

we find all the objects which are in the OCCLUSION state from the object trajectory lists and the objects that have not been matched to current objects are picked. Take Fig. 12 for example.

Although there is only one previous object, there is another object trajectory in OCCLUSION state. Then, the unmatched current objects and the objects we picked are taken into the matching function. Note that because the estimated motion for each individual object is used here. If a good match is found, it implies some occluded objects now split because the previously occluded objects now match to two objects individually. In this case, the split objects go back to the NORMAL state and the occlusion group for the occluded objects is deleted. Then the tracked current objects are appended to their respective object trajectories.

If the number of unmatched current objects is not zero, the second step is performed. In the second step, we try to detect the split events from implicitly occluded objects. The process is quite similar. However, instead of finding objects in OCCLUSION state from the object trajectories, all the previous objects are used for matching here since we cannot find any OCCLUSION flag in implicitly occluded objects. If a good match is found, the implication is that one previous object matches to two current objects, which means the split event of implicitly occluded object. In this case, our algorithm creates a new object trajectory for the object that splits out and the tracked objects before splitting are duplicated.

3.3.3.3 Estimated Objects Matching

Because we do not think that the object is dead soon after it disappears, we append an estimated object to its object trajectory for later matching process. We use the object information in the past few frames to predict the position and the motion of the estimated object. When the temporally disappeared object now reveals again in the current frame, therefore, we must pick up the estimated objects for matching. Fig. 14 shows the situations that an estimated object is used and Fig. 15 shows the diagram of the estimated object matching process.

29

This matching process first finds all the object trajectories in the DYING state and picks up the estimated objects from these object trajectories. Then, these estimated objects and the unmatched current objects are taken into the matching function. If an estimated object successfully matches to an unmatched current object, the current object is appended to the object trajectory of that estimated object and the OBJ_STATE goes back from the DYING state to the NORMAL state.

After all the sub-matching modules are presented, we now illustrate the process of the matching algorithm. The current objects and the previous objects are taken into the matching function, and the matching function terminates when no more good match can be found. If there are any events such as appearance, disappearance splitting and occlusion of objects, some current objects and previous objects will be left unmatched. As shown in Fig. 9, for the unmatched previous objects, first the matching algorithm performs the occluded objects matching process to check whether there are any objects occlusion events and tries to find

?

Fig. 14. The condition that estimated object is appended (a) Time t-2; (b) Time t-1, the object disappears, and an estimated object is appended ;(c) Time t, the object

reveals again and is going to be match to the estimated object

Matching

Fig. 15. The estimated objects matching process

30

matches. If there are still any previous objects that cannot find a match, we think that there are objects disappeared. For the object trajectories of these unmatched previous objects, we make the OBJ_STATE go to the DYING state and estimated objects are appended.

For the unmatched current objects, first it goes to the split objects matching process to check whether there are any object split events and try to find matches. The current objects that still cannot find matches will then go to the estimated objects matching process. After the estimated objects matching process, we will consider the rest of current objects as new objects and new entries for the object trajectories will be created.

Finally, the matching algorithm goes to the refresh trajectory function. In this function, the objects in DYING state are first examined. If the object stays in the DYING state too long, we will consider that the object is really disappeared for\ever and let it go to the DEAD state.

After that, the motion vector of each moving object is re-computed using the position of the newly tracked object position. Finally, based on the tracked object positions and the computed motion vectors, the function examines if any two objects may collide with each other in the near future. Each time after the matching algorithm finishes matching and processing all the objects, the tracking algorithm will pass the object to temporal filtering process.