The last module in the system is the video object abstraction module. The video abstraction is generated using the video abstraction algorithm by selecting the key frames with meaningful semantics. Because the moving objects are the most important parts in surveillance videos, the selection of important key frames is equivalent to the selection of important key objects. Therefore, in our video abstraction algorithm, we will analyze the tracked object trajectories and detect object events to extract representative key objects.
Although the best and the most representative key objects of an object trajectory can be selected after the life cycle of that object is terminated, this kind of approach is not suitable for a real-time tracking system like ours. In order to achieve on-line alarming on real-time tracking system, the key objects must be selected near real-time, which means the delay must be bounded and very small. Therefore, every time a new frame comes in, our algorithm examines the current tracked object in each trajectory and selects it as a new key object if it is representative enough for its trajectory.
One of the criteria for key object selection is based on the object events which are representative for some object states or objects relationships at some time instant. Such events may raise our human’s interests. There are some important object events in general domains, such as appearance and disappearance. Besides, the motions and the positions of the object may also be used as the selection criteria. For example, we may have interests and pay more attention when a new object appears or the moving direction of the object changes because they can represent significant events. Therefore, the analysis of the object trajectories to extract this specific information is required.
The diagram of the abstraction algorithm is shown in Fig. 17. There are three modules in this algorithm. The algorithm takes the object trajectories generated in the video object tracking algorithm as input. The abstraction will be generated by selecting the frames with
33
key objects and output to clients.
3.4.1 Object State Analysis
The object state analysis process detects the general object events such as appearance, disappearance, occlusion and split of objects. Because we have handled and detected these events for object matching in the tracking algorithm, we can directly capture these events by examining the state transition of OBJ_STATE and the OCC_STATE of the objects. The only exception is that we do not directly extract the event when an object appears because the temporal filtering is applied to filter out the ghosts. Therefore, the events of object appearance will only be captured when the object survives for a period of time after it appears.
Object Trajectories
Object State Analysis
Object Trajectory
Analysis
Selected Major and Minor Key Objects
Fig. 17. The abstraction algorithm
34
3.4.2 Object Trajectory Analysis
The object trajectory analysis process tries to analyze the trajectories to find the featured objects as key objects. The featured objects are representative for the changes in the moving speed and direction, the position in the frame view or the object size. Every time when new an object is tracked, our algorithm compares the motion, position and size of that object to those object features of the previously selected key object. To evaluate the motion difference of current object and the previously selected key objects, the motion vector distance function described in section 3.3.2 is used. However, to avoid the zigzag-like paths for non-rigid objects to affect the analysis of motion direction, the motion vector after temporal filtering is used. Fig. 18 shows the analysis process.
3.4.3 Video Abstraction with Selected Key Objects
After the object event detection process and the object trajectory analysis process, the abstraction can be output using the selected key objects. In our algorithm, we define two types of key objects: major key objects and minor key objects. The major key objects represent important event and are always exported. On the contrary, the minor key objects are less important and are exported only when there is no other key object exported recently. All the
MV of
Fig. 18. The trajectory analysis and key object selection process
35
key objects selected in object state analysis process are major key objects. Besides, the objects which change in motion significantly are also selected as major key objects. The key objects which are selected using the position as the criterion are minor key objects. Fig. 19 shows how the algorithm selects the key objects to export.
Major Key Obj
Minor Key Obj
Export Key Obj to client Any Major
Key Obj ?
Any Minor Key Obj ?
Key obj Recently exported?
Export Key Obj to client Y
N
Y Y
Fig. 19. The key objects exporting process
36
Chapter 4
System Architecture and Experiment Result
In this chapter, we will present the system for object-based video tracking and abstraction. In the section 4.1, we will first show an overview of the system architecture. In the sections 4.2 to 4.4, the experiment results of each module will be represented. The implemented system will be presented in the section 4.5.