Complexity Analysis

Complexity can be discussed in two different ways: time and space. Both complexities of the proposed and optimal method are discussed.

The complexity of Farins’ optimal method is divided into two parts: the building of coding cost matrix described in Section 1.5.1, and the optimal partition algorithm described in

83

Section 1.5.2. While building the cost matrix, the coding cost of all sub-sequences beginning at frame i and ending at frame k with reference frame r must be computed. Suppose that the sequence has N frames, the time and space complexity of building a cost matrix will be N³. However, they had developed a method to reduce the space required to N². The optimal partition algorithm using Eq. (1.12) to find the best partition frame-by-frame takes N² time.

The proposed method calculates the accumulated translation and scaling first, and both of them take linear time. The finding of candidate partition points is also linear time because it only observes the changes of accumulated translation and scaling once. Let M be the number of candidate partition points found in Section 4.3.1, finding reference frame for all possible sub-sequences takes M²×N time. Finally, the Farins’ optimal partition is applied. Since only M candidate partition points are involved, it takes only M² time. The accumulated translations and scalings must be hold in memory and the coding-cost matrix must be generated. These will need 2N+M² space.

Table 4.5 shows the complexity of both methods. The proposed method takes

2

2 N M

M N

N + + × + time, i.e. O(M²×N) in time and O(M²)+O(N) in space. Since M is usually very small in contrast to N in practical, for example, M=9 and N=300 in the sequence ‘stefan’, this makes the complexity of the proposed method better than the optimal method.

84

Table 4.5 Complexity comparison.

Time Space

Farin et al.’s optimal

method O(N³) O(N²)

Proposed method O(M²N) O(M²)+O(N)

85

CHAPTER 5

CONCLUSIONS AND FUTURE RESEARCH DIRECTIONS

5.1 Conclusions

An effective sprite generator without segmentation masks is proposed. The method is based on MPEG-4’s framework. A balanced feature point extraction with object removing is introduced to increase the precision of estimated global motion parameters. In order to remove the demand of segmentation masks, an intelligent blending strategy is proposed. It counts the occurrence of pixels and chooses the pixels with higher count to be blended into the sprite. The ghostlike shadows in the sprite generated by conventional reliability-based blending are eliminated, and the average PSNR is increased slightly.

An efficient and fast method for generating multiple sprites is also proposed. In contrast to the conventional sprite generating method that using a single sprite, using multiple sprites reduces the storage space of sprites. Conventional multiple sprite generation method uses an exhaustive search to find the optimal subsequence partition and optimal reference frame of each partition. However, the exhaustive search costs a lot of time. The proposed method consists of a sub-sequence partition algorithm and a fast reference frame selection algorithm, which are developed based on the frame accumulated translations and scalings. By using the proposed methods, a video sequence can be partitioned and reference frame can be selected in a very short time. In order to increase the performance of selected reference frames, two

86

reference frame validation methods are also proposed. The proposed validation methods searches frames close to the result of the fast reference frame selection method and check if better reference frame exists. The experimental results also show that the proposed method greatly increases the executing speed from 10 to 190 times in contrast to the Farins’ optimal method, the total sprite size is slightly higher and the qualities of generated sprites are preserved.

5.2 Future Research Directions

A sprite is a ‘pure’ background generated from a video sequence. This makes generated sprites very useful. For example, in a change-detection-mask based video object detection system, a sprite can be applied as the reference background instead of previous frames to achieve better results. We will work on the moving object detection methods based on generated sprites.

87

REFERENCES

[1] ISO/IEC MPEG Video Group, “MPEG-4 video international standard with amd. 1,”

ISO/IEC 14496-2, Jul. 2000.

[2] F. Dufaux and F. Moschieni, “Background mosaicking for low bit rate coding,” Proc.

IEEE Int. Conf. Image Processing, vol. 1, Sept. 1996, pp.673-676.

[3] M. Irani and P. Anandan, ”Video indexing based on mosaic representations,” Proc.

IEEE, 16(5) , pp. 905-921, May 1998.

[4] R. Szeliski, “Video mosaics for virtual environments,” IEEE Computer Graphics and Applications, 16(2) , pp. 22-30, Mar. 1996.

[5] R. Szeliski and H. Y. Shum, “Creating full view panoramic mosaics and environment maps,” Proc. ACM Conf. SIGGRAPH 97, Aug. 1997, pp. 251-258.

[6] M. Kourogi, T. Kurata, J. Hoshino and Y. Muraoka, “Real-time image mosaicing from a video sequence,” IEEE International Conference on Image Processing (ICIP'99), Oct.

1999, pp. 133-137.

[7] S. Ji and H.W. Park, “Moving object segmentation with adaptive sprite for DCT-based video coder,” IEEE International Conference on Image Processing, Oct. 2001, pp.

566-569.

[8] Y. Lu, W. Ga and F. Wu, “Automatic video segmentation using a novel background model,” IEEE International Symposium on Circuits and Systems. (ISCAS), May 2002, pp. 807-810.

[9] S.Y. Chien, Y.W. Huang, B.Y. Hsieh, S.Y. Ma, and L.G. Chen, “Fast video segmentation algorithm with shadow cancellation, global motion compensation, and adaptive threshold techniques,” IEEE Transactions on Multimedia, vol. 6, no. 5, Oct.

2004, pp. 732–748.

[10] D. Farin, P.H.N. de With, and W. Effelsberg, “Video-object segmentation using multi-sprite background subtraction,” IEEE International Conference on Multimedia and Expo, ICME ‘04, pp. 343-346, Jun. 2004.

[11] A. Krutz, M. Kunter, T. Sikora, M. Mandal and M. Frater, ”Motion-based object segmentation using sprites and anisotropic diffusion,” International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Jun. 2007, pp. 35-38.

[12] S. Kopf, T. Haenselmann, D. Farin, and W. Effelsberg, “Automatic generation of video summaries for historical films,” IEEE International Conference on Multimedia and Expo, ICME ‘04, pp. 2067-2070, Jun. 2004.

88

[13] T. Sikora, “The MPEG-4 video standard verification model,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 7, no. 1, Feb. 1997, pp. 19-31.

[14] ISO/IEC MPEG Video Group, “MPEG-4 video verification model version 18.0,”

N3908, Jan. 2001.

[15] J.J. Moré, “The Levenberg-Marquardt algorithm: implementation and theory,”

Numerical Analysis, ed. G. A. Watson, Lecture Notes in Mathematics 630, Springer Verlag, pp. 105-116, 1977.

[16] S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 13, no.

4, Apr. 1991, pp. 376–380.

[17] G.B. Rath and A. Makur, “Iterative least squares and compression based estimations for a four-parameter linear global motion model and global motion compensation,”

IEEE Trans. on Circuits and System for Video Technology, vol. 9, no. 7, Oct. 1999, pp.1075-1099.

[18] A. Smolić, T. Sikora, and J-R. Ohm, “Long-term global motion estimation and its application for sprite coding, content description, and segmentation,” IEEE Trans.

Circuits and system for video technology, vol. 9, pp. 1227-1242, Dec. 1999.

[19] Y. Lu, W. Gao, and F. Wu, “Fast and robust sprite generation for MPEG-4 video coding,” Proc. IEEE Pacific-Rim Conf. Multimedia, Beijing, China, pp. 118-125, Oct.

2001.

[20] Y. Lu, W. Gao, and F. Wu, “Sprite generation for frame-based video coding,” Proc.

IEEE Int. Conf. Image Processing, vol. 3, Thessaloniki, Greece, pp. 473-476, Oct.

2001.

[21] Y. Lu, W. Gao, and F. Wu, “Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques,” IEEE Trans. Circuits and Systems for Video Technology, vol. 13, no. 5, pp. 394-405, May 2003.

[22] F. Dufaux and J. Konrad, “Efficient, robust and fast global motion estimation for video coding,” IEEE Trans. on Image Processing, vol. 9, no. 3, Mar. 2000, pp. 497-501.

[23] A. Smolić and J-R. Ohm, “Robust global motion estimation using a simplified M-estimator approach,” International Conference on Image Processing (ICIP), Sep.

2000, pp. 868-871.

[24] H. Richter, A. Smolić, B. Stabernack and E. Müller, “Real time global motion estimation for an MPEG-4 video encoder,” Proceedings of Picture Coding Symposium (PCS ’01), Apr. 2001, pp. 401-404.

89

[25] Y. Keller and A. Averbuch, “Fast gradient methods based on global motion estimation for video compression,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 4, Apr. 2003, pp.300-309.

[26] H, Alzoubi and W.D. Pan, “Reducing the complexity of MPEG-4 global motion estimation using pixel subsampling,” Electronics Letters, vol. 44, Jan. 2008, pp.20-22.

[27] A. Smolić, K. Müller and J.-R. Ohm, “Global motion compensation and video mosaicing using different 2-D motion models,” Proceedings of Picture Coding Symposium (PCS ’01), Apr. 2001, pp. 331-334.

[28] L.G. Brown, “A survey of image registration techniques,” ACM Computing Surveys, vol. 24, no. 4, Dec.1992, pp. 325-376.

[29] E.C. Hildreth, “The analysis of visual motion: from computational theory to neural mechanisms,” Annual Review of Neuroscience 10, Dec. 1986, pp. 477-533.

[30] A. Azarbayejani, T. Starner, B. Horowitz, and A. Pentland, “Visually controlled graphics,” IEEE Trans. Pattern Anal. Machine Intell., vol. 15, Jun. 1993, pp. 602-605.

[31] R. C. Gonzalez and R. E. Woods, “Digital Image Processing,” 2nd Edition, Prentice Hall, Jan. 2002.

[32] N. Grammalidis, D. Beletsiotis and M.G. Strintzis, ” Sprite generation and coding in multiview image sequences,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 10, no. 2, Mar. 2000, pp. 302-311.

[33] C.T. Hsu and Y.C. Tsan, “Mosaics of video sequences with moving objects,”

International Conference on Image Processing 2001, Oct. 2001, pp.387-390.

[34] H. Watanabe and K. Jinzenji, “Sprite Coding in Object-based Video Coding Standard:

MPEG-4,” World Multiconference on Systemics, Cybernetics and Informatics (SCI) 2001, Proc. Vol.XIII, pp.420-425, Jul. 2001.

[35] H.K. Cheung, W.C. Siu, “Robust global motion estimation and novel updating strategy for sprite generation,” IET Image Processing, vol. 1, no. 1, Mar. 2007, pp. 13-20.

[36] T. Meier and K.N. Ngan, “Video segmentation for content-based coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, no. 8, Dec. 1999, pp. 1190–1203.

[37] M. Massey and W. Bender, “Salient stills: Process and practice,” IBM Systems Journal, vol.35, no. 3-4, 1996, pp.557-573.

[38] D. Farin, P.H.N. de With, W. Effelsberg, “Minimizing MPEG-4 Sprite Coding-Cost Using Multi-Sprites,” SPIE Visual Communications and Image Processing, Vol. 5308, pp. 234-245, Jan. 2004.

90

[39] D. Farin and P.H.N. de With, “Enabling arbitrary rotational camera motion using multisprites with minimum coding cost,” IEEE Trans. Circuits and system for video technology, vol. 16, no. 4, pp.492-506, Apr. 2006.

[40] S.Y. Chen, C.Y. Chen, Y.W. Huang and L.G. Chen, “Multiple sprites and frame skipping techniques for sprite generation with high subjective quality and fast speed,”

IEEE International Conference on Multimedia and Expo 2002 (ICME ’02), Aug. 2002, pp. 785-788.

[41] S.Y. Chen, C.Y. Chen; W.M. Chao, C.W. Hsu, Y.W Huang and L.G. Chen, “A fast and high subjective quality sprite generation algorithm with frame skipping and multiple sprites techniques,” International Conference on Image Processing. 2002 (ICIP ’02), vol. 1, Sep. 2002, pp. 193-196.

[42] M. Kunter, A. Krutz, M. Drose, M. Frater and T. Sikora, “Object-based multiple sprite coding of unsegmented videos using H.264/AVC,” International Conference on Image Processing 2007 (ICIP ’07), Sep. 2007, pp. 65-68.

91

PUBLICATION LIST

We summarize the publication status of the proposed methods and our research status in the following.

(1) I-Sheng Kuo and Ling-Hwei Chen, “High Visual Quality Sprite Generator using Automatic Segmentation and Intelligent Blending,” Visual Communications & Image Processing 2005 (VCIP ‘05), Proceedings of the SPIE, vol. 5960, Jul. 2005, pp. 2128-2139.

(2) I-Sheng Kuo and Ling-Hwei Chen, “A High Visual Quality Sprite Generator using Intelligent Blending without Segmentation Masks,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 20, no. 8, Dec. 2006, pp. 1139–1158.

(3) I-Sheng Kuo and Ling-Hwei Chen, “A Fast Multi-Sprite Generator with Near-Optimum Coding Bit-Rate,” accepted by International Journal of Pattern Recognition and Artificial Intelligence.