Discussions

After presenting the experimental results, we would like to discuss some issues in concern as follows.

The first issue is the feature point locating process. It is a most important process of the virtual face creation, and users must concentrate on images which the feature points will be located on during this process. They must locate 26 feature points manually, and then adjust the y-coordinates of those points slightly. The system in this study will automatically adjust the x-coordinates of those points to be at symmetric positions. Because the virtual faces we create are realistic, the y-coordinates must be located accurately to get the optimal results.

The second issue is the feature point tracking process. The proposed tracking technique dynamically fits the mouth shapes in the real-face video models and the window sizes are dynamically changed according to the two mouth states we proposed. Because the image information of closed mouths are different from other mouth shapes, including the shape, brightness, and texture, the correction of feature point locations are proposed. For each frame, when we detect any one of the three closed-mouth shapes we proposed, we correct the feature point locations immediately.

By implying the correction, we can ensure that every feature point in each frame of the video models is correct.

Finally, we discuss the virtual-face creation process. In this process, realistic virtual faces are created. We dynamically scale the mouth sizes and adjust the positions of the chins and the mouths according to the mouth shapes of the real-face video models.

The experimental results show that the created virtual faces are natural.

74

Chapter 6

Conclusions and Suggestions for Future Works

6.1 Conclusions

In this study, a system for automatic creation of virtual faces with dynamic mouth movements has been implemented. We have presented a way to automatically create virtual faces from a given neutral facial image, and the mouths of these virtual faces can move dynamically by feature point tracking and mouth size scaling. The system contains three components, including a feature point locator, a feature point tracker, and a virtual face creator.

By the feature point locator, 26 feature points of the input image and those points of the first frame of the video model are located manually at the accurate and symmetrical positions. Mouth feature regions are defined by groups of these points.

Next, using the feature point tracker, the feature points of each frame can be extracted by an image matching technique proposed in this study. Furthermore, mouth-movement information of the video model is analyzed in this study to get mouth states. The mouth states of each frame are detected for image matching to dynamically change window sizes.

However, correction of the feature point locations need be implemented when detecting a closed mouth. This is achieved by the use of a hierarchical bi-level thresholding technique and an edge detection technique.

75

Finally, the virtual face creator creates virtual face sequences with dynamical mouth movements by a proposed mouth shape morphing technique. The mouths and the chins of the virtual faces are created to move naturally, and the skins near them are made to look smooth by a morphing technique. And, a mouth size scaling technique is proposed for the synchronization of mouth movements.

The experimental results shown in the previous chapters have revealed the feasibility of the proposed system.

6.2 Suggestions for Future Works

Several suggestions for future researches are listed as follows.

(1) Automatic detection of feature points of a mouth --- For the convenience of using the proposed system, automatic feature point detection can let users skip the feature point locating process and the operation of the proposed system will be easier.

(2) Integration of eye and eyebrow movements --- With the eye and eyebrow movements, created virtual faces will become more vivid and amusing.

(3) Integration of facial wrinkles --- Just like integration of eye and eyebrow movements, virtual faces with wrinkles, such as those on the forehead, along smile lines, and round the eyes, look more natural and lifelike.

(4) Improvement on feature point tracking --- In order to deal with larger videos, the speed of feature point tracking must be faster to have the ability to handle high-quantity videos.

(5) Improvement on mouth shape detection --- In the proposed system, a person in the video model talk with a medium speed, but the talking speed

76

of people is faster. For wider applications, the mouth shape detection must be improved.

(6) Real-time virtual face creation --- If the facial features can be extracted in real time, the virtual face can synchronize with a speaking person in the proposed system in real time. The proposed system can be used in distance teaching for students to choose a favorite movie star or singer to be the teacher.

77

References

[1] Y. L. Chen and W. H. Tsai, “Automatic generation of talking cartoon faces from image sequences,” Proceedings of 2004 Conference on Computer Vision, Graphics and Image Processing, Hualien, Taiwan, Republic of China, Aug.

2004.

[2] Y. L. Chen and W. H. Tsai, “Automatic real-time generation of talking cartoon faces from image sequences in complicated backgrounds and applications,”

Proceedings of 2006 International Computer Symposium (ICS 2006) - International Workshop on Image Processing, Computer Graphics, and Multimedia Technologies, Taipei, Taiwan, Republic of China, Dec. 2006.

[3] Y. C. Lin, “A study on Virtual talking head animation by 2D Image analysis and voice synchronization techniques,” M. S. Thesis, Department of Computer and Information Science, National Chiao Tung University, Hsinchu, Taiwan, Republic of China, June 2002.

[4] C. J. Lai and W. H. Tsai, “A study on automatic construction of virtual talking faces and applications,” Proceedings of 2004 Conference on Computer Vision, Graphics and Image Processing, Hualien, Taiwan, Republic of China, Aug.

2004.

[5] Y. F. Chang and W. H. Tsai, “Automatic 2D virtual face generation by 3D model transformation techniques and applications,” M. S. Thesis, Institute of Multimedia Engineering, National Chiao Tung University, Hsinchu, Taiwan, Republic of China, June 2007.

[6] C. Bregler, M. Covell, and M. Slaney, “Video rewrite driving visual speech with audio,” ACM Computer Graphics Proc. SIGGRAPH 97, Los Angeles, CA, pp.

353-360, Aug. 1997.

78

[7] E. Cosatto, “Sample-based talking-head synthesis,” Computer Animation 98 Proceeding, Philadelphia, PA, USA, pp. 103-110, June 1998.

[8] I-Chen Lin, et al., “A speech driven talking head system based on a single face image,” Proceedings of the 7th Pacific Conference on Computer Graphics and Applications, Seoul, South Korea, pp. 43-49, Oct. 1999.

[9] J. P. Nedel, “Integration of speech & video applications for lip synch lip movement synthesis & time warping,” M. S. Thesis, Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, May 1999.

[10] T. Ezzat and T. Poggio, “Visual speech synthesis by morphing visemes,”

International Journal of Computer Vision, Vol. 38, issue 1, pp.45-47, June 2000.

[11] I. Buck, et al., “Performance-driven hand-drawn animation,” ACM SIGGRAPH 2006 Courses, No. 25, Boston, Massachusetts, July 30 - Aug. 03, 2006.

[12] Q. Zhang, et al., “Geometry-driven photorealistic facial expression,”

Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, San Diego, California, July 2003.

[13] R. C. Gonzalez and R. E. Woods, “Digital image processing,” 2nd ed., New Jersey: Prentice-Hall, 2002.

[14] J. Gomes, et al., “Warping and morphing of graphical objects,” San Francisco, CA: Morgan Kaufmann, 1998.

[15] T. Beier and S. Neely, “Feature-based image metamorphosis,” ACM SIGGRAPH Computer Graphics, Vol. 26, issue 2, pp.35-42, July 1992.

[16] J. Ostermann, “Animation of synthetic faces in MPEG-4,” Proceedings of the Computer Animation, pp.49, June 1998.

[17] T. Goto, et al., “MPEG-4 based animation with face feature tracking,”

Proceedings of the Erographics Workshop on Computer Animation and

79

Simulation'99, Milano, Italy, Springer, Wien New York, pp.89-98, Sep. 1999.