CHAPTER 4 FEATURE POINT DRIVEN SYSTEM
4.2 RECONSTRUCTION OF PRIMITIVE MODEL
(a) (b) (c)
Figure 20:(a) the natural feature points (b-c) Using the block matrix method to track the feature points
4.2 Reconstruction of Primitive Model
The reconstruction of primitive 3D model can be produced by the camera calibration and stereo triangulation algorithm.
Given the projection matrices M and
M ′
, the corresponding feature 2D points p and p’, we can rewrite the equation p=MP and p’=M’P as:[ ] [ ]
' ' 0 we can solve P easily by a linear least-square method.4.3 Deformation of 3D primitive model
After we get the 3D position of the calibration, we can use the geometry to deform the 3D primitive head model. First, users have to select a set of corresponding pairs {pi, qi}, where pi is the feature point position of our synthesizing expression and qi is the corresponding point position on the generic model. Once the displacement of each feature point ui = qi – pi was calculated, we use scattered data interpolation S(p) to estimate the displacement of other vertices on the original mesh. We adopted the radial basis function as:
( )
p =∑
iciφ(
p−pi) (
+M p−pi)
+tS (17)
where φ is radial symmetric basis function, and ci are displacement coefficients, and M, t are affine terms. To determine ci, M and t, we solve a set of linear equations that includes interpolation constraints ui = S(p). We useφ( )r =
e
−r/32 .When deforming the 3D primitive model, we need to divide the 3D face model into sub-regions such as the forehead, the nose and the mouth…etc. After applying RBF functions locally to deform the sub-region, we can produce primitive 3D animation Fig. 21 shows the deforming result using the local RBF. The detailed facial animation will be introduced in section 5.3.
(a) (b) (c) Figure 21: (a)the netural face (b) Using local RBF functions (c) global RBF
Chapter 5
Experiment and Result
In this chapter, we will describe our experiment and show our result. At the beginning, we introduce the experiment of the input video sequence and analyze the optimized result. Then, we will show the synthetic results where the facial details are included.
5.1 The Experiment of Input Video Sequence
In our system, we use two synchronized video streams to create the difference normal and height maps. In order to acquire the more accurately facial details, our input images are taken under an illumination-controlled environment.
We set a projector as the single light source. Our input data are the two synchronized high-definition video (HDV, 1280*720 pixel resolution) and the frame per second (FPS) is set to the 30 frames per second. Fig 22 shows the two different views of video data.
We put a set of markers ( as shown in Fig 21, 18 markers) on the actor’s face but avoid placing markers on regions of wrinkles or creases.
5.2 The Result of Space-time SFS
In our research, we apply a novel space-time shape-from-shading to reconstruct the 3D shape. We utilize an optimization method to solve the ill-condition of shape-from-shading. This method can optimize the space and reflectance parameters to minimize the cost function. Fig 23 shows the chart of optimizing the motion of
Figure 23: the progress of optimization of the cost function
We also optimize the reflectance parameters for the synthesized images. Fig 24 shows the result of the optimized reflectance parameters. We set the initial shape as the flat shape and the reflectance parameters Kd=0.5, Ks=0.5, alpha=15. The optimized result is close to the accurate value.
(a) (b)
Synthesized data the recovery reflectance data Normalized
Kd Ks Alpha Kd Ks Alpha Kd Ks
teapot 0.7 0.3 15 0.536298 0.153753 15.001 0.7772 0.2228 ball 0.7 0.3 15 0.28751 0.09362 15.004 0.7543 0.2456
Figure 24: the result of the optimized reflectance parameters
Fig 25 shows the progress of optimization phases. In the first phase, we just optimize the diffuse term to get the more accurately initial shape. After the second phase, specular term will include to be optimized. Adding the spatial constraint will smooth the optimized result. The optimization method will stop until the variation of reflectance parameters is small tan the threshold. Another result of shape recovery is show on Fig 26.
(a)
(b)
Figure 25: (a) the wrinkle of the input image (b)the optimized phase
Figure 26: results of shape recovery( forehead and glabella)
5.3 The Synthesized facial details
The 3D head model used in this thesis has 6078 vertices and 6315 polygons. Every vertex has a predefined normal vector. We need to separate the region to apply the local RBF functions. These sub-regions include the forehead, nose and mouth…etc. In order to apply the height map on the face model, we subdivide the polygons and utilize the difference normal map to render the synthesized data. Figure 28 shows the subdivision result for the real-time rendering. Fig 29-34 shows the facial details using normal difference and height map are added on the face model.
(a) (b) (c) Figure 27: (a) original 3D mesh (b) subdivide area( green ) (c) subdivided result
Figure 28:the natural face model
(a) (b)
(c) (d)
Figure 29(a) Raising the forehead without facial details (b-c) adding the facial details (d) the original captured image
(a) (b) Figure 30(a) Applying the height map by subdivision(b) the side view
Figure 31: anger expression.
Figure 32:the facial details by opening the mouth
Figure 33: smile expression
(a)Raising the forehead
(b)anger expression
(c)Opening the mouth
(d)smiling expression Figure 34: the facial animation
Chapter 6
Conclusions and Future Work
6.1 Conclusions
In this thesis, we propose a space-time shape-from-shading (SFS) to reconstruct the facial details. In order to solve the ill-condition environment, we apply the optimization method with adding spatial and temporal constraints to get more reliable results. We utilize the feature point driven system for primitive 3D face model and the estimated facial details are then combined with the face model. To render the height maps, we subdivide the primitive model according to the height values and apply normal difference maps. With the proposed method, we will get the more detailed facial animation.
Our contribution include (1) a novel space-time shape-from-shading for recovering 3D data. (2) Using an optimization method to get the more reliable results for real data.
6.2 Future Work
In this thesis, we adopt Phong model as the reflectance model. Other reflectance models such that Torrance model or BSSRDF which has more physical cues may get more accurately results. And the other hand, we can apply other numerical method such that Fast Marching Method (FMM) to speed up the optimized procedure.
References
[1] Zhang L., Snavely N., Curless B. and Seitz S.M. “Spacetime Faces: High Resolution Capture for Modeling and Animation”, Proc. ACMSIGGRAPH'04, Pages 548-558, 2004
[2] Weyrich T., Matusik W., Pfister H., Lee J., Ngan A., Jensen W. and Gross M.,
“Analysis of Human Faces using a Measurement-Based Skin Reflectance Model”
Proc. ACMSIGGRAPH'06, Pages 1013-1024, 2006.
[3] Huynh D.Q. “Calibration of a Structured Light System: A Projective Approach”, In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition( CVPR’ 97), Pages 225-230, 1997
[4] Hertzmann A. and Seitz S.M.,”Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27 no. 8 , Pages 1254-1264 ,2005
[5] Vogiatzis G., Torr P.H.S and Cipolla R., “Multi-view Stereo via Volumetric Graph-cuts”, In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition( CVPR’ 05), vol. 2, Pages 391-398, 2005
[6] Fang, H., and Hart, J. C. “Textureshop: Texture Synthesis as a Photograph Editing Tool”, Proc. ACMSIGGRAPH'04, Volume 23, Issue 3 (August 2004), Pages 354-359, 2004.
[7] Horn, B.K. 1990. “Height and Gradient from Shading”, International Journal of Computer Vision, Vol. 5(1), Pages 37-75, 1990.
[8] Zeng G., Matsushita Y., Quan L., and Shum H.Y., ”Interactive Shape from Shading”, In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition( CVPR’ 05), vol.1, Pages 343-350, 2005
[9] Han F. and Zhu S.C. “Cloth Representation by Shape from Shading with Shading Primitives”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition ( CVPR’ 05), vol.1, Pages 1203-1210, 2005
[10] Yu T., Xu N. and Ahuja N. ”Recovering Shape and Reflectance Model of Non-lambertian Objects from Multiple Views”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition ( CVPR’ 04), vol.2, Wrinkles on Human Skin”, Pacific Graphics 02, Pages 166-175, 2002.
[13] Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. “Spacetime Faces:
High-Resolution Capture for Modeling and Animation”, ACM Trans. on Graphics, Vol. 23, Issue 3, Pages 548-558, 2004.
[14] Zhang, Q., Liu, Z., Guo, B., Terzopoulos, D., and Shum, H. “Geometry-Driven Photorealistic Facial Expression Synthesis”, IEEE Trans. On Visualization and Computer Graphics, Vol. 12(1), Pages 48-60, 2006.
[15] Zhang, R., Tsai, P.-S., Cryer, J., and Shah, M. “Shape from Shading: A Survey”, IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 21(8), Pages 690-706, 1999.
[16] Zhu L, Lee W.-S., "Facial Expression via Genetic Algorithms", Computer Animation and Social Agents , 2006
[17] Beier, T., and Neely, S. “Feature-based Image Metamorphosis”, Proc. ACM SIGGRAPH'92, Pages 35-42, 1992.
[18] Blanz, V., Basso, C., Poggio, T., and Vetter, T. “Reanimating Faces in Images and Video”, Computer Graphics Forum 22 (3), Pages 641 - 650, 2003.
[19] Blanz, V., and Vetter, T. “A Morphable Model for the Synthesis of 3D Faces”, Proc. ACM SIGGRAPH'99, Pages 187-194, 1999.
[20] Golovinskiy, A., Matusik, W., and Pfister, H. “A Statistical model for Synthesis of Detailed Facial Geometry”, ACM Trans. on Graphics, Volume 25, Issue 3, Pages: 1025-1034, 2006.