以範例為基礎之人臉老化模擬

國

立

交

通

大

學

多媒體工程研究所

碩 士 論 文

以範例為基礎之人臉老化模擬

Example-based Human Face Aging

研 究 生：廖彥凱

指導教授：林奕成 博士

以範例為基礎之人臉老化模擬

Example-based Human Face Aging

研 究 生：廖彥凱

指導教授：林奕成

Student：Yen-Kai Liao

Advisor：Dr. I-Chen Lin

國 立 交 通 大 學

多 媒 體 工 程 研 究 所

碩 士 論 文

A Thesis

Submitted to Institute of Multimedia Engineering College of Computer Science

National Chiao Tung University in partial Fulfillment of the Requirements

for the Degree of Master

in

Computer Science July 2010

Hsinchu, Taiwan, Republic of China

I

以範例為基礎之人臉老化模擬

研究生：廖彥凱 指導教授：林奕成 博士

國立交通大學

多媒體工程研究所

摘要

II

Example-based Human Face Aging

Student: Yen-Kai Liao

Advisor: Dr. I-Chen Lin

Institute of Multimedia Engineering

National Chiao Tung University

Abstract

In this thesis, we propose a two-stage method to simulate the aging process of a human face. We use EM-like algorithm and PCA-based data filling method to estimate the geometric and facial texture variation of a target face from subspace of aging patterns. Since the facial texture predicted by PCA-based method cannot preserve personalized details, we propose using a superimposing method. We establish a facial texture database with high resolution pictures, and synthesis the detailed wrinkles or creases by texture synthesis. Besides, we further analyze the parents' inheritable impact in the aging simulation. Our experiments show that the proposed example-based method is able to provide reasonable simulation of face aging.

III

Acknowledge

本論文之完成，首先要感謝林奕成老師兩年來的細心指導，教導我許多相關的專業知識。 接著，我要感謝所有電腦動畫與互動圖學實驗室的同學們，在我的研究遇到困難時給我 的適時協助，最後我要感謝我的父母，充分的支持我的研究所學業。

IV

Contents

摘要 ... I

Abstract ... II

Acknowledge ... III

Contents ... IV

List of Figure ... VI

List of Table ... VIII

Chapter 1 Introduction ... 1

1.1 Motivation ... 1

1.2 Framework ... 2

Chapter 2 Related Work ... 6

2.1 Previous Work ... 6

2.2 The Face Aging Database ... 9

Chapter 3 Overview ... 10

Chapter 4. Aging Effects Prediction by PCA ... 13

4.1 Aging Pattern Subspace ... 13

4.2 Aging Simulation by A Single Image ... 16

4.3 Adding Parents’ Effect ... 18

Chapter 5. Facial Texture Synthesis ... 20

5.1 Elder Skin Texture Synthesis ... 20

5.2 Details of Skin Transferring ... 24

5.3 Interactive Aging Simulation ... 27

Chapter 6 Experiment Result ... 28

V

6.2 Aging Simulation with Parents' Effect ... 34

Chapter 7 Conclusion and Future Work ... 37

7.1 Conclusion ... 37

7.2 Future Work ... 37

VI

List of Figure

Figure 1. 1 The flowchart of our system ... 5

Figure 2. 1 The aging effect of midface [ZKT01] ...6

Figure 2. 2 Some example sequence in FG-NET. Each row is the face picture sequence of same person from young to elder ... 9

Figure 3. 1 The aging pattern. The missing data in the aging pattern are marked "m" ... 12

Figure 3. 2 The aging pattern data. ... 12

Figure 4. 1 The full-filled aging pattern. The images with red dotted line frames are the missing data full-filled by AGES method. The ages are marked above the image. 15 Figure 4. 2 (a) The full-filled missing image at 17 years old. (b) The ground truth image at 17 years old. ... 15

Figure 4. 3 (a) The result generated by AGES algorithm is blurry. (b) Transferring the double-fold by ERI method from the input image. ... 17

Figure 5. 1 (a) The target image (b) The process of patch-based skin texture synthesis... 21

Figure 5. 2 (a) The skin texture (b) The target image with the transferring skin texture. ... 22

Figure 5. 3 Example of color discontinuous between two adjacent patches. ... 23

Figure 5. 4 (a) The 6 original patches of one eye. (b) The eye image adjusted by Poisson Image Editing... 23

Figure 5. 5 (a) The original patches of one whole face. (b) The face image adjusted by Poisson Image Editing. ... 24

Figure 5. 6 (a) The original skin image generated by patch-based texture synthesis (b) Smooth image generated by Gaussian Blur (c) The Expression Ratio Image ... 26

Figure 5. 7 (a) The original target image (b) The target image multiply ERI. ... 26

Figure 5. 8 (a) ~ (c) The original target face and patch-based texture synthesis (d) The target face with manual wrinkles. (e) and (f) The result with interactive wrinkles editing. ... 27

Figure 6. 1 Aging simulating from 14 to 41. Picture Information : FG-NET No.72 ... 29

Figure 6. 2 Aging simulating from 14 to 41. Picture Information : FG-NET No.01 ... 29

Figure 6. 3 Aging simulating from 14 to 40. Picture Information : FG-NET No.11 ... 30

Figure 6. 4 Aging simulating from 18 to 46. Picture Information : FG-NET No.28... 30

Figure 6. 5 Aging simulating from 18 to 45. Picture Information : FG-NET No. 62... 30

Figure 6. 6 Aging simulating from 18 to 43. Picture Information : FG-NET No. 45 ... 31

VII

Figure 6. 8 Figure 6.8 Aging simulating from 10 to 20. Picture Information : Rupert Grint .. 31

Figure 6. 9 Aging simulating from 22 to 55. Picture Information : Paul McCartney ... 32

Figure 6. 10 Aging simulating from 12 to 23. Picture Information : Kirsten Dunst ... 32

Figure 6. 11 Aging simulating from 22 to 55. Picture Information : Harrison Ford ... 32

Figure 6. 12 Aging simulating from 5 to 25. Picture Information : FG-NET No. 44 ... 33

Figure 6. 13 Aging simulating from 12 to 41. Picture Information : FG-NET No. 8 ... 33

Figure 6. 14 Aging simulating from 24 to 55. Picture Information : Mona Lisa (side view adjusted by view morphing) ... 33

Figure 6. 15 Aging simulating from 34 to 60. Picture Information : Michael Douglas (Father : Kirk Douglas Mother : Diana Dill) ... 35

Figure 6. 16 Aging simulating from 22 to 60. Picture Information : George W. Bush (Father : George H. W. Bush Mother : Barbara Bush) ... 35

Figure 6. 17 Aging simulating from 7 to 20. Picture Information : Prince Henry of Wales (Father : Prince Charles Mother : Diana, Princess of Wales) ... 36

Figure 6. 18 Aging simulating from 27 to 55. Picture Information : Chelsea Clinton (Father : Bill Clinton Mother : Hillary Rodham Clinton) ... 36

VIII

List of Table

1

Chapter 1 Introduction

1.1 Motivation

Simulation of human face aging is a special task in facial editing. It aims at generating an elder face image from a young one. And it has many important applications, such as pursuing criminals, seeking missing children or face recognition system.

If a child has been missing or a criminal has fled for a long time. Their faces might change because of the aging process. The differences between the real person and the old pictures can increase the difficulty of face identification by the law enforcement agencies.

The same problem appears in the automatic face recognition system. Facial pictures in the database of recognition system were established earlier. But those pictures are difficult to be updated as time pass by. The accuracy of the face recognition system will decrease as the members in the database get older.

The goal of the human face aging simulation is to solve the above problems we listed. The technique of simulating the aging effect of face from the young pictures can help the law enforcement agencies to get the more reasonable pictures to pursue criminals or seek missing children. In other hand, the recognition rate of face recognition system will also increase.

The aging simulation can also be used in the aspect of entertainment. There are more and more video games that included the whole life of avatars (e.g. Fable or The Sims). If the video game can let players to set their own photos and the photos would get aging with time, it can

2

make realistic effects.

Compared to the other image editing domains, there are some special characteristics in face aging simulation. For instance, every person's aging process is individual with others. One's aging process cannot be used to simulate other people directly, which means that there may not exist a general function for forecasting of human’s facial aging. Therefore, we propose the example-based algorithm to achieve a reasonable simulation by using an aging database and facial skin database.

1.2 Framework

As aforementioned, everyone has his/her own unique aging process. It is hard to use physical-based method to find a general aging variation for simulating all people. Therefore, we adopt a widely-use statistical method named “Principal Component Analysis” (PCA) [JOLL02] to find the appropriate aging effects in the aging database [FGN02].

The proposed method is inspired by Aging Pattern Subspace (AGES) algorithm [GZS07], presented by Geng et al. They use PCA and EM-like [DLD77] algorithm to extract the aging variation of aging database [FGN02] and full-fill the incomplete aging database for improving the accuracy rate of age estimation system originally.

In our system, we put the young face picture in the aging pattern. The aging database can be trained in the full-filling process in AGES algorithm. By doing this, the elder variation of face can be solved by the least squares solution with the training database. However, only one or two young pictures in the aging pattern is too less to generate the stable result. To solve this problem, we adjust the initial guess of the aging pattern by the available young data. In

3

addition, we analyze the similarities between the target person and his or her parents, and enhance the PCA process by adding reasonable ratio of parents' aging process.

Although the PCA-based algorithm can predict the variation of facial geometries and skin textures, there are two main problems about the facial skin texture simulation. First, the skin textures are very blurry since the dimensional constrain of PCA-based method. The images in the database are composed of only about 5,000 pixels and it is grayscale image. Second, in the process of PCA method, the data of texture will be projected to the subspace which is composed by some important bases. And the facial detail (e.g. wrinkles and other creases) may disappear since the detail of skin is in high frequency domain.

However, the wrinkles and other creases are important visual elements in elder faces. We proposed a patch-based facial texture synthesis method to enhance the detail of facial skin. This method is modified by Visio-lization algorithm [MPK09] proposed by Mohammed et al. It is used for generating the novel facial images by synthesizing the facial patches in the database. We established a skin database with a lot of high resolution images at different ages. And all facial images in database are divided to several patches. The low frequency elder skin generated by PCA will be compared with the skin images of database patch by patch. Then we keep all of the appropriate skin patches and combine them by solving the Poisson equalization [PGB03]. Finally, we extract the skin detail [LSZ01] to transfer to estimated skin image. After the two-stage method, we can generate an elder face image with the aging geometry and appropriate elder facial skin. To enhance the visual effect, we also compare the grayscale values between the target image and the images in the database and transfer the color of most appropriate image to target image finally.

4

This paper is organized as follows. In the chapter 2, we introduce several aging simulation methods presently. the overview in our aging algorithm is described in chapter 3. About the facial aging prediction, we introduce the AGES algorithm and how parents' effects the prediction in the chapter 4. The facial texture synthesis and transferring is shown in the chapter 5. Also, we demonstrate the experiment results in the chapter 6. The chapter 7 is our conclusion and the future work. Figure 1.1 is the flowchart of our system.

5

Figure 1. 1 The flowchart of our system

Input Image

Normalize the

geometries and

skin texture

Put the image in

the aging pattern

Full-fill training data

with PCA and EM-Like

ɛ > threshold

or time < max

Elder face

Geometries

prediction

Elder face

Texture

prediction

Patch-based skin

texture synthesis

Skin Texture

Database

ERI method to extract

and blend the detail

wrinkles

Morphing to aging

prediction geometry

Output Image

Solve the aging variations

by training data

6

Chapter 2 Related Work

2.1 Previous Work

Biological aging is an inevitable process for all creatures on Earth. The aging process makes plenty of changes on facial features of human beings. For instance, due to the loss of facial tissue elasticity, the facial lines and wrinkles will increase in human senescence. Zimbler et al. made a description of the anatomy and pathophysiology of facial aging [ZKT01] from the medical perspective. Figure 2.1 shows the aging effect of midface described by Zimbler. However, it is hard to simulate the aging effect according to theory of medical science since factors of aging process is complicated. There are several researches in simulation of facial aging by other mathematical models in domain of computer vision.

7

There are two main factors of face should be handled in the simulation of face aging : geometry of face and facial skin texture. Most of researches in face aging focus on these two parts. Burt and Perrett [BP95] composited the male facial colors and shape in the same age bracket and extract the difference between the mean of elder of faces and the mean of young faces. Ramanathan and Chellappa [RC06] described the growth related shape variations in human faces by craniofacial growth model. This work was designed for the young ages. Hubball et al. proposed a data-driven approach of aging simulation [HCG08]. They used evolutionary computing and non-uniform radial basis function (NURBF) to simulate the age progress or regression. Suo et al. [SZSC10] proposed a compositional and dynamic model for face aging. They represented faces in the same age group by three-level "And-Or graph". The And-nodes mean decomposition of faces. The faces are divided to several components (e.g. hairs, eyes and mouse) from coarse to fine levels. The Or-nodes mean the alternatives to the each component. Then they learned process of face aging by Morkov chain.

Cootes et al. propose Active Appearance Model (AAM) [CET02] approach. The AAM model is a combined model of facial shape and texture to represent face image. There are several studies used the AAM model and the PCA-based method to simulate aging effect. Lanitis et al. [LTC02] projected the landmarks and grayscales image pixels of face on to 50 parameters in the subspace. Then they calculated the aging function parameters by genetic algorithm to transfer aging effects. Geng et al. [GZS07] proposed a method named Aging Pattern Subspace (AGES). They also projected the shape and image data to the subspace with 200 parameters and sorted these data in time order named Aging Pattern. Then they used PCA to find the main vector of aging pattern in whole database and solved the subspace parameters by least squares solution to full-fill the missing data in aging pattern.

8

caricaturing algorithm to simulate 3D face models’ aging effect. Scherbaum[SSSB07] et al. used 3D Morphable Face Model [BV99] to reconstruct a 3D face model from the input image, and computed the individual aging trajectories for the 3D model. Then they rendered the aging 3D face model back to the 2D image.

Ramanathan et al. [RCB09] did a thorough analysis of various aging simulation methods in 2009.

9

2.2 The Face Aging Database

For our experiments, we use FG-NET (Face and Gesture Recognition Research Network) Aging Database [FGN02] as our training data. FG-NET is an image database contains 1002 photos of 82 Caucasian people’s (35 females and 47 males) pictures at different ages (0 to 69). And there are 68 positions of feature points with each picture in the database. Figure 2.2 is some examples in FG-NET database.

Figure 2. 2 Some example sequence in FG-NET. Each row is the face picture sequence of same person from young to elder

10

Chapter 3 Overview

There are two main stages of our aging simulation algorithm :

1. Geometric and facial texture prediction. We put the input image to the aging pattern, and use PCA-based method to full-fill the training database. Then we can solve the simulating aging data by the least squares solution in subspace. To simulate more reasonable result, we adjust the initial guess of the aging pattern and regard the parents' effects.

2. Detail facial texture synthesis and transfer. Since the skin textures predicted by PCA-based method can lose details of wrinkles, we establish another facial skin database with high resolution face images at different ages. Then we combine the most appropriate skin patches and transfer the details of skin to the target image.

We use FG-NET aging database to be our training data. Here, we briefly introduce the FG-NET database [FGN02]. Each data in the FG-NET can be divided into two parts : the geometry and the face texture. To let the data be comparable. We should normalize two parts of data. Since the sizes of images in the database are inconsistent, we scale all images to the uniform shape with about 5,000 pixels and also transform to grayscale domain. The 68 feature points which are tilted should be rotated to upright positions. Then, we reduce the 5,000 pixels and 68 feature points to 200 parameters in the subspace. It can still preserve about 95% variability of the original data.

After normalizing the face data in the FG-NET, we construct the aging patterns for all people in the database. The aging pattern is a sequence of individual face data sorted chronologically from young age to elder age. However, the age range of FG-NET database is

11

very large (0 to 69) and the database is incomplete. There are only about 10 pictures of the same person. If we divide the age pattern to whole 70 groups, the aging pattern will be too specific and with a lot of missing data. The PCA-based method is also hard to work with highly incomplete aging pattern. To solve this problem we divide 70 ages into 17 age groups according to that aging effects are not obvious in few years. The contrastive table between original age and new age group is shown in Table 3.1.

Table 3. 1 The contrastive table between age and age group.

Original Age New Age group Original Age New Age group

0 to 1 0 22 to 24 9 2 to 3 1 25 to 27 10 4 to 5 2 28 to 30 11 6 to 7 3 31 to 34 12 8 to 9 4 35 to 39 13 10 to 12 5 40 to 49 14 13 to 15 6 50 to 59 15 16 to 18 7 60 to 69 16 19 to 21 8

Classifying the age pattern to 17 groups can reduce the missing data amount in the aging pattern effectively. An example of aging pattern is shown in Figure 3.1. Figure 3.1 only shows the images in the aging pattern. Actually, there are also geometric data with the images in the aging pattern. It is shown in Figure 3.2.

12

Figure 3. 1 The aging pattern. The missing data in the aging pattern are marked "m".

Figure 3. 2 The aging pattern data.

In this chapter, we will describe the process of normalizing the images and geometries data in FG-NET and introduce the definition of an aging pattern in our algorithm. In chapter 4.1, we explain how to use Aging Pattern Subspace to full-fill missing data in aging pattern. Then the aging simulations of our method are described in chapter 4.2. About the detail skin texture synthesis and detail wrinkle transfer are proposed in chapter 5.

13

Chapter 4. Aging Effects Prediction by PCA

4.1 Aging Pattern Subspace

In this chapter, we will describe how to use AGES [GZS07] algorithm to full-filled the aging patterns in the aging database. As the mention in chapter 3, the aging pattern is the face data sequence sorted by age. There are 200 parameters in each face data. Thought we reduce the missing data amount by classify the original 70 different ages into 17 age groups, the each aging pattern in the database is still incomplete. The face data can be classified into two types: the available data and the missing data. The purpose of AGES algorithm is to calculate the appropriate values of missing data by available data in the aging pattern.

The AGES algorithm utilizes PCA [JOLL02] to project the original data to the subspace that represents main variation of the aging pattern. The projection from original data to subspace is calculated by

.

(4-1)

Where is the original data of the aging pattern and is the mean vector of . is the eigenvectors of covariance matrix of , and it is composed of orthogonal the eigenvectors. If there are N aging patterns in the database, the database can be represented as . Each aging pattern can be classified into two parts: the available data and the missing data . The parameters in the subspace projected by can be calculated by (4-1) and the reconstruction of is calculated by (4-2).

14

(4-2)

However, there are no values of parameters in the position of missing data at first, we put the to be an initial guess of missing data. The definition of is the mean vector of available data from other aging patterns in the database. After full-filling the whole missing data by the initial guess, the EM-like (Expectation-Maximization like) [DLD77] algorithm can be used to learn the appropriate missing data by available data.

The EM-like algorithm is to utilize the Estimation step and the Maximization step iteratively to compute the maximizing expected likelihoods of missing data. In the Estimation step, cannot be calculated by projected from directly. It should be solved from the available data in the aging pattern as the least squares solution of

.

(4-3)

Where and are the parts in and , and the is the iterative times. The new parameters can be calculated by (4-2). The missing data is updated by , but the available data need not to update since it is real data. In the Maximization step, the new transformation matrix and the new mean vector is calculated by the updated data with standard PCA. The AGES algorithm uses the mean reconstruction error of the available data and the reconstructive available data to evaluate the iterative times in EM-like. The formula is

15

The EM-like algorithm repeats iteratively until mean reconstruction error is smaller than the threshold or iterative time is over the maximum time. The Figure 4.1 shows the results that missing data in the aging pattern are full-filled by AGES algorithm. The Figure 4.2 is the image generated by Leave-one-out method and the ground truth image.

Figure 4. 1 The full-filled aging pattern. The images with red dotted line frames are the missing data full-filled by AGES method. The ages are marked above the image.

(a) (b)

Figure 4. 2 (a) The full-filled missing image at 17 years old. (b) The ground truth image at 17 years old.

16

4.2 Aging Simulation by A Single Image

The missing data of FG-NET database was full-filled in chapter 4.1. In this chapter, we will describe how to use full-filled FG-NET as the training database to predict the aging effects by the single input image. Since we suppose that the age of the input image is already known, we do not need to estimate the age position of the input image in the aging pattern. Suppose that the age of the input image is in age group, the subspace of aging pattern is solved by

.

(4-5)

Then we can use to reconstruct the whole aging pattern. However, our training data of is only FG-NET database and the data are further separated for the males and females respectively. Thus, the training data is inadequate to generate the reasonable elder face. For making the stable results, we put the new pattern created by input image to the training database. The new aging pattern is defined by

(4-6)

Each of the missing data in the new aging pattern is added the difference of the input image and the mean vector of input age group. By adding new aging pattern to the training step, it can let the training data contain more component of the input face.

In order to let the simulated elder face retains the characteristics of the original face, the color and the skin detail should be transferred to the output image. Therefore, we calculate the parameters in YUV domains of the original image from RGB domains and transfer the UV

17

domain to the target image. The transformation between RGB domains YUV domains is

(4-7)

Transferring the UV domains from original image to target image can let the simulated image preserve the original skin color and skin details. Moreover, the results generated by AGES algorithm are all grayscale images. Adding the color to the result can enhance the visual cues. As mentioned above, the resolution of images generated by AGES algorithm is about 5,000 pixels and EM-like algorithm is calculated in the subspace, the result images will lose a lot of personal facial details (e.g. double-fold eyelid or nasolabial folds). We use ERI method to transfer the personal facial details of input image to the target image. The process of ERI method will be describes in chapter 4. The result is shown in Figure 4.3

(a) (b)

Figure 4. 3 (a) The result generated by AGES algorithm is blurry. (b) Transferring the double-fold by ERI method from the input image.

18

4.3 Adding Parents’ Effect

According to the genetic theory, children’s facial features are inherited from their parents. Therefore, we add the parents’ information to the training step. If we obtain the single picture of target’s father and mother, we can generate two new aging patterns. The definitions of these two patterns as follow.

(4-8)

After establishing the two parents’ patterns, we should find the impact ration of father and mother's data. Therefore we solve the linear combination problem in Equation (9). Where and are the impact ratios of father's data and mother's data.

(4-9)

The new aging patterns of father and mother are added to the training database with the ratios calculated by Equation (9). The new training database can be represented as

19

The aging simulation image can be solved by Equation (4-5) with the new training database adjusted by parents' effect. The experimental results are shown in chapter 6.

20

Chapter 5. Facial Texture Synthesis

In this chapter, we describe the algorithm of Elder Skin Texture Synthesis in chapter 4.1. The skin texture transferring method is introduced in chapter 4.2. The interactive wrinkles editing is proposed in chapter 4.3.

5.1 Elder Skin Texture Synthesis

As mentioned in chapter 3, the resolution of images in AGES algorithm is reduced to about 5,000 pixels. Furthermore, the PCA-based method projects the face data to the subspace retaining major variation. Reasons mentioned above make the simulative results lose the facial details. However, the facial creases and the wrinkles are the significant factors in elder faces. These factors are the main difference between the young faces and elder faces. Thus, we propose a facial texture synthesis algorithm to extract the creases and wrinkles from external images in the facial skin database. We establish this new database with high resolution facial texture at different ages. These images were collected from the Internet. The reason that we do not adopt the images in FG-NET is that a lot of images in FG-NET are too blurry to extract the facial details.

After the new facial skin database was established, the next question is how we select the appropriate facial skins for the target image. If we extract the creases from complete one image, a problem may occur. The whole image may not be appropriate in every parts of target face. For example, the cheek of skin image is fit for the target image, but the shape of nose may be quite different. In order to solve this problem, we propose the patch-based texture synthesis of facial skin.

21

The patch-based texture synthesis of facial skin is inspired by Visio-lization method [MPK09] which is proposed by Mohammed et al. The original purpose of the Visio-lization method is to generate novel facial images. In our application, we use this method to find the most appropriate skin textures for target image. As implied by the name, the patch-based texture synthesis is to divide the all of images in the database to several patches. The target image is also divided into several patches and each patch will be compared with all of the patches at the same position in the skin database. In process of comparing two patches, the costs of error are calculated by the pixels values difference and the gradients difference. The formula of costs of error is

(5-1)

Where is the gradients difference and is the pixel value difference.

The patch with the minimum error value would be selected in comparing process. To keep the symmetry of human faces, a constraint should be set that the two symmetric patches in the right and left sides of a face are selected from the same image.

The process of patch-based synthesis is shown in Figure 5.1.

(a) (b)

22

The elder skin texture image compose of the most appropriate patches is generated by patch-based skin texture synthesis. However, a new problem emerges. There are many obvious edges between two adjacent patches which are selected from different skin images since color of skin and lighting them is different. If we transfer this skin texture to the target image by ERI method directly, the edges will be transferred to the target image. The result is shown in Figure 5.2.

(a) (b)

Figure 5. 2 (a) The skin texture (b) The target image with the transferring skin texture.

To solve this problem, we use Poisson Image Editing [PGB03] to remove the artifact edges between two adjacent patches. Figure 5.3 illustrates an example of color discontinuous between two adjacent patches.

23

Figure 5. 3 Example of color discontinuous between two adjacent patches.

We suppose the source patch is patch A, and pixels of patch B should be adjusted to remove the color discontinuous. We should solve the linear equations as follow:

For pixel p Ω ,

(5-2)

Where is the of adjacent pixels’ number in pixel p, and q is the pixels which are adjacent to p. Ω is the region of patch B. ∂Ω is the region of patch A which is adjacent to patch B. is the gradient value between p and q. Since we focus on the wrinkles and the creases, all patches in the database are transfer to grayscale domain. The results of image are adjusted by Poisson Image Editing is shown in Figure 5.4 and Figure 5.5.

(a) (b)

Figure 5. 4 (a) The 6 original patches of one eye. (b) The eye image adjusted by Poisson Image Editing.

24

(a) (b)

Figure 5. 5 (a) The original patches of one whole face. (b) The face image adjusted by Poisson Image Editing.

5.2 Details of Skin Transferring

In chapter 5.1, the most appropriate skin patches of the target face are generated. The artifacts between adjacent patches are removed by Poisson Image Editing. In chapter 5.2, we will introduce how to transfer the creases and wrinkles of the skin image to the target image. We make use of the Expression Ratio Image (ERI) [LSZ01] of the skin image to transfer to the target image. The ERI method is proposed by Liu et al. The original purpose of ERI is to achieve expression mapping between different people. In our algorithm, we use the ERI to transfer the wrinkles and the creases from the skin image to the target image.

Suppose that two face images A and B are the face of the skin texture and the face of the target person. and mean the smooth skin texture and the skin texture with creases. and mean the target image without detail and target image with the details of . We assume that the two face images that morphed to the same shape. Their normals and lighting directions at the same position are roughly same. It can be represented as

25

, , .

By the definition of the Lambertian model, the intensity at (u, v) in is

(5-3)

Therefore we can assume that the ratio of and are equal to the ratio of and . (u, v) mean the coordinates of a pixel in images. The result can

be represent as

.

(5-4)

Since our goal is to transfer the creases to the target image, we adjust the formula to

.

(5-5)

is the skin image generated in chapter 5.1 and is the original target without

details. However, we do not have the smooth skin image . To solve this problem, we apply the Gaussian Blur skill to the . Then we can generate the smooth skin image .

26

The formula of Gaussian Blur is

_.

(5-6)

Where x and y are the distances from the origin in the horizontal axis and vertical axis. is the standard deviation of the Gaussian distribution. The process of texture transferring is shown in Figure 5.6.

Non-Smooth Image

(a)

Smooth Image

(b)

Expression Ratio Image

(c)

Figure 5. 6 (a) The original skin image generated by patch-based texture synthesis (b) Smooth image generated by Gaussian Blur (c) The Expression Ratio Image ( )

(a) (b)

27

5.3 Interactive Aging Simulation

In chapter 5.1 and 5.2, we use the images that predicted by AGES algorithm to be the base image of skin texture synthesis. However, some of them are too blurry to searching the patches with wrinkles. The few smooth patches in the database may be selected at the positions which should have general wrinkles (e.g. nasolabial folds or pouches). The advantage of patch-based synthesis is that it can support the interactive face editing. Since this patch-based method searches the database patch by patch, the little user-assigned wrinkles of source image can find the appropriate skin patches with real wrinkles easily. The result is shown in Figure 5.8.

(a) (b) (c)

(d) (e) (f)

Figure 5. 8 (a) ~ (c) The original target face and patch-based texture synthesis (d) The target face with manual wrinkles. (e) and (f) The result with interactive wrinkles editing.

28

Chapter 6 Experiment Result

In this chapter, we demonstrate our experiment results of example-based face aging simulation. In section 6.1, we show the aging simulation from a single input image in this section. The targets for comparison are AGES method and Evolutionary Computing method. The results of parents’ data enhancing are shown in section 6.2.

6.1 Aging Simulation from a Single Image

The experiment data are as same as the training data FG-NET aging database. We preserve the picture and feature points’ data at input age and remove other available data of one experimental aging pattern. The simulated results can be compared with the ground turth which are not included in training. Besides, we also use several photos of celebrities (e.g. stars or politicians) to be our experimental data since most of people are more sensitive and familiar with famous people than ordinary people. It can increase the credibility of our experiment results. Figure 6.1 to Figure 6.14 are our experiment results. Comparing the pure AGES method, we can find that our results contain more facial details. The results of old age have more clear wrinkles and creases.

29

Input Image AGES Method Our Result Ground Truth

Age : 14

Age : 41

Age : 41 Age : 41

Hubball 08

Age : 41

Figure 6. 1 Aging simulating from 14 to 41. Picture Information : FG-NET No.72

Input Image AGES Method Our Result Ground Truth

Age : 8

Age : 28

Age : 28 Age : 28

Hubball 08

Age : 28

30

Input Image AGES Our Result Ground Truth

Age : 14 Age 40 Age 40 Age 40

Figure 6. 3 Aging simulating from 14 to 40. Picture Information : FG-NET No.11

Input Image AGES Our Result Ground Truth

Age : 18 Age : 46 Age : 46 Age : 46

Figure 6. 4 Aging simulating from 18 to 46. Picture Information : FG-NET No.28

Input Image AGES Our Result Ground Truth

Age : 18 Age : 45 Age : 45 Age : 45

31

Input Image AGES Our Result Ground Truth

Age : 18 Age : 43 Age : 43 Age : 43

Figure 6. 6 Aging simulating from 18 to 43. Picture Information : FG-NET No. 45

Input Image AGES Our Result Ground Truth

Age : 18 Age : 49 Age : 49 Age : 49

Figure 6. 7 Aging simulating from 18 to 49. Picture Information : FG-NET No. 05

Input Image AGES Our Result Ground Truth

Age : 10 Age : 20 Age : 20 Age : 20

32

Input Image AGES Our Result Ground Truth

Age : 22 Age : 55 Age : 55 Age : 55

Figure 6. 9 Aging simulating from 22 to 55. Picture Information : Paul McCartney

Input Image AGES Our Result Ground Truth

Age : 12 Age : 23 Age : 23 Age : 23

Figure 6. 10 Aging simulating from 12 to 23. Picture Information : Kirsten Dunst

Input Image AGES Our Result Ground Truth

Age : 22 Age : 55 Age : 55 Age : 55

33

Input Image AGES Our Result Ground Truth

Age : 5 Age : 25 Age : 25 Age : 25

Figure 6. 12 Aging simulating from 5 to 25. Picture Information : FG-NET No. 44

Input Image AGES Our Result Ground Truth

Age : 12 Age : 41 Age : 41 Age : 41

Figure 6. 13 Aging simulating from 12 to 41. Picture Information : FG-NET No. 8

Input Image AGES Our Result

Age : 24 Age : 55 Age : 55

Figure 6. 14 Aging simulating from 24 to 55. Picture Information : Mona Lisa (side view adjusted by view morphing)

34

6.2 Aging Simulation with Parents' Effect

In this section, we demonstrate the results of aging simulation with parents' image data. For the reason to increase the credibility. We adopt several famous families of stars and politicians. Figure 6.15 to Figure 6.18 are our experiment results with parents' effect. We use equation 4-9 to calculate the ratios of parents' data. The results are compared with the images that ratio of father's and mother's data are 100% separately. Though that the differences of the images with different ratios of parents' effect are not very obvious. We can still find that the facial shapes of results are more close with the high ratio one. Finding the linear combination rations of parents can make the results stable and reasonable.

35 Input Image Age : 22 Father Age : 50 Mother Age : 20 Ground Truth Age : 60 No Parents’ Effect Age : 60 R f : R m = 100% : 0% Age : 60 R f : R m = 0% : 100% Age : 60 R f : R m = 85% : 15% Age : 60 Figure 6. 15 Aging simulating from 34 to 60. Picture Information : Michael Douglas

(Father : Kirk Douglas Mother : Diana Dill) Input Image Age : 22 Father Age : 65 Mother Age : 74 Ground Truth Age : 60 No Parents’ Effect Age : 60 R f : R m = 100% : 0% Age : 60 R f : R m = 0% : 100% Age : 60 R f : R m = 90% : 10% Age : 60 Figure 6. 16 Aging simulating from 22 to 60. Picture Information : George W. Bush

36 Input Image Age : 7 Father Age : 40 Mother Age : 34 Ground Truth Age : 60 No Parents’ Effect Age : 20 R f : R m = 100% : 0% Age : 20 R f : R m = 0% : 100% Age : 20 R f : R m = 75% 25% Age : 20 Figure 6. 17 Aging simulating from 7 to 20. Picture Information : Prince Henry of Wales

(Father : Prince Charles Mother : Diana, Princess of Wales) Input Image Age : 27 Father Age : 47 Mother Age : 61 No Parents’ Effect Age : 20 R f : R m = 100% : 0% Age : 20 R f : R m = 0% : 100% Age : 20 R f : R m = 45% 55% Age : 20 Figure 6. 18 Aging simulating from 27 to 55. Picture Information : Chelsea Clinton

37

Chapter 7 Conclusion and Future Work

7.1 Conclusion

In this thesis, we propose an example-based face aging algorithm to simulate the aging effects on the single input image. We use AGES algorithm to predict the aging variation of the facial geometries or skin texture. Since the facial skins are regularized and scaled to small size at first and PCA-based method project the data in the subspace, the skin details of images generated by AGES method are unclear. To enhance the creases and wrinkles, we propose the patch-based skin texture synthesis to search the most appropriate elder skin patches in the high resolution skin database. Then we use Poisson Image Editing to reduce the artifacts between patches, and generate ERI to transfer the details of skin to the target images. Besides, we further analyze the parents' inheritable impact in the aging simulation. Our experiments show that the proposed example-based method is able to provide reasonable simulation of face aging.

7.2 Future Work

Several points can be improved in our future work. First, Most of people in the FG-NET are Caucasians. There is no Asiatic or African in the database. Therefore, using FG-NET to be training data may decrease the accuracy of predicting aging effects of the Asiatic or African. Second, our method can only process the input image with frontal face. It cannot process the face photo of side view. If we morph the face of side view to the uniform shape, the face will be not symmetric. We plan to use the method like View morphing or Morphable Model to acquire the 3D information and adjust the face from the side view to the front view to generate reasonable results.

38

References

[BP95] M. Burt and D. Perrett, “Perception of Age in Adult Caucasian Male Faces: Computer Graphic Manipulation of Shape and Color Information,” Proc. Royal Soc. London B:

Biological Sciences, vol. 259, no. 1355, pp. 137-143, 1995.

[BV99] V. Blanz, T. Vetter, “A morphable model for the synthesis of 3D faces,” In Computer Graphics Proc. SIGGRAPH’99 (1999), pp. 187–194.

[CET02] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” IEEE

Transactions on Pattern Analysis and Machine Intelligence, vol. 23(6), pp. 681–685, June

2001.

[DLD77] Arthur Dempster, Nan Laird, and Donald Rubin. “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the Royal Statistical Society, Series B, 39(1):1–38, 1977.

[FGN02] The FG-NET Aging Database, http://www.fgnet.rsunit.com/, 2002.

[GZS07] X. Geng, Z. H. Zhou, and K. Smith-Miles, “Automatic age estimation based on facial aging patterns,” IEEE Pattern Analysis and Machine Intelligence, vol. 29 (12), pp. 2234–2240, December 2007.

[HCG08] D. Hubball, M. Chen and P. W. Grant, “Image-based aging using evolutionary computing,” Computer Graphics Forum, 27 (2):607-616, 2008.

39

[JOLL02] I.T. Jolliffe, Principal Component Analysis, 2nd, Springer, 2002.

[LSZ01] Z. Liu, Y. Shan, Z. Zhang, “Expressive Expression Mapping with Ratio Images,”

Computer Graphics, Annual Conference Series, ACM SIGGRAPH, pp. 271-276, August 2001.

[LTC02] A. Lanitis, C.J. Taylor and T.F. Cootes, “Toward automatic simulation of aging effects on face images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume: 24 Issue: 4, Page(s): 442 - 455, April 2002.

[MPK09] U. Mohammed, S. J. D. Prince, J. Kautz, “Visio-lization: Generating Novel Facial Images,” ACM Transactions on Graphics (Proceedings SIGGRAPH 2009), 28(3), August 2009, pages 57:1-57:8

[PGB03] P. Pérez, M. Gangnet, A. Blake. “Poisson image editing,” ACM Transactions on

Graphics (Proceedings SIGGRAPH 2003), 22(3):313-318, 2003.

[PTK08] U. Park, Y. Tong, and A. K. Kain, “Face recognition with temporal invariance: A 3d aging model,” in Proceedings of the 8th IEEE International Conference on Automatic Face

and Gestures, 2008.

[RC06] Narayanan Ramanathan and Rama Chellappa, “Modeling Age Progression in Young Faces”, IEEE Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 387-394, June 2006, New York.

[RCB09] N. Ramanathan, R. Chellappa and S. Biswas, “Age Progression in Human Faces: A Survey,” Journal of Visual Languages and Computing, April 2009.

40

[SSSB07] K. Scherbaum, M. Sunkel, H.-P. Seidel, and V. Blanz, “Prediction of Individual Non-Linear Aging Trajectories of Faces,” Computer Graphics Forum, 26 (3): 285-294, 2007.

[SZSC10] J. Suo, S.C. Zhu, S. Shan,X. Chen, “A Compositional and Dynamic Model for Dace Aging,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010 Mar;32(3):385-401.

[ZKT01] Marc S. Zimbler, Mimi S. Kokoska, J. Regan Thomas, “Anatomy and pathophysiology of facial aging,” Facial Plastic Surgery Clinics of North America, 2001 May;9(2):179-87, vii.