以範例引導式之網格變形為基礎之高解析度人臉模型年輕化

國立臺灣大學電機資訊學院資訊工程學系 碩士論文

Department of Computer Science and Information Engineering College of Electrical Engineering and Computer Science

National Taiwan University Master Thesis

以範例引導式之網格變形為基礎之高解析度人臉模型年輕化 De-aging High-Resolution 3D Facial Models by Example-Driven

Mesh Deformation

黃詩晏 Hwang Shih-Yen

指導教授：歐陽明 博士 Advisor: Ming Ouhyoung, Ph.D.

中華民國 101 年 6 月

June, 2012

致致致謝謝謝

謝謝歐陽老師一路以來對執導學生的熱情與努力，一再包容總是 出錯的我，並不厭其煩的告訴我們許多做事情應該有的態度。謝謝莊 永裕老師如英雄般的降臨。能夠在cmlab的G組和大家度過碩士生涯，

實在是件幸福到過去完全無法想像的事，雖然辛苦，但是非常值得。

謝謝法蘭科科、winble、老大及其他博班大大們總是不吝對程度其差 無比的我伸出援手。謝謝她口、小鐵、妞妞、毛、歐維斯和竹田，還 好還有你們，讓剛到台大被嚇壞了的我能繼續堅持努力下去，碩一的 生活是段辛苦但卻回味無窮的時光。謝謝在口試前停下腳步為我加油 的妙麗、技巧和強壯。謝謝阿威和哈姆，陪我吃吃喝喝聊天放鬆。謝

謝宇婷和大美，雖然這兩年和你們一起吃飯的次數用一隻手就數的完

了，但是我很珍惜也很開心能擁有和你們聊天的每一刻。謝謝宜庭和 志霖葛格，謝謝你們cover著我糟糕的課業、陪我說話、幫我想著該怎 麼度過危機，很開心能和你們一起當個歐陽寶寶，一起加油。 謝謝漾 漾，從碩一開始，不管是崩潰痛哭還是情緒高漲的時候，妳總是冷靜

但溫暖的在我身邊，讚揚優點之餘也不忘提醒我的缺點，讓我看到自

己可笑的一面和應該驕傲的時刻，有妳堅定而且安穩身影的台北，才

是那個我會想念的地方。 謝謝塗塗，因為有你的吉他debug時間，我才

有了繼續彈吉他的念頭；因為有你親切的帶著對一切陌生的我們認識

學校，一個人來到陌生的地方才不至於驚慌失措；如果要找一個為離 開實驗室難過的理由，沒有在那之前和你有更多交集應該會是第一個 浮出的念頭，謝謝不曾讓我感覺自己像個該被淘汰的失敗者，謝謝你

讓我看到更好的人，謝謝你在口試前陪我在漫漫長夜裡反覆練習，能

有你當我的最佳戰友，真的很幸運。

感謝能在這裡認識這麼多不一樣的人，讓我對生活有了許多新的看

法。 感謝爸爸媽媽對我不遺餘力的支持，當我累了能夠回家，就是人 生最幸福的時候了。

中

中中文文文摘摘摘要要要

臉部特徵是我們認識其他人的第一印象，包含許多個人訊息如年齡 及吸引程度，雖然目前科學研究仍無法明確定義出絕對的臉部美學，

但已有研究指出一個人臉對其他人的吸引程度並不受限於文化或是社

會審美觀(a cross-culture criteria)，而越年輕的人臉也隱含對其他人越具 吸引力。臉部老化有四個重要特徵:眼皮下垂、雙頰凹陷、臉部肌肉鬆 垮以及皺紋，除了膚色外臉部輪廓也會隨著年紀增長而變化，在高解

析度的立體模型(high-resolution 3D model)進一步分析年紀對人臉輪廓

的影響，應該能得到更好的年輕化效果。本篇論文提出一個以引導式

範例為基礎的網格變形方法以增加高解析度人臉模型的吸引力(enhance

facial attractiveness)，在較年長的模型上取出特徵點間的長度，如上述

與年齡相關，再利用學習得到的數學老化模型以求取較年輕的特徵向 量並盡量使之與原始資料相近，該組特徵長度即表示一較年輕模型該

有的臉部比例，以此為根據將較年長人臉模型變形至較年輕人臉，對

最後的結果，我們也提出各種不同的分析圖表。

關關關鍵鍵鍵字字字:年年年輕輕輕化化化、、、高高高解解解析析析度度度人人人臉臉臉立立立體體體模模模型型型、、、增增增進進進臉臉臉部部部吸吸吸引引引力力力、、、美美美 化化化人人人臉臉臉。。。

Abstract

Human face conveys significant characteristics of a person such as age traits and facial attractiveness. The absolute aesthetic value for human faces is still unclear; but meanwhile, facial attractiveness has been considered as cross-culture criteria. De-aging procedure on 3D model geometry can be used to enhance facial attractiveness due to improved shape analysis. We conclude our observation on aging progress into four rules: 1. drooping of eyelids, 2.

lack of elasticity causing hollow cheeks, 3. slack on the middle face and 4.

wrinkles. In this paper we present a framework to beautify (enhance facial attractiveness) realistic facial model by a mesh deformation method based on data-driven approach. For an elder input face, we extract the edges connected among feature points on the face mesh. The edge lengths are used as a feature vector, and an age score is associated with it. We search for a local minimum to obtain a related young vector and consequently maintain the similarity between the elder and young face. Once the young length vector has been determined, we can embed it to our new feature locations and then deform the model to a younger shape. The effectiveness of our work is to rejuvenate an elder face by altering the outline of a facial model.

Keyword: De-aging, Beautification, high-resolution 3D facial model, Enhance facial attractiveness.

Contents

致

致致謝謝謝 iii

中 中

中文文文摘摘摘要要要 v

Abstract vii

1 Introduction 1

1.1 Application . . . 2

1.2 Problems . . . 2

1.3 Motivation . . . 3

1.4 Architecture . . . 4

2 Related Work 7 2.1 Facial Modeling . . . 7

2.2 Facial Attractiveness . . . 8

2.3 De-aging . . . 9

3 System Overview 11 3.1 System Overview . . . 11

3.2 Implementation . . . 14

4 Database and Age/Beauty Model 15 4.1 Western Database . . . 15

4.2 Asian Database . . . 16

4.3 Feature Extraction . . . 18

4.4 Age/Beauty Model . . . 18

5 De-Aging Process and Beautification 21 5.1 Young/Pretty Vector . . . 21

5.2 Adjustment . . . 23

5.3 Geometric Transformation . . . 24

6 Experimental Result 27 6.1 Result . . . 27

6.2 User Study . . . 27

6.2.1 Beauty Model . . . 30

6.2.2 Age Model . . . 30

7 Conclusion and Future Work 33

Bibliography 35

List of Figures

1.1 The Science Behind The Curious Case of Benjamin Button. . . 1 1.2 Photographer: Ulric Collette. As these photo show, there are four major

aging traits on the face: 1. drooping of eyelids, 2. lack of elasticity causing hollow cheeks, 3. slack on the middle face and 4. wrinkles. . . . 4 2.1 Light-Stage equipment. . . 8 2.2 Using Kinect to create realistic facial model. . . 9 3.1 The training part of system. Two different database will be used to train-

ing two SVR model separated. . . 12 3.2 The adjustment part of system. . . 13 4.1 Models in different age and different parameters generated by the BFM

3D Morphable Model database. . . 15 4.2 Asian database. . . 16 4.3 Some examples of facial geometry in the Asian database without texture. 17 4.4 Some examples of facial texture in the Asian database. . . 17 4.5 All 72 feature points are pick up from the subset of Farkas feature points

and label manually. . . 19 4.6 The lengths between facial features are used to represent aging/beauty

level of the face. . . 19 5.1 The new location of feature points that is calculated by our de-aging process. 24 6.1 The front view of results. . . 28 6.2 The side view of results. . . 29 6.3 When the percentage of the original face increase, the age score decrease

which means the face will look elder. . . 30 6.4 The summary of user study of all three facial model de-aging/beautification. 31 6.5 The beauty score which is ranked by users. The lower score means more

attractive. . . 32

6.6 The average age score which is ranked by users. The lower score means younger. . . 32 6.7 Another case for our de-aging process. . . 32

List of Tables

4.1 The mean squared error of the SVR model training by Asian database.

Half of our data in the database will be used to training the SVR model, and others will be used to testing the SVR model. . . 20 4.2 The mean squared error of the SVR model training by Western database. . 20 5.1 Both age and beauty process will be applied on this facial model. A higher

score is better (younger or prettier). . . 23 5.2 Only de-aging process will be applied on this facial model. A higher score

is better (younger). . . 23 5.3 The preliminary result will blend with the original model in order to ob-

tain a gradual de-aging/beautification process. The texture of facial mod- els has been smooth in advance. . . 26

Chapter 1

Introduction

As more and more personal information can be represented in digital forms, high-resolution 3D facial models become the state-of-the-art of displaying our faces. With these realistic models, we can use them to improve security control, assist forensic artist in predicting subject faces or even playing a fancy role in Hollywood movie to experience things that we never have chance to approach. Although creating our own facial model becomes much easier than before, we may still not be able to collect every model in different age of life. But in some network social circumstances, we want to look younger, greater and being more attractive. That is how anti-aging process has received many attentions for over two decades.

Figure 1.1: The Science Behind The Curious Case of Benjamin Button.

1.1 Application

Being the first impression of a person, human face conveys much important information such as gender, age, ethnic origin and personal identity. How to render and interpret hu- man faces with specific attributes, such as gender, age and expression, become the most active fields in both industry and academia. Age synthesis and face beautification espe- cially be noticed in entertainment industry and cosmetology. For example, there is a fa- mous Hollywood film “ The Curious Case of Benjamin Button ” (2008) shows extremely realistic aging and rejuvenating effects on the appearance of leading actor. As we can see in Figure 1.1. From a baby a with 60-year-old looking to a geriatric in a childlike body, Benjamin lives in a world with times flows backward. The facial characteristic of leading actor will appear in whole film consistently; nevertheless, the actor who plays the role of Benjamin Button is in his middle age and the audiences know his face well. These odd- looking of a famous actor could not be played by a stand-in. A lifelike digital facial model and age synthesis techniques help visual special effect artists create fantastic experience and make things possible. For people who wish to rejuvenate through a cosmetic surgery, it may be hard to imagine the effect of surgery without cosmetic background. Beautician or surgeon can use systems with age synthesis technology or beautify process to visualize rejuvenated results and help customer making decisions. They also have more reference in beauty treatment by foreseeing the probable consequence of a surgery.

1.2 Problems

The research of age synthesis tries to infer the most possible consequence what years can do. Using appearance information of particular time of life to predict future or past looking is a big problem. Although academic research is rich in age synthesis, there are still lots of challenges and problems remaining open. Problems such as building person- alized aging model and extracting general aging features while reducing the influence of

individual difference have been considered as two major types of predicament. The age progression of different people can be quite different due to many factors like gender, health condition, living style, working environment and race. Some extreme situations such as drug-abuse, disease processes and dramatic changes in weight would even have more influence than regular age progression. All these circumstances lead researchers to a doubtful condition while predicting aging trajectory. Another difficult task is collecting the complete age progression data of a person. Collecting photos of a single person from all age of life is a very long term project and cannot be done in a couple of years, not to mention to gather photos from different people in different races. This causes the lack of proper database and has become a basic limitation of age related research. The beautifica- tion process which tries to enhance facial attractiveness can be anything but simple. The process here as “ beautification ” denotes a model which is capable of generating more attractiveness than the original face. An absolute definition of aesthetic values is still un- clear. Most of us have the ability to rank people according to our aesthetic, yet most of us do not have the ability to specify which part of face is attractive or not. It is still difficult to define a succinct set of rules that capture beauty. However, lots of experiences reveal that no matter where the rankers come from or their social economic status, gender..., there is always an orient result of facial attractiveness. Thus, there are several studies in psychology indicate that facial attractiveness is a cross-cultural agreement which means that attractive face might be data-driven.

1.3 Motivation

Along with the improvement of facial modeling technology, we may all have our own personal facial model which can be used for 3D games character, social website portrait and many other entertainment applications. Nowadays we can obtain a facial model with high accuracy geometry from many equipments like light-stage, Kinect. Such precise

Figure 1.2: Photographer: Ulric Collette. As these photo show, there are four major aging traits on the face: 1. drooping of eyelids, 2. lack of elasticity causing hollow cheeks, 3.

slack on the middle face and 4. wrinkles.

and fine models give us extremely realistic appearance on both the shape of our face and the texture of our skin. But high-resolution means not only realistic traits but also many aging details will show on result model. Previous work on de-aging and beautification usually focus on images for the texture part. However, many observations say that aging features behave on both wrinkles and slack of facial tissue. For example, compare with a younger person, aging usually means drooping of eyelids, lack of elasticity cause hollow cheeks, slack on middle face and of course wrinkles. All these aging features would affect the lineament of our face. Photographer Ulric Collette create a serial photos with combining the portrait of family member in different age. Figure 1.2 shows some of his work and reveals that the proportion between facial organs imply the perception of age and facial attractiveness. All the aging features will affect not only texture but also the outline of face. Thus in order to get a better result of de-aging and also beautification, deforming the geometry of model can be a novel solution to rejuvenating and enhance facial attractiveness.

1.4 Architecture

The first chapter in this thesis is an introduction to our work; why we want to solve this problem and what problem we want to solve, and what problem we face and what

problems still remain. Because there are many different point of views of this problem that we try to solve, we introduce some related work that have been discussed before in second chapter, and roughly explain the methods used in our system. Chapter 3 is our system overview. The flow chart in chapter 3 is our novel process for de-aging on facial geometry. We will propose a framework and leave the details to later chapters. Chapter 4 gives more details for implementation, such as how we collect our testing data and training data. In the next chapter, we present our method for facial model de-aging in details. Chapter 6 and 7 are the results, user study and future work.

Chapter 2

Related Work

It is crucial to obtain a proper facial model with sufficient aging traits and facial details.

A 3D model can be roughly divide into two parts, texture and geometry. The level of realistic of a facial model can be judged by both texture and geometry. More detail on a facial model can help us to generate more realistic anti-aging result. As people mention rejuvenation, they always imply that they will to be look more prettier and being more attractive. Unlike forensic topic, the goal of de-aging for public is to look better, that is why anti-aging is always related to enhance attractiveness.

2.1 Facial Modeling

At the old days, a fine 3D facial model usually could only be generated by an artist with some special 3D modeling software. With the improvement of technology, more and more new equipment with special techniques have been developed in order to automat- ically generate realistic facial models, and examples are like laser scanner, Light-Stage and Kinect. The technique like Light-Stage [13] developed by University of Southern California can display extremely realistic face models by precise diffuse estimation, ac- curate specular reflection map and high-resolution geometry mesh. Even pores on the faces can be capture and reconstruction through the rendering technique. It is a special

Figure 2.1: Light-Stage equipment.

capture device like Figure 2.1 shows and it flashes camera with specific pattern which can simulate reflection of surface. Another popular facial modeling technique is generated model from Kinect which also produce a quality personal 3D facial model with infra light depth estimation. Like Figure 2.2 shows. The most competitive advantages of Kinect is the price of equipment is much lower than others and with proper set up even for the unprofessional user can generate their own model. Other modeling technology like 3D Morphable facial models [1] can generate personal facial model automatically from their data base with even a single personal photo. But it has suffered from the database collec- tion and we Asian are still hard to reconstruct from western faces. However it provides a new way to obtain facial model and does not any special equipment which mean available for almost every computer-user.

2.2 Facial Attractiveness

Extracting factors that affect facial attractiveness has been studied from many points of views. From the famous “ Averageness Hypothesis ” [9] [15] to the “ Symmetry Hypoth- esis ” [7], academic researches keep telling us that facial attractiveness is cross-culture

Figure 2.2: Using Kinect to create realistic facial model.

and irrespective of the perceiver. No matter the gender, ethnicities and social-economic classes of rankers facial attractiveness seems to have more common agreement. Thus there are two main methods to synthesize a more attractive face from its original. First is the data-driven way, Leyvard et al. [11] propose a data-driven enhancement of facial attractiveness in a 2D image while maintaining a strong similarity between original and synthesized image. Second way is to apply specific rules like Neoclassical Canons, golden ratios and symmetry to human face. For example Liao et al. [12] propose an enhance- ment on the symmetry and proportion of 3D face geometry. Using rules like Neoclassical Canons and golden ratios of human face to enhance facial attractiveness and not only in frontal face also the profile face, we can make sure that resulting face would still be valid.

2.3 De-aging

Age synthesis has attracted many focuses over a decade [5]. There are many methods been proposed in both 2D and 3D fields. The shape of craniofacial mainly changes in infancy to adulthood level, for example the shape of head can be simulated as a mathematical model which transforms with time [18] [20]. The age synthesis of this level focuses

on the difference of shape rather than texture change. After growth, in adulthood, age progression mostly considers as texture changing like wrinkles appealing, the depth of nasolabial fold, and crow’s feet at the corner of eyes. With a large realistic image data base, Suo et al. [21] present a dynamic facial aging model using and-or graph and dynamic Markov process to simulate the relation between different age groups and exhibit highly photorealistic results. But their work is only suitable for 2D image and hard to apply on 3D models. Other work like Guo [8] present a digital anti-aging of face images focusing on removing wrinkles on specific region of face while keeping the edge (e.g. outline of face, lip and hair) shape enough to maintain the realistic of photo. Also considering the consistence of young look, they do the hair and eyebrow dyeing if needed. After process, their anti-aging images are 10-20 years younger than the original and keep the similarity to themselves. Still, it’s hard to manipulate the same process on a 3D model. Golovinskiy et al. [6] propose a statistical model for synthesizing the detailed facial geometry. Using high-resolution facial model data base for ages from 20 to 60, they extract statistics of aging on a tile-by-tile basis and synthesize aged face with aging details. Their work concentrates on analysis of facial details without changing the geometry of the head shape to get a great quality of aging face.

Chapter 3

System Overview

For the input source (relatively old) model, we accept a standard OBJ file to represent a 3D model and if there is any texture or normal map input, it should calculate texture coordinates and store them in the OBJ file in advance. We develop our system in a desktop with Intel core 2 Quad CPU and 4G memory. For communicate with other 3D modeling software, the output of our result model will also be an OBJ file format.

3.1 System Overview

Our key observation is that as we getting older, the lack of elasticity for skin will cause the slack of cheek and eyelids. Such transform will became a main aging trait in a high- resolution model. The proportions between facial features have a great influence on facial attractiveness and the perception of aging. Lengths between facial features imply aging degree or attractiveness of a person. There are two parts of our system to achieve de- aging on a 3D facial model. Seeing Figure 3.1 and Figure 3.2. First, a SVR model is used to train an aging trajectory and a beauty model. These models will be used as a guide to modify our facial model. Then, by model a de-aging/beautification process into an optimization procedure, we can produce a new proportion which involves the facial vector.

Modeled by SVR

Training database

Asian Database

Western Database

western

asian f

f

function : &

Figure 3.1: The training part of system. Two different database will be used to training two SVR model separated.

Rejuvenating/beautifying process

Facial Vector

Find

Younger/prettier Vector

New Model Adjustment

Figure 3.2: The adjustment part of system.

3.2 Implementation

We use two databases to verify our concept. First is Asian database and score with attrac- tiveness. The score is rated manually by graduate students and ranges from 1 to 5 where 5 score for the most attractive face while 1 for the most unattractive one. Second database of our system for training young model is collected from 3D Morphable Model Database where each face is a 320 X 240 image and contains 65536 vertices. Because this mor- phable model already analyzes the performance of age and supplies an age attribute for generating morphable model, we use their age attributes and classify them into 5 levels of age score. The training process can be done in advance and for the user input, only their facial model will be required. The input model needs to extract their facial features in order to calculate its facial vector. Since an automatic model feature extraction is still a difficult problem and is also not our first task to solve, we let the user to notify us their facial feature locations.

Chapter 4

Database and Age/Beauty Model

There are two different races database will be used to construct the SVR models. A west- ern database is generated by a 3D morphable model which is established by the university of Basel. A age model will be training from this western database. Another database is collected by a local company and our laboratory with total 116 people involved. This Asian database will be used to training a beauty model.

4.1 Western Database

The BFM (Basel Face Model) 3DMM (3D Morphable Model) database [16] is published by the Computer Science department of the University of Basel. This BFM morphable

Figure 4.1: Models in different age and different parameters generated by the BFM 3D Morphable Model database.

model database is calculated from registered 3D scans of 100 male and 100 female faces.

With an age attribute provided by BFM database, we generate several random models as our western database and classify them from 1 to 5 according to their age level. For example, the most left model shows in Figure 4.1 is an example of the youngest person we can generate. For each person in western database, five models with different level of age will be generated by BFM to simulating aging trajectory of a person.

4.2 Asian Database

Figure 4.2: Asian database.

This database is captured from structural light 3D scans of 116 adults (92 males and 25 females). We use two cameras and one projector to project structural light patterns on the faces. Each capture takes about 10 seconds. After applying the correspondence algorithm, each model is represented by 65536 vertices and the same number of color values. Figure 4.2 shows a part of facial models in this database with both 3D geometry and texture. Figure 4.3 and Figure 4.4 is some example faces in the database with only geometry and texture respectively.

Figure 4.3: Some examples of facial geometry in the Asian database without texture.

Figure 4.4: Some examples of facial texture in the Asian database.

4.3 Feature Extraction

As we known, 3D facial model alignment is a difficult problem. Without color informa- tion, it is hard to figure out the outline of a face model. Without the depth information, it is hard to detect the influence of pose. Unfortunately, even we use both color and depth information, it is still a tough problem to align a 3D facial model. In order to simplify the situation we met, we extract feature points manually of the subset of a standard feature point set defined by Farkas based on anatomy of the Face. We pick up 72 feature points as the number of BFM data set provided, shown in 4.5. These feature points will be classified to 9 classes: left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, nose, mouth and other. The feature points are distributed all over the face including some parts of ears. If the generated model does not contain ears on its head, for example our Asian database, then the ear part will be omitted. While we extract all feature points from a face model, we connect specific vertex by a Delaunay triangulation to create a triangular mesh. This triangulation tries to maximal the minimum angle of triangles, which means to avoid triangles being clustered too tight. In our case, there will be total 201 edges to connect all 72 feature points like Figure 4.6 shows. Finally, the length of each edge in the triangular mesh will form a 201-dimensional distance vector which we call it a facial vector. Principal Component Analysis (PCA) can help us reducing the dimension of the facial vector. Before any training or optimization process, we will reduce our facial vector to 35 dimensions to speed up the whole process.

4.4 Age/Beauty Model

Support vector Regression (SVR) [16] can fit highly non-linear function by choosing different kernels. For a set of sample (x, y), where x ∈ R^d and y ∈ R, SVR will try to find a proper hyper plane to fit it. In our de-aging/beautify process we use the age/beauty label in the database as the target class. Since we try to use the distance between feature

Figure 4.5: All 72 feature points are pick up from the subset of Farkas feature points and label manually.

Figure 4.6: The lengths between facial features are used to represent aging/beauty level of the face.

points to describe aging traits and attractive proportion of face, the dispersed edges have a great influence on the ability of describing aging level of face vectors. Facial vector is used as the input attribute for SVR model, and the age/beauty label is its corresponding target class. After training, SVR defines a smooth function f : R^d → R, which can estimate the age/beauty label for an input facial vector. We use a RBF kernel function for SVR and perform grid search to find the optional parameters, such as the width r of a RBF kernel. The implementation LIBSVM [3] for  – support vector regression has combined all these works in their software. The mean squared error will be used to evaluating the accuracy of prediction. Table 4.4 is the error of age model which is training by Asian database. Table 4.4 shows that when the data number of database goes to 100 people, the mean squared error become stable.

Table 4.1: The mean squared error of the SVR model training by Asian database. Half of our data in the database will be used to training the SVR model, and others will be used to testing the SVR model.

(Regression) Mean square error

58 for training, 58 for testing 1.05562

Table 4.2: The mean squared error of the SVR model training by Western database.

(Regression) Mean square error

20 people (100 facial models) 0.326835

50 people (250 facial models) 0.507263

100 people (500 facial models) 0.26696

200 people (1000 facial models) 0.276817

500 people (2500 facial models) 0.372647

Chapter 5

De-Aging Process and Beautification

By extract facial vector from every models, we construct the age/beauty model to rep- resent our concept of ”aging” and ”beautification”. An optimization process will be used to generate an young/beauty vector which means an younger or prettier face. The young/pretty vector represent a new proportion of facial features. This new proportion will be used as a guide to decide a new location of all features. Finally, a smooth defor- mation will deform our original facial model to his/her new look.

5.1 Young/Pretty Vector

We process a SVR-based procedure to obtain the young vector while maintaining the sim- ilarity to the original face. The young vector v⁰ is defined as v⁰ = arg min E(u) = −f (u) , where E(u) is our energy function and f (u) is obtained from the SVR training result.

Before solving the optimization, PCA had been performed on our vector in order to reduce the dimension and accelerate the optimization. We use standard no-derivatives Direction Set Method which is also called Powell’s method [17] to numerically approximate the minimum of function E(u). Powell’s method would find a local minimum for us that is exactly what we needed: finding a younger face while maintaining the similarity between them. The basic idea of Powell’s method is using the method of finding minimum in one

dimension and applies to each basis in N-dimension. In each iteration, it chooses the av- erage direction as a good direction for next iteration. Powell’s method will give us a local minimum near the original face vector but this minimum distance vector doesn’t guaran- tee it’s correspond to a valid face. For example, the ratio of eyes and face may be too large so the face will look like a weird baby but with a lower cost. The valid face space is a subspace of our search space. To avoid an unreasonable result, we add a regularization term to this energy function call LP (log-likelihood) term, so the function will look like

E(u) = (α − 1)f (u) − αLP (u), (5.1)

which is similar to the one use in 3D morphable model [1] and we set α to 0.3. α is a parameter to control the importance of the log-likelihood term. The function P (likelihood function) is a multivariate Gaussian distribution

P = 1

2π^N2pQ

iλ_i Y

i

exp(−β_i² 2λi

),

where λi denotes the i − th PCA eigenvalue and βi denotes the i − th component of u.

After projecting u to the PCA space, the log-likelihood term becomes

LP (u) =X−β_i²

2λ_i + const.

The constant term is independent of u and thus can be omitted from the optimization.

With the likelihood term, we can make sure the distance from a valid face by using the Gaussian distribution. By this SVR-based procedure, we can find a young/prettier facial vector with the face still extremely close to the origin face. The result of our testing facial model in Table 5.1 shows the age/beauty score before and after optimization Equation 5.1.

Table 5.1 shows another difference of score under age model.

Table 5.1: Both age and beauty process will be applied on this facial model. A higher score is better (younger or prettier).

Before After

Beauty score 2.91732 2.99806

Age score for 20 people training set 1.63214 7.81515 Age score for 50 people training set 1.77 8.07728 Age score for 100 people training set 0.802941 7.01342

Table 5.2: Only de-aging process will be applied on this facial model. A higher score is better (younger).

Before After Age score for 20 people training set 3.26527 7.83385 Age score for 50 people training set 3.52103 8.07232 Age score for 100 people training set 3.19073 7.01016

5.2 Adjustment

With the new young facial vector v⁰, we now can use these distance as a guide to adjust our feature points. Formally, we define

E(q

, . . ., q

) = X

eij

α

(kq

− q

k

− d

)

, (5.2)

where qi is our new position of feature points; eij is the facial mesh connectivity matrix.

We set α to 10 if i and j belong to the same class and 1 for others. The target dij is the element from young vector that corresponds to edge eij. By minimizing E(q1, q₂, . . . , q_n) we can get our new position of feature points. Levenberg-Marquardt (LM) algorithm [10][14] is an efficient way to solve this energy function and the original facial vector

Figure 5.1: The new location of feature points that is calculated by our de-aging process.

provides us a good initial guess. As shown in Figure 5.1, red points is the original feature points, meanwhile blue points is the position after de-aging/beautification.

5.3 Geometric Transformation

After we get the new position for every feature point, we need to use these displacements as a guide to deform whole model. A transformation called Thin Plate Spline (TPS) warping which is a special case of Radial basic function warping (RBF)[4] is perfectly suitable for us. Because that we are mainly handling a high-resolution model, no matter how we try our best to increase the number of feature points, compared with our model which may contain about 70000 vertices, our feature points are still much less than model vertices. TPS did a great job on this situation. Even we only use less than 100 feature points as a guide, our model, with over 40000 vertices, still maintain its original shape and deforms smoothly. Just as its name suggests, this method can deform a mesh just like adding some stress on a specific position and the other part of the mesh will deform

because of the stress, but it still maintains the smoothness. Formally, we define

x

=

















 x

y

z



















=

















 P

i=1

w

U (x, p

) + a

+ a

x + a

y + a

z P

i=1

w

U (x, p

) + a

+ a

x + a

y + a

z P

i=1

w

U (x, p

) + a

+ a

x + a

y + a

z

















 ,

(5.3)

where (x⁰, y⁰, z⁰) is the position after deformation as Bookstein [2] did, we use the kernel function K = U (x, p_i) = kx − p_ik that minimizes the bending energy of deformation.

We need to solve the coefficients of w and a. Let p is the origin coordinate, q is the target coordinate. The coefficient for TPS can be estimated by solving a linear system











K p

p^T 0



















 w

a











=









 q 0









 .

Just like Rohr et al. did [19], the TPS will calculate a transformation for every vertex, and in the meanwhile ensuring our feature points to exactly match the target position as long as we have the right correspondence of p and q. After TPS deformation, the output model lost some similarity to the original face. Since age progression or beautification is a process which changes gradually, we blend the original model and the output model of this step in different weights as our final result and show it in the experiment chapter.

There’s the preliminary result in Table 5.3 of both age and beauty process.

Table 5.3: The preliminary result will blend with the original model in order to obtain a gradual de-aging/beautification process. The texture of facial models has been smooth in advance.

Original Preliminary Result of Beauty Model

Preliminary Result of Age Model

20 people training set 100 people training set 200 people training set

Chapter 6

Experimental Result

6.1 Result

Since de-aging and beautification is a gradual process, we blend the preliminary result and the original face with some different weights to decrease the gap between them. By applying 30 percent, 50 percent, 80 percent for the weight of preliminary result, we can generate these following results. Because our work is a process on 3D geometry, the frontal portrait and profile view will show different effects of process. The table 6.3 below shows the effect of blending. As we decrease the percentage of the preliminary model, we lost some age score. The following table 6.1 and table 6.2 will show the final result with blending percentage from 0%, 30%, 50%, 80% to 100%.

6.2 User Study

There are 20 people involved in this user study. All of them are students from 22 to 26 years old and half of them are males. For every rank, score one is the best or is the youngest or most attractive and score five for the worst. We show the pictures on the view of both the frontal and 45 degree. Figure 6.4 shows the conclusion of our user study. For beauty model, 85% tested consider one of our 30%, 50% and 80% preliminary

Figure 6.1: The front view of results.

Figure 6.2: The side view of results.

Figure 6.3: When the percentage of the original face increase, the age score decrease which means the face will look elder.

results is more attractive than the original face. For age model work on first case, 100%

testers consider one of our 30%, 50% and 80% preliminary results is younger than original face. For age model work on second case, 25% testers consider one of our results is the youngest one.

6.2.1 Beauty Model

For Asian database with beauty score, we ask user to rank four models for original model, 30 percentage of preliminary model, 50 percentage of preliminary model, 80 percentage of preliminary model. Here we show the average rank among 20 people. Most users consider when we blend 30% of preliminary and 70% original face is the most attractive face. Figure 6.5 shows the result of our user study.

6.2.2 Age Model

80% of testers consider the 80% of preliminary is the youngest face but also mention that there are some artifacts like a skewed nose which will affect judgement. We also ask testers to estimate the age of each face. The best result is 80% of preliminary, and it is thought be 28.63 years younger than original faces on an average. Figure 6.6 is our result

Figure 6.4: The summary of user study of all three facial model de-aging/beautification.

Beauty Age Age

20 15 10 5 0

People who give the best score to 80% preliminary People who give the best score to 50% preliminary People who give the best score to 30% preliminary People who give the best score to original face

of user study.

Many testers consider this set as distinguishable. There are only slightly differences among faces and is hard to tell which is much younger. For the youngest face which is 80% preliminary, it is thought to be 6.1 years younger than original face on an average.

We demonstrate this case in Figure 6.7.

Figure 6.5: The beauty score which is ranked by users. The lower score means more attractive.

Figure 6.6: The average age score which is ranked by users. The lower score means younger.

Figure 6.7: Another case for our de-aging process.

Chapter 7

Conclusion and Future Work

Human beings are born with the desire to be beautiful. No matter men or women, we all wish to be younger and prettier. Human project these desire on many sides. We try to look younger in the real world and this makes cosmetic industry flourish. We try to be looking prettier in the virtual world such as the internet social website which makes entertainment industry such popularity. Due to the different goal of process, de-aging and enhance fa- cial attractiveness in academy researches can be considered from many different points of view. For example, Aging or de-aging researches for forensic medicine focus on the precision of predicting output while for entertainment purpose the accuracy of aging tra- jectory can be more tolerant. The result of aging/de-aging process does not need to be exactly the same as his/her old/young looking. A good looking or a dramatic older effect may be better than the true appearance. Because of the limitation of technology, most research in the past works on 2D which is photos and images. Researches point out that after adulthood the craniofacial will rarely change their shape, so in the past 2D facial de-aging works aim for removing wrinkles on the faces. The slack skin may not change the shape of craniofacial but it will change the outline of faces. Removing only wrinkles on the face will cause inconsistent facial characteristic, which makes result face unreal and creepy.

In this paper, we propose a de-aging and enhance facial attractiveness framework

based on 3D high-resolution facial model. In order to make the final result as consistent as possible with the original face, we ensure our rejuvenating process to focus on the outline which is the geometry part of a 3D model. The improvement of facial age perception or attractiveness can be displayed as both frontal portrait and profile view. For every input face, we provide a possible result of younger and prettier face while keeping the similarity to the original and producing results. By the improvement of shape analysis, we can search for a new viewpoint about age estimation and facial attractiveness researches.

The database plays a major role of age related researches and so as on the human facial perception field. While more and more 2D image databases have been established, 3D facial model database is relatively rare. Thanks to the great progress of facial model- ing technology today, we can generate our personal facial model easier than in the past.

More 3D facial model will be generated and collected which provide researchers more information to reference.

Since we now only process on the geometry of model, adding other non-geometry feature such as texture color into our framework should be an interesting consideration.

Many users point out that no matter the face is young/pretty or not, if the texture or other part of a head doesn’t match the perception of face, they would not give him/her a good score because of the creepy feeling. We believe that this is when uncanny valley happens. Considering both texture and geometry at the same time should decrease the uncomfortable feeling.

Bibliography

[1] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Transac- tions on Graphics (Proc. of SIGGRAPH 1999), 1999.

[2] F. Bookstein. Principal warps: Thin-plate splines and the decomposition of defor- mations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11:567–

585, 1989.

[3] C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm.

[4] J. Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In W. Schempp and K. Zeller, editors, Constructive Theory of Functions of Several Variables, volume 571 of Lecture Notes in Mathematics, pages 85–100. Springer Berlin / Heidelberg, 1977. 10.1007/BFb0086566.

[5] Y. Fu, G.-D. Guo, and T. Huang. Age synthesis and estimation via faces: A survey.

IEEE Trans. on Pattern Analysis and Machine Intelligence, 2010.

[6] A. Golovinskiy, W. Matusik, H. Pfister, S. Rusinkiewicz, and T. Funkhouser. A statistical model for synthesis of detailed facial geometry. ACM Transactions on Graphics (Proc. SIGGRAPH), 25(3), July 2006.

[7] K. Grammer and R. H. Thornhill. facial attractiveness and sexual selection-the role of symmetry and averageness. Journal of Comparative Psychology, pages 233—- 242, 1994.

[8] G. Guo. Digital anti-aging in face images. In ICCV’11, pages 2510–2515, 2011.

[9] J. H. Langlois and L. A. Roggman. Attractive faces are only average. Psychological Science, 1:115–121, 1990.

[10] K. Levenberg. A method for the solution of certain problems in least squares. Quart.

Applied Math., 1944.

[11] T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski. Data-driven enhancement of facial attractiveness. In ACM SIGGRAPH 2008 papers, pages 38:1–38:9, 2008.

[12] Q. Liao, X. Jin, and W. Zeng. Enhancing the symmetry and proportion of 3d face geometry. IEEE Transactions on Visualization and Computer Graphics, 18(to ap- pear):to appear, 2012.

[13] W.-C. Ma, A. Jones, J.-Y. Chiang, T. Hawkins, S. Frederiksen, P. Peers, M. Vukovic, M. Ouhyoung, and P. Debevec. Facial performance synthesis using deformation- driven polynomial displacement maps. In ACM SIGGRAPH Asia 2008 papers, pages 121:1–121:10, 2008.

[14] W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters.

SIAM J. Appl. Math., 1963.

[15] A. J. O’Toole, T. Price, T. Vetter, J. C. Bartlett, and V. Blanz. 3d shape and 2d surface textures of human faces: the role of ”averages” in attractiveness and age.

Image and Vision Computing, 18(1):9–19, 1999.

[16] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. A 3d face model for pose and illumination invariant face recognition. In Proceedings of AVSS 2009, pages 296–301, 2009.

[17] W. H. PRESS, B. P. FLANNERY, S. A. TEUKOLSKY, and W. T. VETTERLING.

Numerical Recipes: The Art of Scientific Computing, 2nd ed. Cambridge University Press., 1992.

[18] N. Ramanathan and R. Chellappa. Modeling age progression in young faces. IEEE Computer Vision and Pattern Recognition, 1:387–394, 2006.

[19] K. Rohr, M. Fornefett, and H. S. Stiehl. Approximating thin-plate splines for elastic registration: Integration of landmark errors and orientation attributes. In In Proc. of IPMI’99, volume 1613 of LNCS, pages 252–265. Springer, 1999.

[20] K. Scherbaum, M. Sunkel, H.-P. Seidel, and V. Blanz. Prediction of individual non-linear aging trajectories of faces. In The European Association for Computer Graphics, 28th Annual Conference, EUROGRAPHICS 2007, volume 26 of Com- puter Graphics Forum, pages 285–294, Prague, Czech Republic, 2007. The Euro- pean Association for Computer Graphics, Blackwell.

[21] J.-L. Suo, F. Min, S. C. Zhu, S. Shan, and X. Chen. A multi-resolution dynamic model for face aging simulation. In CVPR’07, pages –1–1, 2007.

RESUME

黃詩晏













