A Real-Time User Interest Meter Based on Human Cognitive Model and its Applications

A Real-Time User Interest Meter Based on Human

Cognitive Model and its Applications

Chia-Han Chang1, Wei-Ting Peng1, Li-Wei Chan1, Chien-Nan Chou2, Wen-Yan Chang2, Zita Chao-Ling Chen1, Yi-Ping Hung1,2

1_{Graduate Institute of Networking and Multimedia} 2_{Department of Computer Science & Information Engineering}

National Taiwan University [email protected]

Abstract―In this paper, we propose the Interest Meter,

a system allowing computers understand users’ reactions, based on multimodal interfaces for measuring users’ in-terests in real time. The Interest Meter takes account of users’ spontaneous reactions. In this work, we analyze the variations of users’ eye movements, blinks, head motions, and facial expressions when they interact with computers. Furthermore, we propose the method of combining those signals into interest level and verify its reliability in our experiment. There are two integrated applications pre-sented in this work. First, produces each user’s personal memory according to their reactions relate to the Magic Crystal Ball. Second, from user’s reaction, automatically edits the MV-style home video during watching. Experi-mental results have showed that the Interest Meter can measure user’s interest and make a great improvement in the interaction.

Index Terms―Human Computer Interaction, Affective

Computing, Eye Detection, Human Facial Expression Recognition, Human Cognitive Model

I. INTRODUCTION

The goal of Human Computer Interaction is to minimize the barrier between users and computers, that is, to make computers more usable and recep-tive to users’ needs. Communication is an impor-tant social contact to understand each other in hu-man society. The first step during communication is to understand reactions of each other. With these two concepts, we can make computers more recep-tive to users’ needs by making computers under-stand users’ reactions.

AIDA [2] is a marketing theory which describes

a common list of events that happen very often when a person is selling a product or service. 「AIDA」 means Attention, Interest, Desire, and Action. When a person is selling a product or ser-vice, he should attract customers’ attention before making them feel interesting. Once customers are interested in the product, they will generate the de-sire to buy the product. Interest has an influence on determining decision making. Therefore, under-standing users’ interest leads to know users’ reac-tions. Moreover, interest can trigger emotion, at the same time, emotion can reveal interest. Finally, we propose the idea of measuring users’ interest by constructing attention model and emotion model. Attention describes visual focus of the user and emotion de-scribes inner state of the user.

The details of bodily signals (proposed by Argyle, 1988 [6]) were references for the clues of reactions feeling interesting. There are six classes of clues. 1. Facial expression: laughing, more smiling (not false or miserable smile), eyebrows down, circular wrinkles and upward movements of mouth. 2. Gaze: smiling eyes, more gaze, glances, fixation, dilation, less blink, eyes track and look. 3. Gesture: lively movements of hands and shoulders, head-nods. 4. Posture: forward lean, draws back legs. 5. Bodily orientation: more direct, but side by side for some situations of group or pair work. 6. Non-verbal vo-calization (tone of voice): higher pitch, upward pitch con-tours, orotund. We consider the first three classes of clues: facial expression, gaze and gesture in our work. Eye gaze plays an important role in

attention because a speaker usually presents his focus of attention on a listener by looking. In emo-tion, the intuitive and obvious clue of interest is from the facial expression. It has been demonstrat-ed that emotions influence people's attitude towards their current and next action. In adiitoin, there is evidence that emotions play an essential role in ra-tional decision making, perception, learning, and other cognitive functions [9]. Finally, the Interest Meter adopts blinking detection, saccade detection, head motion detection, and facial expression rec-ognition for measuring users’ interest.

In this paper, we have implemented Interest Me-ter in two applications: Magic Crystal Ball and MV-Style Home Video Automatic Editing System. In Magic Crystal Ball, users’ reactions are conti-nuously captured from a color camera during inte-raction process. According to these reactions, In-terest Meter automatically keeps the clips with in-teresting reactions of the user and combines these clips into personal memory. In MV-style Home Video Automatic Editing System, users can con-duct video editing by “watching videos”. When us-ers are watching videos, the Interest Meter indexes the important part of each shot in raw home video according to users’ reactions.

In the experiments, we verify the Interest Meter can measure users’ interest accurately, and dynam-ically adjust attention and emotion weight to obtain better performance.

The paper is organized as follows. In paragraph II, we show an overview of related works. The sys-tem framework is described in paragraph III and paragraph IV goes to the details of the Interest Me-ter implementation and asserts the evidence that the experiment results verify the system effect. Finally, we demonstrate two applications mentioned above in paragraph V, and verdict the conclusion and fu-ture work in paragraph VI.

II. RELATED WORK

The related work can be divided into two parts. First, we describe the comparison of unimodal in-terface and multimodal inin-terface. Second, we refer to some real-time affective multimodal interactive applications.

Multimodal interaction has become a key factor

in developing novel, effective solutions of natural human machine interaction. In many situations, multimodal interfaces are preferred over unimodal interfaces. Oviatt [10] characterizes that multimod-al interfaces satisfy higher user preference levels during interacting with these systems. More flex-ibility, expressiveness and control ability are avail-able in such interfaces. Also, studies have reported enhanced performance when using multimodal in-stead of unimodal interfaces. Therefore, the Interest Meter adopts multimodal interfaces.

Gaze-X [5] is a context-aware affective multi-modal interface that can adapt to the user's emo-tional state interface in an office scenario environ-ment. It uses speech, eye gaze direction, facial ex-pression, and keystroke and mouse movement as input factors. One of the major advantages of such a real-life working application is; it can be used as a research tool, to perform real-life experiments for affect measurement or usability aspects. A multi-modal affective mirror [1] contain vocal and facial affect-sensing modules, and a component fuses the output of these two modules to achieve a user-state assessment, a user state transition model, and a component presenting audiovisual affective feed-back, these should be keep or bring to the user in intended state. The mirror’s interaction is to evoke positive emotions, to let people laughing and to in-crease laughter.

Attention Meter [4], most similar to ours, is a vi-sion-based input toolkit which gives users an anal-ysis of faces found in a given image stream, in-cluding face tracking, head motion detection, facial expression recognition. In face tracking, they use a face detection algorithm using the Intel Open Computer Vision library. This algorithm gives the locations and sizes of all faces in the image. In our case, we use the same face detection algorithm, but instead of tracking all faces in the image, we only detect the largest face in each frame taken from the video stream. In head motion detection part, Atten-tion Meter detects large moAtten-tion, nodding, and shaking by using a finite state machine to analyze sequences of small movements and smaller ges-tures of nodding and shaking. Different to Attention Meter, our approach only detects the head motion. In eye feature detection part, Attention Meter de-tects the blink by using basic knowledge of the

structure of the face and looking for the distinctive brightness gradients of the eye. In our case, we detect not only blinking but also eyes movement by using more precise algorithm described in Section 3. In facial expression recognition part, Attention Meter only detects mouth shape such as open wide, close or smiling. In our case, we consider facial lo-cal regions and holistic face simultaneously.

The whole comparison between Attention Meter and Interest Meter shows in Figure 1. In attention part, the difference is that Interest Meter has sac-cade detection. In emotion part, Interest Meter uses facial expression recognition but Attention Meter use mouth shape recognition.

Figure 1 Comparison of Attention Meter and Interest Meter

III. ATTNTION MODEL IN INTEREST METER

We propose the Interest Meter, a real-time sys-tem to measure a user’s interest level. Figure 2 illu-strates the system framework.

Figure 2 System framework of Interest Meter As mentioned above, in this paragraph, we define the Interest Meter as two models: attention model and emotion model. The attention model contains head motion detection, blinking detection and sac-cade detection. The techniques we used in blinking and saccade detection can be found in [3]. In facial expression recognition, based on our previous work [13], we consider the local and holistic face com-ponents at the same time. Moreover, we focus on information fusion by combining those detection results into attention and emotion score, and dy-namically adjusting attention and emotion weights

of interest score. We also conducted the experiment results to verify the functionality of Interest Meter and the efficiency of two weighting adjustment rules.

The attention model is composed of three parts: head motion detection, blinking detection and sac-cade detection. In head motion detection, we adopt face movement as features. In blinking and saccade detection, we adopt three visual features: the center of the eyeball, two corners of the eye and the upper eye lid. To extract these eye features, face detection [8] is applied in advance for efficiently identifying possible eyes locations. Based on the facial geome-try [7], we further simplify the procedure of eye detection only on the possible regions. As the face detection, the cascaded Adaboost is also used for eye detection. To find the center of the eyeball, we apply the Gaussian filter to the image in order to detect the dark circle of the iris. The location with the minimum value is regarded as the center of an eyeball. To detect the corners of the eye, we use the method proposed in [11], which utilizes Gabor wavelets to localize possible corners. After finish-ing these three detection methods, we can construct the attention score according to these three detec-tion results.

A. Head Motion Detection

Interest Meter monitors the face found in the camera view. Each frame taken from the video stream would run through a face detection algo-rithm [8]. This algoalgo-rithm gives us the locations and sizes of all faces in the image. The first process of head motion detection is to detect the face and cal-culate the face movement. In the second process, we acquire the mapping relationship between face movements and head motion score by adjusting the variance of the Gaussian kernel.

B. Blinking Detection

Figure 3 Definition of blinking

Figure 3 shows the blinking definition, the point Q points to the eyeball center. First, we find the point P by finding first intersection of upper

eyelid and line PQ. When the eyeball center Q is covered by upper eyelid P, it means the user with blinking action. In the next step, we translate blinking signal into blinking score. We open a slid-ing window with one second and analyze this time duration to verify whether multiple blinking hap-pened within one second. If there is more than one blinks found within one second, we label the one-second duration as abnormal blinking state. C. Saccade Detection

Figure 4 Definition of saccade

We define the ratio d2/d1 to represent the eye movement amount. If ratio difference |Rt-Rt-1|

be-tween two adjacent frames is bigger than a preset threshold, it means a saccade happened (see Figure 4). Now, we have to convert saccade signal into saccade score. We open a one second sliding win-dow first and analyze this time duration whether it has any saccade detected. If there is any saccade found in one second, we label the duration as ab-normal saccade state.

D. Saccade Detection Attention Score Computing Attention has two properties in our observation. First, people need a period of time to concentrate on paying attention, but they are distracted easily in seconds. Besides, attention is a continuous state, so the attention value of present frame should be de-termined according to the values of previous adja-cent frames. Figure 5 shows the formula we set for the attention score.

Figure 5 Definition of attention score

The initial score of Sa(t) is set as zero. If there are any blinking, saccade, and head motion reac-tions found, it means the user is attentive to the ob-ject. The score of attention increases smoothly. On

the other hand, the score of attention will suddenly degrade α (one-third in our implementation) time of original attention score when the user is inattentive.

III. EMOTION MODEL IN INTEREST METER

We implement the emotion model by using the facial expression recognition proposed in [13]. A. Facial Expression Recognition

In our work, we only consider two types of emotion, positive and negative. Both local facial components and global face are adopted. We divide a face into seven components including left eye (LE), right eye (RE), middle of eyebrows (ME), nose (NS), mouth and chin (MC), left cheek (LC), and right cheek (RC). In addition, two components, upper face (UF) and holistic face (HF), are also considered. As the method in [13], we adopt mani-fold learning and fusion classifier to integrate the multi-component information for facial expression recognition. Given a face image I , a mapping M : Rd × c→ Rt is constructed by

(1)

,where c is the number of components, mi(．) is an embedding function learned from the manifold of component i, and Ii is a d-dimensional sub-image of the i-th component. Then, the multi-component in-formation is encoded to a t-dimensional feature vector M(I), where t ≥ c. To characterize the signi-ficance of components from the embedded features, a fusion classifier F: Rt → {Positive, Negative} is used based on a binary classifier SVM. By apply-ing this method, users’ emotion can be recognized in our system.

After completing LDE models and SVM model construction, we can start to run the facial expres-sion program. When we get each frame from cam-era, we do face registration and feature extraction of each component. Then, we project each compo-nent’s feature to the corresponding manifold mod-els and calculate the probability of belonging to each class. Finally, combine all probability as a new feature vector and use this new feature vector as the input of SVM classifier to produce the facial

( )

[

( ) ( )

( )

]

expression recognition result. B. Emotion Score Computing

We use the positive probability of facial expres-sion recognition result as the emotion score be-cause we only interested in the positive emotion.

IV. INFRMATION FUSION

As above, we have calculated attention and emo-tion scores. Furthermore, we propose two rules to dynamically adjust weights of attention and emo-tion scores for better interest measurement. The ideas of these two rules come from our observa-tions of 15 recorded videos during the users watching short videos selected from Youtube web-site, and Argyle’s [6] psychology study. Based on psychology research [6], users will have the fol-lowing reactions when they feel interested: laugh-ing, more fixation, less blink, lively movements of shoulders and head-nods. On the other hand, users will have the following reactions when they feel bored: blank face, less fixation and more blink. Based on our observations we conducted two rules: A. Interest = Attention + Emotion

B. Attention usually occurs before emotion In first rule, we discuss the combination of atten-tion and emoatten-tion for high and low situaatten-tions. Fig-ure 6 shows the four situations of people’s emo-tional state.

When attention and emotion are both high, it means that people are interested in the object. When attention is high but emotion is low, it means that people pay attention to the object but with blank face. When attention is low and emotion is high, it means that people is excited. When atten-tion is low and emoatten-tion is low, it means that people feel bored. In second observation, it is intuitive. Before having emotions triggered by an object, people must pay attention to the object first. A. Interest Score Computing

The interest score formula is as below:

(2)

The parameters Wa, Sa, We, Se, Si represent

at-tention weight, atat-tention score, emotion weight, emotion score, and interest score respectively.

Figure 6 Four situations of people’s emotional state B. Weighting Adjustment

We propose two rules for weighting adjustment. Firstly, when the attention score is decreasing, we analyze the emotion score in weighting adjustment. Secondly, when the attention score is increasing, we consider that the user starts to concentrate gradually.

According to the second observation that atten-tion usually occurs before emoatten-tion, we determine attention score first. Why do we adopt decrease and increase policies rather than low and high? Because attention is a continuous reaction, so we consider that adopting decrease and increase policies can measure user’s interest better.

As mentioned before, Figure 6 shows four possi-ble combinations of attention and emotion states. In the first column, with high emotion, shows that no matter what the attention value, people must be in-terested in. In this situation by first rule, we prefer using emotion factor to represent the interest level, so we increase the emotion weight and decrease the attention weight. In the second column, with low emotion, shows that the attention score is the major factor. When the attention score with high value, then people are interested in the object; otherwise, they feel bored with low attention score. In the same way, we prefer using attention factor to represent the interest score, so we increase the at-tention weight and decrease the emotion weight.

How to adjust the attention and emotion weights? We used the equation shown in below:

, e e a a i W S W S S = × + ×

(3)

(4)

,where Wa is attention weight, We is emotion weight, W1 is blinking weight, W2 is saccade weight, W3 is head motion weight and W1+W2+W3=1, Sb is blink-ing score, Ss is saccade score and Sm is head motion score. We use β to control the variance of adjust-ment amount. We define the attention score ac-cording to head motion, blinking and saccade reac-tions. Therefore, when the inattentive reactions oc-cur, the β value is increasing, and then the adjust-ment amount is also increasing. The weights of head motion, blinking and saccade are set with equal weighting.

In the second rule, we assign a higher score to a higher weight, that is, if the attention score is high-er than the emotion score, we set attention factor to represent interest level. The formula shows below:

(5)

Where Wa, We, Sa, Se stand for attention weight, emotion weight, attention score and emotion score respectively.

IV. EXPERIMENT

In the experiment, test videos were shown on a monitor with a screen that is 40-cm wide. Partici-pants were seated at a distance about 40-cm from the screen, and the viewing angle subtended by the screen is approximately 52 degrees. There are 6 participants (4 males and 2 females) volunteered in the experiment, from 20 to 35 years old, and they were not informed about the specific purpose in the experiment.

Figure 7 The structures of two test videos Figure 7 shows the structures of the two test vid-eos. The lengths of these two videos are all 3

mi-nutes and are composed of interesting and boring video clips. All interesting video clips are collected from the Youtube website and the boring video clips are edited from our home videos. All partici-pants are not familiar with the roles appearing in the videos. The difference of these two videos is that the interesting clips in video 1 may trigger par-ticipants with emotion reactions, while video 2 may not.

During participants watching two videos, the In-terest Meter measures their inIn-terest level by ana-lyzing their attention and emotion and calculates the interest score for each frame. There are two goals in our experiments. The first goal is to verify that Interest Meter can measure user’s interest. Figure 8 shows the broken line graphs of two mea-suring results. In Figure 8, the interest scores are lower in the boring segments and higher in the in-teresting segments. Therefore, we can approx-imately divide interesting and boring segments in the broken line graph, that is, the Interest Meter can measure user’s interest.

Figure 8 The broken line graphs of two measurement results

The second goal is to verify that using weighting adjustment rules can measure user’s interest better. Firstly, we produce personal boundary because the same source will trigger different levels and differ-ent kinds of emotion for each person. The personal boundary is used to eliminate people difference in about emotion. The method of producing personal boundary we used is maximum likelihood estima-tion. After having the personal boundary, we de-termine which frame is interesting if the interest score is higher than the boundary.

The motivation of this paper is, to collect the record clip when users are interested in the Magic

(1 ) , where( ) ( ) ( )a,b 1,0or0,1 b a W W W W e a e a ₌       × +       × − =       　 β β , e a e e e a a a S S S W S S S W + = + = 　，　

(

)

(

)

(

)

Crystal Ball application during the interaction. Therefore, we focus on users’ interested responses rather than non-interested reactions. For the reason, we like to know the interesting detection rate of Interest Meter. We define the interesting detection rate as below:

After calculating the intersection rate with the ground truth, we can acquire the interesting detec-tion rate. For users’ interested states lasting with a period of time, we open a window with three second width. If there is more than one second of interesting clips during this interval, it is assumed as interesting clips.

Table 1 shows the data that using weight adjust-ment rules can make better performance in interest measurement. There is a gap in the average im-provement between video 1 and video 2. For that the interesting clips of video 2 may not trigger us-er’s emotion reaction, the improvement of using weight adjustment rules can not be showed in these clips.

Table 1 The interesting detection rate of test videos

V. APPLICATIONS

There are two applications introduced in this work, the Magic Crystal Ball and the MV-style Home Video Automatic Editing System.

A. Magic Crystal Ball

Magic Crystal Ball (MaC Ball) is an interactive visual display system which allows the users to see

a 3D virtual artifact appearing inside a transparent acrylic ball and to manipulate it with bare hands. The interactive components of MaC Ball contain hand motion and touch detections. In hand motion detection, they construct the MaC Ball coordinate by system calibration to obtain precise 3D direc-tions and distances of hand modirec-tions during the in-teraction process. In touch detection, they integrate strain gauge sensors with a cantilever beam struc-ture into the touch detection module to achieve the better stability of detection. Figure 9 shows the steps of collecting users’ interested reactions and displaying their personal memory in the magic crystal ball.

Figure 9 The flowchart of magic crystal ball The equipment of this application contains a camera for capturing the users’ reactions, a trans-parent ball for manipulating the relic and showing the personal memory, and a touch screen for con-trolling the interaction process. When a user mani-pulates the ball with his hands, the Interest Meter keeps the clips shown with high interest scores. Once the user returns to the homepage, his personal memory is presented in the transparent ball. Origi-nally, the Magic Crystal Ball plays a passive role manipulating by the users. After combining with the Interest Meter, it changes to an active role that models users’ interest level and interacts with the users by providing a personal memory.

B. MV-Style Home Video Automatic Editing Sys-tem

MV-style home video automatic editing system, a system developed to automatically analyze video and a user-selected music clip. For video shots, the system eliminates shots with blurred content or

drastic motion. For music, the system detects onset information and estimates tempo of the entire me-lody. With the aids of the editing theory and the concepts of media aesthetics, the system matches selected video shots with music tempo, and there-fore facilitates users to make an MV-style video summary that conforms to editing aesthetics with-out difficulties.

The second application is combing Interest Meter into video editing system for automatically summarizing home video. The Interest Meter takes account of user’s spontaneous behaviors when watching videos. Based on users’ reactions when watching videos, we can construct a systematic framework to automate video summarization. With the aids of Interest Meter, the developed system can automatically generate a more receptive summa-rized video. The system architecture is illustrated in Figure 11 and the detail description can be found in [12].

Figure 10 The system architecture of video editing application

VI. CONCLUSION AND FUTURE WORK

We propose the idea that people’s interests can be measured by the Interest Meter, a computer vi-sion based approach to estimate people’s interests. In this work, we analyze users’ blinks, saccades, head motions and facial expressions when they in-teract with computers and provide interest scores for different applications. Therefore, applications can make a great improvement of the interaction by adjusting the interactive contents according to in-terest scores.

In future work, we will pay attention to incorpo-rate with other human perceptions. For example, considering head orientation recognition or

ex-tending with modularized sensors. Moreover, the Interest Meter can be extended to measure mul-ti-users at the same time in different applications, not only one user.

ACKNOWLEDGEMENTS

This work was supported in part by the Excellent Research Projects of National Taiwan University under grants 98R0062-04.

REFERENCE

[1] A. Melder Willem et al., “Affective

mul-ti-modal mirror: sensing and eliciting laugh-ter,” ACM HCM, 2007.

[2] B. Alec, Glengarry Glen Ross, Panayiotis

Pa-padakis, 1992.

[3] C. N. Chou, “Real-time three-stage eye feature

extraction and its Applications,” Master thesis in CSIE at National Taiwan University, 2009.

[4] L. Chia-Hsun Jackie, J. Wetzel, and T. Selker,

“Enhancing interface design using attentive interaction design toolkit,” ACM SIGGRAPH, Educators program, Boston, Massachusetts, July 30-August 03, 2006.

[5] L. Maat, and M. Pantic, “Gaze-X: adaptive

af-fective multimodal interface for single-user of-fice scenarios,” in Artificial Intelligence for Human Computing, Vol.4451, 251-271, 2007.

[6] M. Argyle, “Bodily communication,” New

York: Methuen & Co. Ltd, 1988.

[7] P. Ekman and W. V. Friesen, “Unmasking the

face,” Prentice-Hall, Englewood Cliffs, 1975.

[8] P. Viola and M. J. Jones, “Robust real-time

face detection,” International Journal of Com-puter Vision 57(2), 137–154, 2004.

[9] R.W. Picard, “Affective computing,” MIT

Press, Cambridge, MA, 1997.

[10] S. Oviatt, “User-centered modeling and

evalu-ation of multimodal interfaces.” in Proceedings of the IEEE, Vol.91 (9), 1457-1468, 2003.

[11] S. Sirohey and A. Rosenfeld, “Eye detection in

a face image using linear and nonlinear filters,” Pattern Recognition 34, 1367–1391, 2001.

[12] W. T. Peng, W. J. Huang, W. T. Chu, C. N.

Chou, W. Y. Chang, C. H. Chang and Y. P. Hung, “A user experience model for home video summarization,” in Proceedings of In-ternational Multimedia Modeling Conference, 2009.

[13] W. Y. Chang, C. S. Chen and Y. P. Hung,

“Analyzing Facial Expression by Fusing Ma-ni-folds,” in Proceedings of Asian Conference