Department of Psychology, National Chengchi University, Taipei, Taiwan Jie-Li Tsai (蔡介立

Department of Psychology,

National Chengchi University, Taipei, Taiwan

Jie-Li Tsai (蔡介立)

Overview

 Eye movements and cognition

 Why the eyes move

 Tracking eye movements

 Factors affecting eye movements

 Eye movements and reading

 Cognitive operations in reading

 Characteristics of eye movements in reading Chinese

 Tracking eye movements in e-learning and e-reading

Why the eyes move

（蔡介立，科學人，2007.09)

Eye tracking systems

When viewing a scene

Cognitive operation and eye movements

• Cognitive activity involves

• Computation (Bottom-up processing)

• Physical features: brightness, contrast, color, etc..

• Symbolic meaning or function

• Constraint (Top-down processing)

• Goal/Cognitive loading

• Task/Context

• Strategies: intention, motivation, preference

• Tasks in daily activities

• Reading, visual search, driving, walking, sports,

making tea, etc…

Cognitive operation and eye movements

• Eye movements are driven by

• The demands of a task in the spatial and temporal dimensions

• Eye movements are affected by

• Stimulus features

• Goal/Task/Context

• Strategies

Reading the mind in the eyes

1. Free examination

2. Estimate the material circumstances 3. Give the ages of the people

4. Surmise what the family had been doing before the visitor arrivals

5. Remember the clothes worn by the people

6. Remember the position of the people and objects in the room.

7. Estimate how long the visitor had been way from the family

(Yarbus, 1967)

Preference and motivation

Preference and reading skill

Eye Movements in Reading

Eye movement control and reading

Perception

visual acuity, physical features

Attention

word by word or processing gradient

Language

Lexical properties Contextual info.

EM control

when & where to move the eyes

地球暖化問題日趨嚴重，節約能源成為全人類的重要課題

Word Recognition

Semantic

Integration

Basic characteristics in reading

 Sequence of fixations and saccades, not moving smoothly.

 Each fixation lasts about 250 msec, with intra- and inter-reader variability.

 Each saccade lasts about 20-40 msec and the saccade length is 2-3 Chinese characters (7-9 English letters) in average.

 Processing all words but not fixating every word, some are skipped

 About 10 to 15% of regressive saccades to reprocess earlier words

0%

2%

4%

6%

8%

10%

12%

0 40 80 12 0 16 0 20 0 24 0 28 0 32 0 360 4 00 4 4 0 4 8 0 520 56 0 60 0 640 68 0 72 0 760 8 00 8 4 0 8 8 0 9 20 9 6 0 10 00 10 4 0 10 8 0 11 20 11 60 12 00

Probability

Duration (msec)

Distribution of All Fixation Duration

Kid_G3(N=15,D=138106)

0%

2%

4%

6%

8%

10%

12%

0 40 80 12 0 16 0 20 0 24 0 28 0 32 0 360 4 00 4 4 0 4 8 0 520 56 0 60 0 640 68 0 72 0 760 8 00 8 4 0 8 8 0 9 20 9 6 0 10 00 10 4 0 10 8 0 11 20 11 60 12 00

Probability

Duration (msec)

Distribution of All Fixation Duration

Kid_G3(N=15,D=138106) Kid_G5(N=14,D= 93271)

0%

2%

4%

6%

8%

10%

12%

0 40 80 12 0 16 0 20 0 24 0 28 0 32 0 360 4 00 4 4 0 4 8 0 520 56 0 60 0 640 680 72 0 760 8 00 8 4 0 8 8 0 9 20 9 6 0 10 00 10 4 0 10 8 0 11 20 11 60 12 00

Probability

Duration (msec)

Distribution of All Fixation Duration

Kid_G3(N=15,D=138106)

Kid_G5(N=14,D= 93271)

Adult(N=48,D=2094238)

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

-10 -9.6 -9.2 -8.8 -8.4 -8 -7.6 -7.2 -6.8 -6.4 -6 -5.6 -5.2 -4.8 -4.4 -4 -3.6 -3.2 -2.8 -2.4 -2 -1.6 -1.2 -0.8 -0.4 0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8 5.2 5.6 6 6.4 6.8 7.2 7.6 8 8.4 8.8 9.2 9.6 10

Probability

Character

Distribution of Saccade Length

Kid_G3(N=15,D=138106)

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

-10 -9.6 -9.2 -8.8 -8.4 -8 -7.6 -7.2 -6.8 -6.4 -6 -5.6 -5.2 -4.8 -4.4 -4 -3.6 -3.2 -2.8 -2.4 -2 -1.6 -1.2 -0.8 -0.4 0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8 5.2 5.6 6 6.4 6.8 7.2 7.6 8 8.4 8.8 9.2 9.6 10

Probability

Character

Distribution of Saccade Length

Kid_G3(N=15,D=138106) Kid_G5(N=14,D= 93271)

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

-10 -9.6 -9.2 -8.8 -8.4 -8 -7.6 -7.2 -6.8 -6.4 -6 -5.6 -5.2 -4.8 -4.4 -4 -3.6 -3.2 -2.8 -2.4 -2 -1.6 -1.2 -0.8 -0.4 0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8 5.2 5.6 6 6.4 6.8 7.2 7.6 8 8.4 8.8 9.2 9.6 10

Probability

Character

Distribution of Saccade Length

Kid_G3(N=15,D=138106) Kid_G5(N=14,D= 93271) Adult(N=48,D=2094238)

Developmental change of reading Chinese

地球暖化問題日趨嚴重，節約能源成為全人類的重要課題

Perceptual span and parafovea preview

，節約能源 成

Parafoveal preview

• Perceptual span

• A limit number of characters can be perceived and processed in a fixation

• Chinese: 1L3R (characters); English 3-4L 14-15R (letters)

• Asymmetric to the right side

• Parafoveal preview

• A word which was parafoveally previewed on F _n-1 could be foveally processed again in F _n .

• Integration of currently fixated word with its preview is necessary.

約能源成為

Perceptual

span

導遊宣稱這次全程下榻五星級飯店或是高級溫泉渡假村

FD ₀

FD ₁ FD ₂

FD ₃ FD ₄

• First Fixation Duration FFD = FD ₁

• Gaze Duration GD = FD ₁ + FD ₂

• Total Viewing Time

TVT = FD ₁ + FD ₂ + FD ₄

Skipping Rate

Regression Rate Refixation Rate

Reread Rate

Means of frequency

Means of Strokes

Predictability

values Example

HF P

91.08 21.10 0.78

UP ^0.01

LF P

1.58 22.08 0.73

UP 0.01

(Tsai, 2012, submitted)

 Fixation durations and probabilities showed the main effects of word predictability and frequency.

 The predictability effect was larger than frequency effect on SFD and FFD.

HF LF HF LF HF LF

FFD SFD GD

P 246.8 250.8 244.8 250.2 266.2 277.6 UP 254.2 262.8 255.8 263.8 276.3 299.8 220

240 260 280 300 320

Dur atio n ( ms)

HF LF HF LF HF LF SKIP ReFIX ReREAD P 25% 22% 9% 14% 9% 11%

UP 22% 18% 10% 18% 14% 18%

5%

10%

15%

20%

25%

30%

Pr o b abil ity

(Tsai, 2012, submitted)

evidence in reading

 Factors affecting eye movements

 Perceptual: typography

 Lexical : word frequency, semantic, syntax

 Contextual: word predictability

 Multiple eye movement indices reflecting processing difficulty in details

 Global measures

 Local measures

 Regressive behaviors

 Temporal sequence

Linking seeing and listening

eye movements

(Altmann & Kamide , 2007)

 The man has drunk the wine

 The man will drink the beer

shared book reading

(Chang & Tsai, 2011; 張雅嵐，政大幼教所碩士論文，2011)

Seeing cooperates with listening for comprehension

Fixation Area Utterance

(Chang & Tsai, 2011; 張雅嵐，政大幼教所碩士論文，2011)

Eye movements and e-learning

 Multimedia learning

 Learning from texts, graphics, animations, and sound.

 Proceeding on a moment-to-moment basis

 Eye movements can provide the evaluation by revealing learner’s attention in two dimensions:

 Spatial

 Fixation durations, density, and probability

 Transitions between different areas

 Temporal /Time course

 Overall

 Comparing different stages /scanpath

 Evaluation of an e-reading system

 Off-line assessments

 Changes of eye movement behaviors

 Eye movements in a ‘off-line’ comprehension test

 Eye movement contingent interaction in an e-reading system

 Detecting the reader’s mental status on-line

 Reading ability

 Attention

 Reading strategy

 Providing the information in time and in space

Eye movements and e-reading

Conclusions

 Seeing is tightly linked to cognitive activity

 Computation

 Constraint

 Cognition coordinates seeing in time with

 Listening

 Speaking

 Action (hands, head, body)

 Providing information in time and in space can help

reading and learning

Thank you !

http://emrlab.nccu.edu.tw