1
行政院國家科學委員會補助專題研究計畫
█成果報告
□期中進度報告
（移動事件的語言與圖像表達和概念化）
計畫類別：
█
個別型計畫 □整合型計畫
計畫編號：NSC 97 -2410-H-004 -111 -MY3
執行期間： 97 年 8 月 1 日至 100 年 7 月 31 日
執行機構及系所：國立政治大學英國語文學系
計畫主持人：徐嘉慧
共同主持人：
計畫參與人員：
成果報告類型(依經費核定清單規定繳交)：□精簡報告
█
完整報告
本計畫除繳交成果報告外，另須繳交以下出國心得報告：
□赴國外出差或研習心得報告
□赴大陸地區出差或研習心得報告
█
出席國際學術會議心得報告
□國際合作研究計畫國外研究報告
處理方式：
除列管計畫及下列情形者外，得立即公開查詢
□涉及專利或其他智慧財產權，□一年□二年後可公開查詢
中 華 民 國 100 年 12 月 14 日
2
國科會補助專題研究計畫成果報告自評表
請就研究內容與原計畫相符程度、達成預期目標情況、研究成果之學術或應用價
值（簡要敘述成果所代表之意義、價值、影響或進一步發展之可能性）
、是否適
合在學術期刊發表或申請專利、主要發現或其他有關價值等，作一綜合評估。
1. 請就研究內容與原計畫相符程度、達成預期目標情況作一綜合評估
█
達成目標
□ 未達成目標（請說明，以 100 字為限）
□ 實驗失敗
□ 因故實驗中斷
□ 其他原因
說明：
The purpose of this project is thus to investigate how Chinese speakers gesture and
conceptualize motion events with linguistic representations in narration and conversation. This project consists of two parts. The first part focuses on the linguistic and imagistic representations of motion events. All the motion events and motion-event gestures in both narrative and
conversational discourse will be identified and analyzed in relation to (1) motion-event
components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. The second part of this project is to discuss the
conceptualization of motion events, based on the use of language and gesture in speaking, as suggested by the findings in the first part of the project.
In the first year, both the speech transcription and the gestural analysis in the transcripts were checked and revised. At the same time, all the motion events and motion-event gestures in
narrative discourse were identified and analyzed in relation to (1) motion-event components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. First, there are five components of a prototypical motion event (Talmy 1975), namely ‘motion’, ‘manner’, ‘path’, ‘figure’ and ‘ground’. They can be linguistically expressed in different ways with “combinations of lexical items and grammatical morphemes in various construction types” (Slobin 2004:220). Then, five types of gestures will be distinguished (Chui 2002): iconic gestures whose meanings correspond to the semantic content of the related speech, metaphoric gestures for abstract ideas, deictic gestures pointing at referents in the immediate speech environment, spatial gestures manipulating the gesture space to depict a spatial relation between the linguistic constituents or between the speaker and the linguistic constituent, and beats indicating the rhythm of speech. Concerning synchronization of speech and gesture, I examined whether the stroke phase of a motion-event gesture occurs at the time the associated linguistic expression is uttered. Finally, a binary given-new distinction were used to specify the information state of the gesture-associated lexical expressions.
3
speech transcription and the gestural analysis in the transcripts continued. On the other hand, motion-event gestures in conversational discourse were identified and analyzed with respect to (1) motion-event components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. The analyses were counted and tabulated using statistics. Then, three issues were discussed: (1) What is the most preferred type of linguistic representation of each motion-event component? (2) How do Chinese speakers gesture the five motion-event components? (3) What is the relationship between the preferred types of linguistic representation and motion-event gestures?
In the third year, checking and revising the transcripts continued. Also, the large amount of data analyses finished in the first and second years was further be counted and tabulated using statistics. Then, based on the analyses and statistics, I examined the theme of the
project—linguistic and imagistic representations of motion events and conceptualization. Another six issues were discussed: (1) Do motion-event gestures tend to synchronize with associated lexical expressions? (2) Do motion-event gestures tend to carry new information? (3) Do the findings differ across conversational and narrative discourse? (4) How do the findings suggest the conceptualization of motion events? (5) Are there any differences among Chinese, English, and Spanish in gesturing and conceptualizing motion events? (6) How do the findings and
cross-linguistic comparisons indicate the interaction among language typology, gesture, and thought?
To summarize, most of the narrative and conversational transcripts were checked and revised. The motion events and motion-event gestures were identified and analyzed. The conversational and narrative data analyses were counted and tabulated using statistics. With the analyses,
findings, and statistics, it was found that the linguistic-imagistic representation of motion events in Chinese differs from that in other languages. The past studies showed that
The findings of the project have also been presented in international conferences and published in international journals. The graduate students who participated in the project have been trained to transcribe spoken data and gestures. They know more about spoken grammar and gesturing in speaking.
MANNER and PATH in English are mentioned within one clause, and the two components can be represented together in one gesture. In Turkish and Japanese, they are expressed separately in two clauses, and two separate gestures – one for MANNER and one for PATH - are produced accordingly. As to Chinese speakers, they predominantly use simple manner verbs to express MANNER and serial verbs and prepositional phrases to convey PATH within a clause. In imagistic representation, speakers prefer to depict path information only, be it carrying new or given information. The cross-linguistic differences demonstrate language specificity in linguistic encodings and manual depictions of motion. Such linguistic-imagistic variation can further suggest language specificity in the conceptualization of motion events.
4
2. 研究成果在學術期刊發表或申請專利等情形：
論文：█已發表 □未發表之文稿 □撰寫中 □無
Chui, Kawai. 2008. Imagistic representation of motion events. The Sixth International
Conference of Cognitive Science (ICCS 2008), Yonsei University, Seoul, Korea, July 27 - 29, 2008.
Chui, Kawai. 2009. Linguistic and imagistic representations of motion events. Journal of Pragmatics 41(9): 1767-1777. (SSCI, AHCI)
Chui, Kawai. 2010. Gesture and Frame Knowledge. The 4th Conference of the International Society for Gesture Studies (ISGS), European University Viadrina Frankfurt/Oder, Germany, July 25 - 30, 2010.
Chui, Kawai. 2011. Do gestures compensate for the omission of motion expression in speech? Chinese Language and Discourse 2(2): 153-167.
Chui, Kawai. 2011. Gesture and embodiment in discourse. The 5th Conference on Language, Discourse and Cognition (CLDC 2011), National Taiwan University, Taipei, April 29 – May 1, 2011.
Chui, Kawai. 2011. Conceptual metonymies in gesture. The 11th International Cognitive
Linguistics Conference (ICLC 11), Xi’an International Studies University, China, July 11 - 17, 2011.
Chui, Kawai. 2011. Language typology and linguistic-gestural conceptualization of motion event. The 9th Biennial Conference of the Association for Linguistic Typology (ALT 9), University of
Hong Kong, Hong Kong, China, July 21 - 24, 2011.
Chui, Kawai. (under review). Cross-Linguistic representation of motion in language and gesture.
3. 請依學術成就、技術創新、社會影響等方面，評估研究成果之學術或應用價
值（簡要敘述成果所代表之意義、價值、影響或進一步發展之可能性）（以
500 字為限）
Based upon the quantitative-qualitative analysis of a large corpus of narrative and conversational data, the study in this project has shown the general tendency of producing motion-event gestures, which represents Chinese speakers’ common gesturing behaviors in speaking. Language and gesture together reflect the conceptualization of motion events in gesture-utterance production. The present study also helps understand the intricate relationship among gesture, language, use, and cognition. Finally, it provides a new aspect for the study of linguistic issues vis-à-vis gesturing in real discourse, and helps formulate a computational model of gesture-speech performance.
Linguistic and imagistic representations of motion events
Kawai Chui*
National Chengchi University, P.O. Box 1-322, Mucha, Taipei 116, Taiwan Received 3 August 2008; accepted 20 April 2009
Abstract
This study investigates the linguistic and imagistic representations of motion events in Chinese discourse. First, manner is most usually conveyed and in the form of single manner verbs, but information of this type is rarely gestured. Second, speakers also mention path very often by means of manner-path-deictic verbs and prepositional phrases indicating location, source, and goal. Speakers more commonly gesture path in contrast to manner. Similar results can be found in English, showing that the way people gesture motion events does not have to do with linguistic typology. However, in a single gesture, while manner and path can be conveyed simultaneously in Chinese and English, manner-ground gestures are found in Spanish. Such difference suggests various conceptualizations of motion event across different languages. Finally, gesturing the reference object and the moving object, being linguistically represented by nominal phrases, has to do with new information.
# 2009 Elsevier B.V. All rights reserved.
Keywords: Gesture; Motion event; Imagistic representation; Linguistic representation; New information
1. Introduction
A prototypical motion event ‘‘consists of one object (the ‘Figure’) moving or located with respect to another object (the reference-object or ‘Ground’). . . the ‘Path’. . .is the course followed or site occupied by the Figure object with respect to the Ground object. ‘Motion’. . .refers to the presence per se in the event of motion or location. . .. In addition to these internal components a Motion event can have a ‘Manner’ or a ‘Cause’’’ (Talmy, 1985:61). Consider the event in Example (1). The characters are running to an eating place. The figure refers to a threesome, namely a bull, Mickey, and Pluto; the path is indicated by dao ‘to’, and the ground is chifan de defang ‘the eating place’. The motion and the manner are realized simultaneously by the verb pao ‘to run’.
(1) W: ..lingwai san zhi. . . gong de niu haiyou. . . miqi haiyou bulutuo. . . jiu. . . another three CL male ASSC cow and Mickey and Pluto then pao dao. . . yecan. . . de. . . chifan de difang
run to picnic ASSC eat ASSC place
W: ‘As to another three, i.e., the bull and Mickey and Pluto, then ran to. . .picnic. . .the eating place.’ ‘‘The language of motion events is a system used to specify the motion of objects through space with respect to other objects’’ (Huang and Tanangkingsing, 2005:336). Then, how do different languages express motion events www.elsevier.com/locate/pragma
Available online at www.sciencedirect.com
Journal of Pragmatics 41 (2009) 1767–1777
* Tel.: +886 2 29393091x88088. E-mail address:[email protected].
0378-2166/$ – see front matter # 2009 Elsevier B.V. All rights reserved.
linguistically?Talmy (1985)regards Chinese and English as satellite-framed languages, since they have a class of verbs that incorporate the motion and manner information, whereas path is an adjunct; Spanish is a verb-framed language, in that motion and path are lexicalized as a single verb, while manner is an adjunct. Based on these different lexicalization patterns and on the further distinction between English being predicate-prominent and Chinese being topic-prominent, McNeill and Duncan (2000) investigated motion-event gestures across English, Spanish, and Chinese. They then made a claim about the relationship among language typology, gesture, and thought: The gestural differences among the three languages are the results of different forms of thinking being realized in their typologically distinct linguistic structures. This study will show that McNeill and Duncan’s claim is problematic. It then investigates the linguistic and imagistic representations of motion events in Chinese, and discusses how the results can shed light on cross-linguistic conceptualizations of motion events.
The next section discusses the motion-event gestures inMcNeill and Duncan’s (2000)study. Section3examines the linguistic and imagistic representations of various motion-event components in Chinese discourse. The last section provides a discussion.
2. McNeill and Duncan’s (2000)cross-linguistic study
McNeill and Duncan (2000)in their cross-linguistic study of motion-event gestures find typological differences in both linguistic and imagistic representations of motion events in English, Spanish, and Mandarin, suggesting various forms of thinking-for-speaking. According to the study, English speakers gesture manner when it is a focus. The stroke phase also synchronizes with the manner verb. To downplay manner, the gesture does not convey manner information, but synchronizes with the path or ground words. Spanish speakers, on the other hand, use path verbs. But they often convey manner in their gestures along with path and/or ground. Finally, Mandarin speakers tend to gesture motion-event components at the utterance-initial position, prior to the production of affiliated words, to form a topic frame. There are two major problems in McNeill and Duncan’s study. The first has to do with the synchronization of speech and gesture. According to McNeill and Duncan, whether the obligatory stroke phase of a gesture synchronizes with the manner information in English and Spanish rests upon the notion of focus.
In English, gesture and verb jointly highlight manner when it is part of the speaker’s focus. When manner is not in focus, gesture does not encode it and need not synchronize with a manner verb, even if one is present. . ..Manner appears in Spanish speech presumably only when it is a focused component, and it is often omitted even when it is potentially significant. (McNeill and Duncan, 2000:151–152)
They provide the following two examples that describe Sylvester being rolled down a drainspout by a ball. In the first utterance (a), a wiggling-hand gesture is produced to synchronize with the verb rolls in the utterance, since the focus is on the rolling manner of the ball. In the second utterance (b) when the focus is not on manner, the speaker’s hand plunges straight down to convey the path information only. The gesture lacks manner and does not synchronize with rolls, but with the path down and the ground drainspout instead.
(a) [but it rolls] him out
(b) [and he rolls. . . down the drainspout]
The problem of the analysis lies in the lack of an independent way of establishing what the ‘‘focus’’ is. In their study, the decision as to what is and what is not a ‘‘focus’’ rests upon the identification of the motion-event component being expressed in gesture, and whether the gestural stroke synchronizes with speech. The whole argument thus becomes circular.
The other problem in McNeill and Duncan’s study is concerned with the relation between language and gesture. Given that Chinese is a topic-prominent language,McNeill and Duncan (2000:152)state that the timing pattern of Chinese gestures resembles the topicalizing structure in Chinese, in that ‘‘the gesture shifts forward in the surface speech stream, in the direction of the utterance-initial position characteristic of topic statements in Chinese speech.’’ This claim is not borne out, since as many as 51% of motion-event gestures (78 out of a total 153) in our Chinese data synchronize with, rather than precede, their co-expressed words. See Example (2). The speaker is describing the way the bull attempts to jump into the sea. The speaker’s right hand, which is high above the head, at the time the associated verb tiao ‘jump’ is produced, descends onto the thigh to signify the motion of jumping (seeFig. 1).
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1768
(2) A: ..mini gen.. miqi zai jianghua.. er ta pangbian de na zhi Minnie with Mickey PROG talk and 3SG side ASSC that CL ..gongniu jiu jiao tamen kan guolai
bull then request 3PL look come here B: ..mhm
BC
A: ..ranhou jiu yi fu hen shenyongde yangzi zheyang. . . zhangkai.. haoxiang then then one CL very brave look like this stretch out look like tiaosanzhuang nayang tiao xiaqu
like parachuting like that jump down
tiao ‘to jump’: right hand above the head (pic.1 inFig. 1) descends onto thigh (pics.2–3 inFig. 1)
A: ‘Minnie was talking with Mickey. The bull beside her then requested them to look at him.’ B: ‘Mhm.’
Z: ‘Then it, looking very brave, stretched out and jumped down, as if he were parachuting.’
Even when the gestural strokes come before the associated words, they do not necessarily function as a ‘sentence-topic’.Chui (2005)finds that strokes coming before the associated words can function to signal that the upcoming new information is noteworthy and deserves attention. In short, the relationship among language typology, gesture, and thought proposed byMcNeill and Duncan (2000)cannot be maintained.
3. Gesturing motion events in Chinese discourse
How do Chinese speakers actually gesture motion events in discourse? This question will be addressed in this section with reference to both linguistic and imagistic representations of various motion-event components.
The database consists of 10 short oral narratives produced by undergraduate students of National Chengchi University in 2002. Each subject viewed a cartoon episode of the Mickey Mouse and Friends series. The soundtrack of the cartoon included music and only a very small amount of dialogue. In the episode, Mickey, Minnie, Pluto and a bull are holding a party at the beach, and eating and playing around. Then, they have a fight with an octopus, which they finally win. After viewing the cartoon, the subject immediately recounted the story from memory to a listener. The subject was filmed by a video camera so that speech and manual movements would be recorded. The subjects were not informed about our particular research interests. The elicited cartoon narrations ranged from 3 to 10 min in length. All of the verbs characterizing motion events in the 10 narratives were tabulated, totaling 245 instances. The gestures examined in the present study are the idiosyncratic spontaneous movements of hands and arms which accompany a speech event with context-dependent meaning and use. They take up 62.4% (153 instances) of all the data.
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1769
Talmy (1985)has categorized Mandarin and English as S-languages. Both languages have a large lexicon of motion-manner verbs. Examples from Chinese are zoulu ‘walk’, pao ‘run’, pa ‘crawl’, and fei ‘fly’. Prepositional phrases also delineate paths in relation to ground elements, as in the following examples (c) and (d):
(c) English: He ran out [of the house]pp.
(d) Chinese: ta pao [dao chifan de difang]pp
3SG run to eat ASSC place ‘He ran to the eating place.’
Slobin (2000)rather regards Chinese as a serial-verb language. Each verb in a series is morphologically unmarked and monosyllabic, such as fei - chu ‘fly exit’. ‘‘Manner is not syntactically subordinated to path. . .[b]ecause the path verbs can occur alone’’ (Slobin, 2000:228).Huang and Tanangkingsing (2005)show that Mandarin is a strongly verb-serializing language, based on a total of 153 Mandarin motion clauses, and 48.4% use the type M(anner)#P(ath)#D(irection) for describing motion event. The present study also considers Chinese as a serial-verb language because its idiosyncratic morphological patterning of motion-event components suggests a typological difference between Mandarin and other languages.
Table 1shows the frequency distribution of all the 153 gestures across the various semantic components of a motion event. These various types of motion-event gestures will be discussed accordingly in the following sub-sections. 3.1. Manner
First, what are the linguistic representations of manner of motion in Chinese? ‘‘A language provides its speakers with a range of ways of describing motion events – combinations of lexical items and grammatical morphemes in various construction types’’ (Slobin, 2004:220). The various lexical forms to convey manner are: manner verbs like you ‘swim’; manner-path verbs like pao-hui ‘run-back’; manner-deictic verbs like pao-lai ‘run-come’; manner-path-deictic verbs like reng-guo-qu ‘throw-across-go’. At the grammatical level, adverbial and phrasal expressions outside the verbs also ‘‘add information about such dimensions as suddenness, rate, force dynamics, inner state, terrain, and so forth – that is, information about factors that suggest manner of movement’’ (Slobin, 2004:232) In our data, 31 instances are of this type, such as the adverbial phrase tai yuan ‘too far away’ modifying the throwing motion in Example (3).
(3) F1: ..ranhou yinwei. . . na ge miao diu tai yuan.. diu dao then because that CL anchor throw too far away throw to F2: (0) mhm
BC F1: ..hai limian
sea inside
F1: ‘Then because the anchor was thrown too far away, it was thrown to,’ F2: ‘Mhm.’
F1: ‘into the sea.’
Of all the 245 motion events in the database, 200 (81.6%) use manner verbs of various types. Similar results can be found inHuang and Tanangkingsing (2005)with a distinct preference for manner verbs, at 83.4%.Table 2, which contains the 31 instances conveying manner without using manner verbs, demonstrates that single manner verbs (147 instances) are the habitual linguistic expressions of manner in Chinese discourse, at 63.6%.
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1770
Table 1
Various types of gestures.
Manner gestures Path gestures Ground gestures Figure gestures Manner and path gestures Total
5 112 14 15 7 153
Then, what are the imagistic representations of manner of motion? In Example (4), when the speaker verbalizes the nominal yangshi ‘backstroke’, about Mickey Mouse swimming backstroke in the sea, both of her hands, facing down, paddle outward two times to signify swimming (seeFig. 2).
(4) F: . . . houlai. . .(1.5)houlai <L2 mickey L2> hoaxiang shi you. . . yangshi ba later later Mickey seem COP swim backstroke PRT
you yangshi ‘swim backstroke’: both hands are on knees (pic.1 inFig. 2), then right hand and two fingers of left hand paddle outward one time (pic.2 inFig. 2), then right hand paddles outward one more time (pic.3 inFig. 2) and comes back (pic.4 inFig. 2). F: ‘Later, later, it seems that Mickey swam backstroke.’
90.2% (138 instances) of all the 153 motion-event gestures that accompany motion events were found to occur in clauses with a manner verb. Nevertheless, just twelve manner gestures (8.7%) were produced (seven instances depict path simultaneously), despite the prevalence of manner. Moreover, McNeill and Levy (1993:365) suggest that ‘‘gestures tend to occur at points of topic shift, such as new narrative episodes or new conversational themes. . .highly presupposed linguistic elements would either lack gestures entirely, or would be accompanied by gestures that are specialized for their cohesive function or form.’’Chui (2005)also demonstrates that Chinese speakers tend not to gesture for old information. A question thus arises: Is the scarcity of manner gestures related to information state? ‘New information’ is what has not been brought up in the previous context at the moment of speaking; ‘given information’ is what has already been mentioned at the moment of utterance.Table 3presents the frequency distribution of given and new information across the various types of manner verbs. The overall proportion of given to new information is about equal, suggesting that speakers are not likely to produce manner gestures even though the information is new. 3.2. Path
Table 1shows that it is very common (73.2%) to convey path information via the imagistic modality. This section investigates the linguistic and imagistic expressions of path. First, just like manner, there are various lexical forms to convey path: path verbs like hui ‘return’; deictic verbs like que ‘go’; path-deictic verbs like chu-lai ‘out-come’; K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1771
Fig. 2. Manner gesture: depiction of the backstroke style of swimming. Table 2
Linguistic representations of manner.
Manner verb Manner-path verb Manner-deictic verb Manner-path-deictic verb Manner outside the verb Total
147 7 4 42 31 231
Table 3
Information state and manner.
Manner verb Manner-path verb Manner-deictic verb Manner-path-deictic verb Total
Given 57 3 2 12 74 53.6%
New 45 2 1 16 64 46.4%
manner-path verbs like pao-hui ‘run-back’; manner-deictic verbs like pao-lai ‘run-come’, and manner-path-deictic verbs like reng-guo-qu ‘throw-across-go’. A deictic word can also come before a motion-manner verb, just like zhui ‘chase after’ in Example (5), or after a prepositional phrase, such as dao hai limian ‘into the sea’ in Example (6). (5) M: . . .bulutuo. . .jiu yibian. . . ta ye yibian
Pluto then on the one hand 3SG also on the one hand
qu zhui a
go chase after PRT
M: ‘Pluto, then, on the one hand, he also went to chase after the sausages.’ (6) M: . . .na zuihou ta jiu.. diu yi chuan xiangchang deshihou
then at last 3SG then throw one string sausage when . . .yinwei diu tai yuan .. diao dao hai limian qu
because throw too far away fall to sea inside go
M: ‘Then, at last, when it, then, threw a string of sausages, because it was thrown too far away, it fell into the sea.’
There are 161 motion events in the database that consist of the path component.Table 4indicates the various linguistic forms conveying path information. Manner-path-deictic verbs (26.7%) and prepositional phrases indicating location, source, and goal (46.6%) are the most frequently used linguistic patterns when speakers talk about path. Next, how is the path of a motion gesturally expressed? Example (7) is about Mickey and Minnie running to the beach. The speaker’s right hand, dangling at waist level on the right side, starts moving leftward at the time the first pao ‘run’ is uttered to depict a route. When she produces the fifth pao, her right hand in the central periphery moves downward to signify the endpoint of the path (seeFig. 3).
(7) F: (0) ranhou tamen jiu. . .(.9) pao pao pao pao pao.. pao dao yi ge haitan then 3PL then run run run run run run to one CL beach
pao ‘run’: right hand on the right (pic.1 inFig. 3) raises to waist level (pic.2 inFig. 3), and moves to central periphery (pic.3 inFig. 3).
F: ‘Then, they ran and ran and ran and ran and ran. They ran to a beach.’ K. Chui / Journal of Pragmatics 41 (2009) 1767–1777
1772 Table 4
Linguistic representations of path.
Path verb Deictic verb Path-deictic verb Manner-path verb Manner-deictic verb
2 10 12 7 7
Manner-path-deictic verb Deictic + manner verb PP PP + deictic Total
43 5 67 8 161
Ninety-one gestures (56.5%) were produced out of a total 161 motion events conveying path information, far outnumbering manner gestures by eighteen to one. Again, are such high occurrences related to information state?
Table 5, similar to the result inTable 3, illustrates that ‘given’ and ‘new’ are almost equally distributed among all the 91 path gestures. Thus, information state does not play a role in the imagistic representations of either path or manner of motion.
Moreover, 21 path gestures were found to accompany pure manner verbs without path information in speech. The frequency distribution of givens and news are about the same: 10 givens (47.6%) and 11 news (52.4%). Finally, concerning the synchronization of path gestures and associated speech, 17 instances were produced prior to the verbs; one was produced after the path constituents. Forty instances (35.7%) synchronize with the path constituents. What is intriguing is that 48.2% (54 instances) of all the 112 path strokes synchronize with the manner verbs rather than with the path words. This will be brought up again in section4.
3.3. Ground and figure
The reference object and the moving object of a motion event are linguistically represented by nominal expressions, such as ye ge zhuzi ‘a pillar’ in Example (8) and the third person pronominal ta in Example (9), respectively. They can also be expressed manually. The speaker depicts the round shape of the pillar to gesture the ground component of the colliding event in Example (8) with a circle made by the thumb and index finger of his left hand at the moment of verbalizing the motion-manner verb zhuangdao ‘collide’ (seeFig. 4).
(8) C: . . . na zhi niu. . . jiu yong na ge. . . x zuo de jiushengquan that CL bull then use that CL REPAIR make ASSC life belt jiu zai hai. . .(2.)hai shang piao a
then on sea sea on float PRT B: ..mm
BC
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1773 Table 5
Information state and path.
Path verb Deictic verb Path-deictic verb Manner-path verb Manner-deictic verb
Given 0 0 0 3 2
New 1 3 8 2 0
1 3 8 5 2
Manner-path-deictic verb Deictic + manner verb PP PP + deictic Total
Given 10 1 24 4 44 58.4%
New 14 1 15 3 47 51.6%
24 2 39 7 91 100.0%
C: ..ranhou piao yi piao. . .jiu. . . zhuangdao yi ge zhuzi then float one float then collide:RESULT one CL pillar
zhuzi ‘pillar’: thumb and index finger of left hand raises from thigh (pic.1 inFig. 4) and form a circle (pic.2 inFig. 4), and then return to thigh (pic.3 inFig. 4).
C: ‘That bull then used a life belt and on the sea. . .floated on the sea.’ B: ‘Mm.’
C: ‘Then, it floated for a while, then it collided with a pillar.’
Example (9) describes a horse using watermelon seeds as weapon to fight against an octopus. The speaker gestures the figure component, i.e., the watermelons seeds, bouncing off a tea-kettle at the side. The thumb and the index finger of the speaker’s left hand form a circle when the pronominal subject ta, which refers to watermelon seeds as a group, is uttered to represent the shape of the watermelons seeds (seeFig. 5).
(9) F: (0) dui ta chi xigua. . . ranhou jiushi tu zi right 3SG eat watermelon then that is spit out seed B: ..heh
BC
F: . . . ranhou jiu.. yizhi tu.. ranhou. . .you. . . ta jiu.. you then then continuously spit out then PRF 3PL then PRF tan dao.. pangbian de nage.. shenme shuihu a.. shenme de bound to side ASSC that whatever water bottle PRT whatever PRT ta ‘they’: thumb and index finger of her left hand form a circle (pic.1 inFig. 5).
tan ‘bound’: the circle gesture raises to waist level (pic.2 inFig. 5) and sweeps down (pic.3 inFig. 5). F: ‘Right, it (i.e., the horse) was eating watermelon. Then, that is, it spit out the seeds.’
B: ‘Heh.’
F: ‘Then, it kept spitting out seeds. Then, they (i.e., the seeds) bounded against the side of a water bottle, or whatever.’
Reference objects and moving objects are not frequently brought up, unlike manner and path, in that just 53 motion events (53 out of 245, 21.6%) include ground information and 67 motion events (67 out of 245, 27.3%) have a figure component. Nevertheless, the number of their respective gestural occurrences is much higher than that of manner gestures. The speakers produced 14 ground gestures (out of 53, 26.4%) and 15 figure gestures (out of 67, 22.4%) in the corpus.Table 6recapitulates the proportions of the total number of motion events including a particular motion-event component to the total number of gestures for that particular component. Gestures for reference objects and moving objects outnumber those for manner by about five times.
Finally, different from manner and path, the occurrences of ground and figure gestures are more likely to be constrained by information state: 71.4% (10 instances) of the reference objects and 73.3% (11 instances) of the moving objects carry new information.
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1774
4. Discussion and conclusion
Based on the findings in the previous section, this section will discuss the relationship among language, gesture, and motion-event conceptualization. It was first addressed by McNeill and Duncan (2000), who claim that English, Spanish, and Chinese are typologically different languages and that the speakers also gesture motion events in different ways because they have their own respective conceptualization (see the detailed discussion in section 2). Their claim is subject to Slobin’s (1987) ‘‘thinking-for-speaking’’ hypothesis. This hypothesis refers to how speakers organize their thinking to meet the demands of linguistic encoding on-line, during acts of speaking. As speakers are thinking ‘‘in terms of a combination of imagery and linguistic categorical content’’ (McNeill and Duncan, 2000:142), their conceptualization of an event can thus be realized simultaneously in speech and gesture.
This study has investigated how Chinese speakers produce manual movements while talking about motion events. Our findings do not bear outMcNeill and Duncan’s (2000)analysis of ‘‘thinking-for-speaking’’ for Mandarin, since the timing pattern of Chinese gestures by no means resembles the topic-comment patterning of the language. Thus, their claim about the relationship among language typology, gesture, and thought cannot be maintained. Nevertheless, we agree that conceptualization of an event includes both imagery and linguistic content, which can be realized simultaneously in speech and gesture, respectively. Thus, the linguistic and imagistic representations of the motion-event components shed light on how motion events are conceptualized by Chinese speakers. The conceptualization is also embodied in daily social interaction. The findings of this study thus represent the common experience of Chinese speakers talking about motion events. First, manner is most usually conveyed and in the form of single manner verbs, but information of this type is rarely gestured. Second, speakers also mention path very often by means of manner-path-deictic verbs and prepositional phrases indicating location, source, and goal. Speakers more commonly gesture path in contrast to manner. Similar results can be found in English: ‘‘Adults, when they describe such motion events, typically produce gestures showing only path. . .or gestures showing in a single gesture both manner and path. . .. Manner without path, however, rarely occurs’’ (McNeill, 2005:185). Since Mandarin and English are typologically different languages, their similarities in gestures for motion events evidence that the imagistic representations of motion events do not have to do with linguistic typology. Gestures enable the speakers to go beyond the structural restriction of language and convey their own thinking about a motion event in verbal communication.
In a single gesture, while manner and path can be conveyed simultaneously in Chinese and English,1manner-ground gestures are found in Spanish (McNeill and Duncan, 2000). Such difference seems to suggest various conceptualizations of motion events across different languages. In Chinese, to incorporate more than one motion-event component in a single gesture is rare. Moreover, the high occurrences of path gestures, despite the fact that speakers often mention manner and path in speech, evidence that ‘path’ is the most salient component in motion-event conceptualization.
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1775 Table 6
Motion-event components and their respective occurrences of gesture.
Motion events including manner Manner gestures Proportion
245 12 4.9%
Motion events including path Path gestures Proportion
161 119 73.9%
Motion events including ground Ground gestures Proportion
53 14 26.4%
Motion events including figure Figure gestures Proportion
67 15 22.4%
1
Since frequency distribution is not provided inMcNeill and Duncan’s (2000)study, whether there is any quantitative difference between Mandarin and English concerning the single gestures conveying both manner and path simultaneously awaits future research.
Finally, gesturing the reference object and the moving object, being linguistically represented by nominal phrases, has to do with information state, in that ground and figure gestures usually convey new information. Conceptualization thus has to consider the flow of information in discourse.
Acknowledgments
This research was funded by grants from the National Science Council (NSC 97-2410-H-004-111-MY3). I would also like to thank the referees for offering valuable comments and suggestions. All errors of interpretation are my own responsibility.
Appendix A. Gesture and speech transcription conventions Transcription of speech [ ] speech overlap . . .(N) long pause . . . medium pause .. short pause (0) latching @ laughter
TSK a click of the tongue <L2 L2> code-switch to English Transcription of gesture
For the representation of gesture in examples, the underlined part of the utterance is the stroke phase (and the hold phase, if there is any); the lexical affiliate(s), if there is any, is/are in boldface. The description of gesture is given under the line of associated speech. In each gestural description, the word(s) before the colon represent(s) the referent a gesture refers to; the description of the manual movement comes after the colon.
Abbreviations of linguistic terms 1PL first person plural 1SG first person singular 2PL second person plural 2SG second person singular 3PL third person plural 3SG third person singular ASSC associative morpheme BA the morpheme BA BC backchannel CL classifier COP copula verb NEG negative morpheme PF pause filler
PRF perfective aspect PROG progressive aspect PRT discourse particle QST question particle REPAIR repair phoneme(s) SELF reflexive morpheme RESULT resultative morpheme
K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1776
References
Chui, Kawai, 2005. Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics 37 (6), 871–887. Huang, Shuanfan, Tanangkingsing, Michael, 2005. Reference to motion events in six western Austronesian languages: toward a semantic typology.
Oceanic Linguistics 44, 307–340.
McNeill, David, Levy, E.T., 1993. Cohesion and gesture. Discourse Processes 16 (4), 363–386.
McNeill, David, Duncan, S., 2000. Growth points in thinking-for-speaking. In: McNeill, David (Ed.), Language and Culture. Cambridge University Press, Cambridge, pp. 141–161.
McNeill, David, 2005. Gesture and Thought. University of Chicago Press, Chicago.
Slobin, D., 1987. Thinking for speaking. In: Aske, J., Beery, N., Michaelis, L., Filip, H. (Eds.), Proceedings of the 13th Annual Meeting of the Berkeley Linguistics Society. Berkeley Linguistics Society, Berkeley, CA, pp. 435–445.
Slobin, D., 2000. Verbalized events: a dynamic approach to linguistic relativity and determinism. In: Niemeier, S., Driven, R. (Eds.), Evidence for Linguistic Relativity. John Benjamins Publishing, Amsterdam, pp. 107–138.
Slobin, D., 2004. The many ways to search for a frog: linguistic typology and the expression of motion events. In: Stro¨mqvist, S., Verhoeven, L. (Eds.), Relating Events in Narrative: Typological and Contextual Perspectives. Lawrence Erlbaum Associates, Mahwah, NJ, pp. 219–257. Talmy, L., 1985. Lexicalization patterns: semantic structure in lexical forms. In: Shopen, T. (Ed.), Language Typology and Syntactic Description,
vol. 3. Cambridge University Press, Cambridge, pp. 57–149.
Kawai Chuiis Professor of Linguistics at National Chengchi University, Taipei. She has been working to build a corpus of spoken Chinese with gestural analysis. Her research interests include gesture and speech communication, and the discourse basis of grammar.
Chinese Language and Discourse 2:2 (2011), 153–167. DOI 10.1075/cld.2.2.01chu
ISSN 1877‑7031 / E‑ISSN 1877‑8798 © John Benjamins Publishing Company
Do gestures compensate for the omission of
motion expression in speech?*
Kawai Chui
National Chengchi University
The present study investigates whether and to what extent motion‑event gestures compensate for the omission of linguistic expression in Chinese discourse and across different languages to understand language‑specificity/language‑univer‑ sality and the coordination of motion information across the two modalities. The Chinese conversational and narrative data consistently show that manner fog (i.e., manner absent from speech but present in gesture) was not found. Chinese speakers also demonstrate a preference for compensation — gestures tend to compensate for the lack of path content in speaking. These results differ from those for English and Turkish which do not prefer path gestures in manner‑only clauses. The cross‑linguistic variation provides evidence for language specificity in gestural compensation. The language‑specific coordination of information in speech and gesture suggests Chinese speakers’ habitual focus of attention on PATH in multimodal communication.
Keywords: gestural compensation, motion event, gesture, linguistic‑imagistic
representation, cross‑linguistic representation
关键词：手势补偿、移动事件、手势、语言─图像表达、跨语言比较
1. Introduction
The use of hands and arms along with speech is indispensable and prevalent in multimodal communication (Goldin‑Meadow, 1999; McNeill, 1992, 2000; Ken‑ don, 2004). “The tremendous overlap between neural structures contributing to language and hand/arm movement may help to explain the prevalence of hand gesture in language” (Glenberg, 2007: 363). During speaking, gestures bear a wide variety of functions. They can, among others, facilitate speech production (Rime & Schiaratura, 1991) and help learning (Alibali & Goldin‑Meadow, 1993), lexical
154 Kawai Chui
retrieval (Krauss et al., 1996, 2000), problem‑solving (Alibali et al., 1999), remem‑ bering more (Goldin‑Meadow et al., 2001), organizing the speaker’s thinking for speaking (Özyürek & Kita, 1999; Alibali, Kita, & Young, 2000; Kita, 2000), pro‑ viding referential meaning (Kendon, 2004), constructing meaning in classroom activity (Singer et al., 2008), and accomplishing conversational coherence (Chui, 2009a).
When the speaker conveys a message, speech and gesture often work in col‑ laboration to express information. Melinger and Levelt’s (2004) experiment showed that speakers use gestures intentionally to convey part of their message. The messages represented by manual configurations can be of various kinds. Mc‑ Neill (1992) found that in narrative discourse iconic gestures and abstract pointing mainly express information contained in narrative clauses; beat gestures indicate the textual shift between the narrative and extranarrative levels; and metaphoric gestures depict information in extranarrative clauses. McNeill also noted that “[s] peech and gesture refer to the same event and are partially overlapping, but the pictures they present are different” (ibid.: 13). In Bavelas et al. (1992: 473) ‘topic gestures’ in conversation were found to enact information directly related to the topic of discourse; ‘interactive gestures’ were used to address to other participants and function “to aid the maintenance of conversation as a social system” in con‑ versation (Bavelas et al., 1992: 470). Some gestures of this kind had parallel verbal references but some did not. With respect to ‘communicative dynamism’, McNeill & Levy (1993) showed that more complex gestures would be produced along with more complex linguistic expressions to depict the information that functions to push the communication forward. Kendon (1995: 247) distinguished between ‘substantive gesturing’ and ‘pragmatic gesturing’. “[The former] contributes to various aspects of the content of the utterance of which it is a part, whether liter‑ ally or metaphorically… [and the latter] expresses aspects of utterance structure, including the status of discourse segments with respect to one another, and the character of the ‘speech act’ or interactional move of the utterance.” In Kendon (2004), he made a distinction between ‘gestures with equivalent verbal expres‑ sions’ and ‘gestures with a non‑matching verbal expression’ in the discussion of contributions that gestures can make to referential meaning. Based on Chinese conversational data, Chui (2008) investigated different kinds of information rep‑ resented by complementary gestures which provide additional meanings to enrich speech events or maintain the continuity of a topic under discussion. Finally, for children, Church & Goldin‑Meadow (1986) studied the mismatches between ges‑ ture and speech in children’s explanations of a concept. The ‘discordant’ children in their study “produced many explanations in which the information conveyed in speech did not match the information conveyed in gestures” (ibid: 43).
Do gestures compensate for the omission of motion expression in speech? 155
The previous studies mostly focused on the kinds of information that can be expressed by gestures. The question as to whether gesture may compensate for the omission of expression in speech was not a main concern. ‘Gestural compensation’ is distinguished from ‘complementarity’, a term used by McNeill (1992). His exam‑ ple was: At the moment the speaker utters she chases him out again, “speech con‑ veys the ideas of pursuit and recurrence while gesture conveys the weapon used (an umbrella)” (ibid.: 13). The form of the speaker’s hand in a shape as though to grip something is not associated with any lexical item, and the utterance is gram‑ matically complete without the gesture. Gestures of this type have also been re‑ ferred to as ‘supplementary’, ‘mismatching’, or ‘non‑redundant’ in other studies (as mentioned in Alibali et al., 2009: 291). They do not necessarily compensate for the absence of certain linguistic expression in speech; they can simply provide non‑ linguistic information via the gestural modality.
Gestural compensation has been mentioned in previous studies with regard to different languages, including Spanish (McNeill & Duncan, 2000), Turkish, Japa‑ nese, and English (Kita & Özyürek, 2003; Özyürek et al., 2005). However, these studies did not show the habitual imagistic representation used to compensate for speech in daily communication or about the ways in which the general lin‑ guistic‑gestural representation and imagistic compensatory representation work together in the coordination of information in speech and gesture. The present study will provide empirical evidence to discuss these two issues with respect to motion events, to understand if a speaker has produced a gesture for the omis‑ sion of linguistic expression and to understand how information is coordinated across the two modalities. The presence or absence in speech can be clearly de‑ termined as the components of motion‑events are (near‑)universal (Talmy, 1985). Cross‑linguistic comparison is also available based on a series of research into the linguistic‑imagistic expression of motion across languages. In this study, I first investigate whether and to what extent motion‑event gestures may compensate for the omission of linguistic expression. The findings in Chinese will be compared across typologically different languages so as to understand language‑specificity/ language‑universality and the coordination of motion information across the two modalities.
The next section introduces the data and preliminaries for the study. Section 3 provides a general representation of motion in speech and gesture in Chinese dis‑ course. Section 4 examines gestural compensation across different languages. Sec‑ tion 5 discusses the coordination of motion information in speech and gesture.
156 Kawai Chui
2. Data and preliminaries
The data for Chinese used in this study is from The NCCU (National Chengchi University) Corpus of Spoken Chinese. This corpus is part of an archive of lan‑ guage documentation which collects the spoken forms of Mandarin, Taiwanese, and Hakka in Taiwan (Chui & Lai, 2008). The sub‑corpus of spoken Mandarin contains short oral narratives and daily face‑to‑face conversations. The cartoon narrations were produced by twenty‑two NCCU undergraduate students in 2002. Each subject viewed a seven‑minute cartoon episode of the ‘Mickey Mouse and Friends’ series. The soundtrack of the cartoon included music and only a very small amount of dialogue. In the episode, Mickey, Minnie, Pluto and a bull are holding a party at the beach, and eating and playing around. They then have a fight with an octopus, which they finally win. The subject immediately recounted the story from memory to a listener after viewing the cartoon. The subject was filmed by a video camera so that speech and manual movements could be re‑ corded. The subjects were not informed about our particular research interests. The elicited cartoon narrations ranged from about two to ten minutes in length; the mean length of narration is four minutes thirty seconds. With regard to con‑ versations, there are two sets of data. The first set was collected during 1994 and 1995; the participants were college students who knew each other. The second set was casual conversations among family members, friends, and colleagues which have been videotaped since 2006, and this portion of data can be accessed online.1
All the participants were paid, and they were not told the particular focus of the research. The participants were free to find and develop topics of common inter‑ est; they were filmed for approximately an hour with a visible camera. One stretch from each talk, of about twenty to forty minutes, in which the participants were comfortable in front of the camera, was then selected for transcription. A further project related to The NCCU Corpus of Spoken Mandarin is a gestural analysis of the transcribed narratives and conversations. The data used in this study consist of ten complete narratives and seven conversational extracts (five from the first set and two from the second set) for a total of 183 minutes of talk. The same nar‑ rative data were used in the study of linguistic‑imagistic representation of motion in narrative discourse (Chui 2009b); however, because of different research issues, the statistics presented here do not totally accord with those in the former study.
For the purpose of the present study, the occurrence of manner and path ges‑ tures will be considered because the lexical‑syntactic packaging of MANNER and PATH constitutes a linguistic typology of motion, and the cross‑linguistic studies of the linguistic‑gestural representation of motion were mainly subject to these two components. The speech and the gesture data relevant for the present study were separately coded by two trained coders. The criterion to identify a motion
Do gestures compensate for the omission of motion expression in speech? 157
event was the linguistic description in the clause which consists of the main predi‑ cate and its argument(s), and/or gestural depiction of a protagonist of a move‑ ment from one place to another.2 The motion‑event gestures are the spontaneous
movements of hands and arms depicting motion components. In the case of dis‑ agreement between the coders, data were re‑analyzed and discussed. Data with‑ out consensus were not used. Agreement was reached for a total of 180 motion events encoded by a single VP in a clause including manner and/or path gestures in conversations and 124 instances in narratives.3 They form the database for the
investigation of gestural compensation during speaking.
3. The representation of motion in speech and gesture
A prototypical motion event “consists of one object (the ‘Figure’) moving or lo‑ cated with respect to another object (the reference‑object or ‘Ground’)… [and] the ‘Path’…is the course followed or site occupied by the Figure object with respect to the Ground object. ‘Motion’…refers to the presence per se in the event of motion or location” (Talmy, 1985: 61). Chinese is a verb‑serializing language in both spoken and written discourse (Slobin, 2000; Huang and Tanangkingsing, 2005; Chen and Guo, 2009, 2010; Chui, 2009b), a type of ‘equipollently‑framed’ languages where ‘‘both manner and path are expressed by ‘equipollent’ elements — that is, elements that are equal in formal linguistic terms, and appear to be equal in force or signifi‑ cance’’ (Slobin, 2004: 228). In Example (1) the last clause about Speaker F’s former classmate who had been walking by her side to attract her attention (Line 2) is concerned with a walking event. The covert subject is the FIGURE, the classmate referred to in the pronominal form tā in the preceding clause (Line 1); the noun phrase in the prepositional phrase — wŏ pángbiān ‘my side’ — is the GROUND; the main serial‑verb provides information about MOTION and MANNER in the form of zŏu ‘walk’, and about PATH and DIRECTION encoded by guò ‘go across’ and qù ‘go’, respectively (Line 2).
(1) 1 F: …(0.5) hěn qíguài o… tā méi cì… xiàkè jiù kāishǐ… very strange PRT 3SG every time after.class then start → 2 zŏu zŏu zŏu zŏu zŏu… a jiù cóng wŏ pángbiān
walk walk walk walk walk PRT just from 1SG side
zŏu‑guò‑qù
walk‑go.across‑go
F: ‘It’s very strange. Every time after class, he started to walk and walk and walk and walk and walk. He walked by my side.’
158 Kawai Chui
In addition to language, gesture also readily and frequently depicts various com‑ ponents of motion during speaking, as illustrated in Example (2). F1 is complain‑ ing about her past summer job at school when she had to keep running up and down stairs all day to make just one photocopy of one or two pieces of paper each time. The manner of the running event, encoded by the pro‑form zhèyàngzi ‘like this’, the adverbial yīzhí ‘continuously’, and the manner verb pǎo ‘run’ in Line 11, is depicted by the hands: At the time the numeral yī ‘one’ is verbalized, F1 starts raising her right hand with the fingers hanging down at waist level; her left hand also rises slightly. These movements prepare for the next running motion during the production of zhèyàngzi ‘like this’: F1 first moves the right hand toward her own body with the left hand flicking slightly outward. Then, both hands alternate the directions two more times successively till the first mention of pǎo has been uttered. These noticeable and discernable gestural configurations, being produced in front of the body for a comparatively long duration of 1.034‑seconds in total, enact the manner of running back and forth somewhere again and again, in this case to a photocopy machine.
(2) 1 F1: …(1.0) zhè yī fēnzhōng… jiào nĭ ná‑qù yìn yí fèn this one minute tell 2SG take‑go print one CL 2 F2: ..Mm
BC
3 F1: ..wŏmen shi pǎo dào yī lóu qù 1PL COP run to first floor go 4 F2: ..Mm
BC
5 F1: ..yìn‑wán le… húi‑lái yǐhòu print‑finish PRF return‑come after 6 F2: …(0.5) zài ná‑qù [yìn yí fèn] again take‑go print one CL
7 F1: [yòu guò méi‑duōjiǔ]… zài ná‑qù yìn…[[ zhè yí cì… again pass NEG‑so.long again take‑go print this one time bú shì yī fèn]]
NEG COP one CL 8 F2: [[o… shénjīngbìng]]
PRT nuts
9 F1: .. yí cì gěi ná gěi nĭ yī zhāng… huò liăng zhāng one time give take to 2SG one CL or two CL 10 F2: ..o [biàntài]
PRT sick
→ 11 F1: [ránhòu… yī tiān nĭ jiù] zhèyàngzi… yīzhí păo then one day 2SG just like.this continuously run
Do gestures compensate for the omission of motion expression in speech? 159
yīzhí păo
continuously run
12 at yī ‘one’, right hand rises with fingers handing down at waist level; left hand rises slightly ((a) to (b) in Figure 1)
13 from zhèyàngzi to first mention of păo, right hand moves toward own body; left hand flicks slightly outward; then right hand out and left hand in; then right hand in and left hand out ((c) to (e) in Figure 1)
14 at the second mention of yīzhí, both hands return to thighs ((f) in Figure 1) F1: ‘At that moment, she told you to take the document and make a copy.’ F2: ‘Mm.’
F1: ‘We had to go down to the first floor.’ F2: ‘Mm.’
F1: ‘After printing, after we had returned,’
F2: ‘She told you to take it and make another copy.’
F1: ‘after a short while, I had to take it to make a copy again. But this time I didn’t make a copy of the whole document,’
F2: ‘Oh, she’s nuts.’
F1: ‘each time she gave me just one or two pieces of paper.’ F2: ‘Oh…she’s sick.’
F1: ‘Then you had to run upstairs and downstairs continuously all day.’
7
F1: ‗At that moment, she told you to take the document and make a copy.‘ F2: ‗Mm.‘
F1: ‗We had to go down to the first floor.‘ F2: ‗Mm.‘
F1: ‗After printing, after we had returned,‘
F2: ‗She told you to take it and make another copy.‘
F1: ‗after a short while, I had to take it to make a copy again. But this time I didn‘t make a copy of the whole document,‘
F2: ‗Oh, she‘s nuts.‘
F1: ‗each time she gave me just one or two pieces of paper.‘ F2: ‗Oh...she‘s sick.‘
F1: ‗Then you had to run upstairs and downstairs continuously all day.‘
Figure 1. Gestural depiction of running continuously.
Table 1 shows the frequency distribution of manner gestures and path gestures with a total of 180 motion events with gestures in conversations and 124 in narratives. Based on these statistics, the next section will investigate the coordination of information in speech and gesture when one of the two components is omitted in speech. Does gesture occur to depict information which was not spoken?
Table 1. Gestures for MANNER and PATH
Conversations Narrations
I Path gestures 114 63.3% 112 90.3%
II Manner gestures 31 17.3% 5 4.0%
III Combined manner &
path gestures 35 19.4% 7 5.7%
Total: 180 100.0% 124 100.0%
4. Gestural compensation across languages
Gestural compensation has been brought up in previous studies. McNeill & Duncan (2000:
(a) Line 12 (b) Line 12 (d) Line 13
(e) Line 13 (f) Line 14
(c) Line 13
Figure 1. Gestural depiction of running continuously.
Table 1 shows the frequency distribution of manner gestures and path gestures with a total of 180 motion events with gestures in conversations and 124 in narratives.
160 Kawai Chui
Based on these statistics, the next section will investigate the coordination of in‑ formation in speech and gesture when one of the two components is omitted in speech. Does gesture occur to depict information which was not spoken?
Table 1. Gestures for MANNER and PATH
Conversations Narrations
I Path gestures 114 63.3% 112 90.3%
II Manner gestures 31 17.3% 5 4.0%
III Combined manner & path gestures 35 19.4% 7 5.7%
Total: 180 100.0% 124 100.0%
4. Gestural compensation across languages
Gestural compensation has been brought up in previous studies. McNeill & Dun‑ can (2000: 150) reported that “[a]lthough Spanish speakers often omit manner from their speech, manner is abundant in their gestures and combines with other linguistic categories, typically path (verb) and/or ground (nominal phrase).” Their study, however, did not provide quantitative data which could be used to com‑ pare their findings with those of other related studies, such as those based on the storytellings of the same Sylvester and Tweety cartoon or of a set of video clips depicting motion events (Özyürek & Kita, 1999; Kita & Özyürek, 2003; Özyürek et al., 2005; Kita et al., 2007). In Kita & Özyürek (2003), Turkish, Japanese, and English speakers were all found to gesture the lateral direction of a swing event and a rolling event that was not verbalized in their narrations. Özyürek et al.’s (2005: 236) experimental data produced by English and Turkish speakers, on the other hand, showed the general tendency that “gestural information was found to fit the semantic encoding of the event rather than compensate or convey mean‑ ing not expressed by speech.” In fact, manner gestures were still found in English and Turkish path‑only clauses (see Figure 2 in Özyürek et al., 2005: 233), and the percentages were even higher for the occurrence of path gestures in manner‑only clauses (see Figure 3 in Özyürek et al., 2005: 234). It is thus not conclusive whether motion‑event gestures compensate for the absence of linguistic representation across typologically different languages.
This section investigates whether there is a compensatory relationship to co‑ ordinate information across the two modalities in Chinese discourse. Example (3) illustrates a path gesture without linguistic expression of PATH produced by F1 while she is talking about a walking event. The scenario is: F1 was in a car with her colleague. As they were going downhill, F1 saw two of her students walking
Do gestures compensate for the omission of motion expression in speech? 161
together. The clause about the students consists of the FIGURE in pronominal form tāmen ‘they’, the quantifier phrase liăng ge ‘two’, and the simple manner verb
zǒulù ‘walk’ (Line 3). Whether the two students were walking uphill is not men‑
tioned in the utterance, but rather expressed in gesture. To prepare for the upward movement, F1, during the production of the verb xiàshān ‘go downhill’ (Line 1), raises her left hand to shoulder level while moving her right hand leftward and downward to depict the action of going downhill. Then, during the 1.044‑sec‑ ond utterance of the next clause in Line 3 about the students walking together, F1 moves her left hand rightward and upward to the central space with noticeable and discernable configurations, signifying the students going uphill.
(3) 1 F1: .. oh… dùi a jiù kāichē xiàshān na PRT right PRT then drive go.downhill PRT
2 at xiàshān, right hand moves leftward and downward; left hand rises to shoulder level ((a) in Figure 3)
→ 3 ..ránhòu tāmen liăng ge zǒulù then 3PL two CL walk
4 from ránhòu to zǒulù, left hand moves rightward and upward ((b) to (c) in Figure 3)
5 ..suǒyĭ wǒmen yíding shì zhèyàng xiàqù so 1PL must COP like.this go.down
6 at suǒyĭ, left hand starts moving leftward and downward ((d) in Figure 3)
F1: ‘Right, we drove downhill. Then, they both walked. So we must have been going down like this.’
9
2 at xiàshān, right hand moves leftward and downward; left hand rises to shoulder level ((a) in Figure 3) 3 ..ránhòu tāmen liăng ge zǒulù
then 3PL two CL walk
4 from ránhòu to zǒulù, left hand moves rightward and upward ((b) to (c) in Figure 3) 5 ..suǒyĭ wǒmen yíding shì zhèyàng xiàqù
so 1PL must COP like.this go.down
6 at suǒyĭ, left hand starts moving leftward and downward ((d) in Figure 3)
F1: ‗Right, we drove downhill. Then, they both walked. So we must have been going down like this.‘
Figure 2. Gestural depiction of upward movement.
Of all of the 180 clauses that include gestures for MANNER and/or PATH in the conversational data (Table 1), 89 (49.4%) of them were either manner-only or path-only clauses. Out of these 89 instances, gestural compensation constituted 42.7% (38 instances). This proportion does not agree with Özyürek et al.‘s (2005: 234) finding in English and Turkish that ―the information expressed both in gesture and speech showed strong parallels.‖ It is still likely, at least in Chinese, that motion components are represented by the manual modality exclusively. More importantly, there is a preference for compensation. First, no manner gestures occurred for all of the 25 path-only clauses without an expression of manner information either lexically or grammatically. In contrast, many more path gestures occurred in only clauses, at 59.4% (38 instances): Among the 64 manner-only clauses without the lexical and grammatical expression of PATH, 23 of which were single path gestures, like the upward direction of walking in Example (3), and 15 of which were manner-path conflated gestures. All these are truly compensatory gestures without path information in speech. In the narrative data, 63.7% of all the 124 clauses (79 instances) were either manner-only or path-only clauses. Gestural compensation occurred 26.6% of the time (21 out of a total of 79). Again, in spite of the quantitative difference, narrators preferred to gesture PATH, rather than MANNER, when it was absent in speech: Path gestures occurred in all of the 21 manner-only clauses without lexical-syntactic expression of PATH. As to the 58 path-only clauses without lexical-syntactic expression of MANNER, none included manner gestures.
(a) Line 2 (b) Line 4 (c) Line 4 (d) Line 6
Figure 2. Gestural depiction of upward movement.
Of all of the 180 clauses that include gestures for MANNER and/or PATH in the conversational data (Table 1), 89 (49.4%) of them were either manner‑only or path‑only clauses. Out of these 89 instances, gestural compensation consti‑ tuted 42.7% (38 instances). This proportion does not agree with Özyürek et al.’s (2005: 234) finding in English and Turkish that “the information expressed both in gesture and speech showed strong parallels.” It is still likely, at least in Chinese, that
162 Kawai Chui
motion components are represented by the manual modality exclusively. More im‑ portantly, there is a preference for compensation. First, no manner gestures oc‑ curred for all of the 25 path‑only clauses without an expression of manner infor‑ mation either lexically or grammatically. In contrast, many more path gestures occurred in manner‑only clauses, at 59.4% (38 instances): Among the 64 manner‑ only clauses without the lexical and grammatical expression of PATH, 23 of which were single path gestures, like the upward direction of walking in Example (3), and 15 of which were manner‑path conflated gestures. All these are truly compensa‑ tory gestures without path information in speech. In the narrative data, 63.7% of all the 124 clauses (79 instances) were either manner‑only or path‑only clauses. Gestural compensation occurred 26.6% of the time (21 out of a total of 79). Again, in spite of the quantitative difference, narrators preferred to gesture PATH, rather than MANNER, when it was absent in speech: Path gestures occurred in all of the 21 manner‑only clauses without lexical‑syntactic expression of PATH. As to the 58 path‑only clauses without lexical‑syntactic expression of MANNER, none included manner gestures.
The consistency in the results for the Chinese conversational and narrative data show that while MANNER is often omitted linguistically but compensated manually in Spanish (McNeill & Duncan, 2000), manner fog (i.e., manner absent from speech but present in gesture, see McNeill, 2005) was not found in Chinese. Nor do the results in Chinese agree with Özyürek et al.’s (2005: 233–4) claim, based on English and Turkish narrations, that
when speakers of both languages expressed only path in their speech they were more likely to use Path gestures. Likewise when they expressed only manner in their speech, they included gestures that contained manner (both Manner and Conflated gestures), but crucially not Path gestures that would mismatch, or com‑ pensate the informational content of the utterance.
However, the occurrence of path gestures in manner‑only clauses is quite likely in Chinese discourse, because of Chinese speakers’ overall preference for Path ges‑ tures.
In short, do gestures compensate for the omission of motion expression in speech? The answer is not simply yes or no. Gesture does not necessarily depict the absent content when either MANNER or PATH is not linguistically expressed. Nevertheless, when gesture does do so, Chinese speakers demonstrate a prefer‑ ence for compensation. What gestures tend to compensate for is the lack of path content in speaking. Since the results in Chinese are different from those in Span‑ ish, English, and Turkish, the compensatory relationship to coordinate informa‑ tion across the two modalities is thus language‑specific.
