移動事件的語言與圖像表達和概念化

行政院國家科學委員會專題研究計畫 成果報告

移動事件的語言與圖像表達和概念化(第 3 年)

研究成果報告(完整版)

中 華 民 國 100 年 12 月 14 日

中 文 摘 要 ： 本研究計畫的目的是探討在日常會話和故事敘述的語境中， 中文如何使用手勢來描繪移動事件，以及圖像和語言如何共 同表達移動事件的認知概念。本計畫為期三年。在本計畫的 第一年，將會檢查、修改與確認敘述語料的轉寫內容和手勢 眼神的轉寫分析。同時開始分析語料和手勢，找出所有被談 論的移動事件和表達移動事件的手勢，然後做進一步的分 析。分析的項目有：（一）移動事件的成分、（二）語言表 達的方式、（三）手勢類型、（四）語言和手勢的共時性， 和（五）訊息狀態。首先，一個典型的移動事件包括五個成 分（Talmy 1975）：「移動動作」、「移動方式」、「移動 途徑」、「移動物」、「移動的相對物」。這些成分在語言 上可以透過不同的語詞、文法，和句型結構來表達（Slobin 2004）。至於手勢的類型，可分五種(Chui 2002)：「圖象手 勢（iconic gestures）」──跟語詞的意義很接近、「比喻 手勢（metaphoric gestures）」──表達抽象的意義、「指 涉手勢（deictic gestures）」──指向說話現場的人事 物、「空間手勢（spatial gestures）」──利用身體空間 來指涉不在談話現場的人事物、和「節拍手勢（beats）」 ──配合語句的節奏來傳達不同的訊息。這些手勢是跟相關 語詞同時產生，還是在說出相關語詞之前還是之後產生，就 是共時性的議題了。最後，訊息狀態分析手勢傳達的是新的 還是舊的訊息。 計畫的第二年以日常會話語料為主。首先繼續檢查、修改與 確認語料的轉寫內容和手勢眼神的轉寫分析。同時開始分析 會話語料和手勢，找出所有被談論的移動事件和表達移動事 件的手勢，做進一步的分析。分析的項目有：（一）移動事 件的成分、（二）語言表達的方式、（三）手勢類型、 （四）語言和手勢的共時性，和（五）訊息狀態。有了故事 敘述和日常會話的分析，第二年也開始資料的統計和議題的 討論。跟本計畫相關的議題有：（一）對於每個移動事件的 成分，在中文最常使用的語言表達方式是什麼？（二）中文 如何透過手勢表達各個移動事件的成分？（三）最常使用的 語言表達方式和圖像表達有什麼關連？ 計畫的第三年除了繼續檢查、修改與確認語料的轉寫內容和 手勢眼神的轉寫分析外，主要將第一年和第二年的各種分析 完成計算和統計的工作。有了所有的分析資料和統計結果， 就可以探討本計畫的主題──移動事件的語言與圖像表達和 概念化，將討論六個議題：（一）表達移動事件的手勢傾向 於跟對應語詞同時產生嗎？（二）表達移動事件的手勢大多 傳達新訊息嗎？（三）各項分析結果在日常會話和口語敘述 中有沒有差異？（四）圖像和語言如何共同表達移動事件的

認知概念？（五）中文、英文和西班牙文對於移動事件的圖 像和語言表達，以及認知概念是否不同？（六）研究結果和 跨語言的比較如何說明語言結構、手勢，和認知概念的相互 關係？ 本計畫的研究成果希望可以讓我們更瞭解手勢、語言、語用 和認知之間的複雜關係，也可以提供一個研究語言和手勢的 新方向，也希望有助於建立一個能處理語言和手勢的運算系 統。 中文關鍵詞： 移動事件，手勢，語言表達，圖像表達，概念化 英 文 摘 要 ： This is a three-year project that studies the

linguistic-imagistic representations and

conceptualization of motion events. In the first year, both the speech transcription and the gestural analysis in the transcripts will be checked and revised. At the same time, all the motion events and motion-event gestures in narrative discourse will be identified and analyzed with respect to (1) motion-event components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. Five types of gestures will be distinguished: iconic gestures, metaphoric gestures, deictic gestures, spatial gestures, and beats. I also examine whether the stroke phase of a motion-event gesture occurs at the time the associated linguistic expression is uttered. Finally, a binary given-new distinction will be used to specify the information state of the gesture-associated lexical expressions. In the second year, the focus will be on conversational data. First, checking and revising the speech transcription and the gestural analysis in the transcripts will continue. Motion-event gestures in conversational discourse will be identified and analyzed. The analyses will be counted and tabulated using

statistics.Three issues will be discussed: (1) What is the most preferred type of linguistic

representation of each motion-event component? (2) How do Chinese speakers gesture the five motion-event components? (3) What is the relationship between the preferred types of linguistic representation and motion-event gestures? In the third year, checking

and revising the transcripts will continue. Also, the large amount of data analyses will further be counted and tabulated using statistics. Then, the following issues will be discussed: (1) Do motion-event

gestures tend to synchronize with associated lexical expressions? (2) Do motion-event gestures tend to carry new information? (3) Do the findings differ across conversational and narrative discourse? (4) How do the findings suggest the conceptualization of motion events? (5) Are there any differences among Chinese, English, and Spanish in gesturing and conceptualizing motion events? (6) How do the findings and cross-linguistic comparisons indicate the interaction among language typology, gesture, and thought?

英文關鍵詞： motion event, gesture, linguistic representation, imagistic representation, conceptualization

1

行政院國家科學委員會補助專題研究計畫

█成果報告

□期中進度報告

（移動事件的語言與圖像表達和概念化）

計畫類別：

█

個別型計畫 □整合型計畫

計畫編號：NSC 97 -2410-H-004 -111 -MY3

執行期間： 97 年 8 月 1 日至 100 年 7 月 31 日

執行機構及系所：國立政治大學英國語文學系

計畫主持人：徐嘉慧

共同主持人：

計畫參與人員：

成果報告類型(依經費核定清單規定繳交)：□精簡報告

█

完整報告

本計畫除繳交成果報告外，另須繳交以下出國心得報告：

□赴國外出差或研習心得報告

□赴大陸地區出差或研習心得報告

█

出席國際學術會議心得報告

□國際合作研究計畫國外研究報告

處理方式：

除列管計畫及下列情形者外，得立即公開查詢

□涉及專利或其他智慧財產權，□一年□二年後可公開查詢

中 華 民 國 100 年 12 月 14 日

2

國科會補助專題研究計畫成果報告自評表

請就研究內容與原計畫相符程度、達成預期目標情況、研究成果之學術或應用價

值（簡要敘述成果所代表之意義、價值、影響或進一步發展之可能性）

、是否適

合在學術期刊發表或申請專利、主要發現或其他有關價值等，作一綜合評估。

1. 請就研究內容與原計畫相符程度、達成預期目標情況作一綜合評估

█

達成目標

□ 未達成目標（請說明，以 100 字為限）

□ 實驗失敗

□ 因故實驗中斷

□ 其他原因

說明：

The purpose of this project is thus to investigate how Chinese speakers gesture and

conceptualize motion events with linguistic representations in narration and conversation. This project consists of two parts. The first part focuses on the linguistic and imagistic representations of motion events. All the motion events and motion-event gestures in both narrative and

conversational discourse will be identified and analyzed in relation to (1) motion-event

components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. The second part of this project is to discuss the

conceptualization of motion events, based on the use of language and gesture in speaking, as suggested by the findings in the first part of the project.

In the first year, both the speech transcription and the gestural analysis in the transcripts were checked and revised. At the same time, all the motion events and motion-event gestures in

narrative discourse were identified and analyzed in relation to (1) motion-event components, (2)

linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. First, there are five components of a prototypical motion event (Talmy 1975),

namely ‘motion’, ‘manner’, ‘path’, ‘figure’ and ‘ground’. They can be linguistically expressed in different ways with “combinations of lexical items and grammatical morphemes in various construction types” (Slobin 2004:220). Then, five types of gestures will be distinguished (Chui 2002): iconic gestures whose meanings correspond to the semantic content of the related speech,

metaphoric gestures for abstract ideas, deictic gestures pointing at referents in the immediate

speech environment, spatial gestures manipulating the gesture space to depict a spatial relation between the linguistic constituents or between the speaker and the linguistic constituent, and

beats indicating the rhythm of speech. Concerning synchronization of speech and gesture, I

examined whether the stroke phase of a motion-event gesture occurs at the time the associated linguistic expression is uttered. Finally, a binary given-new distinction were used to specify the information state of the gesture-associated lexical expressions.

3

speech transcription and the gestural analysis in the transcripts continued. On the other hand, motion-event gestures in conversational discourse were identified and analyzed with respect to (1)

motion-event components, (2) linguistic representation, (2) gestural types, (3) synchronization of speech and gesture, and (4) information state. The analyses were counted and tabulated using

statistics. Then, three issues were discussed: (1) What is the most preferred type of linguistic representation of each motion-event component? (2) How do Chinese speakers gesture the five motion-event components? (3) What is the relationship between the preferred types of linguistic representation and motion-event gestures?

In the third year, checking and revising the transcripts continued. Also, the large amount of data analyses finished in the first and second years was further be counted and tabulated using statistics. Then, based on the analyses and statistics, I examined the theme of the

project—linguistic and imagistic representations of motion events and conceptualization. Another six issues were discussed: (1) Do motion-event gestures tend to synchronize with associated lexical expressions? (2) Do motion-event gestures tend to carry new information? (3) Do the findings differ across conversational and narrative discourse? (4) How do the findings suggest the conceptualization of motion events? (5) Are there any differences among Chinese, English, and Spanish in gesturing and conceptualizing motion events? (6) How do the findings and

cross-linguistic comparisons indicate the interaction among language typology, gesture, and thought?

To summarize, most of the narrative and conversational transcripts were checked and revised. The motion events and motion-event gestures were identified and analyzed. The conversational and narrative data analyses were counted and tabulated using statistics. With the analyses,

findings, and statistics, it was found that the linguistic-imagistic representation of motion events in Chinese differs from that in other languages. The past studies showed that

The findings of the project have also been presented in international conferences and published in international journals. The graduate students who participated in the project have been trained to transcribe spoken data and gestures. They know more about spoken grammar and gesturing in speaking.

MANNER and PATH in English are mentioned within one clause, and the two components can be represented together in one gesture. In Turkish and Japanese, they are expressed separately in two clauses, and two separate gestures – one for MANNER and one for PATH - are produced accordingly. As to Chinese speakers, they predominantly use simple manner verbs to express MANNER and serial verbs and prepositional phrases to convey PATH within a clause. In imagistic representation, speakers prefer to depict path information only, be it carrying new or given information. The cross-linguistic differences demonstrate language specificity in linguistic encodings and manual depictions of motion. Such linguistic-imagistic variation can further suggest language specificity in the conceptualization of motion events.

4

2. 研究成果在學術期刊發表或申請專利等情形：

論文：█已發表 □未發表之文稿 □撰寫中 □無

Chui, Kawai. 2008. Imagistic representation of motion events. The Sixth International

Conference of Cognitive Science (ICCS 2008), Yonsei University, Seoul, Korea, July 27 - 29, 2008.

Chui, Kawai. 2009. Linguistic and imagistic representations of motion events. Journal of

Pragmatics 41(9): 1767-1777. (SSCI, AHCI)

Chui, Kawai. 2010. Gesture and Frame Knowledge. The 4th Conference of the International Society for Gesture Studies (ISGS), European University Viadrina Frankfurt/Oder, Germany, July 25 - 30, 2010.

Chui, Kawai. 2011. Do gestures compensate for the omission of motion expression in speech?

Chinese Language and Discourse 2(2): 153-167.

Chui, Kawai. 2011. Gesture and embodiment in discourse. The 5th Conference on Language, Discourse and Cognition (CLDC 2011), National Taiwan University, Taipei, April 29 – May 1, 2011.

Chui, Kawai. 2011. Conceptual metonymies in gesture. The 11th _{International Cognitive} Linguistics Conference (ICLC 11), Xi’an International Studies University, China, July 11 - 17, 2011.

Chui, Kawai. 2011. Language typology and linguistic-gestural conceptualization of motion event. The 9th _{Biennial Conference of the Association for Linguistic Typology (ALT 9), University of} Hong Kong, Hong Kong, China, July 21 - 24, 2011.

Chui, Kawai. (under review). Cross-Linguistic representation of motion in language and gesture.

3. 請依學術成就、技術創新、社會影響等方面，評估研究成果之學術或應用價

值（簡要敘述成果所代表之意義、價值、影響或進一步發展之可能性）（以

500 字為限）

Based upon the quantitative-qualitative analysis of a large corpus of narrative and conversational data, the study in this project has shown the general tendency of producing motion-event gestures, which represents Chinese speakers’ common gesturing behaviors in speaking. Language and gesture together reflect the conceptualization of motion events in gesture-utterance production. The present study also helps understand the intricate relationship among gesture, language, use, and cognition. Finally, it provides a new aspect for the study of linguistic issues vis-à-vis gesturing in real discourse, and helps formulate a computational model of gesture-speech performance.

Linguistic and imagistic representations of motion events

Kawai Chui*

National Chengchi University, P.O. Box 1-322, Mucha, Taipei 116, Taiwan Received 3 August 2008; accepted 20 April 2009

Abstract

This study investigates the linguistic and imagistic representations of motion events in Chinese discourse. First, manner is most usually conveyed and in the form of single manner verbs, but information of this type is rarely gestured. Second, speakers also mention path very often by means of manner-path-deictic verbs and prepositional phrases indicating location, source, and goal. Speakers more commonly gesture path in contrast to manner. Similar results can be found in English, showing that the way people gesture motion events does not have to do with linguistic typology. However, in a single gesture, while manner and path can be conveyed simultaneously in Chinese and English, manner-ground gestures are found in Spanish. Such difference suggests various conceptualizations of motion event across different languages. Finally, gesturing the reference object and the moving object, being linguistically represented by nominal phrases, has to do with new information.

# 2009 Elsevier B.V. All rights reserved.

Keywords: Gesture; Motion event; Imagistic representation; Linguistic representation; New information

1. Introduction

A prototypical motion event ‘‘consists of one object (the ‘Figure’) moving or located with respect to another object (the reference-object or ‘Ground’). . . the ‘Path’. . .is the course followed or site occupied by the Figure object with respect to the Ground object. ‘Motion’. . .refers to the presence per se in the event of motion or location. . .. In addition to these internal components a Motion event can have a ‘Manner’ or a ‘Cause’’’ (Talmy, 1985:61). Consider the event in Example (1). The characters are running to an eating place. The figure refers to a threesome, namely a bull, Mickey, and Pluto; the path is indicated by dao ‘to’, and the ground is chifan de defang ‘the eating place’. The motion and the manner are realized simultaneously by the verb pao ‘to run’.

(1) W: ..lingwai san zhi. . . gong de niu haiyou. . . miqi haiyou bulutuo. . . jiu. . .

another three CL male ASSC cow and Mickey and Pluto then

pao dao. . . yecan. . . de. . . chifan de difang

run to picnic ASSC eat ASSC place

W: ‘As to another three, i.e., the bull and Mickey and Pluto, then ran to. . .picnic. . .the eating place.’ ‘‘The language of motion events is a system used to specify the motion of objects through space with respect to other objects’’ (Huang and Tanangkingsing, 2005:336). Then, how do different languages express motion events

www.elsevier.com/locate/pragma

Available online at www.sciencedirect.com

Journal of Pragmatics 41 (2009) 1767–1777

* Tel.: +886 2 29393091x88088. E-mail address:[email protected].

0378-2166/$ – see front matter # 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.pragma.2009.04.006

linguistically?Talmy (1985)regards Chinese and English as satellite-framed languages, since they have a class of verbs that incorporate the motion and manner information, whereas path is an adjunct; Spanish is a verb-framed language, in that motion and path are lexicalized as a single verb, while manner is an adjunct. Based on these different lexicalization patterns and on the further distinction between English being predicate-prominent and Chinese being topic-prominent, McNeill and Duncan (2000) investigated motion-event gestures across English, Spanish, and Chinese. They then made a claim about the relationship among language typology, gesture, and thought: The gestural differences among the three languages are the results of different forms of thinking being realized in their typologically distinct linguistic structures. This study will show that McNeill and Duncan’s claim is problematic. It then investigates the linguistic and imagistic representations of motion events in Chinese, and discusses how the results can shed light on cross-linguistic conceptualizations of motion events.

The next section discusses the motion-event gestures inMcNeill and Duncan’s (2000)study. Section3examines the linguistic and imagistic representations of various motion-event components in Chinese discourse. The last section provides a discussion.

2. McNeill and Duncan’s (2000)cross-linguistic study

McNeill and Duncan (2000)in their cross-linguistic study of motion-event gestures find typological differences in both linguistic and imagistic representations of motion events in English, Spanish, and Mandarin, suggesting various forms of thinking-for-speaking. According to the study, English speakers gesture manner when it is a focus. The stroke phase also synchronizes with the manner verb. To downplay manner, the gesture does not convey manner information, but synchronizes with the path or ground words. Spanish speakers, on the other hand, use path verbs. But they often convey manner in their gestures along with path and/or ground. Finally, Mandarin speakers tend to gesture motion-event components at the utterance-initial position, prior to the production of affiliated words, to form a topic frame. There are two major problems in McNeill and Duncan’s study. The first has to do with the synchronization of speech and gesture. According to McNeill and Duncan, whether the obligatory stroke phase of a gesture synchronizes with the manner information in English and Spanish rests upon the notion of focus.

In English, gesture and verb jointly highlight manner when it is part of the speaker’s focus. When manner is not in focus, gesture does not encode it and need not synchronize with a manner verb, even if one is present. . ..Manner appears in Spanish speech presumably only when it is a focused component, and it is often omitted even when it is potentially significant. (McNeill and Duncan, 2000:151–152)

They provide the following two examples that describe Sylvester being rolled down a drainspout by a ball. In the first utterance (a), a wiggling-hand gesture is produced to synchronize with the verb rolls in the utterance, since the focus is on the rolling manner of the ball. In the second utterance (b) when the focus is not on manner, the speaker’s hand plunges straight down to convey the path information only. The gesture lacks manner and does not synchronize with rolls, but with the path down and the ground drainspout instead.

(a) [but it rolls] him out

(b) [and he rolls. . . down the drainspout]

The problem of the analysis lies in the lack of an independent way of establishing what the ‘‘focus’’ is. In their study, the decision as to what is and what is not a ‘‘focus’’ rests upon the identification of the motion-event component being expressed in gesture, and whether the gestural stroke synchronizes with speech. The whole argument thus becomes circular.

The other problem in McNeill and Duncan’s study is concerned with the relation between language and gesture. Given that Chinese is a topic-prominent language,McNeill and Duncan (2000:152)state that the timing pattern of Chinese gestures resembles the topicalizing structure in Chinese, in that ‘‘the gesture shifts forward in the surface speech stream, in the direction of the utterance-initial position characteristic of topic statements in Chinese speech.’’ This claim is not borne out, since as many as 51% of motion-event gestures (78 out of a total 153) in our Chinese data synchronize with, rather than precede, their co-expressed words. See Example (2). The speaker is describing the way the bull attempts to jump into the sea. The speaker’s right hand, which is high above the head, at the time the associated verb tiao ‘jump’ is produced, descends onto the thigh to signify the motion of jumping (seeFig. 1).

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1768

(2) A: ..mini gen.. miqi zai jianghua.. er ta pangbian de na zhi Minnie with Mickey PROG talk and 3SG side ASSC that CL ..gongniu jiu jiao tamen kan guolai

bull then request 3PL look come here B: ..mhm

BC

A: ..ranhou jiu yi fu hen shenyongde yangzi zheyang. . . zhangkai.. haoxiang then then one CL very brave look like this stretch out look like tiaosanzhuang nayang tiao xiaqu

like parachuting like that jump down

tiao ‘to jump’: right hand above the head (pic.1 inFig. 1) descends onto thigh (pics.2–3 inFig. 1)

A: ‘Minnie was talking with Mickey. The bull beside her then requested them to look at him.’ B: ‘Mhm.’

Z: ‘Then it, looking very brave, stretched out and jumped down, as if he were parachuting.’

Even when the gestural strokes come before the associated words, they do not necessarily function as a ‘sentence-topic’.Chui (2005)finds that strokes coming before the associated words can function to signal that the upcoming new information is noteworthy and deserves attention. In short, the relationship among language typology, gesture, and thought proposed byMcNeill and Duncan (2000)cannot be maintained.

3. Gesturing motion events in Chinese discourse

How do Chinese speakers actually gesture motion events in discourse? This question will be addressed in this section with reference to both linguistic and imagistic representations of various motion-event components.

The database consists of 10 short oral narratives produced by undergraduate students of National Chengchi University in 2002. Each subject viewed a cartoon episode of the Mickey Mouse and Friends series. The soundtrack of the cartoon included music and only a very small amount of dialogue. In the episode, Mickey, Minnie, Pluto and a bull are holding a party at the beach, and eating and playing around. Then, they have a fight with an octopus, which they finally win. After viewing the cartoon, the subject immediately recounted the story from memory to a listener. The subject was filmed by a video camera so that speech and manual movements would be recorded. The subjects were not informed about our particular research interests. The elicited cartoon narrations ranged from 3 to 10 min in length. All of the verbs characterizing motion events in the 10 narratives were tabulated, totaling 245 instances. The gestures examined in the present study are the idiosyncratic spontaneous movements of hands and arms which accompany a speech event with context-dependent meaning and use. They take up 62.4% (153 instances) of all the data.

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1769

Talmy (1985)has categorized Mandarin and English as S-languages. Both languages have a large lexicon of motion-manner verbs. Examples from Chinese are zoulu ‘walk’, pao ‘run’, pa ‘crawl’, and fei ‘fly’. Prepositional phrases also delineate paths in relation to ground elements, as in the following examples (c) and (d):

(c) English: He ran out [of the house]pp.

(d) Chinese: ta pao [dao chifan de difang]pp

3SG run to eat ASSC place

‘He ran to the eating place.’

Slobin (2000)rather regards Chinese as a serial-verb language. Each verb in a series is morphologically unmarked and monosyllabic, such as fei - chu ‘fly exit’. ‘‘Manner is not syntactically subordinated to path. . .[b]ecause the path verbs can occur alone’’ (Slobin, 2000:228).Huang and Tanangkingsing (2005)show that Mandarin is a strongly verb-serializing language, based on a total of 153 Mandarin motion clauses, and 48.4% use the type M(anner)#P(ath)#D(irection) for describing motion event. The present study also considers Chinese as a serial-verb language because its idiosyncratic morphological patterning of motion-event components suggests a typological difference between Mandarin and other languages.

Table 1shows the frequency distribution of all the 153 gestures across the various semantic components of a motion event. These various types of motion-event gestures will be discussed accordingly in the following sub-sections.

3.1. Manner

First, what are the linguistic representations of manner of motion in Chinese? ‘‘A language provides its speakers with a range of ways of describing motion events – combinations of lexical items and grammatical morphemes in various construction types’’ (Slobin, 2004:220). The various lexical forms to convey manner are: manner verbs like you ‘swim’; manner-path verbs like pao-hui ‘run-back’; manner-deictic verbs like pao-lai ‘run-come’; manner-path-deictic verbs like reng-guo-qu ‘throw-across-go’. At the grammatical level, adverbial and phrasal expressions outside the verbs also ‘‘add information about such dimensions as suddenness, rate, force dynamics, inner state, terrain, and so forth – that is, information about factors that suggest manner of movement’’ (Slobin, 2004:232) In our data, 31 instances are of this type, such as the adverbial phrase tai yuan ‘too far away’ modifying the throwing motion in Example (3).

(3) F1: ..ranhou yinwei. . . na ge miao diu tai yuan.. diu dao then because that CL anchor throw too far away throw to F2: (0) mhm

BC F1: ..hai limian

sea inside

F1: ‘Then because the anchor was thrown too far away, it was thrown to,’ F2: ‘Mhm.’

F1: ‘into the sea.’

Of all the 245 motion events in the database, 200 (81.6%) use manner verbs of various types. Similar results can be found inHuang and Tanangkingsing (2005)with a distinct preference for manner verbs, at 83.4%.Table 2, which contains the 31 instances conveying manner without using manner verbs, demonstrates that single manner verbs (147 instances) are the habitual linguistic expressions of manner in Chinese discourse, at 63.6%.

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1770

Table 1

Various types of gestures.

Manner gestures Path gestures Ground gestures Figure gestures Manner and path gestures Total

5 112 14 15 7 153

Then, what are the imagistic representations of manner of motion? In Example (4), when the speaker verbalizes the nominal yangshi ‘backstroke’, about Mickey Mouse swimming backstroke in the sea, both of her hands, facing down, paddle outward two times to signify swimming (seeFig. 2).

(4) F: . . . houlai. . .(1.5)houlai <L2 mickey L2> hoaxiang shi you. . . yangshi ba

later later Mickey seem COP swim backstroke PRT

you yangshi ‘swim backstroke’: both hands are on knees (pic.1 inFig. 2), then right hand and two fingers of left hand paddle outward one time (pic.2 inFig. 2), then right hand paddles outward one more time (pic.3 inFig. 2) and comes back (pic.4 inFig. 2). F: ‘Later, later, it seems that Mickey swam backstroke.’

90.2% (138 instances) of all the 153 motion-event gestures that accompany motion events were found to occur in clauses with a manner verb. Nevertheless, just twelve manner gestures (8.7%) were produced (seven instances depict path simultaneously), despite the prevalence of manner. Moreover, McNeill and Levy (1993:365) suggest that ‘‘gestures tend to occur at points of topic shift, such as new narrative episodes or new conversational themes. . .highly presupposed linguistic elements would either lack gestures entirely, or would be accompanied by gestures that are specialized for their cohesive function or form.’’Chui (2005)also demonstrates that Chinese speakers tend not to gesture for old information. A question thus arises: Is the scarcity of manner gestures related to information state? ‘New information’ is what has not been brought up in the previous context at the moment of speaking; ‘given information’ is what has already been mentioned at the moment of utterance.Table 3presents the frequency distribution of given and new information across the various types of manner verbs. The overall proportion of given to new information is about equal, suggesting that speakers are not likely to produce manner gestures even though the information is new.

3.2. Path

Table 1shows that it is very common (73.2%) to convey path information via the imagistic modality. This section investigates the linguistic and imagistic expressions of path. First, just like manner, there are various lexical forms to convey path: path verbs like hui ‘return’; deictic verbs like que ‘go’; path-deictic verbs like chu-lai ‘out-come’;

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1771

Fig. 2. Manner gesture: depiction of the backstroke style of swimming. Table 2

Linguistic representations of manner.

Manner verb Manner-path verb Manner-deictic verb Manner-path-deictic verb Manner outside the verb Total

147 7 4 42 31 231

Table 3

Information state and manner.

Manner verb Manner-path verb Manner-deictic verb Manner-path-deictic verb Total

Given 57 3 2 12 74 53.6%

New 45 2 1 16 64 46.4%

manner-path verbs like pao-hui ‘run-back’; manner-deictic verbs like pao-lai ‘run-come’, and manner-path-deictic verbs like reng-guo-qu ‘throw-across-go’. A deictic word can also come before a motion-manner verb, just like zhui ‘chase after’ in Example (5), or after a prepositional phrase, such as dao hai limian ‘into the sea’ in Example (6). (5) M: . . .bulutuo. . .jiu yibian. . . ta ye yibian

Pluto then on the one hand 3SG also on the one hand

qu zhui a

go chase after PRT

M: ‘Pluto, then, on the one hand, he also went to chase after the sausages.’ (6) M: . . .na zuihou ta jiu.. diu yi chuan xiangchang deshihou

then at last 3SG then throw one string sausage when . . .yinwei diu tai yuan .. diao dao hai limian qu

because throw too far away fall to sea inside go

M: ‘Then, at last, when it, then, threw a string of sausages, because it was thrown too far away, it fell into the sea.’

There are 161 motion events in the database that consist of the path component.Table 4indicates the various linguistic forms conveying path information. Manner-path-deictic verbs (26.7%) and prepositional phrases indicating location, source, and goal (46.6%) are the most frequently used linguistic patterns when speakers talk about path. Next, how is the path of a motion gesturally expressed? Example (7) is about Mickey and Minnie running to the beach. The speaker’s right hand, dangling at waist level on the right side, starts moving leftward at the time the first pao ‘run’ is uttered to depict a route. When she produces the fifth pao, her right hand in the central periphery moves downward to signify the endpoint of the path (seeFig. 3).

(7) F: (0) ranhou tamen jiu. . .(.9) pao pao pao pao pao.. pao dao yi ge haitan then 3PL then run run run run run run to one CL beach

pao ‘run’: right hand on the right (pic.1 inFig. 3) raises to waist level (pic.2 inFig. 3), and moves to central periphery (pic.3 inFig. 3).

F: ‘Then, they ran and ran and ran and ran and ran. They ran to a beach.’

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1772

Table 4

Linguistic representations of path.

Path verb Deictic verb Path-deictic verb Manner-path verb Manner-deictic verb

2 10 12 7 7

Manner-path-deictic verb Deictic + manner verb PP PP + deictic Total

43 5 67 8 161

Ninety-one gestures (56.5%) were produced out of a total 161 motion events conveying path information, far outnumbering manner gestures by eighteen to one. Again, are such high occurrences related to information state? Table 5, similar to the result inTable 3, illustrates that ‘given’ and ‘new’ are almost equally distributed among all the 91 path gestures. Thus, information state does not play a role in the imagistic representations of either path or manner of motion.

Moreover, 21 path gestures were found to accompany pure manner verbs without path information in speech. The frequency distribution of givens and news are about the same: 10 givens (47.6%) and 11 news (52.4%). Finally, concerning the synchronization of path gestures and associated speech, 17 instances were produced prior to the verbs; one was produced after the path constituents. Forty instances (35.7%) synchronize with the path constituents. What is intriguing is that 48.2% (54 instances) of all the 112 path strokes synchronize with the manner verbs rather than with the path words. This will be brought up again in section4.

3.3. Ground and figure

The reference object and the moving object of a motion event are linguistically represented by nominal expressions, such as ye ge zhuzi ‘a pillar’ in Example (8) and the third person pronominal ta in Example (9), respectively. They can also be expressed manually. The speaker depicts the round shape of the pillar to gesture the ground component of the colliding event in Example (8) with a circle made by the thumb and index finger of his left hand at the moment of verbalizing the motion-manner verb zhuangdao ‘collide’ (seeFig. 4).

(8) C: . . . na zhi niu. . . jiu yong na ge. . . x zuo de jiushengquan that CL bull then use that CL REPAIR make ASSC life belt jiu zai hai. . .(2.)hai shang piao a

then on sea sea on float PRT

B: ..mm

BC

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1773 Table 5

Information state and path.

Path verb Deictic verb Path-deictic verb Manner-path verb Manner-deictic verb

Given 0 0 0 3 2

New 1 3 8 2 0

1 3 8 5 2

Manner-path-deictic verb Deictic + manner verb PP PP + deictic Total

Given 10 1 24 4 44 58.4%

New 14 1 15 3 47 51.6%

24 2 39 7 91 100.0%

C: ..ranhou piao yi piao. . .jiu. . . zhuangdao yi ge zhuzi then float one float then collide:RESULT one CL pillar

zhuzi ‘pillar’: thumb and index finger of left hand raises from thigh (pic.1 inFig. 4) and form a circle (pic.2 inFig. 4), and then return to thigh (pic.3 inFig. 4).

C: ‘That bull then used a life belt and on the sea. . .floated on the sea.’

B: ‘Mm.’

C: ‘Then, it floated for a while, then it collided with a pillar.’

Example (9) describes a horse using watermelon seeds as weapon to fight against an octopus. The speaker gestures the figure component, i.e., the watermelons seeds, bouncing off a tea-kettle at the side. The thumb and the index finger of the speaker’s left hand form a circle when the pronominal subject ta, which refers to watermelon seeds as a group, is uttered to represent the shape of the watermelons seeds (seeFig. 5).

(9) F: (0) dui ta chi xigua. . . ranhou jiushi tu zi right 3SG eat watermelon then that is spit out seed B: ..heh

BC

F: . . . ranhou jiu.. yizhi tu.. ranhou. . .you. . . ta jiu.. you then then continuously spit out then PRF 3PL then PRF tan dao.. pangbian de nage.. shenme shuihu a.. shenme de bound to side ASSC that whatever water bottle PRT whatever PRT ta ‘they’: thumb and index finger of her left hand form a circle (pic.1 inFig. 5).

tan ‘bound’: the circle gesture raises to waist level (pic.2 inFig. 5) and sweeps down (pic.3 inFig. 5). F: ‘Right, it (i.e., the horse) was eating watermelon. Then, that is, it spit out the seeds.’

B: ‘Heh.’

F: ‘Then, it kept spitting out seeds. Then, they (i.e., the seeds) bounded against the side of a water bottle, or whatever.’

Reference objects and moving objects are not frequently brought up, unlike manner and path, in that just 53 motion events (53 out of 245, 21.6%) include ground information and 67 motion events (67 out of 245, 27.3%) have a figure component. Nevertheless, the number of their respective gestural occurrences is much higher than that of manner gestures. The speakers produced 14 ground gestures (out of 53, 26.4%) and 15 figure gestures (out of 67, 22.4%) in the corpus.Table 6recapitulates the proportions of the total number of motion events including a particular motion-event component to the total number of gestures for that particular component. Gestures for reference objects and moving objects outnumber those for manner by about five times.

Finally, different from manner and path, the occurrences of ground and figure gestures are more likely to be constrained by information state: 71.4% (10 instances) of the reference objects and 73.3% (11 instances) of the moving objects carry new information.

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1774

4. Discussion and conclusion

Based on the findings in the previous section, this section will discuss the relationship among language, gesture, and motion-event conceptualization. It was first addressed by McNeill and Duncan (2000), who claim that English, Spanish, and Chinese are typologically different languages and that the speakers also gesture motion events in different ways because they have their own respective conceptualization (see the detailed discussion in section 2). Their claim is subject to Slobin’s (1987) ‘‘thinking-for-speaking’’ hypothesis. This hypothesis refers to how speakers organize their thinking to meet the demands of linguistic encoding on-line, during acts of speaking. As speakers are thinking ‘‘in terms of a combination of imagery and linguistic categorical content’’ (McNeill and Duncan, 2000:142), their conceptualization of an event can thus be realized simultaneously in speech and gesture.

This study has investigated how Chinese speakers produce manual movements while talking about motion events. Our findings do not bear outMcNeill and Duncan’s (2000)analysis of ‘‘thinking-for-speaking’’ for Mandarin, since the timing pattern of Chinese gestures by no means resembles the topic-comment patterning of the language. Thus, their claim about the relationship among language typology, gesture, and thought cannot be maintained. Nevertheless, we agree that conceptualization of an event includes both imagery and linguistic content, which can be realized simultaneously in speech and gesture, respectively. Thus, the linguistic and imagistic representations of the motion-event components shed light on how motion events are conceptualized by Chinese speakers. The conceptualization is also embodied in daily social interaction. The findings of this study thus represent the common experience of Chinese speakers talking about motion events. First, manner is most usually conveyed and in the form of single manner verbs, but information of this type is rarely gestured. Second, speakers also mention path very often by means of manner-path-deictic verbs and prepositional phrases indicating location, source, and goal. Speakers more commonly gesture path in contrast to manner. Similar results can be found in English: ‘‘Adults, when they describe such motion events, typically produce gestures showing only path. . .or gestures showing in a single gesture both manner and path. . .. Manner without path, however, rarely occurs’’ (McNeill, 2005:185). Since Mandarin and English are typologically different languages, their similarities in gestures for motion events evidence that the imagistic representations of motion events do not have to do with linguistic typology. Gestures enable the speakers to go beyond the structural restriction of language and convey their own thinking about a motion event in verbal communication.

In a single gesture, while manner and path can be conveyed simultaneously in Chinese and English,1manner-ground gestures are found in Spanish (McNeill and Duncan, 2000). Such difference seems to suggest various conceptualizations of motion events across different languages. In Chinese, to incorporate more than one motion-event component in a single gesture is rare. Moreover, the high occurrences of path gestures, despite the fact that speakers often mention manner and path in speech, evidence that ‘path’ is the most salient component in motion-event conceptualization.

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1775 Table 6

Motion-event components and their respective occurrences of gesture.

Motion events including manner Manner gestures Proportion

245 12 4.9%

Motion events including path Path gestures Proportion

161 119 73.9%

Motion events including ground Ground gestures Proportion

53 14 26.4%

Motion events including figure Figure gestures Proportion

67 15 22.4%

1

Since frequency distribution is not provided inMcNeill and Duncan’s (2000)study, whether there is any quantitative difference between Mandarin and English concerning the single gestures conveying both manner and path simultaneously awaits future research.

Finally, gesturing the reference object and the moving object, being linguistically represented by nominal phrases, has to do with information state, in that ground and figure gestures usually convey new information. Conceptualization thus has to consider the flow of information in discourse.

Acknowledgments

This research was funded by grants from the National Science Council (NSC 97-2410-H-004-111-MY3). I would also like to thank the referees for offering valuable comments and suggestions. All errors of interpretation are my own responsibility.

Appendix A. Gesture and speech transcription conventions

Transcription of speech [ ] speech overlap . . .(N) long pause . . . medium pause .. short pause (0) latching @ laughter

TSK a click of the tongue <L2 L2> code-switch to English

Transcription of gesture

For the representation of gesture in examples, the underlined part of the utterance is the stroke phase (and the hold phase, if there is any); the lexical affiliate(s), if there is any, is/are in boldface. The description of gesture is given under the line of associated speech. In each gestural description, the word(s) before the colon represent(s) the referent a gesture refers to; the description of the manual movement comes after the colon.

Abbreviations of linguistic terms

1PL first person plural 1SG first person singular

2PL second person plural

2SG second person singular 3PL third person plural 3SG third person singular

ASSC associative morpheme

BA the morpheme BA

BC backchannel

CL classifier

COP copula verb

NEG negative morpheme

PF pause filler

PRF perfective aspect

PROG progressive aspect

PRT discourse particle

QST question particle

REPAIR repair phoneme(s)

SELF reflexive morpheme

RESULT resultative morpheme

K. Chui / Journal of Pragmatics 41 (2009) 1767–1777 1776

References

Chui, Kawai, 2005. Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics 37 (6), 871–887. Huang, Shuanfan, Tanangkingsing, Michael, 2005. Reference to motion events in six western Austronesian languages: toward a semantic typology.

Oceanic Linguistics 44, 307–340.

McNeill, David, Levy, E.T., 1993. Cohesion and gesture. Discourse Processes 16 (4), 363–386.

McNeill, David, Duncan, S., 2000. Growth points in thinking-for-speaking. In: McNeill, David (Ed.), Language and Culture. Cambridge University Press, Cambridge, pp. 141–161.

McNeill, David, 2005. Gesture and Thought. University of Chicago Press, Chicago.

Slobin, D., 1987. Thinking for speaking. In: Aske, J., Beery, N., Michaelis, L., Filip, H. (Eds.), Proceedings of the 13th Annual Meeting of the Berkeley Linguistics Society. Berkeley Linguistics Society, Berkeley, CA, pp. 435–445.

Slobin, D., 2000. Verbalized events: a dynamic approach to linguistic relativity and determinism. In: Niemeier, S., Driven, R. (Eds.), Evidence for Linguistic Relativity. John Benjamins Publishing, Amsterdam, pp. 107–138.

Slobin, D., 2004. The many ways to search for a frog: linguistic typology and the expression of motion events. In: Stro¨mqvist, S., Verhoeven, L. (Eds.), Relating Events in Narrative: Typological and Contextual Perspectives. Lawrence Erlbaum Associates, Mahwah, NJ, pp. 219–257. Talmy, L., 1985. Lexicalization patterns: semantic structure in lexical forms. In: Shopen, T. (Ed.), Language Typology and Syntactic Description,

vol. 3. Cambridge University Press, Cambridge, pp. 57–149.

Kawai Chuiis Professor of Linguistics at National Chengchi University, Taipei. She has been working to build a corpus of spoken Chinese with gestural analysis. Her research interests include gesture and speech communication, and the discourse basis of grammar.

Chinese Language and Discourse 2:2 (2011), 153–167. DOI 10.1075/cld.2.2.01chu

ISSN 1877‑7031 / E‑ISSN 1877‑8798 © John Benjamins Publishing Company

Do gestures compensate for the omission of

motion expression in speech?*

Kawai Chui

National Chengchi University

The present study investigates whether and to what extent motion‑event gestures compensate for the omission of linguistic expression in Chinese discourse and across different languages to understand language‑specificity/language‑univer‑ sality and the coordination of motion information across the two modalities. The Chinese conversational and narrative data consistently show that manner fog (i.e., manner absent from speech but present in gesture) was not found. Chinese speakers also demonstrate a preference for compensation — gestures tend to compensate for the lack of path content in speaking. These results differ from those for English and Turkish which do not prefer path gestures in manner‑only clauses. The cross‑linguistic variation provides evidence for language specificity in gestural compensation. The language‑specific coordination of information in speech and gesture suggests Chinese speakers’ habitual focus of attention on PATH in multimodal communication.

Keywords: gestural compensation, motion event, gesture, linguistic‑imagistic

representation, cross‑linguistic representation

关键词：手势补偿、移动事件、手势、语言─图像表达、跨语言比较

1. Introduction

The use of hands and arms along with speech is indispensable and prevalent in multimodal communication (Goldin‑Meadow, 1999; McNeill, 1992, 2000; Ken‑ don, 2004). “The tremendous overlap between neural structures contributing to language and hand/arm movement may help to explain the prevalence of hand gesture in language” (Glenberg, 2007: 363). During speaking, gestures bear a wide variety of functions. They can, among others, facilitate speech production (Rime & Schiaratura, 1991) and help learning (Alibali & Goldin‑Meadow, 1993), lexical

154 Kawai Chui

retrieval (Krauss et al., 1996, 2000), problem‑solving (Alibali et al., 1999), remem‑ bering more (Goldin‑Meadow et al., 2001), organizing the speaker’s thinking for speaking (Özyürek & Kita, 1999; Alibali, Kita, & Young, 2000; Kita, 2000), pro‑ viding referential meaning (Kendon, 2004), constructing meaning in classroom activity (Singer et al., 2008), and accomplishing conversational coherence (Chui, 2009a).

When the speaker conveys a message, speech and gesture often work in col‑ laboration to express information. Melinger and Levelt’s (2004) experiment showed that speakers use gestures intentionally to convey part of their message. The messages represented by manual configurations can be of various kinds. Mc‑ Neill (1992) found that in narrative discourse iconic gestures and abstract pointing mainly express information contained in narrative clauses; beat gestures indicate the textual shift between the narrative and extranarrative levels; and metaphoric gestures depict information in extranarrative clauses. McNeill also noted that “[s] peech and gesture refer to the same event and are partially overlapping, but the pictures they present are different” (ibid.: 13). In Bavelas et al. (1992: 473) ‘topic gestures’ in conversation were found to enact information directly related to the topic of discourse; ‘interactive gestures’ were used to address to other participants and function “to aid the maintenance of conversation as a social system” in con‑ versation (Bavelas et al., 1992: 470). Some gestures of this kind had parallel verbal references but some did not. With respect to ‘communicative dynamism’, McNeill & Levy (1993) showed that more complex gestures would be produced along with more complex linguistic expressions to depict the information that functions to push the communication forward. Kendon (1995: 247) distinguished between ‘substantive gesturing’ and ‘pragmatic gesturing’. “[The former] contributes to various aspects of the content of the utterance of which it is a part, whether liter‑ ally or metaphorically… [and the latter] expresses aspects of utterance structure, including the status of discourse segments with respect to one another, and the character of the ‘speech act’ or interactional move of the utterance.” In Kendon (2004), he made a distinction between ‘gestures with equivalent verbal expres‑ sions’ and ‘gestures with a non‑matching verbal expression’ in the discussion of contributions that gestures can make to referential meaning. Based on Chinese conversational data, Chui (2008) investigated different kinds of information rep‑ resented by complementary gestures which provide additional meanings to enrich speech events or maintain the continuity of a topic under discussion. Finally, for children, Church & Goldin‑Meadow (1986) studied the mismatches between ges‑ ture and speech in children’s explanations of a concept. The ‘discordant’ children in their study “produced many explanations in which the information conveyed in speech did not match the information conveyed in gestures” (ibid: 43).

Do gestures compensate for the omission of motion expression in speech? 155

The previous studies mostly focused on the kinds of information that can be expressed by gestures. The question as to whether gesture may compensate for the omission of expression in speech was not a main concern. ‘Gestural compensation’ is distinguished from ‘complementarity’, a term used by McNeill (1992). His exam‑ ple was: At the moment the speaker utters she chases him out again, “speech con‑ veys the ideas of pursuit and recurrence while gesture conveys the weapon used (an umbrella)” (ibid.: 13). The form of the speaker’s hand in a shape as though to grip something is not associated with any lexical item, and the utterance is gram‑ matically complete without the gesture. Gestures of this type have also been re‑ ferred to as ‘supplementary’, ‘mismatching’, or ‘non‑redundant’ in other studies (as mentioned in Alibali et al., 2009: 291). They do not necessarily compensate for the absence of certain linguistic expression in speech; they can simply provide non‑ linguistic information via the gestural modality.

Gestural compensation has been mentioned in previous studies with regard to different languages, including Spanish (McNeill & Duncan, 2000), Turkish, Japa‑ nese, and English (Kita & Özyürek, 2003; Özyürek et al., 2005). However, these studies did not show the habitual imagistic representation used to compensate for speech in daily communication or about the ways in which the general lin‑ guistic‑gestural representation and imagistic compensatory representation work together in the coordination of information in speech and gesture. The present study will provide empirical evidence to discuss these two issues with respect to motion events, to understand if a speaker has produced a gesture for the omis‑ sion of linguistic expression and to understand how information is coordinated across the two modalities. The presence or absence in speech can be clearly de‑ termined as the components of motion‑events are (near‑)universal (Talmy, 1985). Cross‑linguistic comparison is also available based on a series of research into the linguistic‑imagistic expression of motion across languages. In this study, I first investigate whether and to what extent motion‑event gestures may compensate for the omission of linguistic expression. The findings in Chinese will be compared across typologically different languages so as to understand language‑specificity/ language‑universality and the coordination of motion information across the two modalities.

The next section introduces the data and preliminaries for the study. Section 3 provides a general representation of motion in speech and gesture in Chinese dis‑ course. Section 4 examines gestural compensation across different languages. Sec‑ tion 5 discusses the coordination of motion information in speech and gesture.

156 Kawai Chui

2. Data and preliminaries

The data for Chinese used in this study is from The NCCU (National Chengchi University) Corpus of Spoken Chinese. This corpus is part of an archive of lan‑ guage documentation which collects the spoken forms of Mandarin, Taiwanese, and Hakka in Taiwan (Chui & Lai, 2008). The sub‑corpus of spoken Mandarin contains short oral narratives and daily face‑to‑face conversations. The cartoon narrations were produced by twenty‑two NCCU undergraduate students in 2002. Each subject viewed a seven‑minute cartoon episode of the ‘Mickey Mouse and Friends’ series. The soundtrack of the cartoon included music and only a very small amount of dialogue. In the episode, Mickey, Minnie, Pluto and a bull are holding a party at the beach, and eating and playing around. They then have a fight with an octopus, which they finally win. The subject immediately recounted the story from memory to a listener after viewing the cartoon. The subject was filmed by a video camera so that speech and manual movements could be re‑ corded. The subjects were not informed about our particular research interests. The elicited cartoon narrations ranged from about two to ten minutes in length; the mean length of narration is four minutes thirty seconds. With regard to con‑ versations, there are two sets of data. The first set was collected during 1994 and 1995; the participants were college students who knew each other. The second set was casual conversations among family members, friends, and colleagues which have been videotaped since 2006, and this portion of data can be accessed online.1

All the participants were paid, and they were not told the particular focus of the research. The participants were free to find and develop topics of common inter‑ est; they were filmed for approximately an hour with a visible camera. One stretch from each talk, of about twenty to forty minutes, in which the participants were comfortable in front of the camera, was then selected for transcription. A further project related to The NCCU Corpus of Spoken Mandarin is a gestural analysis of the transcribed narratives and conversations. The data used in this study consist of ten complete narratives and seven conversational extracts (five from the first set and two from the second set) for a total of 183 minutes of talk. The same nar‑ rative data were used in the study of linguistic‑imagistic representation of motion in narrative discourse (Chui 2009b); however, because of different research issues, the statistics presented here do not totally accord with those in the former study.

For the purpose of the present study, the occurrence of manner and path ges‑ tures will be considered because the lexical‑syntactic packaging of MANNER and PATH constitutes a linguistic typology of motion, and the cross‑linguistic studies of the linguistic‑gestural representation of motion were mainly subject to these two components. The speech and the gesture data relevant for the present study were separately coded by two trained coders. The criterion to identify a motion

Do gestures compensate for the omission of motion expression in speech? 157

event was the linguistic description in the clause which consists of the main predi‑ cate and its argument(s), and/or gestural depiction of a protagonist of a move‑ ment from one place to another.2 _{The motion‑event gestures are the spontaneous}

movements of hands and arms depicting motion components. In the case of dis‑ agreement between the coders, data were re‑analyzed and discussed. Data with‑ out consensus were not used. Agreement was reached for a total of 180 motion events encoded by a single VP in a clause including manner and/or path gestures in conversations and 124 instances in narratives.3 _{They form the database for the}

investigation of gestural compensation during speaking.

3. The representation of motion in speech and gesture

A prototypical motion event “consists of one object (the ‘Figure’) moving or lo‑ cated with respect to another object (the reference‑object or ‘Ground’)… [and] the ‘Path’…is the course followed or site occupied by the Figure object with respect to the Ground object. ‘Motion’…refers to the presence per se in the event of motion or location” (Talmy, 1985: 61). Chinese is a verb‑serializing language in both spoken and written discourse (Slobin, 2000; Huang and Tanangkingsing, 2005; Chen and Guo, 2009, 2010; Chui, 2009b), a type of ‘equipollently‑framed’ languages where ‘‘both manner and path are expressed by ‘equipollent’ elements — that is, elements that are equal in formal linguistic terms, and appear to be equal in force or signifi‑ cance’’ (Slobin, 2004: 228). In Example (1) the last clause about Speaker F’s former classmate who had been walking by her side to attract her attention (Line 2) is concerned with a walking event. The covert subject is the FIGURE, the classmate referred to in the pronominal form tā in the preceding clause (Line 1); the noun phrase in the prepositional phrase — wŏ pángbiān ‘my side’ — is the GROUND; the main serial‑verb provides information about MOTION and MANNER in the form of zŏu ‘walk’, and about PATH and DIRECTION encoded by guò ‘go across’ and qù ‘go’, respectively (Line 2).

(1) 1 F: …(0.5) hěn qíguài o… tā méi cì… xiàkè jiù kāishǐ… very strange PRT 3SG every time after.class then start → 2 zŏu zŏu zŏu zŏu zŏu… a jiù cóng wŏ pángbiān

walk walk walk walk walk PRT just from 1SG side

zŏu‑guò‑qù

walk‑go.across‑go

F: ‘It’s very strange. Every time after class, he started to walk and walk and walk and walk and walk. He walked by my side.’

158 Kawai Chui

In addition to language, gesture also readily and frequently depicts various com‑ ponents of motion during speaking, as illustrated in Example (2). F1 is complain‑ ing about her past summer job at school when she had to keep running up and down stairs all day to make just one photocopy of one or two pieces of paper each time. The manner of the running event, encoded by the pro‑form zhèyàngzi ‘like this’, the adverbial yīzhí ‘continuously’, and the manner verb pǎo ‘run’ in Line 11, is depicted by the hands: At the time the numeral yī ‘one’ is verbalized, F1 starts raising her right hand with the fingers hanging down at waist level; her left hand also rises slightly. These movements prepare for the next running motion during the production of zhèyàngzi ‘like this’: F1 first moves the right hand toward her own body with the left hand flicking slightly outward. Then, both hands alternate the directions two more times successively till the first mention of pǎo has been uttered. These noticeable and discernable gestural configurations, being produced in front of the body for a comparatively long duration of 1.034‑seconds in total, enact the manner of running back and forth somewhere again and again, in this case to a photocopy machine.

(2) 1 F1: …(1.0) zhè yī fēnzhōng… jiào nĭ ná‑qù yìn yí fèn

this one minute tell 2SG take‑go print one CL

2 F2: ..Mm

BC

3 F1: ..wŏmen shi pǎo dào yī lóu qù

1PL COP run to first floor go

4 F2: ..Mm

BC

5 F1: ..yìn‑wán le… húi‑lái yǐhòu print‑finish PRF return‑come after 6 F2: …(0.5) zài ná‑qù [yìn yí fèn]

again take‑go print one CL

7 F1: [yòu guò méi‑duōjiǔ]… zài ná‑qù yìn…[[ zhè yí cì… again pass NEG‑so.long again take‑go print this one time bú shì yī fèn]]

NEG COP one CL

8 F2: [[o… shénjīngbìng]]

PRT nuts

9 F1: .. yí cì gěi ná gěi nĭ yī zhāng… huò liăng zhāng

one time give take to 2SG one CL or two CL

10 F2: ..o [biàntài]

PRT sick

→ 11 F1: [ránhòu… yī tiān nĭ jiù] zhèyàngzi… yīzhí păo then one day 2SG just like.this continuously run

Do gestures compensate for the omission of motion expression in speech? 159

yīzhí păo

continuously run

12 at yī ‘one’, right hand rises with fingers handing down at waist level; left hand rises slightly ((a) to (b) in Figure 1)

13 from zhèyàngzi to first mention of păo, right hand moves toward own body; left hand flicks slightly outward; then right hand out and left hand in; then right hand in and left hand out ((c) to (e) in Figure 1)

14 at the second mention of yīzhí, both hands return to thighs ((f) in Figure 1) F1: ‘At that moment, she told you to take the document and make a copy.’ F2: ‘Mm.’

F1: ‘We had to go down to the first floor.’ F2: ‘Mm.’

F1: ‘After printing, after we had returned,’

F2: ‘She told you to take it and make another copy.’

F1: ‘after a short while, I had to take it to make a copy again. But this time I didn’t make a copy of the whole document,’

F2: ‘Oh, she’s nuts.’

F1: ‘each time she gave me just one or two pieces of paper.’ F2: ‘Oh…she’s sick.’

F1: ‘Then you had to run upstairs and downstairs continuously all day.’

7

F1: ‗At that moment, she told you to take the document and make a copy.‘ F2: ‗Mm.‘

F1: ‗We had to go down to the first floor.‘ F2: ‗Mm.‘

F1: ‗After printing, after we had returned,‘

F2: ‗She told you to take it and make another copy.‘

F1: ‗after a short while, I had to take it to make a copy again. But this time I didn‘t make a copy of the whole document,‘

F2: ‗Oh, she‘s nuts.‘

F1: ‗each time she gave me just one or two pieces of paper.‘ F2: ‗Oh...she‘s sick.‘

F1: ‗Then you had to run upstairs and downstairs continuously all day.‘

Figure 1. Gestural depiction of running continuously.

Table 1 shows the frequency distribution of manner gestures and path gestures with a total of 180 motion events with gestures in conversations and 124 in narratives. Based on these statistics, the next section will investigate the coordination of information in speech and gesture when one of the two components is omitted in speech. Does gesture occur to depict information which was not spoken?

Table 1. Gestures for MANNER and PATH

Conversations Narrations

I Path gestures 114 63.3% _{112 90.3%}

II Manner gestures 31 17.3% _{5 4.0%}

III Combined manner &

path gestures 35 19.4% 7 5.7%

Total: 180 100.0% 124 100.0%

4. Gestural compensation across languages

Gestural compensation has been brought up in previous studies. McNeill & Duncan (2000:

(a) Line 12 (b) Line 12 (d) Line 13

(e) Line 13 (f) Line 14

(c) Line 13

Figure 1. Gestural depiction of running continuously.

Table 1 shows the frequency distribution of manner gestures and path gestures with a total of 180 motion events with gestures in conversations and 124 in narratives.

160 Kawai Chui

Based on these statistics, the next section will investigate the coordination of in‑ formation in speech and gesture when one of the two components is omitted in speech. Does gesture occur to depict information which was not spoken?

Table 1. Gestures for MANNER and PATH

Conversations Narrations

I Path gestures 114 63.3% 112 90.3% II Manner gestures 31 17.3% 5 4.0% III Combined manner & path gestures 35 19.4% 7 5.7% Total: 180 100.0% 124 100.0%

4. Gestural compensation across languages

Gestural compensation has been brought up in previous studies. McNeill & Dun‑ can (2000: 150) reported that “[a]lthough Spanish speakers often omit manner from their speech, manner is abundant in their gestures and combines with other linguistic categories, typically path (verb) and/or ground (nominal phrase).” Their study, however, did not provide quantitative data which could be used to com‑ pare their findings with those of other related studies, such as those based on the storytellings of the same Sylvester and Tweety cartoon or of a set of video clips depicting motion events (Özyürek & Kita, 1999; Kita & Özyürek, 2003; Özyürek et al., 2005; Kita et al., 2007). In Kita & Özyürek (2003), Turkish, Japanese, and English speakers were all found to gesture the lateral direction of a swing event and a rolling event that was not verbalized in their narrations. Özyürek et al.’s (2005: 236) experimental data produced by English and Turkish speakers, on the other hand, showed the general tendency that “gestural information was found to fit the semantic encoding of the event rather than compensate or convey mean‑ ing not expressed by speech.” In fact, manner gestures were still found in English and Turkish path‑only clauses (see Figure 2 in Özyürek et al., 2005: 233), and the percentages were even higher for the occurrence of path gestures in manner‑only clauses (see Figure 3 in Özyürek et al., 2005: 234). It is thus not conclusive whether motion‑event gestures compensate for the absence of linguistic representation across typologically different languages.

This section investigates whether there is a compensatory relationship to co‑ ordinate information across the two modalities in Chinese discourse. Example (3) illustrates a path gesture without linguistic expression of PATH produced by F1 while she is talking about a walking event. The scenario is: F1 was in a car with her colleague. As they were going downhill, F1 saw two of her students walking