主題性與手勢之互動(3/3)

行政院國家科學委員會專題研究計畫 成果報告

主題性與手勢之互動(3/3)

中 華 民 國 94 年 10 月 12 日

行政院國家科學委員會補助專題研究計畫

█

成 果 報 告

□期中進度報告

主題性與手勢之互動 (Topicality and Gesticulation)

計畫類別：

█

個別型計畫 □

整合型計畫

計畫編號：

NSC 91-2411-H-004-021；NSC 92-2411-H-004-005;

NSC 93-2411-H-004-003

執行期間： 91 年 8 月 1 日 至 94 年 7 月 31 日

計畫主持人：徐嘉慧

共同主持人：

計畫參與人員：

成果報告類型(依經費核定清單規定繳交)：□精簡報告

█

完整報告

本成果報告包括以下應繳交之附件：

□赴國外出差或研習心得報告一份

□赴大陸地區出差或研習心得報告一份

□出席國際學術會議心得報告及發表之論文各一份

□國際合作研究計畫國外研究報告書一份

處理方式：除產學合作研究計畫、提升產業技術及人才培育研究計畫、列

管計畫及下列情形者外，得立即公開查詢

□涉及專利或其他智慧財產權，□一年□二年後可公開查詢

執行單位：國立政治大學英國語文學系

中 華 民 國 94 年 10 月 9 日

中文摘要 本計畫為期三年，主要探討日常會話和口語敘述中語句主題性與手勢的關 係。首先會話語料顯示手勢的類型跟語句主題性沒有關係，然而，在傳達主題性 的新訊息時，說話者較常使用那種跟語詞有直接語意關係的手勢，而且這類手勢 大多包括準備階段，或者手勢會出現在對應語詞之前，以提示聽話者注意即將傳 達的新訊息。口語敘述的語料呈現不一樣的結果，就是敘述者在傳達主題訊息時 會使用較多手勢，尤其是那種跟語詞有直接語意關係的手勢。這類手勢，還有表 達節奏的手勢，通常跟對應的、又帶有新訊息的語詞一併出現。最後，在主題語 境中，在對應語詞前出現的手勢有較多的準備階段，同樣可以提示聽話者注意即 將傳達的新訊息。總之，手勢的使用在不同的說話環境中有所不同；不過根據那 些在日常會話和口語敘述中所呈現的共同使用手勢的行為，是有「主題性手勢」 的，也就是那種跟語詞有直接語意關係的手勢出現在主題語境中而對應的語詞是 帶有新訊息。 關鍵字：語句的主題性, 手勢類型, 新/舊訊息 Abstract

This is a three-year project that studied the speech-associated manual gestures in relation to the topicality of the utterances in both narrative and conversational discourse, based on the semantic and pragmatic aspects of information on the part of the speaker’s verbalization. In daily chats, it is found that gestural types do not distinguish topical and non-topical information. However, when the information state of the associated referents is considered, speakers tend to gesture iconically for new information in topical context. Moreover, either onsets tend to occur in topical clauses or strokes would come before topical referents, to signal that the upcoming new information is noteworthy and deserves attention. Different from conversation, the narrators have more hand movements while unfolding topical information. Iconic gestures are also more commonly used in storytelling. They predominate in topical context, and usually synchronize with new topical referents. Both iconics and beats tend to co-occur with new topical information. Finally, onsets are inclined to accompany preceding iconic gestures in topical context, also signaling that the upcoming new information is noteworthy and deserves attention. Thus, gesturing behaviors are not always the same across different communication environments. But based on the consistent gesturing behaviors in conversation and narratives, topical gestures can refer to iconic hand movements that accompany new referents in topical clauses.

Table of Content

中文摘要... I Abstract ... I

1. Introduction...1

2. Database and preliminaries ...1

2.1. Temporal phases of gesturing ...3

2.2. Types of gestures ...3

2.3. Topicality of utterances...4

3. Findings...7

3.1. Gesture and topicality ...7

3.2. Information state, gesture, and topicality...8

3.3. Temporal synchrony, gesture, and topicality ...10

4. General discussion ...12

5. Self-evaluation ...14

References...15

Appendix A: Abbreviations of Linguistic Terms ...17

Topicality and Gesticulation

1. Introduction

When people talk face-to-face, they often produce idiosyncratic spontaneous gestures through use of hands and arms with context-dependent meaning and use. The past research has shown that people use this kind of manual actions in different ways to distinguish various aspects of information conveyed by the utterances. Based on storytelling data, McNeill (1992:93) finds that “iconics occur overwhelmingly in narrative clauses, while beats can occur in both narrative and extranarrative clauses [storytelling act not on the plot line;]…abstract pointing occurs chiefly with narrative clauses, whereas metaphorics appear chiefly with extranarrative clauses.” Bavelas et al. (1992:473) rather distinguished topic gestures and interactive gestures in conversation to characterize, respectively, the information directly related to the topic of discourse, and the information referring “to some aspect of the process of conversing with another person.” They further suggest that interactive gestures may function to cite the other’s previous contribution, to seek agreement, understanding, or help, to deliver new versus shared information, and to present events around the speaker turn. With respect to communicative dynamism, Levy and McNeill (1992) and McNeill and Levy (1993) show that the quantity and complexity of gestures would increase along with complex linguistic expressions as the information conveyed by the utterances functions to push the communication forward. Concerning given and new information, McNeill and Levy (1993:365) suggest that “gestures tend to occur at points of topic shift, such as new narrative episodes or new conversational themes…highly presupposed linguistic elements would either lack gestures entirely, or would be accompanied by gestures that are specialized for their cohesive function or form.” Finally, Kendon (1995:247) differentiated substantive gesturing from

pragmatic gesturing, in that “[the former] contributes to various aspects of the content

of the utterance of which it is a part, whether literally or metaphorically… [whereas the latter] expresses aspects of utterance structure, including the status of discourse segments with respect to one another, and the character of the ‘speech act’ or interactional move of the utterance.”

In this three-year project, I studied the speech-associated manual gestures in relation to the topicality of utterances in both conversational and narrative discourse, based on the semantic and pragmatic aspects of information on the part of the speaker’s verbalization, rather than on the interaction between the speaker and the addressee. The questions to be addressed are: Do the speakers gesture for topical and non-topical information? In other words, is there a distinction between topical

gestures and non-topical gestures? Does the initiation of topical and non-topical

gestures relate to the information state of referents? How does the timing of gesturing coordinate with speech production in topical and non-topical contexts? Is there any difference between gesturing while talking with friends and gesturing in storytelling?

2. Database and preliminaries

The database for this study consists of daily chats and narratives. First, five casual unpremeditated, multi-party conversations took place in 1994 and 1995 among college students who knew each other. They were free to find topics of common interest without knowing our particular research interests. Subjects were filmed for approximately an hour with a visible camera for full-body shot. One section from

each conversation, about twenty minutes of talk, in which students were more comfortable in front of the camera, was then extracted.

Oral narratives were audio-video-taped in ‘a social/monologue condition’ with a silent listener, and in ‘a social/dialogue condition’ with an active and contributing listener. The gender of the subjects was also considered. See the details in Table 1. Each of the 48 subjects watched a Mickey Mouse and Friends cartoon which was about ten minutes long. Then, the subject would immediately tell the story from memory to an interviewer/listener who remained silent in the social/monologue condition, but raised questions in the social/dialogue condition, as the performance was videotaped. Half of the data were analyzed for the present study. Moreover, for the purpose of comparing the results in conversational discourse, gender variation and the two different narrative conditions were not distinguished here.

Table 1. Narrative data

The social/monologue condition: 23 subjects

Interviewer Subjects Interviewer Subjects Interviewer Subjects Interviewer Subjects (silent) (silent) (silent) (silent)

M M M F F F F M M F F M M F F M M F F M M F F M 5 5 F M F 6 7

The social/dialogue condition: 25 subjects

Interviewer Subjects Interviewer Subjects Interviewer Subjects Interviewer Subjects (raise qsts) (raise qsts) (raise qsts) (raise qsts)

M M M F F F F M M F F M M F F M M F F M M F F M M 5 F M 6 6 M M 8

Gestures were analyzed on computer using MediaStudio Pro which has frame-by-frame advance and varying slow-motion capabilities with no muting of sound so that sound could be heard as the images were advanced. Thus, movement at a given moment in time can be matched with the simultaneously uttered syllable or with silence.

2.1. Temporal phases of gesturing

According to McNeill (1992:83), there are mainly three phases of gesticulation:

preparation, stroke, and retraction. The preparation phase, also known as onset, refers

to “the limbs mov[ing] away from their rest position to a position in gesture space where stroke begins.” In the stroke phase, “the meaning of the gesture is expressed.” Finally, the retraction phase is the “return of the hand to a rest position.” Both the preparation and the retraction are optional; the stroke is obligatory. Gestural strokes can further be distinguished into three types, based on their temporal realization vis-à-vis the associated words, namely synchronizing gestures being simultaneous with related words, preceding gestures coming before related words, and following

gestures coming after related words. See Chui (2005) for a detailed discussion of the

temporal issue in Chinese conversation.

Since the preparation and the stroke phase are relevant to unfolding information in speaking, both will be analyzed in this study. Whether the hands return to the rest position, i.e., the retraction phase, right after the stroke is not the main concern here.

2.2. Types of gestures

The obligatory stroke phase determines the interpretation and categorization of hand movements. I have distinguished five types of gestures: iconic gestures whose meanings correspond to the semantic content of the related speech, metaphoric

gestures for abstract ideas, deictic gestures pointing at referents in the immediate

speech environment, spatial gestures manipulating the gesture space to depict a spatial relation between the linguistic constituents, or between the speaker and the linguistic constituent, and beats indicating the rhythm of speech. A detailed discussion of categorizing gestures in conversation can be found in Chui (2003).

The five stretches of conversation yielded 1475 instances of gesture associated with single words; the narratives had 598 instances. Their respective frequency distribution across the five gestural types is presented in Table 2.

Table 2. Gestural types

conversation narratives Iconic gestures 343 23.3% 366 61.2% Metaphoric gestures 27 1.8% 0 0.0% Deictic gestures 158 10.7% 4 0.7% Spatial gestures 167 11.3% 19 3.2% Beats 780 52.9% 209 34.9% total: 1475 100% 598 100%

conversations, the conversational data in the present study maintains similar tendencies: Beats also constitute the majority (52.9%). Of the two types closely related to the semantic content of the affiliated referents, iconics outnumber metaphorics to a large extent, by thirteen times here. However, no metaphoric gestures were found in the narrative data, and iconic gestures are rather the majority, outnumbering beats almost by two times.

2.3. Topicality of utterances

Chui (2001) has proposed topic chain to characterize the structuring of spontaneous conversational topics.

Topic chain as a discourse category characterizes text organization based on the semantic relationship between utterances. A sequence of clauses about the same subject matter introduces a topic; a topic, or more commonly, a number of semantically-connected topics then form a topic chain; the topic chains sharing the identical theme further constitute a larger topic chain. (Chui 2001:27)

Topicality of utterances is thus identified within a topic chain. Within a (sub-) topic chain domain, topical clauses function to introduce new topics and move the discourse forward; non-topical clauses are orientations, descriptions, and digressions concerning a particular topic.

Identifying topical and non-topical clauses is the first indispensable step to study the role gesture plays in conveying different types of information. Gestures taking place when the speaker unfolds topical information are separated from those that occur as the speaker conveys non-topical information. In example (1), the subject for talk is how hard speaker A’s part-time job was at the library, as expressed by the topical clauses from IU1 to IU6 (see Appendix B for the definition of IU), indicating that the speaker had to send a document to different places, from the library to Administration Building, then back to the library, and then to Social Sciences Information Center. The manual forms that accompany these topical clauses include two spatial gestures and two iconic hand shapes for the verbs na ‘hold’ and song ‘send’. First, at the moment of uttering the verb na (IU2), both A’s hands rise from the thighs and the arms cross at belly level as if holding a document. In gesturing song in IU3, the speaker’s both hands move to the upper right periphery, conveying the idea of sending a document to Administration Building. They move back a bit for the second mention of the same verb in IU5, depicting the action of sending the document back to the library. Finally, the speaker also produces two spatial gestures, locating

xingzhengdalou ‘Administration Building’ in the right periphery (IU4) by moving

both hands downward once on her right side, and shezizhongxin ‘Social Sciences Information Center’ in the left periphery (IU6) as both hands move in the same way on her left side.

(1) 1 A: …Ranhou wo jiu%,_ then 1SG thus 2 Æ …(.7) na yi ge gongwen,_

hold one CL document 3 Æ …(.6) song dao%,\

4 Æ …(.7) xingzhengdalou,_

Administration Building 5 Æ …(.6) song huilai,\

send back

6 Æ … song qu shezizhongxin,_

send to Social Sciences Information Center

A: ‘Then I took a document, sent (it) to Administration Building. (After that I) sent (it) back (to the library). (Then I) sent (it) to Social Sciences Information Center.’

The following excerpt (2) exemplifies a metaphoric gesture. The talk centers on the power struggle between the teacher and students in class. It has two major events moving the topics from the classroom being an arena of power (IU1-2) to the need to have the power balanced (IU4, 5, 7). For speaker A to move her left hand level to the left periphery during the 0.7-second pause (IU7), with the palm facing down and fingers extended, for the verb-to-be-uttered pingheng ‘balance’, illustrates a metaphorical extension from the domain of making something level, as literally represented by ping of pingheng, to the domain of making something equal.

(2) 1 A: … Nanguai renjia dou gen wo shuo,_ no wonder people all with me say 2 ..jiaoshi shi yi ge quanlichangyu,\ classroom COP one CL arena of power 3 D: (0)<P m= P>,_ Mm 4 A: (0) Ni%,\ 2SG 5 .. jiu[shi%],\ that is 6 D: (0)[ hm],\ Hm

7 Æ A: …(.7)[[yao qu]] pingheng ta,_

have to go balance 3SG 8 D: [dui],\

right 9 ..dui,\

right

A: ‘No wonder people say to me (that the) classroom is an arena of power,’ D: ‘Mm.’

A: ‘You, that is,’ D: ‘Hm.’

A: ‘have to balance it.’ D: ‘Right, right.’

Different from the topical gestures just mentioned, those in the following examples are produced when the speakers are conveying non-topical information. First, the participants in (3) are discussing how to organize the departmental gathering to be held at the end of the semester, and finding a good place is a major subject matter. The clause about Qixian Building which allows students to cook (IU4-6) is non-topical, since it functions as a reason, being marked by the conjunctive yinwei ‘because’, for the previous topical statement (IU1-3) about changing the place to Qixian Building. There are two hand shapes for the co-expressive verb jian ‘fry’ in the causal clause: Every time the speaker produces jian, her right hand at shoulder level sweeps downward to the left side one time with the fingers closed in, as if to fry something with a cooking utensil. These ‘frying’ gestures are iconic, but unlike the

ones for na ‘hold’ and song ‘send’ in (1), they were counted as non-topical gestures.

(3) 1 C: … Ni <A zhidao A>,_ 2SG know

2 ..women xianzai xiang yao huandao,\ 1PL now think want change 3 ..^qixianlou,_

Qixian Building 4 ...Yinwei qixianlou,_

because Qixian Building 5 Æ ..keyi jian dan%,\

can fry egg 6 Æ ...jian huotui,\

fry ham

C: ‘You know, we now want to change (the place) to Qixian Building, because (we) can fry eggs and ham at Qixian Building.’

In the following stretch of speech (4), the participants are talking about Chinese with mixed blood. For speaker A to recall how people reacted when she went out with such kind of person (IU1-4) is a topical event in this subject of conversation. The next rhetorical question (IU6-8) functions to elaborate on people’s reaction, explaining that people usually felt strange toward Chinese kids going out with a foreign kid. As the speaker elucidates, two co-occurring hand movements are performed to depict a spatial relationship between yi qun zhongguo xiaohaizi ‘a group of Chinese kids’ and

ge waiguo xiaohaizi ‘a foreign kid’: The speaker splits the gesture space with her

hands rising from the thighs to chest level. She first locates the Chinese kids by moving the hands to the space on her right side while uttering the first syllable zhong of the nominal zhongguo xiaohaizi. Then she locates the foreign child by moving the hands to the space on her left side at the time she produces the verbal dai ‘take’. These two spatial gestures are also non-topical.

(4) 1 A: …(.5) Women xiaoshihou a,_ 1PL be.little PRT

2 .. mei ci guangjie,_

every time go window-shopping 3 .. ranhou gen ta chuqu wan a,\ then with 3SG go out play PRT 4 ..[ bieren] dou hui juede hen qiguai.\ others all will feel very be.strange 5 B: [ hm],\

Hm

6 Æ A: .. Wei<A shenme yi qun zhongguo xiaohaizi,\

why one CL Chinese kid 7 Æ .. dai ge waiguo xiaohaizi,\

take CL foreign kid 8 .. ni zhidao ma?/

2SG know QST

A: ‘(When) we were little, every time (we) went window-shopping…(we) went out with her for fun, people (would) feel very strange (about it).’

B: ‘Hm.’

A: ‘(They would think) why a group of Chinese kids took a foreign kid, you know.’

well as one metaphoric gesture in a non-topical clause. The conversational topic here is on nationalism. While the speaker talks about the topical information that the future is very important for realizing nationalism in Taiwan (IU1-2), two topical beats are performed by flicking the speaker’s left hand down from chest level for the pronominal wo ‘I’ (IU1), and then to the right for the adverbial ye ‘also’ (IU2). Non-topical information can be found in the next clauses (IU3-10) which, as a whole, function as a reason, also marked by yinwei ‘because’, to account for the previous main idea. The reason is that Taiwan should have a prospect of a new culture, despite the fact that people also want to find their roots. The non-topical hand movement for the nominal genyuan ‘root; origin’ (IU6) conveys a metaphorical extension from the domain of plant to the domain of human origin. While speaker A verbalizes the first syllable of genyuan, the fingers of her left hand are wide open downward to represent the roots of a plant growing into the soil, just like the human origin forming the base of a person.

(5) 1 Æ A: … ^Wo juede%,_ 1SG feel

2 Æ …(.7) wanghou kan ye shi hen zhongyao,\ future see also COP very important 3 Æ .. Yinwei ni yao,\ because 2SG want 4 …(.6) ni shi yao% -- 2SG EMP want 5 … ^qu,| go

6 Æ .. xunhui ni de genyuan meicuo,\

find.finish 2SG POSS root be.NEG.wrong 7 .. Keshi ni,\

but 2SG 8 .. ye%,\

also

9 .. tongshi ni ye yao,\ at the same time 2SG also should 10 .. qu zhanwang xin wenhua,\ go have a prospect new culture

A: ‘I think (it is) also very important to face the future, because you want…you want to find your root, (which) is not wrong, but you also…also, at the same time, should have a prospect of a new culture.’

The examples provided in this section show clearly that the five types of gestures can take place in both topical and non-topical contexts, despite the fact that the narrators did not gesture metaphorically in the present data base. Whether the various types of hand movement exhibit a preference for a certain kind of information will be pursued in the next section.

3. Findings

3.1. Gesture and topicality

How do Chinese speakers gesture while presenting topical and non-topical information? The frequency distribution of the five types of speech-accompanying hand shapes vis-à-vis topicality is given in Table 3. The statistics show that the speaker gestures more, at 63.4% on average, while uttering non-topical information in

conversation, regardless of gestural types. But it is the opposite in storytelling, since gesturing in topical context predominates at 66.2% on average.

Table 3. Topicality and gestural types conversations

Topical Non-topical total Iconic gestures 140 40.8% 203 59.2% 343 100.0% Metaphoric gestures 5 18.5% 22 81.5% 27 100.0% Deictic gestures 52 32.9% 106 67.1% 158 100.0% Spatial gestures 77 46.1% 90 53.9% 167 100.0% Beats 266 34.1% 514 65.9% 780 100.0% total: 540 36.6% 935 63.4% 1475 100.0% narratives

Topical Non-topical total Iconic gestures 265 72.4% 101 27.6% 366 100.0% Metaphoric gestures 0 0.0% 0 0.0% 0 0.0% Deictic gestures 2 50.0% 2 50.0% 4 100.0% Spatial gestures 14 73.7% 5 26.3% 19 100.0% Beats 115 55.0% 94 45.0% 209 100.0% total: 396 66.2% 202 33.8% 598 100.0%

The statistics in Table 3 further bear out McNeill’s (1992) finding that iconic gestures in his narrative data occur overwhelmingly in narrative (i.e., topical) clauses. Nevertheless, the result is not universal, as iconics produced for topical referents in Chinese conversation are just a minority (40.8%). Concerning the idea of

communicative dynamism proposed by Levy and McNeill (1992) and McNeill and

Levy (1993), utterances that function to push the communication forward can be equivalent to topical clauses in the present study, while gestural complexity can refer to iconic and metaphoric gestures in particular. But contrary to what they claim, these two types of hand movement, together with the other three types, do not predominate in topical contexts in Chinese conversation; most of them co-occur with utterances that do not “push the communication forward.” In short, gesturing behaviors are not consistent across different communication environments.

3.2. Information state, gesture, and topicality

McNeill and Levy (1993) have proposed that gesturing can be affected by the information state of associated referents. This section thus investigates the relationship among information state, gesture, and topicality. The information state of nominal and verbal referents in the main clause will be considered. The flow of information throughout a discourse is a dynamic process, in that the varied aspects of information might change in the course of communication. A referent was analyzed as

new, if it had never been brought up in the previous context at the moment of

the moment of utterance. In this study, nouns and verbs were analyzed and combined for tabulation.

Does the given-new distinction differentiate gesturing for topical and non-topical information? Table 4 presents the frequency distribution of given and new referents across the five types of gestures vis-à-vis topicality. The data were analyzed using chi-square tests.

Table 4. Gesture, topicality, and information state

Conversations: Topical clauses

Given New Total Iconic gestures 29 22.7% 99 77.3% 128 100% Metaphoric gestures 1 20.0% 4 80.0% 5 100% Deictic gestures 42 84.0% 8 16.0% 50 100% Spatial gestures 34 50.7% 33 49.3% 67 100% Beats 74 46.0% 87 54.0% 161 100% total: 180 43.8% 231 56.2% 411 100%

Conversations: Non-topical clauses

Given New Total Iconic gestures 78 40.4% 115 59.6% 193 100% Metaphoric gestures 7 38.9% 11 61.1% 18 100% Deictic gestures 86 81.9% 19 18.1% 105 100% Spatial gestures 42 49.4% 43 50.6% 85 100% Beats 141 50.0% 141 50.0% 282 100% total: 354 51.8% 329 48.2% 683 100%

Narratives: Topical clauses

Given New Total Iconic gestures 49 18.5% 216 81.5% 265 100% Metaphoric gestures 0 0.0% 0 0.0% 0 100% Deictic gestures 2 100% 0 0.0% 2 100% Spatial gestures 5 35.7% 9 64.3% 14 100% Beats 41 35.7% 74 64.3% 115 100% total: 97 24.5% 299 75.5% 396 100%

Narratives: Non-topical clauses

Given New Total Iconic gestures 52 51.5% 49 81.5% 101 100% Metaphoric gestures 0 0.0% 0 0.0% 0 100% Deictic gestures 1 50.0% 1 50.0% 2 100% Spatial gestures 2 40.0% 3 60.0% 5 100% Beats 54 57.4% 40 42.6% 94 100% total: 109 54.0% 93 46.0% 202 100%

The correlation between information state and topicality in conversational discourse is not statistically significant for metaphoric, deictic, spatial, and beat gestures.1 In fact, regardless of topicality, metaphoric gestures mainly accompany new referents (65.2%, 15 out of a total 23); the overwhelming majority of deictic gestures co-occur with given referents (82.6%, 128 out of 155); spatial and beat gestures do not show a distinct preference, since the proportion of given to new referents is about equal. Concerning iconic gestures, the X2 values are highly significant in both genres.2 In conversation, 77.3% of all gestured referents in topical utterances convey new information; the percentage is reduced to 59.6% in non-topical utterances; in storytelling, the percentage is reduced from 81.5% to 48.5%. In short, speakers are inclined to initiate manual actions while presenting new information. What kind of new information they would gesture for depends on topicality, since speakers are even more likely to produce gestures for new topical information. The narrative discourse, again, exhibits inconsistency. The main difference between the two genres lies in beats, which also tend to co-occur with new referents in topical narrative context.3

3.3. Temporal synchrony, gesture, and topicality

Based on the finding that the information state of referents is relevant to the use of iconic gestures in topical clauses across the two text types, this section only focuses on new information and iconic hand shapes, investigating how gestures are patterned temporally with the affiliated words vis-à-vis the preparation and the stroke phase in both topical and non-topical contexts. While new topical referents are more noteworthy, the questions are: Do gestural strokes and the affiliated topical speech tend to be produced at the same time to increase expressivity? Or do they often come before speech to signal that the upcoming referents deserve attention? For the same reason, are onsets more likely to occur in topical clauses?

First of all, not every hand shape includes the preparation phase because of its

1

The Chi-square tests for the distribution of four types of gestures accompanying given and new referents in topical and non-topical clauses in conversation are: metaphoric gestures, X2.95(1)=0.615;

deictic gestures, X2.95(1)=0.103; spatial gestures, X2.95(1)=0.027; beats, X2.95(1)=0.669. 2

The Chi-square test for the distribution of iconic gestures accompanying given and new referents in topical and non-topical clauses is: X2.95(1)=*10.92 in conversation; X2.95(1)=*39.844 in narratives. 3

The Chi-square tests for the distribution of four types of gestures accompanying given and new referents in topical and non-topical clauses in narratives are: deictic gestures, X2.95(1)=1.3333; spatial

optionality. Of a total of 99 iconics accompanying new topical referents in conversation, and of 216 iconics in narratives, onsets constitute 29.3% (29 instances) and 35.6% (77 instances) respectively. In non-topical clauses, the percentages are reduced to 20.9% (24 instances) among all 115 new referents in daily chats, and to 28.6% (14 instances) among all 49 new referents in storytelling.

At the obligatory stroke phase, gestural strokes can further be distinguished into three types, based on their temporal realization vis-à-vis the accompanying words (see section 2.1). Of a total of 214 iconics in conversation and of 265 iconics in narratives, the respective frequency distribution of synchronizing gestures, preceding gestures, and following gestures across topical and non-topical clauses is shown in Table 5. When the statistics in topical utterances and those in non-topical utterances are compared, the X2

value is found insignificant in conversation.4 The simultaneous realization of speech and gesture comprises major number of instances of gesture, irrespective of topicality. However, the X2

value is significant in narratives5: iconic gestures tend to synchronize with new topical information.

Table 5. Synchronization of iconic gestures and new referents conversations

topical clauses non-topical clauses Synchronizing gestures 61 61.6% 67 58.3% Preceding gestures 34 34.3% 44 38.3% Following gestures 4 4.0% 4 3.5%

total 99 100.0% 115 100.0%

narratives

topical clauses non-topical clauses Synchronizing gestures 158 73.1% 26 53.1% Preceding gestures 54 25.0% 22 44.9%

Following gestures 4 1.9% 1 2.0%

total 216 100.0% 49 100.0%

Moreover, topicality can be seen to play a role in timing speech and gesture by considering the preparation and the stroke phase together. Table 6 shows the occurrences of onsets that synchronizing, preceding, and following gestures have in the two types of clauses. While the new topical information is noteworthy, onsets or strokes prior to speech should be more common in topical utterances to signal that the upcoming information deserves attention. This is evidenced by the statistics in Table 6. First, in conversation there are a lot more onsets for synchronizing gestures in topical

4

The Chi-square test for the distribution of synchronizing, preceding, and following gestures accompanying new referents in topical and non-topical clauses in conversation is: X2.95(2)= 0.369. 5

The Chi-square test for the distribution of synchronizing, preceding, and following gestures accompanying new referents in topical and non-topical clauses in narratives is: X2.95(2)= *7.842.

clauses (29.5% vs. 17.9%). Second, despite the fact that more onsets for preceding gestures occur in non-topical clauses (25% vs. 17.6%), for the strokes to be produced before the referents already fulfils the function. Finally, when the strokes come after speech, most of them have onsets in topical clauses (75% vs. 25%). In short, the timing of iconic gestures and new referents distinguishes topical and non-topical information in conversation. In storytelling, however, the synchronizing gestures do not distinguish topical and non-topical information. Rather, onsets for the preceding iconic gestures are more common to present topical information.

Table 6. Synchronization of iconic gestures, new referents, and onsets

Conversations: Topical clauses

total with onset Synchronizing gestures 61 18 29.5%

Preceding gestures 34 6 17.6% Following gestures 4 3 75.0%

Conversations: Non-topical clauses

total with onset Synchronizing gestures 67 12 17.9%

Preceding gestures 44 11 25.0% Following gestures 4 1 25.0%

Narratives: Topical clauses

total with onset Synchronizing gestures 158 59 37.3%

Preceding gestures 54 15 27.8% Following gestures 4 3 75.0%

Narratives: Non-topical clauses

total with onset Synchronizing gestures 26 9 34.6%

Preceding gestures 22 4 18.2% Following gestures 1 1 100.0%

4. General discussion

Topicality is a fundamental universal property of text organization. This study has examined the relationship between topicality of utterances and gestural use in Chinese conversation and storytelling. The results are not consistent across the two text genres. In daily chats, it is found that gestural types per se do not distinguish topical and non-topical information, since their respective occurrences in topical and non-topical contexts are similar. However, when the information state of the

associated referents is considered, the use of iconic gestures is found to be related to this textual notion: Chinese speakers mainly gesture iconically for new information in topical clauses. The patterning of given and new information accompanying metaphoric, deictic, spatial, and beat gestures, however, is similar across the two types of clauses. Finally, speakers rarely produce manual movements while conveying given information, be it topical or not. Based on the finding that the information state of referents is relevant to the use of gesture, the temporal patterning of iconic hand shapes and the associated new referents further distinguished topical and non-topical information: Either onsets tend to occur in topical clauses or strokes would come before topical referents. Both function to signal that the upcoming new information is noteworthy and deserves attention, and the iconic gestures tend to synchronize with the associated referents.

Different from conversation, the narrators have more hand movements while unfolding topical information. Iconic gestures are also more commonly used in storytelling. They predominate in topical context, and usually synchronize with new topical referents. Moreover, both iconics and beats tend to co-occur with new topical information. Finally, onsets are inclined to accompany preceding iconic gestures in topical context, also signaling that the upcoming new information is noteworthy and deserves attention.

Thus, gesturing behaviors are not consistent across different communication environments. Then, do we have topical gestures? Despite the genre differences mentioned above, gesturing in topical context is, by and large, different from that in non-topical context. Based on the consistent gesturing behaviors in conversation and narratives, topical gestures can refer to iconic hand movements that accompany new referents in topical clauses.

The results suggest a close relationship between speech and gesture. Chui (2001) has shown that foregrounded topical clauses are structured differently from backgrounded non-topical clauses. In other words, while speech structures in various ways in accordance with different types of textual information, it is no coincidence that the use of speech-associated iconic gestures also varies in topical and non-topical contexts. Speaking and gesturing must be different manifestations of a single underlying process of utterance production: They are in close association in terms of phonemic clauses, tone units, breath groups, or syntagmata (Kendon 1983), or based upon the very close temporal, semantic, pragmatic, pathological, and developmental parallels between speech and gesture (McNeill 1985, 1992).

Furthermore, many studies have shown that discourse plays a role in the unified system of gesture-utterance formation. As mentioned in section 1, the functions of interactive gestures go beyond the semantic content of speech, and lie mainly in the interaction between the speaker and the addressee (Bavelas et al. 1992). Gestures can also signal topic shift and information state (McNeill and Levy 1993), or facilitate lexical retrieval (Butterworth & Hadar 1989; Hadar & Butterworth 1997; Morrel-Samuels & Krauss 1992). Pragmatic gestures in Kendon’s (1995) study are concerned with utterance structure or speech acts. Examining gesture and speech together “can track the changes in a speaker’s ongoing contextual thinking throughout a discourse” (Dray and McNeill 1990:477). Streeck (1995:87) claims that ‘moment by moment, the speaker’s gestures prefigure the next moment, allowing the participants to negotiate joint courses of action until, finally, a communication problem is solved collaboratively.’ The findings of the present study also suggest a pragmatic dimension

for the imagistic-linguistic conceptual framework in determining what gestural type would be used and how it coordinates with speech in communication. While iconic gestures exhibit images that are semantically parallel to the associated speech, providing conceptual imagery for the content of the utterance-in-progress, their use in Chinese discourse has to take into account whether the speaker is uttering topical or non-topical information, whether the related referent carries new or old information, and whether the information is newsworthy or not. These various aspects of information for verbalization are bound to context, and they have to be incorporated in the process of producing speech-associated gestures along with speech.

Gestures are part of the discourse (McNeill 1992). Studying gesture in spontaneous speech is thus crucial to understanding the intricate relationship among gesturing, linguistic expressions, and language use, through which we can know more about communication and the cognitive unity of speech and gesture.

5. Self-evaluation

In this three-year project, establishing a corpus of oral narratives was the major task in the first year. I audio-video-taped narratives in both ‘a social/monologue condition’ with a silent listener, and in ‘a social/dialogue condition’ with an active and contributing listener. The gender of the subjects was also considered. Then, both the speech and the speech-associated gestures were transcribed for analysis. In the second year, the gesture transcription continued. Moreover, Chui’s research on conversational gesturing was compared to narrative gesturing concerning (1) gesture types, (2) temporal synchrony, (3) semantic aspects of gesticulation, and (4) pragmatic aspects of gesticulation. In the second half of the year, the textual organization of the utterances in the two genres were also analyzed with regard to ‘topic chain (TC)’. In the third year, the gesture transcription and the analysis of topicality continued. At the same time, I used half of all the data, which had been fully transcribed and analyzed for tabulation, to examine the theme of the project—the relationship between topicality of the utterances and the coverbal gestures in Chinese narrative and conversational discourse—was studied, based on the analyses and findings in the first two years. Four issues were discussed: (1) Do topical and non-topical clauses involve different kinds of gestures? (2) If they do, what are the functions of the gestures? (3) How do gestures and speech cooperate in introducing new events and new participants in the two types of clauses? (4) How do they cooperate in maintaining old events and old participants?

The major contribution of this study is the realization that gesturing in conversation is not totally identical to gesturing in narration. This project also helps understand the nature of speech-related manual gestures in Chinese discourse, including the temporal, semantic, and pragmatic relationship between gesticulation and speech, as well as how topicality, speech, gestures, and genres interact.

In future, the potential gender difference in gesturing, and whether the two different taping conditions would affect gesturing should be examined.

References

Armstrong, David F. 1995. Gesture and the Nature of Language. Cambridge: Cambridge University Press.

Bavelas, Janet Beavin, Nicole Chovil, Douglas A. Lawrie, and Allan Wade. 1992. Interactive gestures. Discourse Processes 15: 469-489.

Butterworth, Brian, and Uri Hadar. 1989. Gesture, speech, and computational stages: a reply to McNeill. Psychological Review 96: 168-174.

Chui, Kawai. 2001. Topic Chains and Grounding in Chinese Discourse. Taipei: Crane Publishing.

_____. 2003. Categorization of gestures in communication. Form and function:

linguistic studies in honor of Shuanfan Huang, ed. by Lily I-wen Su, Chinfa

Lien, and Kawai Chui, 105-129. Taipei: Crane Publishing.

_____. 2005. Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics 37 (6): 871-887.

Dray, N.L., and David McNeil. 1990. Gestures during discourse: the contextual structuring of thought. Meaning and prototypes: studies in linguistic

categorization, ed. by S. L. Tsohatzidis, 466-488. New York: Routledge.

Du Bois, John W., Stephan Schuetze-Coburn, Susanna Cumming, and Danae Paolino. 1993. Outline of discourse transcription. Talking data: transcription and coding

in discourse research, ed. by Jane A. Edwards, and Martin D. Lampert, 45-89.

N.J.: Lawrence Erlbaum Associates.

Hadar, Uri, and Brian Butterworth. 1997. Iconic gestures, imagery, and word retrieval in speech. Semiotica 115: 147-172.

_____, Krauss, R. Wenkert-Olenik, and N. Soroker. 1998. Gesture and the processing of speech: neuropsychological evidence. Brain and Language 62: 107-126. Kendon, Adam. 1983. Gesture and speech: how they interact. Nonverbal interaction,

ed. by John Wiemann and Randall Harrison, 13-45. Beverly Hills, CA: Sage Publications.

_____. 1995. Gestures as illocutionary and discourse structure makers in Southern Italian conversation. Journal of Pragmatics 23: 247-279.

Levy, Elena T., and David McNeill. 1992. Speech, gesture, and discourse. Discourse

Processes 15: 277-301.

McNeill, David, and Elena T. Levy. 1993. Cohesion and gesture. Discourse Processes 16: 363-386.

_____. 1985. So you think gestures are nonverbal? Psychological Review 92: 350-371.

University of Chicago Press.

Morrel-Samuels, P., and R. M. Krauss. 1992. Word familiarity predicts the temporal asynchrony of hand gestures and speech. Learning Memory and Cognition 18: 615-622.

Streeck, Jürgen. 1995. On projection. Social intelligence and interaction: expressions

and implications of the social bias in human intelligence, ed. by Esther N.

Appendix A: Abbreviations of Linguistic Terms

1PL first person plural 1SG first person singular 2PL second person plural 2SG second person singular 3PL third person plural 3SG third person singular ACMPL accomplishment aspect ASSC associative morpheme BA the morpheme BA BC backchannel CL classifier COMPARE compare morpheme COMPL complementizer COP copula verb DLM delimitative aspect EMP emphatic adverbial EXP experiential aspect NEG negative morpheme PF pause filler

POSS possessive PRF perfective aspect PROG progressive aspect PRT discourse particle QST question particle REPAIR repair phoneme(s) SELF reflexive morpheme

Appendix B: Transcription Conventions

‘Intonation unit’ is defined as a stretch of speech uttered under a single coherent intonation contour, which tends to be marked by a pause, a change of pitch, and a lengthening of the final syllable (Du Bois et al. 1993).

Relevant expressions in examples are in boldface; the lines where the relevant expressions in question appear are marked by the arrow sign ‘Æ’.

Units

{carriage return} intonation unit

-- truncated intonation unit {space} word

- truncated word Speakers

: speaker identity/turn start

[] speech overlap Transitional continuity

. final , continuing ? appeal Terminal pitch direction

\ fall / rise _ level Accent and lengthening

^ primary accent = lengthening Pause ...(N) long ... medium .. short (0) latching Vocal noises (H) inhalation % glottal stop @ laughter Quality <@ @> laugh quality <A A> allegro: rapid speech <P P> piano: soft