手勢之補足性質與認知處理 (3/3)
計畫類別：
■
個別型計畫 □ 整合型計畫
計畫編號：NSC 96 －2411 －H － 004 － 004 －
執行期間： 96 年 08 月 01 日至 97 年 08 月 31 日
計畫主持人：徐嘉慧
共同主持人：
計畫參與人員：
成果報告類型(依經費核定清單規定繳交)：□精簡報告
■
完整報告
本成果報告包括以下應繳交之附件：
□赴國外出差或研習心得報告一份
□赴大陸地區出差或研習心得報告一份
□出席國際學術會議心得報告及發表之論文各一份
□國際合作研究計畫國外研究報告書一份
處理方式：除產學合作研究計畫、提升產業技術及人才培育研究計畫、
列管計畫及下列情形者外，得立即公開查詢
□涉及專利或其他智慧財產權，
■
一年□二年後可公開查詢
執行單位：
國立政治大學英國語文學系
中 華 民 國 97 年 9 月 5 日
□ 可申請專利 □ 可技術移轉
日期： 年 月 日國科會補助計畫
計畫名稱：
計畫主持人：
計畫編號： 學門領域：
技術/創作名稱
發明人/創作人
中文：
（100~500 字）
技術說明
英文：
可利用之產業
及
可開發之產品
技術特點
推廣及運用的價值
※ 1.每項研發成果請填寫一式二份，一份隨成果報告送繳本會，一份送 貴單位
研發成果推廣單位（如技術移轉中心）
。
Complementary Gestures and Information Types
∗Kawai Chui
National Chengchi University
This paper investigates what kinds of information can be conveyed by complementary gestures that do not have a direct syntactic and semantic relation with particular linguistic constituents, what discourse functions the gestures have in communication, and how the relationship between gesturing and speaking can help in understanding the organization of information in Chinese daily conversation. The findings show that complementary gestures can convey attitudinal, script-evoked, and topical information. With regard to what all the participants are talking about, these various types of information are not the immediate concern of the participants at the moment of utterance; nonetheless, the complementary gestures provide additional meanings to enrich speech events or maintain the continuity of a topic under discussion.
Key words: complementary gestures, information types, focus of interest, Chinese discourse
1. Introduction
Hand gestures spontaneously produced in face-to-face interaction can be associ-ated with linguistic constituents. These manual movements play an important role in communication as they work in co-ordination with speech to express meaning. This study examines hand gestures in Chinese daily conversation, which can be illustrated in Example (1). The subject for talk in Example (1) is the way in which Speaker A’s part-time job at the library was so hard: the speaker had to send a document to a number of different places, from the library to the Administration Building, then back to the library, and then to the Social Sciences Information Center. The manual forms include iconic gestures for the verbs. First, at the moment of uttering the verb na ‘take’ (IU2, see Appendix B for the definition of Intonation Unit), both A’s hands rise from the thighs, and the arms cross at belly level as if to hold a document. Another iconic gesture
∗ This research was funded by grants from the National Science Council (NSC
94-2411-H-004-031; NSC 95-2411-H-004-008; NSC 96-2411-H-004-004). I would also like to thank the two anonymous referees for offering valuable comments and suggestions. All errors of inter-pretation are my own responsibility.
is produced for the verb song ‘send’ at IU3: both hands move to the upper right periphery, conveying the idea of sending a document to the Administration Building. They move back a little for the second mention of the same verb at IU5, depicting the action of sending the document back to the library.
(1) 1 A: … Ranhou wo jiu%,_
then 1SG thus
2 …(.7) na yi ge gongwen,_
take one CL document
na ‘to take’: both hands rise from thighs, and arms cross at belly level 3 …(.6) song dao%,\
send to
song ‘to send’: both hands move to upper right periphery 4 …(.7) xingzhengdalou,_
Administration Building
5 …(.6) Song huilai,\
send back
song ‘to send’: both hands move back a little 6 … Song qu shezizhongxin,_
send to Social Sciences Information Center
A: ‘Then I took a document, and sent (it) to the Administration Building. (After that I) sent (it) back (to the library). (Then I) sent (it) to the Social Sciences Information Center.’
The gestures being discussed in Example (1) all rest upon their respective lexical items for interpretation. However, some manual forms are not affiliated with speech. In the following Example (2), Speaker A is complaining about her former roommate, who often stayed out overnight and sometimes might come back at six in the morning and knock at the door. The idea of opening the door for the roommate is not realized in speech, but rather expressed by hand movements after uttering the causal clause about starting her part-time work at eight in the morning (IU13): both of A’s hands rise from the thighs to waist level and her right hand sweeps weakly to the right periphery, as if to open the door while she was not fully awake. This example illustrates that gesture and speech can be complementary in conveying information. The linguistic information alone does not always provide a complete view of the message that the speaker intends to communicate; “gestures are an integral part of language as much as are words, phrases, and sentences” (McNeill 1992:2).
(2) 1 A: … (.7) Ta jiu%,\
3SG then
2 … (.8) <A chuqu A>,_
go out
3 ..Ranhou <A jingchang wanshang hen wan huilai A>,\ then often at night very late come back
4 ..Buran jiushi%,_ or else that is
5 … zaoshang cai huilai,_ morning then come back
6 …(.5) Na wo ye bu zhidao weishenme?\ PRT 1SG also NEG know why 7 …(.5) <P Ranhou wo jiushi%,\
then 1SG that is 8 … a jiu shui shui P>,\ PRT then sleep sleep
9 ..Youshihou zaoshang liu dian zhong,\ sometimes morning six CL o’clock 10 B: …(.5)<PP M PP>,_
Mm
11 A: (0)Ko% ko% ^ko%,\ knock knock knock 12 …(.6)<HI aiyo= HI>,/
PRT
((2 IUs omitted))
13 … Yinwei wo ba dian yao dagong=,_ because 1SG eight CL have to do part-time work
at the end of the clause/IU: both hands rise from thighs to waist level, and right hand sweeps weakly to right periphery
A: ‘She then went out. Then, (she) often came back very late at night. Or else, (she would) come back in the morning. Well, I didn’t know why. Then, I just slept and slept. Sometimes, (she) came back at six in the morning,’
B: ‘Mm.’
A: ‘(she) knocked and knocked and knocked (at the door). ((2 IUs omitted)) I had to start (my) part-time work at eight.’
‘Complementarity’ is a term used by McNeill (1992:13) to characterize the fact that “[s]peech and gesture refer to the same event and are partially overlapping, but the pictures they present are different.” McNeill’s example is: at the moment the speaker utters ‘she chases him out again’, “speech conveys the ideas of pursuit and recurrence while gesture conveys the weapon used (an umbrella)” (ibid.). In other words, the form of the speaker’s hand in a shape as though to grip something is not coordinated with any
lexical item, and the utterance is grammatically complete without the gesture. This kind of gesture has also been mentioned in other studies. Bavelas et al. (1992, 1995) identified a type of conversational manual forms named ‘interactive gestures’. They are used to address to other participants and function “to aid the maintenance of conversation as a social system” (Bavelas et al. 1992:470). Within this category, some gestures have parallel verbal references but some do not. Moreover, ‘pragmatic gesturing’ in Kendon’s (1995:247) study “expresses aspects of utterance structure, including the status of discourse segments with respect to one another, and the character of the ‘speech act’ or interactional move of the utterance.” The gestures at issue refer to the conventional emblems. Finally, Church & Goldin-Meadow (1986) investigated the mismatches between gesture and speech in children’s explanations of a concept. The ‘discordant’ children in their study “produced many explanations in which the information conveyed in speech did not match the information conveyed in gestures” (ibid: 43). Again, some gestures of this kind had linguistic references, whereas the others “conveyed more information than was conveyed in speech” (ibid: 51). Because of different perspectives and research purposes, gestures that lack references in speech have been categorized in various ways together with those having references. They were not singled out for analysis and discussion.
In the present study, spontaneous hand gestures that are not associated semantically with any particular linguistic constituents in their accompanying clauses are called ‘complementary gestures’. They are idiosyncratic and spontaneous with a context-dependent meaning and use. The study examines what kinds of information can be conveyed in the gesture modality alone, what discourse functions the complementary gestures have in communication, and how the relationship between complementary gestures and speech can help in understanding the organization of information in Chinese daily conversation.
The next section introduces the database used in this study. Section 3 investigates the types of information and functions carried by complementary gestures. Based on the findings, §4 will discuss the organization of information in communication, followed by a conclusion in §5.
2. The database
The data used here include four casual, unpremeditated, multi-party conversations which took place in 1994 and 1995 among college students who knew each other. The subjects were free to find topics of common interest; they were not told the particular focus of the research. The subjects were filmed for approximately an hour, with a visible camera and in full-body shot. One section from each conversation, about twenty
minutes of talk, in which students were more comfortable in front of the camera, was then extracted. Gestures were analyzed on computer, using MediaStudio Pro, a program package which features frame-by-frame advance and varying slow-motion capabilities with no muting, so that the sound could be heard as the images were advanced. Thus, movement at a given moment in time could be matched with a simultaneously uttered syllable or with silence.
According to McNeill (1992:83), there are mainly three phases of gesticulation: ‘preparation’, ‘stroke’, and ‘retraction’. The preparation phase (also known as ‘onset’) refers to “the limbs mov[ing] away from their rest position to a position in gesture space where stroke begins.” In the stroke phase, “the meaning of the gesture is expressed” (ibid.). Finally, the retraction phase is the “return of the hand to a rest position” (ibid.). In the examples of the present study, only the obligatory stroke phase is described in words; the linguistic expression that synchronizes with the gestural stroke is underlined and in bold.
In the database, there are other kinds of gesture that also carry information not conveyed in speech. First, some gestures were used to make the meaning of the associated constituents clear. The iconic gesture in Example (3) clarifies what the accompanying pro-adverb zheme ‘like this’ (IU4) means in discussing the size and height of a gas can. Speaker B, at the time she utters the first syllable zhe of the pro-adverbial zheme, raises her right hand to shoulder level with the palm open and facing downward while her left hand is at waist level with the palm open yet facing upward. The distance between the palms thus represents the height of a gas can.
(3) 1 A: … Shenme shi wasi^guan?\ what COP can of gas
2 C: … Jiu shi xiao guan <A zheyangzi% A>,_ EMP COP small can like this
3 A: (0)<P Duo xiao% P>?\
how small
4 B: …(.9)Zheme gao ba,\ like this high PRT
zheme ‘like this’: right hand raises to shoulder level with palm open and downward; left hand is at waist level with palm open and upward
A: ‘What is a can of gas?’ C: ‘(It) is (a) small can.’ A: ‘How small?’
Second, gestures can also replace covert arguments. In Example (4), as the participants are talking about how using the same kind of shampoo continuously will make people lose hair, Speaker B asks A if she has the same problem (IU6-7). The question involves an unrealized argument—the ‘direct object’ of the verb diao ‘lose’ (IU7). Although the referent of the direct object, ‘hair’, is not verbalized, it can be understood by means of two deictic gestures: A’s right index finger points at her own hair on the right side while uttering you ‘have’, and then points at her hair on the left side while producing the first syllable kai of the verb kaishi ‘start’.
(4) 1 B: … Ruguo ni yizhi yong a,_ if 2SG continuously use PRT 2 .. ranhou,\ then 3 … kai[shi%],\ start 4 A: ..[m=],/ Mm
5 B: ..toufa jiu hui ^diao,\ hair then will lose
((6 IUs omitted))
6 …<F Ni <A xianzai% A>F>,\ 2SG now
7 ..you mei you kaishi zai diao?\ have NEG have start PROG lose
you ‘have’: right index finger points at own hair on right periphery kaishi ‘to start’: right index finger points at own hair on left periphery 8 A: ..Meiyou a,\
NEG PRT
B: ‘If you keep using (the same kind of shampoo), then (you would) start…’
A: ‘Mm.’
B: ‘losing (your) hair. ((6 IUs omitted)) Now, are (you) starting to lose (hair)?’
A: ‘No.’
Finally, some gestures not bound to any constituents can function to address somebody, as exemplified by the interaction in (5). The conversational topic is shampoo, and Speaker C is telling what she knows about the green Vidal Sassoon shampoo, so the subject of the first clause (IU1-2) is the first person singular wo. To present the information particularly to Speaker A, C points at the addressee with her left index finger at the time she utters the pronominal form wo ‘I’. Hand movements of this kind were categorized as ‘interactive gestures’ by Bavelas et al. (1992, 1995).
(5) 1 C: … Wo <A zhidao% A>,\ 1SG know
wo ‘I’: left index finger points at A
2 ..lüse shaxuan shi ganma de la,\ green Vidal Sassoon COP do:what PRT PRT 3 ..Wo tongxue shuo%,_
1SG classmate say
4 .. lüse shaxuan shi gei,_ green Vidal Sassoon EMP for
5 ..<A na zhong meitian xitou de ren yong A>.\ that kind every day wash:hair DE people use
6 A: ..Zhende a=?/
really QST
7 C: …^Dui,\ right
C: ‘I know what (the) green Vidal Sassoon (shampoo) is. My classmates said (the) green Vidal Sassoon is for the kind of people who wash (their) hair every day.’
A: ‘Really?’
C: ‘Right.’
All the various uses of gesture mentioned above were excluded in the present study, since the manual forms are either bound to particular lexical items, be it overt or covert, or the gestured addressing function is not directly related to the content of talk. The complementary gestures to be studied here are not associated with any lexical constituents in their immediate clausal contexts; nor do they substitute for any missing arguments. Utterances are grammatically complete without the complementary gestures. Although the gestural information is not encoded in speech, it is still part of the message.
Since complementary gestures do not parallel meaning that is expressed in words, their context-dependent meanings and uses were checked and confirmed by three analysts.
3. Types of information conveyed by complementary gestures
Despite the fact that the idea expressed by a complementary gesture is not realized linguistically, as in Example (2) in the hand movement of opening the door produced by Speaker A, the interpretation of the gesture is still based on the literal verbalization of the speech event at issue. The conveyed information is explicit in context. Then, what kinds of information would speakers convey only by gesturing? And, what discourse functions do the gestures have in communication? These questions will be discussed in this section.
3.1 Attitudinal information
The following example (6) has to do with shampoo. What Speaker B explains in IU1, 3 and 4 is that she used a particular brand of shampoo because there was a bottle at home. The clause in IU4 does not indicate her attitude toward the event of using the shampoo. It is rather depicted by a complementary gesture: both the speaker’s elbows move outward while uttering the verbs na ‘take’ and yong ‘use’, signifying that using that kind of shampoo was ‘nothing’.
(6) 1 B: …Fanzheng%,\ anyway
2 …w% --
REPAIR
3 …wo jia jiu you yi ping a,/ 1SG home EMP have one bottle PRT 4 …<A Wo [jiu A> nalai yong a],\
1SG then take use PRT na ‘to take’ & yong ‘to use’: both elbows move outward 5 A: [Oh oh=]./ PRT PRT 6 B: …(.7) Ranhou wo juede%,_ then 1SG feel 7 ..hai bucuo,\ still NEG:bad
B: ‘Anyway, there was a bottle (of shampoo) at home. So, I used (it).’
A: ‘Uhuh.’
B: ‘And then, I thought (it) was not bad.’
Example (7) includes another attitudinal gesture that is not tied to any verbal component of the utterance. During a discussion about foreigners dining at a Chinese restaurant, what Speaker A tells is that two Chinese of Mainland origin considered the habits of foreign people strange. The contemptuous attitude of the Chinese, again, is only conveyed manually, as the speaker’s left hand sweeps outward once while producing the nominal waiguoren ‘foreigner’ (IU9).
(7) 1 A: …Jiushi ta zai nage,_ that is 3SG be.at that 2 ..zhongguocheng ma,_
Chinatown PRT
3 …Ranhou jiu you liang ge%,_ then then there.be two CL
4 ..waishengren na zhong,_ Chinese of Mainland origin that CL
5 ..yijing yimin hen jiu [le],\ already emigrate very long PRF 6 B: [m m=],_
BC BC
7 A: ..<A ta shuo% A>,\ 3SG say
8 ..zai na ge zhongguo caiguan zai nabian,\ at that CL Chinese restaurant at there 9 ..tamen ^waiguoren a,\
3PL foreigner PRT
waiguoren ‘foreigner’: left hand sweeps outward 10 ..xiguan jiushi guai.\
habit EMP be.strange
A: ‘That is, she was at the Chinatown. Then, there were two Chinese of Mainland origin, the kind that had been emigrated (to the US for) a long time.’
B: ‘Mm.’
A: ‘She said (that) at the Chinese restaurant (the Chinese said) the habits of the foreigners were strange.’
The two gestures in these two examples are not emblems because they are not standard forms like ‘okay’, and their interpretations rest upon the linguistic contexts (see the definition of emblems in Ekman & Friesen 1969 and Goldin-Meadow 1999). The attitudinal information being gestured is comparable to linguistic attitudinal adverbials. Nevertheless, the gestures do not substitute for any covert constituents of the accompanying clauses. Without relating to any overt lexical constituents, complementary gestures of this type function to add to the total meaning of the speech event as a whole.
3.2 Script-evoked information
The second type of information that also embroiders the entire speech event is evoked by a script being mentioned linguistically in the context. Scripts are stored knowledge that consists of sequences to guide people to move from one activity to
another (Schank & Abelson 1977). People having knowledge of a script are familiar with the sequence of events involved in a particular situation. In communication, such knowledge can be expressed just by gesturing.
The subject for conversation in the following example (8) has to do with food utensils. What the speakers are discussing are the different kinds of utensils that can be used to serve food at a departmental gathering, including jiazi ‘food tongs’ (IU2), kuaizi ‘chopsticks’ (IU6), and tangpiao ‘soup spoons’ (IU10). The things that people do with these various kinds of tools is the familiar knowledge stored in the script of ‘using food utensils’, and this knowledge is activated by the concepts being verbally represented by the nominals jiazi, kuaizi, and tangpiao. The script-evoked information is depicted by three complementary gestures produced by Speaker A. The first is produced while uttering the first syllable tang of tangpiao ‘soup spoon’: the speaker’s right hand moves downward from chest level to form a bowl shape as if to dish the corn up with a soup spoon. The second gesture occurs when shengcai ‘lettuce’ is verbalized (IU12): the speaker, whose right index finger and middle finger have already formed a chopstick gesture at IU11, starts moving the fingers forward and downward to depict picking up vegetables with chopsticks. The last gesture takes place at the moment of uttering the modal keyi ‘can’ within the same IU12: the speaker moves the chopstick gesture back to depict putting the lettuce onto her own plate. In this example, the participants are talking about soup spoons and chopsticks. What people do with these two kinds of food utensils is script-evoked information being expressed in the gesture modality alone.
(8) 1 B: …Ni keyi,\ 2SG can
2 ..duo nong ji zhi jia[zi a]._ more get few CL food tongs PRT
3 C: [Qi- -- REPAIR
4 ..qu nali] ^nong?\ go where get 5 A: …Wo juede% -- 1SG think 6 ..yong kuaizi,\ use chopstick ((2 IUs omitted)) 7 …(1.1)Yinwei ni kan%,\ because 2SG see 8 …xiang%,\ like
9 ..^yumi na zhong dongxi,_ corn that kind thing
10 …jiu yong tangpiao,_
then use soup spoon
tangpiao ‘soup spoon’: right hand moves downward from chest level 11 …(.7)Ranhou%,\
then
ranhou ‘then’: right index finger and middle finger form chopstick gesture 12 …shengcai keyi yong kuaizi.\
lettuce can use chopstick
shengcai ‘lettuce’: chopstick gesture starts moving forward and downward keyi ‘can’: chopstick gesture moves back
B: ‘You could get more food tongs.’ C: ‘Where (can I) get (them)?’
A: ‘I think (you can) use chopsticks, ((2 IUs omitted)) because, you see, for the kind of thing like corn, (you can) use soup spoons. Then, (for) lettuce, (you) can use chopsticks.’
In Example (9), Speaker B tells how she could not find her friend’s article in the journal of Cheng Kung University. There is a lot of knowledge stored in this ‘find-an-article-from-a-school journal’ script that is partly verbalized and partly gestured. The utterances have to do with her requesting a cousin to borrow for her a particular issue of the university journal (IU1, 2, and 3), having borrowed the issue (IU4), then realizing that the article that she had thought would be in it was not there (IU5). Before learning that the article was not included in the journal, the first thing the speaker needed to do was to open it to check. The knowledge that guides the speaker to perform the action is included in the script, and evoked by the first two actions being verbalized by the two clauses in IU1, 2 and 3. The evoked information, again, is conveyed by a complementary gesture: at the time the negation word meiyou is produced (IU5), both of Speaker B’s hands rise up to the front of the chest with closed fingers, appearing to hold the university journal. Then, the hands move side-to-side, as if opening the journal to check.
(9) 1 B: …Wo baituo wo biaoge,_ 1SG request someone to do a favor 1SG cousin 2 ..qu bang wo gen%,/
go help 1SG with
3 …chenggong de ren ^jie ma.\ Cheng Kung University DE people borrow PRT
4 …Jieguo jiedao na yi qi a,_ as a result borrow.ACMPL that CL issue PRT
5 …(1.)Meiyou ta de zuopin,\ NEG.have 3SG ASSC work
meiyou ‘do not have’: both hands rise up to chest front with closed fingers, then move side-to-side
B: ‘I requested my cousin to do me a favor: to borrow (the journal of Cheng Kung University) from a student of Cheng Kung University. As a result, (he) borrowed that (particular) issue, (but) his work was not (included).’
The script-evoked information, just like the attitudinal information, is not incorpo-rated into the associated utterances. Moreover, the complementary gestures mentioned in Examples (8) and (9) do not embroider single utterances, but enrich both of the entire speech events of choosing food utensils and of looking for an article.
3.3 Information that maintains the continuity of topic
The last of the three types of complementary gesture found in the database carries information indicating the continuity of a topic at issue. The gestural information is the topic per se, rather than the information related to the topic of discourse as conveyed by ‘topic gestures’ in Bavelas et al. (1992).
In daily interaction, conversation proceeds as people bring up different topics. Once a topic is established, the participants talk about it with orientations, descriptions, and elaborations. In Example (10), the subject for talk is wearing long hair in high school. After the topic has been brought up by the question in IU1 and 2, what Speaker A informs the other participants is that not until senior high could students at her school wear long hair (IU4). The topical referent toufa ‘hair’ is not verbalized but rather gestured: the speaker’s left hand touches her own long hair on her right shoulder at the time she utters gaozhong ‘senior high’.
(10) 1 B: ….Nimen guozhong toufa,_ 2PL junior high hair
2 ..keyi liu name chang ma?/ can wear such long QST 3 C: …Keyi.\
can ((2 IUs omitted))
4 A: ..Women dao gaozhong./
1PL till senior high
B: ‘Could you wear such long hair in junior high?’ C: ‘(We) could.’
((2 IUs omitted))
A: ‘We (could do that only when we went to) senior high.’
In Example (11), Speaker C brings up the topic of hitting cockroaches in the first clause (IU1-4). This topical idea is manually performed as Speaker C’s right hand with closed fingers at chest level sweeps downward to signify the hitting action while uttering the nominal zhanglang ‘cockroach’ in IU3. Then, she elaborates a lot on this topic, sharing her experiences of using different things to hit cockroaches, consisting of
tuoxie ‘slippers’ (IU5), yi ge hen zhong de shu ‘a heavy book’ (IU7), baozhi ‘a
newspaper’ (IU9), and bijiao chang de dongxi ‘things that are comparatively long’ (IU11). While mentioning the final object in IU11, it is the idea of holding a long thing that the speaker is uttering. The topic of hitting the cockroach is rather conveyed manually: at the moment the adjective chang ‘long’ is produced, C’s right hand descends from the face to the shoulder with the fingers curled, sweeping downward to depict the hitting action.
(11) 1 C: (0)^Wo yiqian ^gan,\ 1SG before dare 2 …jiushi nage%,_ that is that 3 ..zhanglang s- -- cockroach REPAIR 4 …ba ta ^dasi,\ BA 3SG hit to death 5 ..Na tuoxie dasi,\
take slipper hit to death ((25 IUs omitted))
6 …Buran jiu diu yi ge%,_ or else then throw one CL
7 …hen zhong de shu,\ very heavy ASSC book
((5 IUs omitted))
8 …Yaoburan jiu%,\ or else then
9 …zhao baozhi a,_ find newspaper PRT ((28 IUs omitted))
10 …Wo dou hui na%,_ 1SG all will take
11 …bijiao ^chang de dongxi,_ comparatively long DE thing
chang ‘long’: right hand descends from face to shoulder with fingers curled, and sweep downward
12 …Wo juede zheyang,/ 1SG think like this
13 …bijiao%.. ^yuan,\ comparatively far
14 …Jiu bu hui,\ thus NEG will
15 …^kan-dao,\ see:ACMPL
C: ‘I used to dare to hit cockroaches before. (I) used slippers to hit (them) to death. (25 IUs omitted) Or else (I would) throw a heavy book. (5 IUs omitted) Or else (I would) find a newspaper. (28 IUs omitted) I would take something longer. By doing this, I think, (I would be) farther away (from the cockroach). (I) thus would not see (them).’
McNeill & Levy (1993:364) claim that “linguistic elements that are continuous with their context are at times accompanied by gesture… [to] maintain continuity with respect to their location in space, handedness, and/or form.” The topical ideas mentioned in this section, namely ‘wearing long hair’ in Example (10) and ‘hitting cockroaches’ in Example (11), though not being expressed in words, are gestured to maintain their continuity, reminding the participants of the subject matter under discussion while talking about another, though related, theme.
This section has discussed three types of information carried by complementary gestures that neither correspond to any overt lexical constituents, nor substitute for covert constituents. Yet, their meanings and functions are fully understood in context. Their occurrences provide evidence that not everything that a person intends to communicate has to be expressed in speech. As far as the present corpus is concerned, the gestural channel alone can express attitudinal, script-evoked, and topical information that functions to provide additional meaning for an entire speech event or to maintain the continuity of a topic at issue. Based on these findings, the next section will discuss what complementary gestures can tell us about the organization of information in communication.
4. Organization of information in discourse
with respect to how the speaker organizes information. In daily face-to-face conversation, the most common type of linguistic activity is to ‘communicate information’, talking about events concerning time, place, participants, and states-of-affairs. The speaker has to make a choice and provide the information relevant to the discussion among the participants. Information is also organized into various types and conveyed either in words or by gesturing. These two channels of communication form three ways through which different kinds of information can be expressed. The first way to convey information is via the linguistic channel exclusively, not using gesture as part of the utterance.
The second way is using gesture as a way of expressing something similar to what is expressed in the associated speech. Many previous studies have provided evidence that the use of gesture can tell the way speakers organize information, since manual movements can mark different units of talk in discourse. Based on storytelling data, McNeill (1992:93) finds that “iconics occur overwhelmingly in narrative clauses, while beats can occur in both narrative and extranarrative clauses … abstract pointing occurs chiefly with narrative clauses, whereas metaphorics appear chiefly with extranarrative clauses.” Bavelas et al. (1992:473) distinguish ‘topic gestures’ and ‘interactive gestures’ in conversation to characterize, respectively, the information directly related to the topic of discourse, and the information referring “to some aspect of the process of conversing with another person.” They further suggest that interactive gestures may function to cite the other’s previous contribution, to seek agreement, understanding, or help, to deliver new versus shared information, and to present events around the speaker turn. With respect to ‘communicative dynamism’, Levy & McNeill (1992) and McNeill & Levy (1993) show that the quantity and complexity of gestures would increase along with complex linguistic expressions as the information conveyed by the utterances functions to push the communication forward. Concerning given and new information, McNeill & Levy (1993:365) suggest that “gestures tend to occur at points of topic shift, such as new narrative episodes or new conversational themes … highly presupposed linguistic elements would either lack gestures entirely, or would be accompanied by gestures that are specialized for their cohesive function or form.” Finally, Kendon (1995:247), based on Southern Italian conversation, differentiated ‘substantive gesturing’ from ‘pragmatic gesturing’, in that “[the former] contributes to various aspects of the content of the utterance of which it is a part, whether literally or metaphorically… [whereas the latter] expresses aspects of utterance structure.” He further identified two ‘discourse unit marker gestures’ that mark ‘topic’ and ‘focality’ of what is being said respectively. In Chinese discourse, Chui (2005) finds the distinction between foreground and background information in the use of iconic gestures. First, speakers incline to gesture for new information in topical clauses. The temporal patterning of iconic gestures and the
associated new referents serve to further distinguish topical and non-topical information, in that either onsets or strokes would come before topical referents to signal that the upcoming new information is noteworthy and deserves attention. In short, people use manual actions in different ways to distinguish various aspects of information conveyed by the utterances.
The third way to express what the speaker has chosen to talk about is including gesture in the utterance but expressing something in gesture that is not expressed in the speech. Section 3 has already discussed three types of information depicted by comple-mentary gestures. The reason why they are gestured without concomitant speech lies in whether the information is being organized and presented as the ‘focus of interest’. The linguistic contexts provide evidence that the attitudinal, script-evoked, and topic-maintaining information depicted by complementary gestures is not the focus of interest, because it is not the immediate concern of the participants at the time the gesture is produced. What the speakers are focusing on is what they are uttering. In Examples (6) and (7), the common interests of the participants are different brands of shampoo and the habits of foreigners respectively, so Speaker B’s attitude toward a particular brand and the two Chinese’s attitude toward foreigners are subsidiary and conveyed by gesturing. Similarly, the gestured script-evoked information is also not the center of interest: in Example (8) the speakers focus on the different kinds of food utensils to be used at a departmental gathering, rather than on how to use the utensils as gestured. In Example (9), the focus of interest is whether Speaker B can find her friend’s article in the Cheng Kung University journal. The gestural information indicating opening the journal to check is irrelevant to the talk. Finally, once a topic has been set up, speakers would talk about it with a new focus. In Example (10), the center of attention in speaker B’s utterance is ‘the time’, rather than the topic ‘wearing long hair’; the center of attention in Speaker C’s turn in Example (11) is ‘the things’, instead of the topic ‘hitting cockroaches’. Not being organized and presented as the focus of interest, the topical information can be expressed by complementary gestures.
5. Implications
While information being conveyed in communication can be organized and presented in various ways, the findings of the present study bear out a distinct way of organizing the speaker’s ideas into different units of talk with respect to ‘focus of interest’. Whether the information is the focus of interest or not is negotiated and determined by the participants in the linguistic interaction; the utterances indicate what the immediate common interest of the participants is. Two channels of communication have been shown to work together and show a ‘division of labor’ in conveying these
two aspects of information. Complementary gestures further contribute to adding more information to enrich a speech event and maintaining the continuity of a topic. In the future, I hope that data of various kinds are available to attest to the findings reported here.
Finally, speech and gesture are now widely accepted as different manifestations of a single underlying process of utterance production. Nevertheless, there are different views toward the cognitive unity of speech and gesture. Kendon (1983:19) holds that speech and gesture are separately generated; both may represent “different aspects of the single conceptual unit by which they are being guided.” Butterworth & Hadar (1989) claim that gestures are also very closely linked to speech production, but that both share more than one computational stage. De Ruiter (2000) proposes the Sketch Model for gesturing, assuming that gesture and speech are part of the same communicative intention, and are planned by the same process. He extends Levelt’s (1989) model of speech production to incorporate the processing of gesture. McNeill & Duncan (2000) claim that the same idea unit from which both gesture and speech originate from is a growth
point which combines both imagery and linguistic content. Contrary to McNeill and
Duncan, Kita (2000) regards gesture and speech as conceptually parallel but basically separate streams of cognition. Speech is concerned with analytic thinking; gesture, spatio-motoric thinking. In short, these various cognitive models of gesture-speech integration are mainly concerned with whether gesture and speech constitute single idea units, or whether they are separate yet closely related units. However, formulating a computational model of gesture-utterance performance that simulates the way Chinese speakers use complementary gestures along with speech in face-to-face communication also requires the linguistic details. It is also my hope that the linguistic details provided in the present study can contribute to the modeling of complementary gestures and speaking in Chinese discourse.
Appendix A: Abbreviations of linguistic terms
1PL first person plural1SG first person singular 2PL second person plural
2SG second person singular 3PL third person plural
3SG third person singular
ACMPL accomplishment aspect ASSC associative morpheme BA the morpheme BA
BC backchannel CL classifier
COMPARE compare morpheme COMPL complementizer
COP copula verb
DLM delimitative aspect EMP emphatic adverbial EXP experiential aspect NEG negative morpheme PF pause filler
POSS possessive PRF perfective aspect PROG progressive aspect
PRT discourse particle QST question particle REPAIR repair phoneme(s) SELF reflexive morpheme
Appendix B: Transcription conventions
An ‘Intonation Unit’ (IU) is defined as a stretch of speech uttered under a single coherent intonation contour, which tends to be marked by a pause, a change of pitch, and a lengthening of the final syllable (Du Bois et al. 1993).
Units
{carriage return} intonation unit
-- truncated intonation unit
{space} word
- truncated word
Speakers
: speaker identity/turn start
[] speech overlap
Transitional continuity
. final
, continuing ? appeal Terminal pitch direction
\ fall / rise _ level Accent and lengthening
^ primary accent = lengthening Pause ...(N) long ... medium .. short (0) latching Vocal noises (H) inhalation % glottal stop @ laughter Quality <@ @> laugh quality <A A> allegro: rapid speech <P P> piano: soft
References
Bavelas, Janet Beavin, Nicole Chovil, Douglas A. Lawrie, and Allan Wade. 1992. Interactive gestures. Discourse Processes 15.4:469-489.
Bavelas, Janet Beavin, Nicole Chovil, Linda Coates, and Lori Roe. 1995. Gestures specialized for dialogue. Personality and Social Psychology Bulletin 21.4:394-405. Butterworth, Brian, and Uri Hadar. 1989. Gesture, speech, and computational stages: a
reply to McNeill. Psychological Review 96.1:168-174.
Chui, Kawai. 2005. Topicality and gesture in Chinese conversational discourse.
Language and Linguistics 6.4:635-654.
Church, R. Breckinridge, and Susan Goldin-Meadow. 1986. The mismatch between gesture and speech as an index of transitional knowledge. Cognition 23.1:43-71. De Ruiter, Jan Peter. 2000. The production of gesture and speech. Language and
Gesture, ed. by David McNeill, 284-311. Cambridge & New York: Cambridge
University Press.
Du Bois, John W., Stephan Schuetze-Coburn, Susanna Cumming, and Danae Paolino. 1993. Outline of discourse transcription. Talking Data: Transcription and Coding
in Discourse Research, ed. by Jane A. Edwards & Martin D. Lampert, 45-89.
Hillsdale: Lawrence Erlbaum Associates.
Ekman, Paul, and Wallace Friesen. 1969. The repertoire of nonverbal behavior: categories, origins, usage, and coding. Semiotica 1:49-98.
Goldin-Meadow, Susan. 1999. The role of gesture in communication and thinking.
Trends in Cognitive Sciences 3.11:419-429.
Kendon, Adam. 1983. Gesture and speech: how they interact. Nonverbal Interaction, ed. by John Wiemann & Randall Harrison, 13-45. Beverly Hills: Sage.
Kendon, Adam. 1995. Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics 23.3:247-279.
Kita, Sotaro. 2000. How representational gestures help speaking. Language and
Gesture, ed. by David McNeill, 162-185. Cambridge & New York: Cambridge
University Press.
Levelt, Willem J. M. 1989. Speaking: From Intention to Articulation. Cambridge: MIT Press.
Levy, Elena T., and David McNeill. 1992. Speech, gesture, and discourse. Discourse
Processes 15.3:277-301.
McNeill, David. 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
McNeill, David, and Elena T. Levy. 1993. Cohesion and gesture. Discourse Processes 16.4:363-386.
McNeill, David, and Susan D. Duncan. 2000. Growth points in thinking-for-speaking.
Language and Gesture, ed. by David McNeill, 141-161. Cambridge & New York:
Cambridge University Press.
Schank, Roger C., and Robert P. Abelson. 1977. Scripts, Plans, Goals, and
Under-standing: An Inquiry into Human Knowledge Structures. Hillsdale: Lawrence
Erlbaum Associates.
[Received 30 July 2007; revised 7 December 2007; accepted 10 December 2007] Department of English
National Chengchi University 64, Sec.2, Chih-nan Road Taipei 116, Taiwan [email protected]
