The present research aims to examine the cohesive relation which is Intersemiotic Parallel Structure between textual and visual modes in the dialogues of EFL 5th grade of elementary school English textbooks in three editions: Han-lin, Kang-hsuan, and Hess. To address the research questions, three phases of analysis were conducted, including textual analysis, visual analysis and intersemiotic analysis.
Process Types of Textual Elements in the Dialogues
As the first step of analysis in the current research, texts in the dialogues were analyzed following Halliday’s (2004) Transitivity System. Process types which textual elements belong to were identified, including Material, Mental, Relational, Verbal, Behavioral and Existential. The percentage of process type of text in the dialogues achieved in each unit among Han-lin, Kang-hsuan, and Hess edition is presented in the following figures.
Figure 10 displays the percentage of process type in the dialogues realized in Han-lin edition. Relational process (42%) accounted for the highest proportion among all. The second most frequent process found was Material (39%), followed by Mental (16%). Other process like Existential (2%) and Behavioral (1%) were seldom utilized.
There was no Verbal process in the dialogues of Han-lin edition.
46
Figure10. Percentage of process type of text in Han-lin edition
The percentage of process type of text in Kang-hsuan edition is showed in Figure 11. As can be seen, Relational process (53%) is considerably higher than other processes of texts. Mental process (18%) was realized almost as much as Material process (17%). Behavioral process which accounted for 10% is used more frequently than other two publishers. Existential process (2%) was seldom employed.
Han-lin Total
47
Figure 11. Percentage of process type of text in Kang-hsuan edition
Figure 12 demonstrates the percentage of process type in the dialogues realized in Hess edition. In Hess edition, different from the other two editions, the highest percentage was found in Material process which took up 35%, followed by Mental (33%) and Relational (29%) process. Existential process (3%) was seldom realized, the same as in Han-in and Kang-hsuan. There was no Behavioral and Verbal process in the dialogues in Hess edition. In addition, it can be observed that Material, Mental and Relational process share the similar percentage and are evenly distributed in Hess edition.
48
Figure 12. Percentage of process type of text in Hess edition
Hess Total
Overall Distribution of Process Types of Textual Elements in the Three Editions To find out the overall distribution of each process type of texts in the dialogues among the three editions, the data obtained from above was organized and presented in Figure 13. It can be observed that Relational process which accounts for 42% took up the highest percentage in the dialogues. Material process which indicated 29%
occurrence is the second most frequently adopted process, followed by Mental process (22%). Not many proportions of Behavioral process (3%) and Existential process (2%) were identified. There was no Verbal process utilized in the present study.
49
Figure 13. Overall Distribution of Process Type of Texts among Three Editions
The result demonstrates that Relational, Material and Mental process were the three prevalent processes in the dialogues of English textbooks. The functions of these textual processes are presented below.
First of all, Relational process is mainly adopted in the dialogues of English textbooks to describe (1) identity of the character or object, (2) attribute of the activity or object and (3) location of the character and object. Figure 14 and 15 demonstrate the examples of Relational process which functions to identify the character and object in the dialogues. In Figure 14, the sentence “He’s Sir Isaac Newton” provides the identity of the man who is sitting under the apple tree. In Figure 15, the sentence
“It’s not a tiger” and “It’s a cat” provide the identification of the animal.
Material Behavioral Mental Verbal Relational Existential
% 29 3 22 0 42 2
0 5 10 15 20 25 30 35 40 45
50
Figure 14. A screenshot from L1 of Kang-hsuan edition Figure 15. A screenshot from L2 of Hess edition
As exemplified in Figure 16 and 17, Relational process can be utilized to describe the attribute of the activity and object in the dialogues. In Figure 16, the character is playing with water in the zoo. The sentence “It’s so cool” which is Relational process is employed to describe the attribute of the activity. In Figure 17,
“The animals are small and cute” is utilized to identify the feature of the animals in the zoo.
Figure 16. A screenshot from L3 of Han-lin edition Figure 17. A screenshot from L2 of Hess edition
Figure 18 and 19 displays the example of Relational process which functions to identify the location of the character and object in the dialogues. In Figure 18, the boy on the left takes the flashlight and makes exploration in the pyramid. Relational process is utilized to identify the location of characters in the dialogues through the
51
sentence “Maybe we are in Egypt”. In Figure 19, the man on the right says “It’s on the blue box”, which is used to present the location of the woman’s bag.
Figure 18. A screenshot from L4 of Han-lin edition Figure 19. A screenshot from L2 of Kang-hsuan edition
Second, Material process utilized in the dialogues of English textbooks functions to make the story plot more dynamic, including (1) transition of the scene, (2) actions and activity of the characters. For instance, in Figure 20, the boy on the right talks to the other character and invites him to follow with himself. The sentence “I’m going to the park” which is Material process implies the characters’ movement from the library to the park. Figure 21 and 22 provides examples of Material process which functions to present the physical and tangible actions carried out by the characters in the dialogues. In Figure 21, “Let’s sing and dance” presents the characters’ activity which they are going to do. In Figure 22, “Can you help me?” and “I’m going to the library”
depicts the man’s action in the dialogues.
Figure20. A screenshot from L1 of Han-lin edition
52
Figure 21. A screenshot from L2 of Han-lin edition Figure 22. A screenshot from L2 of Hess edition
Third, Mental process in the dialogues of English textbooks functions to describe the characters’ inner thinking or feeling. For example, in figure 23, Material process is used to present the old woman’s need for apples. As can be seen in figure 24, Material process reflects the boy’s mental state which is happy.
Figure 23. A screenshot from L3 of Hess edition Figure 24. A screenshot from L4 of Hess edition
After the process types of texts, the next section reports and discusses the process types of visuals in the dialogues.
53
Process Types of Visual Elements in the Dialogues
As the second phase of analysis in the present research, pictures in the dialogues was explored following Kress and Van Leeuwen’s (1996) Visual Grammar. There are two types of representational dimension of visuals: Narrative and Conceptual.
Narrative process is the presence of a vector which builds the interaction between participants. It can be differentiated based on the kinds of vector and the number of participants involved, including Action process, Reaction process, Mental process, and Verbal process. With respect to Conceptual process, it is identified when there are no vectors present, including Analytical process, Classification process and Symbolic process. The percentage of process type of visuals in the dialogues realized in each unit among Han-lin, Kang-hsuan, and Hess edition is presented in the following figures.
Figure 25. Percentage of Process Type of Visuals in Han-lin Edition
Han-lin Total
54
Figure 25 displays the percentage of process type achieved in Han-lin edition.
Action process (44%) was the most frequently adopted process. The second most frequent process found was Reaction process (32%). Analytical process (10%) was almost as frequent as Symbolic process (9%). Mental (5%) process was seldom utilized. Different from the other two publishers, there was no case of Classification process in the dialogue.
Figure 26. Percentage of Process Type of Visuals in Kang-hsuan Edition
Kang-hsuan Total (17%) was used more frequently than other two publishers. Mental process (3%) and Classificational process (1%) were seldom employed.
55
Figure 27 shows the percentage of process type in the dialogues achieved in Hess edition. As the same as in Han-lin edition, the highest percentage was found in Action process which took up 38%, followed by Reaction (32%) and Analytical (15%) process. The percentage of Symbolic process (6%) and Mental process (5%) was almost the same. Classificational process accounted for the least percentage.
Figure27. Percentage of Process Type of Visuals in Hess Edition
Overall Distribution of Process Types of Visual Elements in the Three Editions To illustrate the overall distribution of each process type of pictures among three editions, the results obtained from each edition were summarized and presented in
56
As Figure 28 demonstrated, Action process which accounted for 35% was the most prevalent process in the three editions. Reaction process (34%) which shared the similar percentage with Action process also showed high frequency. Analytical process (16%) was the third most frequently employed process in the dialogues, followed by Symbolic process (11%). Not many proportions of Classificational process (6%) and Mental process (4%) were identified in the present study.
Figure 28. Overall Proportions of Process Type of Visuals among Three Editions
The result showed that Action process and Reaction process were the two main dominant processes in the dialogues of English textbooks. The functions of these two visual processes are presented below.
Action process is mainly adopted in the dialogues of English textbooks to depict the characters’ physical, concrete and tangible actions. For instance, as shown in Figure 29, there is a woman standing in front of a stove and stirring food in the pot, which can be transcoded as “The woman is cooking.” Action process is formed where the woman is “Actor” (active participant who do the action) and the utensil is “Goal”
(passive participant at which the action is directed).
Action Reaction Mental Analytical Classifica
-ion Symbolic
% 35 34 4 16 6 11
0 5 10 15 20 25 30 35 40
57
Figure 29. Example of Action process from L4 of Kang-hsuan Edition
Reaction process is mainly utilized in the dialogues of English textbooks to illustrate the bidirectional interaction between the characters. For example, two boys in Figure 30 form Action process, which can be transcoded as “Owen (the boy on the right) holds Danny’s (the boy on the left) arm and back”. Owen is presented as “Actor”
while Danny is “Goal”. This Action process then becomes the “Phenomenon”
(passive participant at which the eyeline is directed) in Reaction process, in which the girl is “Reacter” (active participant whose direction of the glance at the two boys creates the eyeline).
Figure 30. Example of Reaction process from L3 of Han-Lin Edition
After the process types of visuals, the next section reports and discusses the Parallel Structure between textual and visual components in the dialogues.
Goal
Actor
Phenomenon
Reacter
58
Intersemiotic Parallel Structure between Textual and Visual Elements in the Dialogues
After the textual and visual analysis, intersemiotic analysis was conducted based on Liu and O’Halloran’s (2009) Intersemiotic Cohesive Device. Parallel Structure, an important text-constructing element in multimodal texts but attracts little attention as one of Intersemiotic Cohesive Devices, is the target of this study. The degree of Intersemiotic Parallel Structure achieved between text and picture among the three editions was investigated, including Parallelism, Partial Parallelism and No Parallelism.
The degree of Intersemiotic Parallel Structure is defined depending on if the text and image possess a similar form and whether participant and process in the text are portrayed in the image. Parallelism is achieved when visual and textual element share analogous transitivity structure and elements in the text are exactly presented in the picture. Partial Parallelism is realized when (1) visual and textual element do not share analogous transitivity structure and elements in the text are portrayed in the picture, or (2) visual and textual element share analogous transitivity structure and elements in the text are not portrayed in the picture. No Parallelism means if visual and textual elements do not share analogous transitivity structure and elements in the text are not portrayed in the picture. The results of intersemiotic analysis of each edition are presented in the next subsections.
Figure 31 illustrated the percentage of Intersemiotic Parallel Structure achieved in Han-lin edition. The line shows that Parallelism and Partial Parallelism remains similar percentage and varies slightly in the four units, whereas No Parallelism accounts for the least proportion. In terms of Parallelism, the percentage ranges from 38% to 53%. The highest percentage was found in Unit 3 (53%). The lowest
59
percentage was found in Unit 1 (38%). With respect to Partial Parallelism, the percentage ranges from 33% to 47%. The highest percentage (47%) was found in Unit 3 and 4, whereas the lowest percentage was found in Unit 1 (33%). In terms of No Parallelism, the percentage ranges from 1% to 29%. The highest percentage (29%) was found in Unit 1. The lowest percentage (1%) was found in Unit 3.
Figure 31. Percentage of Intersemiotic Parallel Structure in Han-lin edition
In Unit 3 in Han-lin edition, it can be observed that the percentage of Parallelism is highest in the four units (53%). Possible reason may be the topic which is animal transitivity structure. Additionally, the participant and process in the text (e.g. see, go, count, animals) are concrete and can be easily portrayed in the picture, which
60
facilitates the reinforcement of the central topic. Students might be able to comprehend the reading materials through receiving the identical meaning across both components. Both text and picture share similar transitivity processes and thus Parallelism is achieved.
Figure 32. A screenshot from L3 of Han-lin edition
It is noticeable that the percentage of No Parallelism in unit 1 is relatively higher than other units (29%). In many cases, textual and visual components in the dialogues of Han-lin edition are incompatible with each other. For instance, in Figure 33, the sentence in the dialogue “It’s sunny” is presented as Relational process. As displayed in the picture, the character is flying and talking on the phone, which can be transcoded as Action process. Thus, Relational process in the text in not convergent with Action process presented in the picture. Additionally, the weather “sunny” is not portrayed in the image. Both the text and the picture do not share similar Transitivity processes and No Parallelism is achieved.
61
Figure 33. A screenshot from L1 of Han-lin edition
Figure 34 demonstrated the percentage of Intersemiotic Parallel Structure in Kang-hsuan edition. The data indicated that the percentage of Intersemiotic Parallel Structure varies slightly in each unit, except for unit 3. Parallelism and Partial Parallelism remains similar percentage in unit1, 2 and 4, whereas No Parallelism accounts for the least proportion. Parallelism was the dominant structure among the four units, ranging from 72% to 50%. The highest percentage (72%) was found in Unit 4. The lowest percentage (50%) was found in Unit 2. In terms of Partial Parallelism, the highest percentage was found in Unit 1 (29%), while the lowest percentage was found in Unit 3 (9%). With respect to No Parallelism, the percentage ranges from 8% to 32%. The highest percentage was found in Unit 3 (32%). The lowest percentage was found in Unit 4 (8%).
Figure 34. Percentage of Intersemiotic Parallel Structure in Kang-hsuan edition
Kang-hsuan Total
62
In Unit 3, the percentage of No Parallelism (32%) is highest in the four units, even higher than Partial Parallelism (9%). In many cases in unit 3, textual and visual components in the dialogues are incompatible with each other. In Figure 35, the sentence in the dialogue “It’s Wednesday” are presented as Relational process. As displayed in the picture, the girl on the right side is talking to the woman, which can be transcoded as Action process. Thus, Relational process presented by picture is in not convergent with Action process in the text. Additionally, “Wednesday” or any calendar which can show the time is not portrayed in the picture. Both the text and the picture do not share similar Transitivity processes and No Parallelism is achieved. In this way, readers may have distraction, exceeding cognitive overload, and thus fail to comprehend the reading materials (Bezemer& Jewitt, 2010; Guichon& McLornan, 2008).
Figure 35. A screenshot from L3 of Kang-hsuan edition
Figure 36 demonstrates the percentage of Intersemiotic Parallel Structure in Hess edition. The data indicated that the percentage of Parallelism and Partial Parallelism varies greatly in each unit, while No Parallelism remains similar in the four units. In terms of Parallelism, the percentage ranges from 47% to 81%. The highest percentage was found in Unit 2 (81%). The lowest percentage was found in Unit 3 (47%). With
63
respect to Partial Parallelism, the percentage ranges from 14% to 42%. The highest percentage (42%) was found in Unit 3, whereas the lowest percentage was found in Unit 2 (14%). No Parallelism, which ranges from 5% to 11%, remained similar
Figure 36. Percentage of Intersemiotic Parallel Structure in Hess edition
The spikes in the figure reveal that there is an obvious variation of the degree between Parallelism and Partial Parallelism, especially in Unit 2 and Unit 4, where Parallelism (81%, 80%) was considerably higher than Partial Parallelism (14%, 15%). However, in Unit 1 and Unit 3, the variation between the two is quite little. In these two units, Parallelism (50%, 47%) was slightly higher than Partial Parallelism (41%, 42%).
Possible reason for why Parallelism was relatively higher than Partial Parallelism in Unit 2 and Unit 4 may be the target sentence patterns in the dialogue.
In Unit 2, the sentence pattern is “What do you see? I see .” In Unit 4, the sentence pattern is “How do you feel? I feel .” These two target sentences belong to Perception process, one of Mental process which deals with characters’
sensing (see, hear, taste, etc) which is tangible and concrete. The participant and
64
process in the text can be easily portrayed in the image and thus reached Parallelism between the text and picture.
Figure 37. A screenshot from L2 of Hess edition
Take Unit 2 in Hess edition for example. The main sentence pattern “What do you see? I see .” which classified as Mental process repeatedly occurs in the dialogues. In Figure 37, the man on the right looks at monkeys climbing in the tree, which is Reaction process. The Mental process in written text is in convergence with Reaction process presented in the picture. The elements in the text (e.g. monkeys, in the tree) are exactly portrayed in the picture. Students might be able to comprehend the reading materials through receiving the identical meaning across both components.
Both text and picture share similar transitivity processes and thus Parallelism is achieved.
Overall Distribution of Intersemiotic Parallel Structure of Each Edition To have a full representation of Intersemiotic Parallel Structure achieved among Han-lin, Kang-hsuan, and Hess edition, the data were organized and presented in Figure 38. Overall, as displayed in Figure 38, Parallelism which accounted for 58%
was predominantly achieved in the dialogues among the three editions, followed by Partial Parallelism (30%). No Parallelism took up the least percentage in the three editions (12%).
65
Figure 38. Overall Distribution of Intersemiotic Parallel Structure among the three editions
The result showed that Parallelism was the main dominant process in the dialogues of English textbooks. Take Unit 1 in Hess edition for example. In the picture (see Figure 39), the boy on the right takes the VR machine and keeps going forward. The thought bubble on the right shows the route and the destination “zoo”.
The Action process presented in the picture is in convergence with Material process in written text “We’re going to the zoo.” Additionally, the participant and process in the text are exactly portrayed in the picture, which facilitates the reinforcement of the central topic. Students might be able to comprehend the reading materials through
The Action process presented in the picture is in convergence with Material process in written text “We’re going to the zoo.” Additionally, the participant and process in the text are exactly portrayed in the picture, which facilitates the reinforcement of the central topic. Students might be able to comprehend the reading materials through