The current study aims to investigate the degree of intersemiotic ideational and interpersonal complementarity in Book 5 of Joy and Kang-hsuan junior high school English textbooks. The chapter reports and discusses the results with respect to three research questions. The first section focuses on the results of intersemiotic ideational analyses. The second section centers on the results of intersemiotic interpersonal analyses. The final section then compares the degree of ideational and interpersonal complementarity and addresses the issue of genre in connection to multimodal analyses.
Intersemiotic Ideational Complementarity
The first research question explores to what extent ideational intersemiotic complementarity is achieved in Book 5 of Joy and Kang-hsuan junior high school English textbooks in Taiwan. Six sense relations: repetition, synonymy, antonymy, hyponymy, meronymy and collocation were used to interpret the relation between VMEs identified from the pictures and the lexical items in the verbal texts. Table 11 displays the percentage of matched sentences, the number of total tokens of sense relations and the density of token distribution in the two editions of textbooks. The percentage of matched sentences means among all the sentences, how many of them have matched sense relations occurring and the density of token distribution was obtained by dividing the number of tokens by the percentage of matched sentences.
49
Table 11. Intersemiotic Ideational Complementarity in Joy and Kang-hsuan Edition Percentage Joy Kang-hsuan Joy Kang-hsuan Joy Kang-hsuan
L1 29% 17% 6 6 0.21 0.35
Figure 8. Percentage of Matched Sentences in the Two Editions of Textbooks Figure 8 demonstrates the tendency of the percentage of matched sentences. The fluctuating lines show that the percentage of matched sentences varies greatly, even in the same edition. In Joy edition, the percentage ranges from 12% to 69 %. L4, with 53 tokens of matched sense relations, has the highest percentage of matched sentences
L1 L2 L3 L4 L5 L6 L7 L8 L9 Mean
50
(69%). L5, with only 3 tokens of matched sense relations, has the lowest percentage of matched sentences (12%). In Kang-hsuan edition, the range is even greater, from 7% to 83%. The highest percentage of was found in L4, which contains 25 tokens of sense relations. The lowest percentage was found in L9, which contains 5 tokens. The findings suggest that in both editions, there is a discrepancy among lessons in terms of intersemiotic ideational complementarity. In some lessons, complementarity is well achieved, but not in others.
A closer examination reveals that the relation between the percentage of matched sentences and the number of tokens is not linear, which is influenced by the
distribution of the tokens. The density of token distribution was calculated to show how the matched tokens are distributed: higher density signifies that the tokens are more concentrated in some sentences. For example, in Joy edition, the percentage of matched sentences in L1 and L2 is the same (29%), but while L1 has only 6 tokens, L2 has 17 tokens. The density of token distribution is higher in L2 (0.59) than in L1 (0.21), indicating that the tokens in L2 are more concentrated.
This phenomenon is also reflected in the mean percentage and total tokens of the two textbooks. Generally, the degree of ideational complementarity is a little higher in Kang-hsuan edition given that the mean percentage of matched sentences in
Kang-hsuan edition (43.7%) is higher than that in Joy edition (39%). But when looking at the total tokens, Joy edition (222 tokens) has much more tokens than Kang-hsuan edition (130 tokens), showing that the density of Joy edition is higher than that of Kang-hsuan edition. The matched tokens in Joy edition are more concentrated while in Kang-hsuan edition are spread more diffusely in different sentences.
After the overview of the ideational complementarity, the following section
51
80.8%
2.3%
1.5% 8.5%
6.9%
Kang-hsuan
R S M H C
reports the use of six sense relations.
The Use of Six Sense Relations
Six types of intersemiotic sense relations: repetition, synonymy, antonymy, hyponymy, meronymy and collocation were investigated between the visuals and the texts. The numbers of tokens of each sense relation were shown in Figure 9.
Figure 9. Proportions of the Sense Relations in Joy and Kang-hsuan Edition Note. R=Repetition, S=Synonymy, A=Antonymy, M=Meronymy, H=Hyponymy,
C=Collocate
It can be observed that in both editions, the most frequently used sense relation is repetition. In Kang-hsuan edition, the predominance of repetition is especially
significant. No instances of antonymy were found in both editions. In Joy edition, repetition accounts for about 39.2% of total tokens, followed by hyponymy (29.7%), collocate (24.8%), meronymy (4%) and synonymy (2.3%). In Kang-hsuan edition, repetition takes up 80.8% of total tokens, followed by hyponymy (8.5%), collocate (6.9%), synonymy (2.3%) and meronymy (1.5%). The share of the six sense relations is relatively evener in Joy edition.
Tables 12 and 13 below detail the frequency of occurrence of each sense relation
52
in each lesson of the two textbooks. The percentage in red is the highest percentage in each lesson, meaning the sense relation is most frequently utilized. As displayed in the tables, of the 9 lessons, repetition is the most frequently utilized sense relation in 6 lessons in Joy edition and 8 lessons in Kang-hsuan edition. In L2 of Joy edition, the frequency of repetition and meronymy are the same. There are 2 lessons in which collocate is the most employed sense relation: L6 in Joy edition and L7 in
Kang-hsuan edition. Hyponymy is dominantly used in L7 of Joy edition.
Table 12. The Frequency of Occurrence of the Sense Relations in Joy Edition Lesson Sense Relations Total Tokens
53
Table 13. The Frequency of Occurrence of the Sense Relations in Kang-hsuan Edition
Lesson Sense Relations Total Tokens
R S A M H C
Note. Per. =Percentage (the proportion of the tokens of each sense relation among the
total tokens), R=Repetition, S=Synonymy, A=Antonymy, M=Meronymy, H=Hyponymy, C=Collocation, red color code= sense relation with highest percentage
The Predominant Use of Repetition
The finding that repetition is predominantly used in multimodal texts is in line with several previous studies (Bowcher, 2012; Bowcher & Liang, 2013; Leonzini, 2012; Royce, 1999). The relatively high occurrence of repetition in junior high school English textbooks may be associated with EFL students’ proficiency level and
cognitive development (Paivio, 1986). Visual images presenting vivid and concrete
54
details of the texts could facilitate students’ comprehension of the texts (Rowsell &
Hamilton, 2012; Royce, 1999).
Take Kang-hsuan edition, L9, for example (see Figure 10). There is 1 picture (P1) and 3 sentences in the first paragraph of this lesson, which depicts a psychological test. As exhibited in Figure 10, P1 illustrates the setting and all the four options in sentence 3, realizing 5 tokens of repetition. The participants and process in the visual are
identically represented in the texts, helping to reinforce the central topic of the text.
Students may be able to obtain a more direct and robust interconnection between the texts and images and visualize the reading content by receiving repeated messages multimodally. But, one point has to be considered is that, for students with higher proficiency level, consistently receive repeated meaning multimodally may lead to redundancy.
Figure 10. A Screenshot from L1 of Kang-hsuan Edition
55
Dissimilar to repetition, not many tokens of synonymy were found in these 18 texts. One possible reason is that it is easier for junior high school students to comprehend the reading content by obtaining the identical meaning across both modes (Mayer & Anderson, 1992). Also, due to students’ comparatively small vocabulary size, words are often used repeatedly without replacing with other synonymous words. If the word is the target vocabulary item, it is likely that it appears repeatedly in the text. By contrast, in textbooks for higher educational stages or for more advanced learners, more tokens of synonymy may be yielded as a result of greater lexical variation.
Likewise, not many tokens of meronymy were identified in the present study, which does not parallel with Royce’s (2002) findings that meronymy was prevalently used in an article on water cycle. One possible explanation is discipline difference. In scientific texts, the whole passage may focus on explaining a single figure, like charts, diagrams and tables. Elements and icons within the whole figure were elaborated in the textual part, therefore, forming more part-whole relation than English textbooks.
In this study, no instances of antonymy were observed in both editions, which is in accordance with some earlier research that the visual-verbal ideational
complementarity is achieved mainly in terms of intersemiotic repetition, synonymy, hyponymy, meronymy and collocation, but not antonymy (Bowcher, 2012; Leonzini, 2002; Royce, 2002).The finding indicates that ideationally visuals in junior high school textbooks are inclined to accurately represent the content of the texts.
The Functions of Hyponymy and Collocate
In these two textbooks, hyponymy was found to be frequently deployed when exemplifying. The texts point out the general term or superordinate class and the visuals illustrate the sub-class items as examples. As shown in Figure 11, The
56
VMEs of picture one (P1) and picture two (P2) are “two male workers” and “a female worker” which can be subsumed into the lexical item “working-class people” in sentence○1 , forming a hyponymy relation.
Figure 11. A Screenshot from L8 of Joy Edition
In these 18 texts, it was discovered that intersemiotic collocate is realized in three major ways. One is that the pictures are designed like decorative frames, arranged around the texts. They are not related directly to the specific content of the text, but the elements are suggestive of the main topic. An example can be seen in Figure 12. Picture one (P1) and picture two (P2) are not in relation to particular sentences in the text, but when talking about Halloween, these kinds of decoration, or icons like the VME of P1, “ skeleton” and the VME of P2, “ a balloon with a skeleton pirate” naturally co-occur. The visuals and texts form instances of intersemiotic collocate.
57
Figure 12. A Screenshot from L3 of Joy Edition
The second way is that the VMEs are not explicitly connected to the lexical items in the text, but they are reasonably expected in the context. Take Figure 13, for example. One VME of picture three (P3) is “iron chains”, which collocates with the lexical item, “prisoner” in sentence○16. There is a high probability that the VMEs of the visuals and lexical items of the text co-occur in the specific subject area.
Figure 13. A Screenshot from L9 of Joy Edition
The third way is that the pictures capture the topic of the lesson, but they are not linked to specific sentences in the text. In these 18 texts, this way is often used in lessons about true story of people or places, like Joy edition, L1, about Meinong town,
58
Kang-hsuan edition, L7 about Nick Vujicic and L8 about the Stockholm. An example was given in Figure 14, no sentences depict picture two (P2) and picture three (P3) directly, but these two pictures are relevant to the topic, Nick Vujicic.
Figure 14. A Screenshot from L7 of Kang-hsuan Edition Effectively connecting the visual and textual mode through intersemiotic collocate may help set up the context and provide pertinent background information for readers (Hibbing & Rankin-Erickson, 2003). Teachers could make good use of the visual-textual connection to offer students supplementary information about the topic.
The unbalanced proportion of the six sense relations indicates that when
illustrating or choosing visual images, textbook designers tend to faithfully represent what the texts depict, leading to the extensive use of repetition. As visual and textual resources are governed by distinct logics, namely, textual mode is mainly structured by temporal sequence whereas visual mode is arranged by spatial relation (Guo, 2004; Kress, 2000), more diverse relations could be established between the two modes to fully exploit their unique potential to provide readers with more integrated meaning. If students could receive various relations and layers of meaning
59
constructed in visual and textual resources, their reading may go beyond literal comprehension to use critical thinking and interpretative ability (Vethamani, 2007).
After the use of six sense relations, the next section reports and discusses the occurrences of represented participants and processes.
Presentation of Represented Participant and Process
Represented participants refer to who or what in the visuals and processes refer to actions taking place. In the process of intersemiotic ideational analysis, VMEs of represented participants and processes identified from the visuals were linked to the lexical items in the texts. To investigate how the represented participants and
processes are presented in textbooks, the tokens were counted. Figure 15 exhibits the amounts of matched represented participants and processes in the two textbooks. A marked difference was observed between the number of matched represented participants and processes. In both editions, the tokens of represented participants outnumbered that of processes. Among all the 18 texts, only 3 texts have more tokens of processes than represented participants: L9 in Joy edition, L5 and L6 in
Kang-hsuan edition.
Figure 15. Amounts of Represented Participant and Process in Joy and Kang-hsuan Edition
60
To understand the distribution of six process types, a further analysis was
conducted. As shown in Table 14, taken together, totally 43 tokens of material process, 6 tokens of behavioral process, 1 token of mental process and 1 token of verbal
process were identified. This distribution of process type indicates that the two process types, material and behavioral, having to do with physical action, are presented more often in the visuals. On the contrary, the other four types, verbal, mental, relational and existential, despite their presence in the textual mode, are nearly absent in the visual mode.
Table 14. The Distribution of Process Type in Joy and Kang-hsuan Edition
Joy Kang-hsuan
Total Tokens Process Type Total Tokens Process Type
L1 0 0 0 0
L2 0 0 5 4 material, 1 mental
L3 3 3 material 2 2 material
L4 11 11 material 3 2 material, 1 verbal
L5 0 0 3 3 material
L6 2 2 material 5 5 behavioral
L7 1 1 material 0 0
L8 10 10 material 0 0
L9 5 5 material 1 1 behavioral
Total material
32
material behavioral verbal mental 11 6 1 1
Two exceptions are 1 token of mental process in L2 and 1 token of verbal process in L4 of Kang-hsuan edition. The mental process, “enjoy” in sentence○9 of L2 is repeated in picture two (P2) by illustrating the person’s happy face (see Figure 16). The verbal process, “talk” in sentence○9 of L4 is represented in picture four (P4)
61
by illustrating the girl holding the cellphone with her mouth open (see Figure 17).
Figure 16. A Screenshot from L2 of Kang-hsuan Edition
Figure 17. A Screenshot from L4 of Kang-hsuan Edition
The findings that represented participants and those process types associated with movement are illustrated more correspond to Bowcher and Liang (2013) on tourist site tickets, which also found that the sense relations achieved were mainly on the represented participants. The findings also testify Liu and Qu’s (2014) result that compared to visuals in science textbooks, visuals in EFL textbooks tended to make less abstract meaning. The result is expected since technically, concrete items are
62
more feasible for picture illustration (Ellis & Beaton, 1993; Tan, 2012). However, previous research has found that language learners may not understand the symbolic or metaphorical meaning even though they have no problem comprehending the literal meaning of the words (Johnson, 1981; Picken, 2001). For EFL junior high school students, abstract things may be more challenging to comprehend and
conceptualize than concrete ones (Paivio, 1986). If the visuals only deal with concrete items, it is likely that students may have difficulty understanding the content if they fail to comprehend the texts. More effort is needed to provide readers with multiple entry points into reading abstract concepts by utilizing different semiotic resources.
This section presents the results of ideational analyses of the 18 texts, including the degree of complementarity of the two editions, the distribution of six sense
relations, the represented participants and process types. The next section presents and discusses the findings of interpersonal analyses.
Intersemiotic Interpersonal Complementarity
The second research question explores to what extent interpersonal intersemiotic complementarity is achieved in Book 5 of Joy and Kang-hsuan junior high school English textbooks in Taiwan. Reinforcement of address in the 18 texts was examined to determine the degree of interpersonal complementarity. Table 15 presents the percentage of matched sentences, the number of total tokens and the density of token distribution in Book 5 of the two editions. For clearer comparison, Figure 18
demonstrates the tendency of the percentage of matched sentences.
Table 15. Intersemiotic Interpersonal Complementarity in Joy and Kang-hsuan Edition Percentage
Joy Kang-hsuan Joy Kang-hsuan Joy Kang-hsuan
63
Figure 18. Percentage of Matched Sentences in the Two Editions of Textbooks The spikes in Figure 18 reveal that, similar to ideational complementarity, there is considerable variation in the degree of interpersonal complementarity, even within the same edition. In Joy edition, the range of percentage of matched sentences is from 0% in L3 to 50 % in L8. In Kang-hsuan edition, the range is from 15 % in L6 to 59%
in L8. One instance of no complementarity was found in L3 of Joy edition. Overall, the degree of interpersonal complementarity is slightly higher in Kang-hsuan edition as the mean percentage in Joy edition is 20.8 % and in Kang-hsuan edition, 29.6%.
64
There are 7 lessons in which Kang-hsuan edition has higher percentage.
Moreover, as mentioned in the previous section, the relation between the
percentage of matched sentences and the number of tokens is not linear, which is also true with respect to interpersonal complementarity. The percentage of matched sentences shows how many sentences catch the interactive messages conveyed by the visuals and the density of token distribution manifests the how these tokens are spread among the sentences. In L2 of Joy edition, L7 of Kang-hsuan edition, the density is the highest, meaning that in the two lessons, the tokens are more concentrated.
After the overview of the interpersonal complementarity in the two editions, the following section expounds the two ways of reinforcement of address in more depth.
Two Ways of Reinforcement of Address
Reinforcement of address means the visuals and the texts complement mutually to interact with the reader in the same way. In this study, the reinforcement of address is realized in two ways: (1) between the gaze in the visual mode and the Mood structure in the textual mode (GM) and (2) between the horizontal angle in the visual mode and the use of personal pronouns in the textual mode (AP). The amounts of these two ways in Joy and Kang-hsuan edition were given in Figure 19. As revealed in the Figure, GM, with more tokens than AP, is realized more in both editions, but the difference is modest, suggesting that these two ways are used evenly to deliver interactive messages.
65
Figure 19. Amounts of GM and AP of Reinforcement of Address in Joy and Kang-hsuan Edition
The results show that Kang-hsuan edition has relatively higher degree of interpersonal complementarity than Joy edition. Nevertheless, the mean percentages in both editions are rather low, implying that reinforcement of address is not much achieved. The interpersonal messages delivered by the visual and textual mode are inconsistent. There are two kinds of situation for reader address and two for reader involvement, as exemplified and discussed below.
Incoherence with respect to reader address appears when one mode is rather interpersonal, whereas the other mode is simply providing objects of observation and items of information, not addressing the reader. For example, in Kang-hsuan edition, L4 (see Figure 20), the represented participant in picture one (P1) does not look at the reader, suggesting no reader address. In picture two (P2), the Walkman is
impersonified object, showing no reader address as well. But the Mood structure of sentence○1 and ○2 is interrogative, explicitly addressing and acknowledging the
impersonified object, showing no reader address as well. But the Mood structure of sentence○1 and ○2 is interrogative, explicitly addressing and acknowledging the