The notion of textbook is evolving due to the advancements in multimedia and digital technology (Ajayi, 2012). This trend has created opportunities to combine different modes into textbooks. The textbooks shifted from print material which is made out of words to the integration of diverse modes, including images, words, colors and audio. Emphasizing the mixture of multimodal resources, the phenomenon has promoted scholars to explore the multimodality of textbooks in different subjects.
Multimodal Studies on non-ELL Textbooks
O’Hallaron (1999a, 1999b, 2005) investigated mathematical discourse in his publication Mathematical Discourse: Language, Symbolism and Visual Images depending on Halliday’s social semiotic theory. The study analyzed written mathematical texts and examined the mathematical symbolism, visual images, and intersemiosis between language, visual images and symbolism. The results give implications for other multimodal studies based on social semiotic approach.
The research conducted by Lemke (2000) was concerned with the interaction of verbal and literacies in science textbooks. Scientific concepts in textbooks are usually presented “simultaneously and essentially verbal, mathematical, visual-graphical, and actional operational”. Visual-graphical representatives (e.g. diagrams, graphs, tables, equations and drawings) play an important role in science textbooks. The findings revealed that images and texts deal with different aspects of meaning. Texts
25
accompanied with images are more comprehensible for students to read.
Drawing on his own intersemiotic complementarily framework (Royce, 1998), Royce (2002) investigated the intersemiotic ideational meaning encode in high school science textbook, Science and Life: Work, Leisure, Technology and the Environment.
An extract with the topic of water cycle from the textbook was analyzed. The analysis indicated that the visual and verbal semiotic systems complement each other to produce a coherent multimodal text.
Guo (2004) conducted another study, exploring meaning making of various semiotic resources (e.g. graphs, images and language) in biology textbook, Essential Cell Biology. Based on Halliday’s systemic functional perspective, the texts were analyzed in terms of three dimensions: ideational, interpersonal, and textual meta-functions. The results of analysis suggested that visuals and language are interdependent and co-contextualized mutually. Visuals supplement and extend the language. Language “anchored and constrained the many possible meanings made in the visuals.”
Roehrich (2013) scrutinized the usage of visuals in undergraduate science textbooks in order to determine the intersemiotic conjunction between different media.
The results revealed the functional relationship between visuals and language. Visuals provide not only linguistic explanation bot also aesthetic value in textual enhancement.
The findings further showed that the choice of visuals has a profound influence on the science author’s linguistic choices, which directs reader’s interpretation of the articles.
In sum, these studies demonstrated the relation among multiple semiotic modes in the textbooks of mathematics (O’Hallaron, 1999a, 1999b, 2005), biology (Guo, 2004) and science (Lemke, 2000; Royce, 2002; Roehrich, 2013). Recently, English language learning (ELL) textbooks has also attracted researchers’ attention (Baldry,
26
2000). In the following paragraphs, English language textbooks which encompass multiple modes of semiotic resources and were viewed as multimodal texts will be explored.
Multimodal Studies on ELL Textbooks
Giaschi (2000) investigated the affordance of visual and textual modes in English as second/foreign language (ESL/EFL) textbooks in the U.K. The findings revealed that some messages (e.g. power, gender and social issues) that were carried through the images across textbooks may not be relayed by the written texts. The results further suggested ESL/EFL teachers that should note the distinct affordance permitted by different semiotic resources.
Two studies carried out by Chen (2009, 2010) were concerned with the interpersonal meaning in EFL primary and secondary school textbooks in China.
Chen (2009) explored how verbal and visual semiotic resources were cooperated to form interpersonal meaning. The data analysis showed that visuals implied the attitude readers should assume, while written texts explained to readers how they should perform. In the design of textbooks, visual features along with Linguistic properties should be taken into consideration.
In 2010, Chen examined the arrangement of multimodal resources in textbooks and the dialogic engagement with readers. The findings revealed that one kind of multimodal resource may make different engagement meanings and one kind of engagement meaning may be achieved by different multimodal resources. The dialogic engagement was closely related with devices, including dialogue balloons, texts, and illustrations.
Bezemer and Kress (2010) analyzed and compared the historical changes in
27
English textbooks of 1930s, 1980s and 2000s versions in England. The analysis indicated that the use of multimodal modes has altered between 1930s and 2000s. All modes in textbooks, including image, writing, layout and typography, contribute to meaning and potential for learning. Social relation between designers and users of textbooks also altered in the way that both of them share the responsibility for coherence equally, which was previously the particular domain of designers. Further, the production and construction of knowledge has changed into a more interactive and participatory process.
A multimodal research conducted by Tan (2012) attempted to investigate English textbooks Advance with English for senior high school students in China. The results indicated that the co-existence of the text and image make meaning, which facilitates students’ literacy learning and stimulated their interest of participation.
Liu and Qu (2014)’s study examined the multimodality of two EFL textbook series in China, Experiencing English (EE) and New Century College English (NCCE).
Through comparing the visual and verbal semiotic modes in textbooks, the findings showed that EE and NCCE were visually-verbally coherent with respect to Royce’s (1999) intersemiotic complementarity. Pictures and margin styles were two main differences between EE and NCCE. Such differences are related with target learners’
English proficiency and the differences of language difficulty. The results further gave suggestions for textbook designers to consider target learners’ English proficiency and adopt multimodal resources to achieve optimal intersemiotic complementarity.
In an investigation on English textbooks World Link for college students in South Korean, Torres (2015) explored the representational and interactive patterns based on Kress and van Leeuweun (2006)’s visual grammar. The researcher probed into the
28
embedded ideologies and their relation to the purpose of the textbook. The data analysis demonstrated where the verbal and the visual contradict with each other and how these contradictions disclose the embedded ideologies.
Tsai (2016) carried out another comparative study, investigating the difference of intersemiotic ideational and interpersonal complementarity between two junior high school English textbooks in Taiwan. Drawing on Kress and Van Leeuwen’s (1996) Visual Grammar, Halliday’s (1985, 1994) SFL and Royce’s (1999) intersemiotic complementarity, all the texts were analyzed through three steps : (1) intersemiotic ideational analysis, (2) intersemiotic interpersonal analysis and (3) comparison the degree of intersemiotic ideational and interpersonal complementarity. The findings revealed that ideational meaning rather than interpersonal meaning has higher degree of complementarity in both editions. Further, genres may make influence on visual design and intersemiotic complementarity between visual and verbal modes.
To sum up, a great number of researchers (Giaschi, 2000; Chen, 2009 ; Chen, 2010; Bezemer and Kress, 2010; Tan, 2012; Liu and Qu, 2014; Attar, 2014; Torres, 2015; Tsai, 2016) have examined the multimodality in English textbooks. However, some issues remain unexplored.
First of all, a great number of multimodal studies investigated English textbooks for secondary (Giaschi, 2000; Bezemer and Kress, 2010; Tan, 2012; Attar, 2014; Tsai, 2016) and college (Liu and Qu, 2014; Torres, 2015) students; nevertheless, few of them target on the level of elementary school (Chen, 2010). Nowadays, language learning textbooks are multimodal which contain different semiotic modes. Reading comprehension thus involves understanding of not only literal meaning but also interaction between various modes that coexist in the text. Fewer words and less complex syntax are used because of children’s limited vocabulary, syntax and world
29
knowledge (Chen, 2006; Hsieh, 2010). Images, relatively concrete and familiar for young readers, occupy more layout than texts which are more abstract and unfamiliar (Yang, 2011). How to understand and establish the meaning of literal texts through the references of pictures is an important issue for elementary school students. Given that visuals have become a salient feature of English textbooks especially in primary school (Shokouhi & Parvaresh, 2010), more multimodal research examining the verbal and visual components in the elementary school English textbooks is needed.
In addition, to date, previous studies (Giaschi, 2000; Chen, 2009; Chen, 2010;
Bezemer and Kress, 2010; Tan, 2012; Liu and Qu, 2014; Attar, 2014; Torres, 2015;
Tsai, 2016) have explored the image-text relations in English textbooks. These studies proved that language and image are equally important in meaning-making of texts (Kress, 2001). However, the merely co-occurrence of visual and verbal elements is not a sufficient evidence for a semantically integrated multimodal text (Liu and O’Halloran, 2009). According to Liu and O’Halloran (2009), one vital feature of multimodal texts is to create “integration of words and pictures rather than a mere linkage between the two modes.” To construct and deliver comprehensive meaning, the arrangement and degree of coherence between different semiotic resources are important (O’Halloran, 2005; Liu and O’Halloran, 2009).
So far, limited research has been done to examine not only the nature of intersemiotic semantic relations between visual and verbal modes but also the feature which renders multimodal text “visually-verbally coherent” (Royce, 2007).
Intersemiotic Parallel Structure, which is an essential text forming property in multi-semiotic texts and intertwines different semiotic modes together into a coherent unit deserves more investigations. In order to fill this gap, the present study aims at investigating the Intersemiotic Parallel Structure of visual and textual semiotic
30
resources in elementary school English textbooks.
Table 3. Synthesis of Multimodal research on English textbooks
Author Materials Level Research Focus Theoretical
Framework
secondary school comparison the changes of
senior high school relation of visual and verbal modes
college school intersemiotic complementarity
31