台灣國民小學英語教科書對話之多模態分析

全文

(1)國立臺灣師範大學文學院英語學系碩士論文 Department of English, College of Liberal Arts. National Taiwan Normal University Master’s Thesis. 台灣國民小學英語教科書對話之多模態分析 A Multimodal Analysis of Dialogues in Taiwanese Elementary School English Textbooks. 魏佩儀 Wei, Pei-Yi 指導教授: 張珮青博士. 中華民國 109 年 8 月 August 2020.

(2) 摘要運用不同的符號建構並傳達意義是當代溝通的特色。隨著多種符號大量並用，教科書也呈現多模態的趨勢，而閱讀理解的概念也與以往不同，圖文連貫在閱讀過程中扮演相當重要的角色。然而，關於探究國小教材圖文連貫的研究較少，此外各版本教科書圖文呈現的方式也有所不同。因此，本研究旨在探究翰林、康軒及佳音版國民小學英語教科書第五冊課文對話中圖文符際平行結構。本研究採用系統功能語法、視覺語法及符際銜接語法進行分析，文本分析總數為 12 篇，分析過程包含三個階段:文字分析、圖片分析及符際分析。本研究的發現如下:(1)文字分析中，關係過程、物質過程及心理過程最常在三個版本課文對話中出現，(2)圖片分析中，行動過程及反應過程最常在三個版本課文對話中出現，(3)符際分析中，完全平行結構最常在三個版本課文對話中出現，接著是部分平行結構，無平行結構的比例最少。本研究可提供英語教學者及學習者建議，教師在閱讀理解教學中，應提升學生的圖文意識、強化圖文概念，期望這些建議能提供英語教學以及未來多模態教科書研究參考。. 關鍵字: 多模態、多模態分析、系統功能語法、視覺語法、符際銜接語法、符際平行結構、國小英語教科書. i.

(3) ABSTRACT With the landscape of contemporary communication, meaning is constructed through the integration of multimodal modes. Following the prevalent employment of different modalities, textbooks have been more multimodal. The concept of reading comprehension has evolved with the emergence of multimodal texts. How visual and textual modes interact with each other to reach textual-visual cohesion is important in reading process. Nevertheless, little multimodal research was conducted to investigate the cohesive relation between textual and visual element (i.e. Intersemiotic Parallel Structure) in the elementary school context. Therefore, the current research aims to examine the Intersemiotic Parallel Structure in the dialogues of elementary school English textbooks for the 5th grade among Han-lin, Kang-hsuan, and Hess editions. A total of 12 dialogues were collected and underwent three phases of analysis. Halliday’s (2004) Transitivity System was utilized to conduct textual analysis, Kress and Van Leeuwen’s (1996) Visual Grammar was employed to do visual analysis and Liu and O’Halloran’s (2009) Intersemiotic Cohesive Device was adopted to implement intersemiotic analysis. The major findings are summarized as followings: (1) Relational, Material and Mental process were the three main process types in the texts of dialogues in the three editions. (2) Action and Reaction process were the two prevalent process types in the pictures of dialogues in the three editions. (3) Parallelism was predominantly achieved in the dialogues among the three editions, followed by Partial Parallelism. No Parallelism took up the least percentage among all. The present study yields significant implications for EFL instructors and learners. Language pedagogy for reading comprehension should be broadened to deal with textual and visual connection. It is hoped that this research could provide reference for. ii.

(4) second language teaching and learning as well as future multimodal studies on textbooks.. Keywords: Multimodality, Multimodal Discourse Analysis, Transitivity System, Visual Grammar, Intersemiotic Cohesive Device, Intersemiotic Parallel Structure, Elementary School English Textbooks, EFL. iii.

(5) ACKNOWLEDGEMENT This research could not have been completed without the support and assistance of a great number of people. First and foremost, I would like to express my utmost gratitude to my advisor, Dr. Pei-chin Chang. I am extremely grateful for her instruction on this thesis. She provided several concrete directions and valuable opinions to improve my study. Her insightful comments and guidance assisted me overcome the difficulties I met. My deepest gratitude also goes to my thesis committee members: Dr. Jun-Jie Tseng and Dr. Ya-ming Tai. Their helpful insights and constructive suggestions considerably polish the quality and enrich the content of my thesis. Moreover, I am thankful for the professors in the Department of English at NTNU. Their instruction has inspired me, broadened my horizon and enabled me to become a competent researcher as well as English teacher. Last but not least, I would like to express my sincere thanks to my beloved family, who offered encouragement and strengthened my belief throughout the process of thesis writing. Thanks for your support, understanding and unwavering love.. iv.

(6) TABLE OF CONTENTS CHINESE ABSTRACT .................................................................................................. i ABSTRACT ................................................................................................................... ii ACKNOWLEDGEMENT ............................................................................................ iv TABLE OF CONTENTS ............................................................................................... v LIST OF TABLES ....................................................................................................... vii LIST OF FIGURES ....................................................................................................viii CHAPTER ONE INTRODUCTION ............................................................................. 1 Background of the Study ........................................................................................ 1 Purpose of the Study ............................................................................................... 4 CHAPTER 2 LITERATURE REVIEW ........................................................................ 5 Systemic Functional Linguistics ............................................................................. 5 An Overview .................................................................................................. 5 Systemic Functional Approach to Multimodal Discourse Analysis .................... 6 Intersemiotic Cohesive Devices ..................................................................... 8 Parallel Structure .......................................................................................... 10 Transitivity System ............................................................................ 13 Visual Grammar ................................................................................... 17 Summary ...................................................................................................... 21 Studies on Multimodal Discourse Analysis ....................................................... 22 Multimodal studies on Textbooks................................................................ 24 Multimodal Studies on Non-ELL Textbooks .................................... 24 Multimodal Studies on ELL Textbooks............................................... 26 v.

(7) Elementary School English Textbooks in Taiwan ............................................... 31 Research Question ............................................................................................... 33 CHAPTER THREE METHODOLOGY ..................................................................... 35 Materials for Textbook Analysis ......................................................................... 35 Data Analysis ....................................................................................................... 36 CHAPTER FOUR RESULTS ..................................................................................... 45 Process Types of Textual Elements in the Dialogues .............................................. 45 Overall Distribution of Process Types of Textual Elements of Each Edition . 48 Process Types of Visual Elements in the Dialogues ................................................ 53 Overall Distribution of Process Types of Visual Elements of Each Edition ... 55 Intersemiotic Parallel Structure between Texts and Visuals in the Dialogues ......... 58 Overall Distribution of Intersemiotic Parallel Structure of Each Edition ........ 64 Summary .................................................................................................................. 66 CHAPTER FIVE DISCUSSION AND CONCLUSION ............................................ 67 Higher Frequency of Relational, Material and Mental Process of the Texts ....... 67 Prevalence of Action and Reaction Process of the Visuals .................................. 70 Predominance of Parallelism between the Texts and Visuals ........................... 71 Summary of Discussion ....................................................................................... 73 Pedagogical Implications ..................................................................................... 74 Limitation and Suggestions for Future Research ................................................. 74 Conclusion ........................................................................................................... 75 REFERENCES ............................................................................................................ 76. vi.

(8) LIST OF TABLES Table 1. Summary of Intersemiotic Cohesive Devices in the texts ............................... 9 Table 2. Correspondence between Visual Process and Textual Process ...................... 12 Table 3. Synthesis of Multimodal research on English textbooks ............................... 30 Table 4. Lessons in Han-lin, Kang-hsuan, and Hess edition ....................................... 36. vii.

(9) LIST OF FIGURES Figure 1. Example of Inter-semiotic Parallel Structure ............................................... 11 Figure 2. Six Process Types of Transitivity System .................................................... 13 Figure 3. Correspondence between Halliday’s (1994) SFL and Kress & Van Leeuwen’s (1996) Visual Grammar ............................................................................. 17 Figure 4. Variables of Representational Analysis in Visuals ...................................... 18 Figure 5. Theoretical Framework of the present study ................................................ 22 Figure 6. Procedure of Intersemiotic Cohesion Analysis ............................................ 37 Figure 7. Example of Parallelism (Unit3 from Han-Lin Edition) ............................... 39 Figure 8. Example of Partial Parallelism (Unit3 from Han-Lin Edition)................... 42 Figure 9. Example of No Parallelism (Unit3 from Kang-Shuan Edition) ................. 43 Figure 10. Percentage of process type of text in Han-lin edition ................................ 46 Figure 11. Percentage of process type of text in Kang-hsuan edition ......................... 47 Figure 12. Percentage of process type of text in Hess edition ..................................... 48 Figure 13. Overall Distribution of Process Type of Texts among Three Editions ...... 49 Figure 14. A screenshot from L1 of Kang-hsuan edition ............................................ 50 Figure 15. A screenshot from L2 of Hess edition ........................................................ 50 Figure 16. A screenshot from L3 of Han-lin edition ................................................... 50 Figure 17. A screenshot from L2 of Hess edition ........................................................ 50 Figure 18. A screenshot from L4 of Han-lin edition ................................................... 51 Figure 19. A screenshot from L2 of Kang-hsuan edition ............................................ 51 Figure 20. A screenshot from L1 of Han-lin edition ................................................... 51 Figure 21. A screenshot from L2 of Han-lin edition ................................................... 52 Figure 22. A screenshot from L2 of Hess edition ........................................................ 52 Figure 23. A screenshot from L3 of Hess edition ........................................................ 52 viii.

(10) Figure 24. A screenshot from L4 of Hess edition ........................................................ 52 Figure 25. Percentage of Process Type of Visuals in Han-lin Edition ........................ 53 Figure 26. Percentage of Process Type of Visuals in Kang-hsuan Edition ................. 54 Figure 27. Percentage of Process Type of Visuals in Hess Edition ............................. 55 Figure 28. Overall Proportions of Process Type of Visuals among Three Editions .... 56 Figure 29. Example of Action process from L4 of Kang-hsuan Edition ..................... 57 Figure 30. Example of Reaction process from L3 of Han-Lin Edition ....................... 57 Figure 31. Percentage of Intersemiotic Parallel Structure in Han-lin edition.............. 59 Figure 32. A screenshot from L3 of Han-lin edition ................................................... 60 Figure 33. A screenshot from L1 of Han-lin edition ................................................... 61 Figure 34. Percentage of Intersemiotic Parallel Structure in Kang-hsuan edition ...... 61 Figure 35. A screenshot from L3 of Kang-hsuan edition ............................................ 62 Figure 36. Percentage of Intersemiotic Parallel Structure in Hess edition .................. 63 Figure 37. A screenshot from L2 of Hess edition ........................................................ 64 Figure 38. Overall Distribution of Intersemiotic Parallel Structure among the three editions ......................................................................................................................... 65 Figure 39. Example of Parallelism from L1 of Hess Edition ...................................... 66. ix.

(11) CHAPTER ONE INTRODUCTION Background of the Study With the rapid improvement of multimedia and digital technology, contemporary communication has been increasingly multimodal (Ajayi, 2009). Due to the use of multiple modes, meaning is constructed not only through language but also the combination of multiple semiotic resources such as textual, visual, audio and spatial (Baldry, 2000; Baldry & Thibault, 2006; Hagan, 2007; Kress & Van Leeuwen, 2001; Matthiessen, 2007; Murray, 2013). Rather than entertainment and decoration, images and other non-verbal resources have become important in communication and meaning-making (Liu, 2013). The last few decades have witnessed the growing popularity of examining how multiple modes interact with each other to generate meaning (Fairclough, 2000). Following the prevalent employment of different modalities, textbooks, like many other materials, have been more multimodal (Ajayi, 2009; Chen, 2010; Early &Marshall, 2008).. Kress (2000) stated that it is now almost impossible for readers. to make sense of a text without referring to other semiotic resources which co-exist. Visual elements are incorporated into the textbooks in the ways of pictures, illustrations, charts and graphics (Jewitt, 2005). These visuals are of great significant in activating prior knowledge, organizing reading inputs, retrieving information and constructing meanings (Arnold& Fonseca, 2009; Cappello & Walker, 2016; Pan & Pan, 2009). Both textual and visual components interplay in multimodal texts work together to promote overall understandings, facilitate comprehension and further expedite learning (Ajayi, 2009; Jewitt, 2008; Kress & Van Leeuwen, 1996; Unsworth, 2001). Take the textbook of Taiwan for example. Elementary school English textbook 1.

(12) in Taiwan is multimodal which composed of visual and textual semiotic modes. Visual interfaces usually occupy more space than written parts. Each unit starts with a dialogue where the written text is inserted in the speech bubble as a part of the image. Language learners comprehend the dialogue by decoding both the pictorial components and linguistic elements. The dialogue functions as instructional scaffold for learners to acquire target vocabulary and sentence structures in that unit. Over the past few decades, the concept of reading comprehension has evolved with the emergence of multimodal texts (Albers, 2008; Anstey & Bull, 2006). Rather than merely decoding the written words alone, reading comprehension involve the processing of other modes in text (Serafini, 2012). The incorporation of different semiotics has promoted readers to interpret and analyze the texts in more integrative way (Unsworth, 2002; Royce, 2002). To effectively comprehend the multimodal texts, readers deal with image, language as well as image-text relations to connect with their background knowledge (Hibbing & Rankin-Erickson, 2003; Kramsch & Andersen, 1999; Royce, 2002). Given the fact that textual and visual modes afford different meaning and need different intellectual work, it may be difficult for some readers to integrate multiple modes to make meaning (Ajayi, 2012; Bezemer& Jewitt, 2010; Guichon& McLornan, 2008; Siegel, 2006). Such challenge may induce readers’ anxiety and consequently inhibit understanding of the text (Matsuda & Gobel, 2004; Saito& Horwitz, 1999). Additionally, when multiple resources did not complement each other in apprehensible way, readers’ attention would be distracted (Ajayi, 2012; Watkins, Miller & Brubaker, 2004). Different modes in multimodal texts may cause readers’ exceeding cognitive load and further impede comprehension (Bezemer& Jewitt, 2010; Guichon& McLornan, 2008; Jewitt, 2008). To address the difficulties and challenges readers may encounter, there is a dire. 2.

(13) need to examine the interconnection of various modes in text (Cappello & Walker, 2016; Carney & Levin, 2002; Kress, 2000). How multimodal resources complement with each other is an essential factor which influences reader’s comprehension (Royce, 2007). To develop a comprehensible text, the deployment and degree of coherence between different semiotic modes is of great significance (O’Halloran, 2005; Liu and O’Halloran, 2009). Intersemiotic Parallel Structure, which refers to cohesive relation integrating language and image, plays an important role in developing a comprehensible and meaningful text (Liu and O’Halloran, 2009). When reading the text, readers will find parallel meanings between picture and written text (Short, 1996; Mayer & Anderson, 1992; Mayer, 2004). If visual and textual information are not compatible, it might cause readers’ distraction, cognitive overload and further impediment of comprehension (Bezemer& Jewitt, 2010; Guichon& McLornan, 2008). Previous multimodal studies have focused on the combination and relation of various range of semiotic resources presented simultaneously, including affordance of visual and textual modes (Giaschi, 2000), co-existence of visuals and texts (Tan, 2012), interpersonal meaning constructed by verbal and visual semiotic resources (Chen, 2009), dialogic engagement (Chen, 2010). However, as pointed out by Royce (2007), Intersemiotic Parallel Structure of different semiotic resources has yet to get enough attention. In order to fill this gap, the present study aims at exploring the Intersemiotic Parallel Structure of visual and textual components in the multimodal text. In addition to the above issue, there has been a lack of research attention to the Intersemiotic Parallel Structure of visual and textual elements in elementary school English textbooks in Taiwan. Previous multimodal studies on textbooks were mostly carried out in higher education level such as secondary school (Giaschi, 2000; Bezemer and Kress, 2010; Tan, 2012; Attar, 2014; Tsai, 2016) and college (Liu and. 3.

(14) Qu, 2014; Torres, 2015). Nevertheless, little multimodal research was conducted in elementary school context. Furthermore, previous research on elementary school English textbook focused on single semiotic mode, which is text (Ma, 2003; Yang, 2004; Chen, 2009; Kou, 2009; Hsu, 2008; Huang, 2008; Yang, 2011; Yen, 2012; Teng, 2013) or image (Hsieh, 2010) only. Given that the prevalent use of multimodal elements is one main feature of textbooks (Chen, 2010), more studies examining multiple semiotic resources in English textbooks and their relationship is needed.. Purpose of the Study With the landscape of contemporary communication, the traditional concept of textbook has been changed (Kress, 2000). The radical shift of textbook has come from where language was the dominant bearer of meaning to where multiple modes combined to convey information (Baldry, 2000; Baldry & Thibault, 2006; Hagan, 2007; Kress & Van Leeuwen, 2001; Matthiessen, 2007; Murray, 2013). With image and language composing an integral part of textbooks, how the visual and textual components interacted with each other to make multimodal text visually-verbally coherent warrants much attention. Therefore, the present study aimed at exploring the visual and textual elements and Intersemiotic Parallel Structure between these two semiotic resources in the dialogues of elementary school English textbooks in Taiwan. Parallel Structure is an important property of multimodal texts which contributes to interaction between different semiotic resources (Liu and O’ Halloran, 2009). Since the parallel meaning across textual and visual components is vital in the reading process (Short, 1996; Mayer & Anderson, 1992; Mayer, 2004), examining the texts, visuals and Parallel Structure between these two modes is an important issue.. 4.

(15) CHAPTER TWO LITERATURE REVIEW The purpose of this study is to investigate the Intersemiotic Parallel Structure achieved in dialogues of elementary school English textbooks in Taiwan. To this end, this chapter reviews four aspects of literature which are relevant to the current study. First, the theoretical framework of the present research is described. Second, previous studies of multimodal discourse analysis on textbooks are reviewed and summarized. Then, the present situation of elementary school English textbooks in Taiwan is provided and related studies are reviewed. Finally, based on these literature reviews, three research questions are generated.. Systemic Functional Linguistics An Overview Systemic Functional Linguistics (SFL) which was mainly developed by Michael Halliday is a linguistic school that emerged in the 1960s. SFL has two pivotal components: systemic linguistics and functional linguistics. Instead of being simply associated with each other, these two components are closely joined together to construct a complete theoretical framework. Systemic linguistics deals with how language is formed for people to use (Eggins, 2004). Language is a complex network composed of systems and several subsystems. Each subsystem presents a set of choices for people to express meaning depending on the specific context. Functional linguistics, on the other hand, refers to how people utilize language in different occasions (Eggins, 2004). The use of language plays a role of social interaction and achieves different purposes in accordance with the context (Eggins, 2004; Fries,. 5.

(16) Gregory & Halliday, 1995). Instead of grammatical rules and forms, more attention is drawn to functions and semantics of language (Halliday& Webster, 2008). Language system performs diversified functions. Halliday (1994) proposed three metafunctions of language: ideational, interpersonal and textual metafunctions. The ideational metafunction refers to “the speaker’s experience of the real world” (Halliday, 1970). Halliday (1985) proposed that it conveys “our experience of the world that lies about us, and also inside us, the world of our imagination.” This metafunction involves two constituents, which are the logical and experiential meanings (Eggins, 2004). The former is related to the logical connection between the clauses while the latter is concerned with the presentation of the context. The interpersonal metafunction reflects “meaning as a form of action: the speaker or writer doing something to the listener or reader by means of language” (p.53). This metafunction involves the interactional meaning between producer and listener/reader who have intercommunication through “giving or demanding” of information in a communicative context. The textual metafunction speaks to the organization of the text. As Halliday (1985) contended, it carries the meaning of “the relevance to the context: both the proceeding and following texts and context of situation”. This metafunction serves to construct a coherent text through integrating ideational and interpersonal meanings and creating correlation between text and context.. Systemic Functional Approach to Multimodal Discourse Analysis Following the rising concept of multimodal communication, language is not the only modality to convey meaning (Chen, 2010; Cope & Kalantzis, 2009; Hyland, 2011; Jewitt, 2003; Lemke, 2000; New London Group, 1996). Meaning is realized through the combination of multiple semiotic resources (Baldry, 2000; Baldry & 6.

(17) Thibault, 2006; Hagan, 2007; Kress & Van Leeuwen, 2001; Matthiessen, 2007). Kress (2000) stated that, “It is now impossible to make sense of texts, even of their linguistic parts alone, without having a clear idea of what these other features might be contributing to the meaning of a text”. Communication has involved various semiotic modes to construct meaning and is inherently multimodal (Ajayi, 2009; Hyland, 2011). Halliday (1994) declared that “There are many other modes of meaning, in any culture, which are outside the realm of language.” Initially developed for the analysis of language, Halliday (1994)’s SFL has been applied to other semiotic modes (Halliday, 1994; Kress & Van Leeuwen, 1996; Lemke, 2002; O’Halloran, 1999; O’Toole, 1994; Unsworth & Wheeler, 2002). In the late 1990s, there has been a paradigmatic shift from the focus of Systemic Functional framework to Multimodal Discourse Analysis (SF-MDA). Discourse analysts have found the need to account for the meaning constructed through multiple semiotic resources in various media (Baldry and Thibault, 2006; O’Halloran 2004, 2005; Royce 1998, 2007; Ventola et al. 2004). According to Halliday (1994), SFL theory offers theoretical tools for MDA because it is a social semiotic theory where the meaning is context-dependent. Pioneering work of O’Toole’s (1994)’s Language of Displayed Art and Kress and Van Leeuwen (1996)’s Reading Images: The Grammar of Visual Design established the foundations for multimodal studies which expand SF approach to visuals (O’Halloran, 2011). Following O’Toole’s (1994) and Kress and Van Leeuwen’s (1996) research, interest in SFL approach to MDA has greatly increased (Baldry and Thibault, 2006a; Bateman, 2008; Iedema, 2001a, 2003; Kress and Van Leeuwen, 2001; Kress et al., 2001; Martin, 2002; Martinec, 2005; O’Halloran, 2004a, 2005; Royce, 1998; Unsworth, 2001; Van Leeuwen, 1999, 2005; Ventola et al., 2004).. 7.

(18) One approach of SF-MDA, specifically concerned with the interaction across of textual and visual components, makes discourse viewed as a whole and analyzes the meaning created by each component of the text. This integration is called “intersemiosis”, which involves different semiotic resources intertwine together to construct meaning in multi-semiotic texts. Given that the barely co-existence of linguistic and pictorial components is insufficient to construct a semantically integrated multimodal text, the degree of coherence between different semiotic modes is significant to create comprehensive meaning (O’Halloran, 2005; Liu and O’Halloran, 2009). Liu and O’ Halloran (2009)’s Intersemiotic Cohesive Devices, one SF-MDA approach investigating intersemiotic cohesion across textual and visual modes, will be elucidated below.. Intersemiotic Cohesive Devices Following the developing interaction of language and image for meaning-making, understanding how these various modes separately and concurrently construct different dimensions of meaning is important (Unsworth, 2001). Since 1970s, there has been a great amount of research exploring the interaction of linguistic elements with other semiotic resources (Barthes,1964, 1977; Royce, 1998, 1999a, 1999b; Knox, 2007; Guijarro& Sanz, 2008; Guijarro, 2010; Leonzini, 2012; Yang& Zhang , 2014; Francesconi, 2011; Bowcher & Liang, 2013; Liu, 2014). Inspired by Hasan’s (1985) Parallelism for semantic association of texts, Liu and O’ Halloran (2009) proposed Intersemiotic Cohesive Devices, examining the semantic relation between textual and visual semiotic modes in texts. These Intersemiotic Cohesive Devices are investigated “in terms of the Meta functions for meaning construal in discourse stratum” to analyze the intersemiotic texture of multimodal texts (Liu and O’ Halloran, 2009).. 8.

(19) Intersemiotic Cohesive Devices are essential text forming resources which contribute to interaction and negotiation between different elements in multi-semiotic texts. They are crucial properties of multimodal texts which integrate linguistic and pictorial components into a coherent semantic unit, instead of only linkage between the two modes. Table 1 presents the Intersemiotic Cohesive Devices in multimodal texts as presented by Liu and O’Halloran (2009). This framework focuses on the relationship between image and written text concerning ideational and textual aspect. In terms of ideational aspect, two types of meta-functions are suggested: Logical (Implication Sequences) and Experiential (Correspondence, Antonymy, Hyponymy, Meronymy, Collocation, and Polysemy). With regard to textual aspect, four meta-functions are proposed, including Reference, Theme-Rheme Development, Given-New organization, and Parallel Structure. Table 1. Summary of Intersemiotic Cohesive Devices in multi-semiotic texts Liu and O’ Halloran (2009) Stratum. Discourse. Meta-function Ideational. Logical Experiential. Implication Sequences (O’Halloran 2005) Correspondence (Jones 2007) Antonymy Hyponymy Meronymy Collocation (Royce 1998; see also Inter-semiotic Ideation, O’Halloran 2005) Polysemy. 9.

(20) Textual. Reference (see Inter-semiotic Identification, O’Halloran 2005) Theme-Rheme Development (see Framing and Salience, Kress and van Leeuwen 2006) Given-New Organization (see Information Value, Kress and van Leeuwen 2006) Parallel Structures (Liu and O’ Halloran, 2009). Among these Intersemiotic Cohesive Devices, Parallel Structure is a significant text-constructing device because learners will find parallel meanings across linguistic and pictorial elements when reading (Short, 1996; Mayer & Anderson, 1992; Mayer, 2004). To comprehend the multimodal texts, learners deal with picture and text as well as visual-textual relation to establish connections with background knowledge. Both image and language utilized in multimodal texts work in tandem to construct meaning and facilitate learners’ reading comprehension (Ajayi, 2009; Jewitt, 2008; Kress & Van Leeuwen, 1996; Unsworth, 2001). When visual and textual information are incompatible with each other, learners might be distracted their attention, get much cognitive load and fail to reach comprehension (Bezemer& Jewitt, 2010; Guichon& McLornan, 2008). Nevertheless, little attention has been paid to parallelism between textual and visual semiotic modes (Liu and O’Halloran, 2009). Parallel Structure, which is important but attracts least attention as a cohesive device, worth of more investigation.. Parallel Structure Gaining insights from Hasan’s (1985) model of parallelism for semantic relation of linguistic texts, Parallel Structure is presented by Liu and O’Halloran (2009) as one of Intersemiotic Cohesive Devices. Parallel Structure refers to cohesive relation which 10.

(21) integrates language and image when the two semiotic resources possess a similar form. According to Liu and O’Halloran (2009), textual and visual semiotic modes construe the world of experience through Transitivity structures. Textual semiotic modes is analyzed according to Halliday (1985)’s Transitivity System and visual semiotic resources is examined based on Kress and Van Leeuwen (2006)’s Visual Grammar. Intersemiotic Parallel Structures will take effect when the two semiotic modes bare analogous Transitivity configuration. According to Mayer & Anderson (1992), learners can make sense of reading content more easily when they gain identical meaning across both modes. When the participants and process in the visual components are identically represented in the linguistic elements, learners can receive stronger connection across image and text and visualize the reading materials through repeated information multimodally (Hibbing & Rankin-Erickson, 2003; Rowsell & Hamilton, 2012).. Figure 1. Example of Inter-semiotic Parallel Structure (Retrieved from Liu and O’Halloran, 2009, 13). 11.

(22) As evidenced by Figure 1, following Halliday (1985), the sentence Israeli army dog attacks Palestinian woman is presented as a Material process where Israeli army dog functions as Actor while Palestinian woman plays the role of Goal. The image which can be trans-coded as A dog attacks a Muslim woman is classified as an Action process (Kress and van Leeuwen, 2006). Consequently, both the text and picture share analogous Transitivity process, which leads to Intersemiotic Parallel Structures. The next part presents the two theories which the analysis of Parallel Structure is based on: Halliday (2004)’s Transitivity System and Kress and Van Leeuwen (1996)’s Visual Grammar.. Table 2. Correspondence between Visual Process and Textual Process (Kress and Van Leeuwen, 2006) Visual Process. Textual Process. Action Process. Material process. Reaction Process (Non-transactional). Behavioral Process. Reaction Process (transactional). Mental Process (perception). Mental Process. Mental Process (Cognition, affection). Verbal Process. Verbal Process. Analytical process. Possessive Attributive process. Classificational process. Intensive Attributive process. Symbolic process. Identifying Relational process. Symbolic process (suggestive). Existential Process. 12.

(23) Transitivity System Among the three metafunctions of language in Halliday (1994)’s SFL, ideational metafuction reflects people’s experience of outer and inner world. People are capable of describing outside events and representing inside feelings with language. The meaning of ideational metafunction can be analyzed by the transitivity system. Transitivity system is regarded as the pattern of experience. Halliday & Matthiessen (2004) maintained that the structure of the transitivity system is composed of three components: process, participant and circumstance. The “process” is expressed by a verbal group in a clause, referring to what is happening. The “participant” is encoded via nominal groups in a clause, signifying people and things included in the process. The “circumstance” is usually presented by adverbial groups or prepositional groups, identifying the accompanying facts related to the process including time, space, etc (Halliday & Matthiessen, 2013).. Figure 2. Six Process Types of Transitivity System (Halliday & Matthiessen, 2013, p.216). 13.

(24) Halliday (2004) noted that “our most powerful impression of experience is that it consists of a flow of events, or ‘goings-on’. This flow of events is chunked into quanta of change by the grammar of the clause”. Halliday and Matthiessen (1999) suggested that “the quantum of change is modeled as a figure – a figure of happening, doing, sensing, saying, being or having”. The transitivity system is regarded by Halliday (1994) as a system developing the world of inner and outer experience into different types of processes. Material, Mental and Relational processes are the main types in transitivity system. The other three subsidiary ones involve Verbal, Existential and Behavioral processes. These six process types construe the Transitivity system (see Figure 2). The definitions and detailed examples of each process are presented below.. . Material Process. Material process deals with clauses of “doing and happening” (Halliday and Matthiensen, 2004), which interprets the tangible and concrete events. Material clauses are entangled with the “Actor” who performs the action and brings about the changes on other participant called “Goal”. For example, in the sentence The lion caught the tourist, the lion is “Actor” and the tourist is “Goal”.. . Mental Process. Mental process are process of sensing which concerned with “our experience of the world of our consciousness” (Halliday and Matthiensen, 2004). “Sensor” is the participant which represents the conscious beings that can sense – feel, think, want and perceive. “Phenomenon” is the other participant which is felt, perceived, wanted or thought. Mental process is subcategorized into Perception (sense, see, notice, etc.),. 14.

(25) Cognition (think, believe, suppose, etc.), Desideration (wish, want, desire, etc.) and Emotion (rejoice, fancy, love, etc.). For instance, in the sentence Mary liked the gift, Mary is “Sensor” and the gift is “Phenomenon”.. . Relational Process. Relational process refers to clauses of “being or having” (Halliday and Matthiensen, 2004), which makes an association between two different entities and “serves to characterize and to identify”. There are three main types of relational process, including “Intensive”, “Possessive” and “Circumstantial”. Each of these comes in two distinct modes of being – “Attributive” and “Identifying”. Reversibility is one of important differences between the two modes. The “Identifying” one is reversible, which means “x” and “a” can be exchanged: Sarah is the leader/the leader is Sarah. On the other hand, the “Attributive” one is not reversible: there is no form wise is Sarah which is agnate to Sarah is wise. In the “Attributive” mode, an entity has some class ascribed or attributed to it. The class is labeled as “Attribute”, and the entity to which it is ascribed is “Carrier”. In the “Identifying” mode, entity has an identity allotted to it. “Value” is the one used to identify another and “serves as identity”, while “Token” represents “what is being defined”. . Verbal Process. Verbal process refers to “the process of saying” (Halliday and Matthiensen, 2004), which are utilized to convey “messages through language” (Thompson, 2004). Four elements involved in such process, including Sayer, Verbiage, Receiver, and Target. “Sayer” is the vital participant who produces the utterances, like my watch in My watch says it’s half past ten. “Verbiage” refers to “what is said”; e.g. what in What. 15.

(26) did you say? “Receiver” is the one to whom the saying is directed and is accompanied by the prepositions to and of; e.g. your parents in did you repeat that to your parents? “Target” is the participant “that is targeted by the process of saying”, which may be a person, an object or an abstraction; e.g. Krishan Kant in He accused Krishan Kant of conspiring with Bansi Lal.. . Behavioral Process. Behavioral process depicts “physiological and psychological behaviors” and possesses the characteristics of Material and Mental processes (Halliday and Matthiensen, 2004). The participant who behaves is called “Behaver”, which is “typically a conscious being”. The most symbolic pattern is a clause comprises “Behaver” and “Process”; e.g. No one’s listening, He’s always grumbling. Common variant is that the process precedes a participant labeled as “Phenomenon”; e.g. she sang a song.. . Existential Process. Existential process presents “the existence of a phenomenon” (Halliday and Matthiensen, 2004), which shares the features of Relational and Material process. The word there serves to signify the feature of existence and plays the role of subject in the clause (Eggins, 2004). The “Existent” refers to “the entity or event that is being said to exist” (Halliday and Matthiensen, 2004); e.g. a storm in there was a storm. The existential clause sometimes involves a circumstantial element (time or place), as on the wall in there was a picture on the wall.. 16.

(27) Visual Grammar Derived from Halliday’s SFL model (1994), “Visual Grammar” was developed by Kress and van Leeuwen (1996, 2006) to analyze visuals to account for other semiotic meanings than only from language. Expanded from Halliday’s (1994) metafunctional perspective of language, Visual Grammar is presented to explain three corresponding meaning of visuals, including representational, interactional and compositional. Representational meaning corresponds to the ideational metafunction, dealing with the participants, activities, attributes or the characteristics of the participants and circumstances involved. Interactional meaning reflects the interpersonal metafunction, examining the interaction (visual address, level of involvement, social distance and power relation) built between the viewers and what is viewed. Compositional meaning echoes the textual metafunction, exploring the page layout which determines the extent to which coherence in the text is achieved between the visual and verbal elements.. Figure 3. Correspondence between Halliday’s (1994) SFL and Kress & Van Leeuwen’s (1996) Visual Grammar (Attar, 2014). 17.

(28) Figure 3 represents the correspondence between Halliday’s (1994) SFL and Kress & Van Leeuwen’s (1996) Visual Grammar. The present study adopted the Visual Grammar framework to analyze representational perspective of visual components in the dialogue. Representation stands for the represented participants displayed in the image and how these participants associate with one another in meaningful ways. Drawing upon Kress & Van Leeuwen’s (1996) visual grammar, representational dimension of visual elements are realized through the represented participants, the processes and circumstances. People, things and places are identified as represented participants. There are two types of representational dimension of visual elements, Narrative and Conceptual (Figure 4).. Action process Narrative process Representational meaning (Kress and Van Leeuwen,1996). Reaction process Mental process Verbal process Analytical process. Conceptual process. Classificational process. Symbolic process. Figure 4. Variables of Representational Analysis in Visuals. Narrative process is similar to the Material processes in the system of transitivity. It possesses participants called “Actor” and “Goal”. Actor means “the participants from whom or which the vector departs and which may be fused with the vector to different degrees” (Kress and van Leeuwen, 2006) while Goal refers to the other participant, whom the vector is pointed. Narrative process is formed when the two 18.

(29) participants are “connected by a vector, represented as doing something to or for each other” (Kress and van Leeuwen, 2006). Narrative process can be differentiated based on the kinds of vector and the number of participants involved, including Action, Reaction, Mental, and Verbal process. The definition and example of each process is listed below.. . Action Process. In the Action process, “Actor” refers to the active participant “from which the vector emanates or which itself, in whole or in part, forms the vector” (Kress and Van Leeuwen, 2006). The passive participant “at which the vector is directed” is labeled as “Goal”.. . Reaction Process. Reaction process is constituted when the vector is created by the eyeline of one or more participants. “Reacter” is the active participant whose direction of the glance creates the eyeline. “Phenomenon” is passive participant “at which the eyeline is directed” (Kress and Van Leeuwen, 2006).. . Mental Process. When a vector is formed by a thought bubble that connects thinkers and their thought, Mental process is established. The participant who produces the thought bubble is “Senser”. What enclosed in the thought bubble functions as “Phenomenon”.. . Verbal Process. Verbal process refers to “a vector formed by the arrow-like protrusion of a ‘dialogue. 19.

(30) balloon’ or similar device connects two participants, a Sayer and an Utterance” (Kress and Van Leeuwen, 2006). “Sayer” is the participant from whom the dialogue balloon originated, while “Utterance” is the participant (verbal) contained in the dialogue balloon.. The other representational process is Conceptual process, which represents “participants in terms of class, structure or meaning, in other words in terms of their more generalized and more or less stable and timeless essence” (Kress and van Leeuwen, 2006). Conceptual process resembles to the Attributive process in the system of transitivity which displays “process of being” (Halliday, 2004; Kress and van Leeuwen, 2006). Different from the Narrative process, there is no vector which determines conceptual process of an image. Basically, conceptual process includes three types: Analytical, Classification and Symbolic. The following section will provide the definition of each process.. . Analytical process. In Analytical process, participants are associated in terms of the part-whole structure. Two kinds of participants involved, including “Carrier” and “Possessive Attributes”. “Carrier” denotes the participant which symbolizes the whole, while “Possessive Attributes” signifies other participants which are the parts of the whole.. . Classificational process. Classificational process connects participants with each other regarding a kind of relation, or taxonomy. In this process, one participant plays the role of “Superordinate” and the other is “Subordinate” of that participant.. 20.

(31) . Symbolic process. Symbolic processes are about “what a participant means or is” (Kress and Van Leeuwen, 2006). In this structure, “Carrier” denotes the identified participant, while “Symbolic Attribute” pertains to the identity of the participant.. Summary To conclude, Halliday’s (1994) SFL has offered a comprehensive approach for multimodal discourse analysis (MDA), not only written text (Halliday, 2002, 2004; Halliday and Matthiessen, 1999; Martin, 1992; Martin and Rose, 2003) but also extension to other semiotic resources in multimodal context (Halliday, 1994; Kress & Van Leeuwen, 1996; Lemke, 2002; O’Halloran, 1999; O’Toole, 1994; Unsworth & Wheeler, 2002). Following the paradigmatic shift from the study of language alone to “intersemosis” concerning the integration of language with other semiotic resources, how different modes intertwine together to construct a multimodal texts received much attention (Lemke, 2000; Liu and O’ Halloran, 2009; O’ Halloran, 2005; Royce, 1998). The theoretical foundation of the research is Liu and O’Halloran’s (2009) Intersemiotic Cohesive Device which investigating semantic relations among linguistic and pictorial components and thus integrate different modalities into a semantically coherent text. The present study targets on Intersemiotic Parallel Structure, one of Cohesive Devices which integrates textual and visual semiotic resources. Halliday (2004)’s Transitivity System is adopted to examine the ideational metafunction of textual modes, which includes six types of processes. Kress and Van Leeuwen (1996)’s Visual Grammar is utilized to analyze the representational aspect of visual components. Intersemiotic Parallel Structures will take effect when the two. 21.

(32) semiotic modes possess analogous Transitivity form. After introducing the major theoretical frameworks of the present multimodal research (see Figure 5), empirical multimodal studies will be reviewed in the next part.. Figure 5. Theoretical Framework of the present study. Studies on Multimodal Discourse Analysis Multimodal discourse analysis has become a new trend of discourse studies, because of its focus on the complete communicative aspects of discourse (Roehrich, 2013). Based on the premise that textual structure is realized not only linguistically but also through verbal means in contemporary communicative context, multimodal discourse analysis expands the focus of study from language only to language combined with other resources, such as images, scientific symbolism, gesture, action, music and sound (O’Halloran, 2011; Liu, 2013). As stated by Van Leeuwen & Jewitt (2001), multimodal discourses analysis can be conducted in many ways such as content analysis, conversation analysis, social and semiotic analysis. The pioneer study of multimodal discourse analysis was Barthes’s (1977) research, The Rhetoric of the Image. Three possible image-text relations were defined,. 22.

(33) including anchorage, illustration and relay. Anchorage refers that text supports image. Illustration means image supports text. Relay is defined that text and image are equal. The meaning of image mainly depends on the accompanied verbal text. Afterward, multimodal discourse analyses on various kinds of materials have gained much more attention (Duncum, 2004; Serafini, 2012). Magazines have been the target of multimodal discourse analyses since 1990s. Royce (1998, 1999a, 19999b) and Leonzini (2012) investigated the visual and verbal intersemiotic complementarity in The Economist. Yang& Zhang (2014) explored the representation meaning of multimodal discourse in The Economist. Picture books are one kind of materials in which pictures occupy more page layout. Guijarro & Sanz, (2008) scrutinized the compositional, interpersonal and representational meanings in the picture book, Guess how much I love you. Guijarro’s (2010) conducted another multimodal analysis, exploring interpersonal meanings expressed by image and text in the picture book, The Tale of Peter Rabbit. Informational texts have been the focus of multimodal discourse analyses as well. Knox (2007) investigated the visual, verbal and visual– verbal communication of three online English newspapers homepages and found that consistency between visual and verbal elements was important. Francesconi (2011) probed into the relationship of text and image in tourist brochures, including heritage, hospitality and diversity. The visual-verbal partnership is shown in Bowcher and Liang’s (2013) research, which explored multimodal resources of tourist site entry tickets and recipes. Liu (2014)’s multimodal research analyzed visual, verbal and visual–verbal cohesion of English majors’ English Presentation slide. Results from previous studies have implied that visual elements in multimodal texts not only carry out the function of illustration, but also play an important role in construction of meaning, through interaction of verbal elements or as an independent semiotic system.. 23.

(34) As many other materials mentioned above, the coexistence of visual and verbal communication appears in contemporary textbooks (Royce, 2002). Multimodal studies on textbook will be reviewed in the following section.. Multimodal studies on Textbooks The notion of textbook is evolving due to the advancements in multimedia and digital technology (Ajayi, 2012). This trend has created opportunities to combine different modes into textbooks. The textbooks shifted from print material which is made out of words to the integration of diverse modes, including images, words, colors and audio. Emphasizing the mixture of multimodal resources, the phenomenon has promoted scholars to explore the multimodality of textbooks in different subjects.. Multimodal Studies on non-ELL Textbooks O’Hallaron (1999a, 1999b, 2005) investigated mathematical discourse in his publication Mathematical Discourse: Language, Symbolism and Visual Images depending on Halliday’s social semiotic theory. The study analyzed written mathematical texts and examined the mathematical symbolism, visual images, and intersemiosis between language, visual images and symbolism. The results give implications for other multimodal studies based on social semiotic approach. The research conducted by Lemke (2000) was concerned with the interaction of verbal and literacies in science textbooks. Scientific concepts in textbooks are usually presented “simultaneously and essentially verbal, mathematical, visual-graphical, and actional operational”. Visual-graphical representatives (e.g. diagrams, graphs, tables, equations and drawings) play an important role in science textbooks. The findings revealed that images and texts deal with different aspects of meaning. Texts. 24.

(35) accompanied with images are more comprehensible for students to read. Drawing on his own intersemiotic complementarily framework (Royce, 1998), Royce (2002) investigated the intersemiotic ideational meaning encode in high school science textbook, Science and Life: Work, Leisure, Technology and the Environment. An extract with the topic of water cycle from the textbook was analyzed. The analysis indicated that the visual and verbal semiotic systems complement each other to produce a coherent multimodal text. Guo (2004) conducted another study, exploring meaning making of various semiotic resources (e.g. graphs, images and language) in biology textbook, Essential Cell Biology. Based on Halliday’s systemic functional perspective, the texts were analyzed in terms of three dimensions: ideational, interpersonal, and textual meta-functions. The results of analysis suggested that visuals and language are interdependent and co-contextualized mutually. Visuals supplement and extend the language. Language “anchored and constrained the many possible meanings made in the visuals.” Roehrich (2013) scrutinized the usage of visuals in undergraduate science textbooks in order to determine the intersemiotic conjunction between different media. The results revealed the functional relationship between visuals and language. Visuals provide not only linguistic explanation bot also aesthetic value in textual enhancement. The findings further showed that the choice of visuals has a profound influence on the science author’s linguistic choices, which directs reader’s interpretation of the articles. In sum, these studies demonstrated the relation among multiple semiotic modes in the textbooks of mathematics (O’Hallaron, 1999a, 1999b, 2005), biology (Guo, 2004) and science (Lemke, 2000; Royce, 2002; Roehrich, 2013). Recently, English language learning (ELL) textbooks has also attracted researchers’ attention (Baldry,. 25.

(36) 2000). In the following paragraphs, English language textbooks which encompass multiple modes of semiotic resources and were viewed as multimodal texts will be explored.. Multimodal Studies on ELL Textbooks Giaschi (2000) investigated the affordance of visual and textual modes in English as second/foreign language (ESL/EFL) textbooks in the U.K. The findings revealed that some messages (e.g. power, gender and social issues) that were carried through the images across textbooks may not be relayed by the written texts. The results further suggested ESL/EFL teachers that should note the distinct affordance permitted by different semiotic resources. Two studies carried out by Chen (2009, 2010) were concerned with the interpersonal meaning in EFL primary and secondary school textbooks in China. Chen (2009) explored how verbal and visual semiotic resources were cooperated to form interpersonal meaning. The data analysis showed that visuals implied the attitude readers should assume, while written texts explained to readers how they should perform. In the design of textbooks, visual features along with Linguistic properties should be taken into consideration. In 2010, Chen examined the arrangement of multimodal resources in textbooks and the dialogic engagement with readers. The findings revealed that one kind of multimodal resource may make different engagement meanings and one kind of engagement meaning may be achieved by different multimodal resources. The dialogic engagement was closely related with devices, including dialogue balloons, texts, and illustrations. Bezemer and Kress (2010) analyzed and compared the historical changes in. 26.

(37) English textbooks of 1930s, 1980s and 2000s versions in England. The analysis indicated that the use of multimodal modes has altered between 1930s and 2000s. All modes in textbooks, including image, writing, layout and typography, contribute to meaning and potential for learning. Social relation between designers and users of textbooks also altered in the way that both of them share the responsibility for coherence equally, which was previously the particular domain of designers. Further, the production and construction of knowledge has changed into a more interactive and participatory process. A multimodal research conducted by Tan (2012) attempted to investigate English textbooks Advance with English for senior high school students in China. The results indicated that the co-existence of the text and image make meaning, which facilitates students’ literacy learning and stimulated their interest of participation. Liu and Qu (2014)’s study examined the multimodality of two EFL textbook series in China, Experiencing English (EE) and New Century College English (NCCE). Through comparing the visual and verbal semiotic modes in textbooks, the findings showed that EE and NCCE were visually-verbally coherent with respect to Royce’s (1999) intersemiotic complementarity. Pictures and margin styles were two main differences between EE and NCCE. Such differences are related with target learners’ English proficiency and the differences of language difficulty. The results further gave suggestions for textbook designers to consider target learners’ English proficiency and adopt multimodal resources to achieve optimal intersemiotic complementarity. In an investigation on English textbooks World Link for college students in South Korean, Torres (2015) explored the representational and interactive patterns based on Kress and van Leeuweun (2006)’s visual grammar. The researcher probed into the. 27.

(38) embedded ideologies and their relation to the purpose of the textbook. The data analysis demonstrated where the verbal and the visual contradict with each other and how these contradictions disclose the embedded ideologies. Tsai (2016) carried out another comparative study, investigating the difference of intersemiotic ideational and interpersonal complementarity between two junior high school English textbooks in Taiwan. Drawing on Kress and Van Leeuwen’s (1996) Visual Grammar, Halliday’s (1985, 1994) SFL and Royce’s (1999) intersemiotic complementarity, all the texts were analyzed through three steps : (1) intersemiotic ideational analysis, (2) intersemiotic interpersonal analysis and (3) comparison the degree of intersemiotic ideational and interpersonal complementarity. The findings revealed that ideational meaning rather than interpersonal meaning has higher degree of complementarity in both editions. Further, genres may make influence on visual design and intersemiotic complementarity between visual and verbal modes. To sum up, a great number of researchers (Giaschi, 2000; Chen, 2009 ; Chen, 2010; Bezemer and Kress, 2010; Tan, 2012; Liu and Qu, 2014; Attar, 2014; Torres, 2015; Tsai, 2016) have examined the multimodality in English textbooks. However, some issues remain unexplored. First of all, a great number of multimodal studies investigated English textbooks for secondary (Giaschi, 2000; Bezemer and Kress, 2010; Tan, 2012; Attar, 2014; Tsai, 2016) and college (Liu and Qu, 2014; Torres, 2015) students; nevertheless, few of them target on the level of elementary school (Chen, 2010). Nowadays, language learning textbooks are multimodal which contain different semiotic modes. Reading comprehension thus involves understanding of not only literal meaning but also interaction between various modes that coexist in the text. Fewer words and less complex syntax are used because of children’s limited vocabulary, syntax and world. 28.

(39) knowledge (Chen, 2006; Hsieh, 2010). Images, relatively concrete and familiar for young readers, occupy more layout than texts which are more abstract and unfamiliar (Yang, 2011). How to understand and establish the meaning of literal texts through the references of pictures is an important issue for elementary school students. Given that visuals have become a salient feature of English textbooks especially in primary school (Shokouhi & Parvaresh, 2010), more multimodal research examining the verbal and visual components in the elementary school English textbooks is needed. In addition, to date, previous studies (Giaschi, 2000; Chen, 2009; Chen, 2010; Bezemer and Kress, 2010; Tan, 2012; Liu and Qu, 2014; Attar, 2014; Torres, 2015; Tsai, 2016) have explored the image-text relations in English textbooks. These studies proved that language and image are equally important in meaning-making of texts (Kress, 2001). However, the merely co-occurrence of visual and verbal elements is not a sufficient evidence for a semantically integrated multimodal text (Liu and O’Halloran, 2009). According to Liu and O’Halloran (2009), one vital feature of multimodal texts is to create “integration of words and pictures rather than a mere linkage between the two modes.” To construct and deliver comprehensive meaning, the arrangement and degree of coherence between different semiotic resources are important (O’Halloran, 2005; Liu and O’Halloran, 2009). So far, limited research has been done to examine not only the nature of intersemiotic semantic relations between visual and verbal modes but also the feature which renders multimodal text “visually-verbally coherent” (Royce, 2007). Intersemiotic Parallel Structure, which is an essential text forming property in multi-semiotic texts and intertwines different semiotic modes together into a coherent unit deserves more investigations. In order to fill this gap, the present study aims at investigating the Intersemiotic Parallel Structure of visual and textual semiotic. 29.

(40) resources in elementary school English textbooks. Table 3. Synthesis of Multimodal research on English textbooks Author. Materials. Level. Research Focus. Theoretical Framework. Giaschi. ESL/EFL textbooks. full range of. affordance of visual. SFL, Visual. (2000). Headway series and. teaching levels. and verbal modes. Grammar. 4th Dimension in the U.K. Chen. EFL textbooks in. primary and. interpersonal meaning SFL, Social. (2009). China. secondary school. between visual and. semiotic. verbal modes. approach. Chen. EFL textbooks in. primary and. arrangement of. Social semiotic. (2010). China. secondary school. multimodal resources. approach. the dialogic engagement with readers Bezemer. English textbooks in. and. England. secondary school. comparison the. Social semiotic. changes of. approach. Kress. multimodal modes of. (2010). 1930s, 1980s and 2000s versions. Tan. English textbooks. (2012). Advance with English. senior high school. relation of visual and. SFL, Social. verbal modes. semiotic. in China Liu and. two EFL textbook. Qu (2014). approach college school. . intersemiotic. SFL, Visual. series in China,. complementarity. Grammar,. Experiencing English. between visual. Intersemiotic. (EE) and New. and verbal. Complementarity. Century College. modes . English (NCCE). comparison the difference of two versions. 30.

(41) Attar. Iranian English. (2014). textbooks. middle school. college school. textual cohesive. SFL, Visual. devices between. Grammar,. visual and verbal. Intersemiotic. modes. Cohesive Device. representational and. Visual Grammar. Torres. English textbooks. (2015). World Link in South. interactive patterns of. Korean. visual and verbal modes. Tsai. two EFL textbook. (2016). junior high school. . intersemiotic. SFL, Visual. series in Taiwan , Joy. complementarity. Grammar,. and Kang-hsuan. between visual. Intersemiotic. and verbal. Complementarity. modes . comparison the difference of two versions. Elementary School English Textbooks in Taiwan A few decades ago, the National Institute for Compilation and Translation (NICT) was the only organization which edited and published the textbooks. After the implementation of the One Guide-Multiple Text Policy in 1996, the textbook market was liberalized. The Ministry of Education (MOE) issued the openness of designing textbooks for private publishers in elementary schools and secondary schools. These non-government publishers develop textbooks following MOE’s curriculum guidelines. From then on, the NICT has changed its character from the only authorized publisher to reviewer and regulator of textbooks. This policy has resulted in both positive and negative impacts (Chu, 2008). On the one side, releasing the restriction of only one version of English textbooks is great. On the other side, selecting the ideal teaching materials is challenging. As a result, the evaluation of 31.

(42) English textbooks has been an important issue nowadays. After the liberalization of the textbook market, English textbook relevant issues have been an active area of research (Hsieh, 2015). Textbook is regarded as an important teaching aid for teachers and learning materials for students (Brewster, Ellis & Girard, 2002). Therefore, supervising the quality of elementary school English textbooks becomes an essential issue (Huang 2008). Previous studies on elementary school English textbooks can be classified into three groups: content analysis, content analysis plus interview, and corpus study (Wang, 2012). Content analysis studies targeted at certain learning components in textbooks: (1) topic design (e.g. Ma, 2003; Yang, 2004; Chen, 2009; Yang, 2011), (2) communicative activities (e.g. Kuo, 2009; Chen, 2009), (3) vocabulary (e.g. Yang, 2004; Chen, 2009), (4) sentence patterns (e.g. Yang, 2011), (5) song and rhymes (e.g. Teng, 2013) and (6) illustrations (Hsieh, 1996; Chen, 2003; Chen, 2006; Hsieh, 2010). Some researchers conducted content analysis research through interviewing material-users, including: (1) student (e.g. Lu, 2006), and (2) teacher (e.g. Chang, 2006). Some carried out corpus studies centered on particular learning elements in textbooks, including: (1) vocabulary (e.g. Hsu, 2008, Huang, 2008), (2) sentence patterns (e.g. Hsu, 2008). Among research on content analysis of elementary school English textbooks in Taiwan, most of them targeted focused on merely textual elements (Ma, 2003; Yang, 2004; Chen, 2009; Kou, 2009; Hsu, 2008; Huang, 2008; Yang, 2011; Yen, 2012; Teng, 2013) or visual components (Hsieh, 1996; Chen, 2003; Chen, 2006; Hsieh, 2010). Nowadays, English learning textbooks have been multimodal which encompass multiple semiotic resources (Baldry, 2000). Reading involves construction of different semiotic modes to make meaning, more than ability to decode image or texts (Hubbard & Siskin, 2004; Nelson, 2006). In addition to conventional linguistic ability,. 32.

(43) readers need multimodal skills to interpret and understand the information constructed by different modes in the textbooks. Additionally, within among the elementary school English textbooks in Taiwan, each unit usually starts with a dialogue which is multimodal, composed of pictures and speech bubbles with written texts inserted. When reading the dialogue, pictures function as comprehensible input for language learners to acquire academic target vocabulary and sentence patterns in the lesson (Cappello & Walker, 2016; Carney & Levin, 2002; Kress, 2000). Visual and textual elements deployed in the dialogue intertwine and complement with each other to realize meanings and enhance learners’ reading comprehension (Hibbing & Rankin-Erickson, 2003; Kost, Foss & Lenzini, 1999; Kramsch & Andersen, 1999; Royce, 2002). However, multimodal research examining the connections of different semiotic codes in dialogue remains unexplored. Hence, the present study aims to explore the cohesive relation of textual and visual components of dialogue in Taiwanese elementary school English textbooks. A more complete understanding of text-image relation in textbooks and applications for future textbook compilation is expected to be provided.. Research Questions To address the issue mentioned above, three research questions were listed as below. 1.. What is the overall distribution of process types of texts in the dialogues of elementary school English textbooks in Han-lin, Kang-hsuan, and Hess edition?. 2.. What is the overall distribution of process types of visuals in the dialogues of elementary school English textbooks in Han-lin, Kang-hsuan, and Hess edition? 33.

(44) 3.. What is the overall distribution of Intersemiotic Parallel Structure between texts and visuals in the dialogues of elementary school English textbooks in Han-lin, Kang-hsuan, and Hess edition?. 34.

(45) CHAPTER. THREE. METHODOLOGY The major goal of the present study is to investigate the visual and textual elements as well as the Intersemiotic Parallel Structure between these two semiotic resources in the 5th grade Taiwanese elementary school English textbooks. This chapter contains two parts, including (1) Materials for textbook analysis (2) Data Analysis.. Materials for Textbook Analysis The materials under analysis are from Book 5 of three major editions of elementary school English textbooks, Han-lin, Kang-hsuan, and Hess. These three editions have better market share than other editions of textbooks in Taiwan (Wang, 2012). Each of them is composed of four lessons. A total of 12 dialogues were collected. Each lesson of these English textbooks starts with a short dialogue which usually portrays situations and several characters engaging in a conversation. The dialogue is multimodal which includes picture and speech bubble with texts inserted in it. The function of dialogue is to introduce the usage of vocabulary and target sentences intended to be learned in that unit. These dialogues are essential because they draw attention from students to the target words and grammatical structures of the sentences (Attar, 2014). The table 4 below shows an overview of the lessons from each edition, including the topic of each unit, number of sentences, words and pictures as well as average number of sentences and words.. 35.

(46) Table 4. Lessons in Han-lin, Kang-hsuan, and Hess edition Han-lin Edition Book 5. Unit. Kang-hsuan Edition Book 5. Topic. Number of sentences. Number of words. Number of pictures. Topic. Number of sentences. Number of words. Number of pictures. Topic. Number of sentences. Number of words. Number of pictures. How’s the Weather?. 23. 76. 9. You Are the Apple of My Eye. 21. 75. 10. Where Are You Going?. 23. 89. 14. Where Are You Going?. 21. 77. 8. This is My Cup of Tea. 24. 106. 10. What Do You See?. 21. 92. 10. How Many Lions Are There?. 18. 77. 8. It’s not My Day!. 25. 77. 11. What Would You Like for Dinner?. 21. 80. 9. What’s Wrong?. 20. 62. 8. He’s Sleeping like a Log. 25. 86. 9. How Do You Feel?. 22. 91. 11. 20.5. 73. 8. 23.75. 86. 10. 21.75. 88. 10. 1. Unit 2 Unit 3. Unit 4 Average. Hess Edition Book 5. Data Analysis The present study sets out to investigate the Intersemiotic Parallel Structure between textual and visual modes in the dialogues of elementary English textbooks among Han-lin, Kang-hsuan, and Hess edition, three main publishers which have higher market share in Taiwan (Wang, 2012). In this research, texts of the dialogue are analyzed according to ideational aspect of Halliday’s (2004) Transitivity System. Process type which the text belongs to is identified. Representational dimension of pictures in the dialogues is explored based on Kress and Van Leeuwen’s (1996) Visual Grammar. The transcoding of the image is presented. Then, following Liu and O’Halloran’s (2009) Intersemiotic Cohesive Device, the degree of Intersemiotic Parallel Structure achieved between text and image is analyzed, including Parallelism, 36.

(47) Partial Parallelism and No Parallelism. Figure 6. Procedure of Intersemiotic Cohesion Analysis. 37.

(48) The procedure of Intersemiotic Cohesion Analysis is shown in Figure 6. First of all, the textual element in the dialogues is identified following Halliday’s (2004) Transitivity System, which includes six types of process: Material (the process of doing and happening), Mental (the process of how people feel and think), Relational (the process of identifying and characterizing), Verbal (the process of saying), Behavioral (the process of physiological and psychological behaviors) and Existential (the process conveys that something exists or happens). Second, the visual element extracted from the dialogues is transcoded and analyzed based on Kress and Van Leeuwen’s (1996) Visual Grammar. There are two types of representational dimension of visual elements, Narrative and Conceptual. Narrative process is formed when the two participants are “connected by a vector, represented as doing something to or for each other”. Narrative process can be established according to the kinds of vector as well as the type and number of the involved participants, including Action process (A vector which is shaped by either an arrow or illustrated element to relate Actor and Goal), Reaction process (the eyeline of the a participant who is Reactor forms the vector, so the other participant is called Phenomenon), Mental process (A thought bubble forms the vector which connects Senser and Phenomenon), Verbal process (An arrow-like protrusion of a dialogue balloon shapes the vector and connects the Sayer to Utterance). Conceptual process includes three kinds of process: Analytical, Classification and Symbolic. In Analytical process, participants are related regarding the part-whole structure. Carrier refers to the participant representing the whole, while Possessive Attributes pertain to other participants which are the parts of the whole. In Classificational process, participants are associated with each other concerning a kind of relation, or taxonomy. In this structure, one participant plays the role of Superordinate and the other is Subordinate. 38.