台灣國民中學英語教科書之多模態言談分析

全文

(1)國立臺灣師範大學英語學系碩. 士. 論. 文. Master’s Thesis Department of English National Taiwan Normal University. 台灣國民中學英語教科書之多模態言談分析. Multimodal Discourse Analyses of Junior High School English Textbooks in Taiwan. 指導教授：張珮青. 博士. Advisor: Dr. Pei-Chin Chang 研究生：蔡欣蓉 Student: Hsin-Jung Tsai. 中華民國一百零五年六月 June 2016.

(2) 中文摘要運用多種不同的符號資源來傳達意義已成為當代溝通的常態，許多素材包括教科書，因圖文的大量並用，也呈現越來越多模態的趨勢。多模態的概念在過去幾十年被廣泛地研究，多模態言談分析也應用到各式各樣的素材，然而，關於探究圖文在多模態文本中如何共同建構各種不同層面意義的文獻卻較少，少數針對此議題的研究又侷限於單一層面意義的分析與詮釋，有鑑於此，一個跨層面的多模態分析實屬必要。再者，在台灣，與英語教科書相關的研究，都只著重圖或文的單模態分析，多模態仍是一個備受忽略的議題。因此，本研究旨在探究佳音及康軒版第五冊國民中學英語教科書中圖文的概念意義協調性及人際意義協調性，並進一步比較這兩層面意義協調性的差異。本研究採用系統功能語法、視覺文法及符際理論進行文本分析，總文本分析數為 18 篇，分析過程分為三階段：(一) 概念意義協調性分析、(二)人際意義協調性分析、(三)概念與人際意義協調性比較。研究結果顯示：(一)康軒版概念意義協調性高於佳音版、(二) 在六種符際關係中，圖文之間以重複關係最多；相反地，反義關係則未使用、(三) 兩版本人際意義協調性都不高，但以康軒版高於佳音版、(四) 整體來說，概念意義協調性高於人際意義協調性、(五) 進一步分析文類，發現文類影響圖像設計及圖文符際協調性。依據本研究發現，圖像系統具有其獨特意義建構功能，在義意傳達上扮演不可或缺的角色，因此，研究者建議教師應跳脫過去以文字為主的單模態教學，而將多模態教學納入課程設計，以培養學生多模態溝通能力，教科書設計者也應善用圖像資源、提高圖文之間協調性以提供學習者更豐富多元且完整的閱讀內容。期望這些建議能為台灣英文學習略盡棉薄之力。 i.

(3) 關鍵字: 多模態、多模態言談分析、符際協調、概念意義、人際意義、系統功能語法、視覺文法、國民中學英語教科書、文類。. ii.

(4) ABSTRACT The increasingly integrative employment of multimodal semiotic resources to project meaning has become a norm in contemporary communication. Textbooks, like many other materials, have been more multimodal due to the prevalent use of both textual and visual semiotic modes. Over the past few decades, the concept of multimodality has been an area of expanding research interest and multimodal discourse analyses of various materials have been conducted. However, limited research examined the ways textual and visual resources work conjointly to construct various dimensions of meaning in multimodal texts. Since meaning is simultaneously encoded in ideational, interpersonal and textual meaning, a more integrated approach to analysis is required. Additionally, most English textbook analyses in Taiwan have been mono-modal, tackling texts or images only. There has been a lack of pedagogical attention to visual-textual complementarity in English textbooks in Taiwan. Therefore, the present study aims to explore to what extent intersemiotic ideational and interpersonal complementarity are achieved in Book 5 of Joy and Kang-hsuan junior high school English textbooks and compare the difference between the degrees of complementarity of these two dimensions of meaning. Totally 18 texts were collected. Kress and Van Leeuwen’s (1996) Visual Grammar was employed to conduct visual analyses, Halliday’s (1985, 1994) Systemic Functional Linguistics (SFL) was utilized to do textual analyses and Royce’s (1999) intersemiotic complementarity was further used to combine the visual and textual analyses. Each text underwent three phases of analysis: (1) intersemiotic ideational analysis, (2) intersemiotic interpersonal analysis and (3) comparing the degree of intersemiotic ideational and interpersonal complementarity. The major findings are summarized as follows: (1) Ideational meaning was iii.

(5) found to achieve slightly higher degree of complementarity in Kang-hsuan edition. (2) Among the six sense relations, repetition was employed most frequently and no tokens of antonymy were found. (3) Similar to ideational meaning, the degree of interpersonal intersemiotic complementarity is higher in Kang-hsuan edition despite that the degrees in both editions are not high. (4) Comparison between the two dimensions of meaning demonstrated that ideational meaning has higher degree of complementarity in both editions than interpersonal meaning. (5) Genres may affect visual design and visual-textual complementarity. Based on the findings, some pedagogical implications are proposed: (1) Teachers need to go beyond traditional mono-modal curriculum and find potential integration of multimodality into English classrooms and their teaching repertoires. (2) Students’ multimodal literacy and multimodal communicative competence is suggested to be developed. (3) Textbook writers should devote due attention to visual design and strive to ensure the complementarity between the visuals and the texts. It is hoped that this study could inform a more comprehensive understanding of textual-visual complementarity, alternative multimodal approaches to language instruction and future textbook compilation.. Key Words: Multimodality, Multimodal Discourse Analysis, Intersemiotic Complementarity, Ideational Meaning, Interpersonal Meaning, Systemic Functional Linguistics, Visual Grammar, Junior High School English Textbooks, Genre. iv.

(6) ACKNOWLEDGEMENT This thesis could not have been possible without the support and assistance of many people. First of all, I would like to express my utmost gratitude to my advisor, Dr. Pei-Chin Chang, for her expertise and invaluable comments on this thesis. Her insightful advice and careful guidance helped me get through the difficulties I encountered during my thesis writing. Without her support, this thesis would not have been possible. My sincere gratitude also goes to the honorable committee members, Dr. Hsi-Nan Yeh and Dr. Hsiu-Sui Chang, who carefully proofread my thesis and offered many constructive suggestions on enriching the content of my thesis. Their invaluable advice and comments helped me to polish the quality of the thesis, for which I am truly grateful. Also, I would like to thank the professors in the Department of English who have taught me during the past two years. Their instruction has broadened my horizon, enabling me to be a better and more competent English teacher. My special thanks also go to my dearest classmates and friends: Amy, Abby, and Vivian, who encouraged me, gave me support and cheered me up. Without their generous help, this thesis would not be possible. Last but not least, I would like to thank my beloved parents and husband for they were very supportive throughout the process of thesis writing. Their love and support inspired confidence in myself to overcome every difficulty during my graduate study years and the conduction of this research, for which I am heartily thankful. This thesis is especially dedicated to them.. v.

(7) TABLE OF CONTENTS. CHINESE ABSTRACT ................................................................................................i ABSTRACT ................................................................................................................ iii ACKNOWLEDGEMENT……………………… ………………… … ……….….. v LIST OF TABLES.................................................................................................... viii LIST OF FIGURES ....................................................................................................ix CHAPTER ONE INTRODUCTION ......................................................................... 1 Background of the Study ..................................................................................... 1 Motivation of the Study ....................................................................................... 3 Purpose of the Study ............................................................................................ 5 Research Questions .............................................................................................. 7. CHAPTER TWO LITERATURE REVIEW ............................................................ 8 Systemic Functional Linguistics ......................................................................... 8 An Overview .................................................................................................. 8 Ideational Meaning and Transitivity System ............................................... 10 Interpersonal Meaning and Mood System ................................................... 13 Mood System ........................................................................................ 13 Personal Pronoun ................................................................................ 14 Theories and Frameworks of Multimodal Discourse Analysis ...................... 15 SFL Approach to Visual Analysis ................................................................ 16 Royce’s Intersemiotic Complementarity ..................................................... 17 Studies on Multimodal Discourse Analysis ...................................................... 19 Multimodal Studies on Non-textbook Materials ......................................... 19 Multimodal Studies on Textbooks ............................................................... 22 Junior High School English Textbooks in Taiwan .......................................... 26. CHAPTER THREE METHOD ................................................................................ 29 Materials ............................................................................................................. 29 Theoretical Frameworks ................................................................................... 30 Visual Analysis ............................................................................................ 31 vi.

(8) Textual Analysis ........................................................................................... 34 Intersemiotic Complementarity: Combining the Visuals and Texts ............ 36 Procedures of Multimodal Analysis.................................................................. 39 Example .............................................................................................................. 42. CHAPTER FOUR RESULTS AND DISCUSSION ................................................ 48 Intersemiotic Ideational Complementarity ..................................................... 48 The Use of Six Sense Relations ................................................................... 51 The Predominant Use of Repetition ..................................................... 53 The Functions of Hyponymy and Collocate ......................................... 55 Presentation of Represented Participant and Process ................................... 59 Intersemiotic Interpersonal Complementarity ............................................... 62 Two Ways of Reinforcement of Address ..................................................... 64 Comparison of Ideational and Interpersonal Complementarity ................... 68 The Influence of Genre on Intersemiotic Complementarity ........................ 70 Genre Difference in Ideational Meaning ............................................. 71 Genre Difference in Interpersonal Meaning ........................................ 73. CHAPTER FIVE CONCLUSION ........................................................................... 76 Major Findings of the Study ............................................................................. 76 Pedagogical Implications ................................................................................... 77 Limitation and Suggestions for Future Research ........................................... 79. REFERENCES........................................................................................................... 81. vii.

(9) LIST OF TABLES Table 1. Definition and examples of each process .................................................. 12 Table 2. Speech Functions and Mood Structures .................................................... 14 Table 3. Lessons in Joy and Kang-hsuan Editions .................................................. 29 Table 4. The Definition of Six Intersemiotic Sense Relations ................................ 36 Table 5. Interpersonal Functions in the Visual and Textual Modes ......................... 38 Table 6. Genre Classification (adapted from Macken-Horarik, 2002, p.23) ........... 41 Table 7. VMEs of Pictures One to Four .................................................................. 43 Table 8. Analysis of the Intersemiotic Ideational Complementarity of Lesson 7 of Joy Edition ................................................................................................ 44 Table 9. Interpersonal Visual Analysis of Pictures One to Four .............................. 46 Table 10. Analysis of the Intersemiotic Interpersonal Complementarity of Lesson 7 of Joy Edition .......................................................................................... 46 Table 11. Intersemiotic Ideational Complementarity in Joy and Kang-hsuan Edition ................................................................................................................. 49 Table 12. The Frequency of Occurrence of the Sense Relations in Joy Edition ..... 52 Table 13. The Frequency of Occurrence of the Sense Relations in Kang-hsuan Edition ..................................................................................................... 53 Table 14. The Distribution of Process Type in Joy and Kang-hsuan Edition.......... 60 Table 15. Intersemiotic Interpersonal Complementarity in Joy and Kang-hsuan Edition ..................................................................................................... 62 Table 16. Ideational and Interpersonal Complementarity of Different Genres in Joy and Kang-hsuan Edition. ......................................................................... 70. viii.

(10) LIST OF FIGURES Figure 1.Central and Peripheral Elements in the Experiential Structure of the Clause ..................................................................................................... 11 Figure 2.The Grammar of Experience: Types of Process in English .................... 11 Figure 3.The ATM Screen ..................................................................................... 32 Figure 4. A Screenshot from Book 4 Lesson 4 of Kang-hsuan Edition................ 33 Figure 5. A Picture from Book 4 Lesson 7 of Joy Edition .................................... 34 Figure 6.A Picture from Book 4 Lesson 5 of Joy Edition..................................... 39 Figure 7. Procedures of Multimodal Analysis ...................................................... 40 Figure 8. Percentage of Matched Sentences in the Two Editions of Textbooks ... 49 Figure 9. Proportions of the Sense Relations in Joy and Kang-hsuan Edition ..... 51 Figure 10. A Screenshot from L1 of Kang-hsuan Edition .................................... 54 Figure 11. A Screenshot from L8 of Joy Edition .................................................. 56 Figure 12. A Screenshot from L3 of Joy Edition .................................................. 57 Figure 13. A Screenshot from L9 of Joy Edition .................................................. 57 Figure 14. A Screenshot from L7 of Kang-hsuan Edition .................................... 58 Figure 15. Amounts of Represented Participant and Process in Joy and Kang-hsuan Edition ............................................................................. 59 Figure 16. A Screenshot from L2 of Kang-hsuan Edition .................................... 61 Figure 17. A Screenshot from L4 of Kang-hsuan Edition .................................... 61 Figure 18. Percentage of Matched Sentences in the Two Editions of Textbooks . 63 Figure 19.Amounts of GM and AP of Reinforcement of Address in Joy and Kang-hsuan Edition ............................................................................. 65 Figure 20.A Screenshot from L4 of Kang-hsuan Edition ..................................... 66 Figure 21. A Screenshot from L7 of Joy Edition .................................................. 66 ix.

(11) Figure 22. A Screenshot from L2 of Kang-hsuan Edition .................................... 67 Figure 23. A Screenshot from L7 of Joy Edition .................................................. 68 Figure 24. Percentage of Matched Sentences of Ideational and Interpersonal Meaning in the Two Editions of Textbooks ......................................... 69 Figure 25. A Screenshot from L5 of Joy Edition .................................................. 72. x.

(12) CHAPTER ONE INTRODUCTION Background of the Study Contemporary communication practices frequently involve multiple modes of resources to project meaning (Kress, 2000). For example, when we take a look at a newspaper page, we can find that in addition to texts, a lot more space is covered by images. Textual and visual resources play an equally critical role in providing information. If either written words or images are absent, comprehension of the content may become partial. Likewise, when watching films, animated movements, sounds, light effects and subtitles all contribute to delivering meaning to audience. With soundtrack being turned off, the film-watching experience may become incomplete (Duncum, 2004). Kress (2000) pointed out that, “It is now impossible to make sense of texts, even of their linguistic parts alone, without having a clear idea of what these other features might be contributing to the meaning of a text”(p. 337). With the improvement of multimedia and digital technologies, communication has been increasingly multimodal (Ajayi, 2009). Multimodality means the multiplicity and interconnection of various modes of communication that are recruited to make meaning (The New London Group, 1996). Languages, images, gestures and sounds are all essential bearers of meaning (Baldry, 2000; Baldry & Thibault, 2006; Hagan, 2007; Kress & Van Leeuwen, 2001; Matthiessen, 2007). Over the past few decades, the concept of multimodality has been extensively examined and discussed by researchers to understand how these multimodal modes complement or interact with each other to create meaning in context. One line of researchers has adopted a linguistically-derived theory, Systemic 1.

(13) Functional Linguistics (SFL) to investigate non-linguistic modes (Royce, 1999). SFL serves as a suitable theoretical tool for multimodal discourse analyses because it centers on the notion of language as a functional socio-semiotic system, which exchanges meanings in a socio-cultural context (Halliday, 1978, 1985). Halliday (1978) proposed that language encodes three general functions, or metafunctions at the same time: ideational is about experiential meaning, interpersonal is about social relations and textual is about discourse organization. In the past, these metafunctions are mainly realized through textual mode, but nowadays, both textual and visual modes are considered to be vital for the construction of meaning (Baldry & Thibault 2006; Ventola & Guijarro, 2009). Therefore, this tri-metafunctional principle of communication has been applied to analyzing the functionalities of various semiotic resources, especially texts and images, and exploring the ways in which semiotic choices are utilized in multimodal discourses to accomplish specific functions (Kress & Van Leeuwen, 1996; Royce, 1999). Multimodal analyses have been conducted for print media (e.g. Bowcher & Liang, 2013; Guijarro, 2011; Guijarro & Pinar Sanz; 2008; Guo, 2004; Kress & Van Leeuwen, 1998; O'Halloran, 1999; Royce, 1998), digital media (e.g. Eisenlauer, 2011, 2013; Knox, 2007, 2009; Sindoni, 2013; Tan, 2010; Van Leeuwen, 2005), photography (e.g. Bell, 2001; Caple, 2009, 2013; Machin, 2004), and film (e.g. Bateman, 2007; O'Halloran, 2004; Tan, 2009; Thibault, 2000; Wildfeuer, 2012). These previous studies have revealed that languages, along with other semiotics, synthesize the ideational meaning of “material reality”, the interpersonal meaning of “social reality” and cohere into textual configurations to accomplish “semiotic reality” (Unsworth, 2001, p.18).. 2.

(14) Motivation of the Study Since students are exposed to multimodal modes of communication with greater frequency in and out of school settings (Serafini, 2009), transaction with these multimodal texts has raised sharp questions about knowledge construction in educational contexts, especially in language learning classes (New London Group, 1996). First of all, multimodality has reshaped traditional conception of literacy and language competence. Evidence can be found in language learning textbooks, which have been more visually-dense due to the prevalent use of both verbal (textual) and visual semiotic modes (Royce, 2002). Since different modes afford different meaning and require different intellectual work from learners, what language learners have to read is not only language, but also a range of graphs, dialogue balloons, labelling, color, and font potential for conveying meaning (Chen, 2010; Cope & Kalantzis, 2009; Jewitt, 2003; Lemke, 2000; New London Group, 1996). The meaning of “text” has now extended beyond written words to incorporate a variety of representational and communicational modes like visual images, multimedia and digital technologies. This shift from a linguistic emphasis to a multimodal one necessitates learners to comprehend, navigate, interpret and analyze texts in more integrative and interactive ways (Anstey & Bull, 2006; Unsworth, 2002). Readers, especially novice readers may find it difficult working across multiple semiotics to make meaning (Siegel, 2006). In this situation, language pedagogy cannot be restricted to linguistic knowledge. Language teachers need to be able to exploit all semiotic resources co-present in texts to maximize the effectiveness of teaching and learning and to assist language learners in developing a new set of reading skills and strategies to deal with multimodal texts. Cope and Kalantzis (2012) advocated that it is important to “supplement traditional 3.

(15) reading and writing skills with multimodal communications” (p. 2). Multimodal literacy and competence include “the knowledge and use of language concerning the visual, gestural, audio and spatial dimensions of communication” (Heberle, 2010, p.102). To inform new pedagogy and support new frameworks for reading multimodal texts, it is necessary to investigate how different semiotics are deployed to communicate ideas and to initiate various kinds of interactions with readers (Kress & Van Leeuwen, 1996; Machin, 2013). Second, although the prominence of multimodality has led a good deal of research to address issues of affordance, multimedia and multiliteracy advocacy (e.g. Andrews, 2004; Bezemer & Kress, 2008; Hull & Nelson, 2005; Kamil, Intrator, & Kim, 2000; Leu, Kinzer, Corio, Castek, & Henry, 2013; Luke, 2003; Mayer, 2008; Richards, 2000), limited research examined text-image relations and the ways textual and visual resources work together in multimodal texts (Martinec & Salway, 2005; Unsworth, 2014). As multimodal texts have become the norm, more concerted effort to present an improved theorization of text-image relation is needed. Indeed, this lack of research concerning text-image relation is even more obvious in textbook research. Among a few researchers who explored the text-image relation of textbooks (Bezemer & Kress, 2010; Chen, 2009, 2010; Giaschi, 2000; Guo & Feng, 2015; Liu & Qu, 2014), most of them have dealt with single dimension of metafunctional meaning, or interpreted them separately. A more integrated approach to analysis is required because meaning is simultaneously encoded in three metafunctions (Halliday, 2002). Complementarity, which means the visual and textual modes complement each other to distribute coherent meaning, may be achieved in one single dimension of meaning but not in another (Royce, 1999). Hence, the present study seeks to quantitatively and qualitatively explore the degrees of complementarity in two dimensions of 4.

(16) metafunctional meaning: (1) ideational and (2) interpersonal. Ideational meaning is the resource for representation of readers’ mental picture of reality and experience, the meaning of “content”, which influences students’ text comprehension. Interpersonal meaning is the resource for establishing interactive relationship between the texts or visuals and the readers, the meaning of “action”, which affects students’ text interpretation (Halliday, 1985, p.53; Royce, 2007). These two dimensions of meaning are assumed to be more pertinent to junior high school students’ language learning. Third, there has been a lack of research attention to verbal-visual relations in English textbooks in Taiwan. Most English textbook analyses have been mono-modal, tackling texts or images only (e.g. Chang, 2006; He, 2004; Hsu, 2010; Lin, 2007; Lin, 2008; Peng, 2008; Wang, 2013; Yen, 2011). A few multimodality studies conducted in language education discipline have been concerned with multimedia, including digital stories (Lia, 2013), presentation slides (Liu, 2014) and multimodal glosses (Hsu, 2008). However, besides multimedia, there is a concomitant need to invigorate and consolidate research in the area of printed materials, which still dominate in classroom setting. The absence of systematic investigations of the verbal-visual relations in textbooks may pose challenges for teachers and students to work with multimodality in textbooks. Given that Taiwan is an English as a foreign language (EFL) context, textbooks have been dominant and indispensable learning and teaching materials. They offer important data for exploring how meaning is delivered to learners through verbal and visual resources and whether verbal and visual features are coherent. Purpose of the Study The page layout of reading sections in junior high school English textbooks in Taiwan exhibits two interconnected interfaces. Verbal interfaces usually occupy the 5.

(17) middle of the pages and visual interfaces are placed around. With visual and textual codes forming an integral part of textbooks, how the linguistic and visual resources are coordinated to make meaning warrants greater research attention. The purpose of the present study is three-fold. One is to determine whether the texts and images of reading sections in textbooks achieve coherence in terms of ideational meaning. Ideational meaning relates directly to the content, including the nature of events, participants (people, animals, objects) and processes (actions taking place). When reading a text, ideational meaning may be the first layer of meaning readers decode. Namely, the reader may first identify the participants, the processes as well as the relationships between all these elements involved (Royce, 2007). Whether elements in the texts can be systematically matched to the images is essential to helping readers expand understanding of the reading content. The second purpose is to investigate the text-image coherence with respect to interpersonal perspective. Interpersonal meaning specifies social roles and social relations between individuals engaged in communication (Halliday, 2002). The existence of reader address, the level of reader involvement, social distance and power relation can be derived from both texts and images (Kress & Van Leeuwen, 1996). Whether texts and images express consistent and coherent interpersonal messages to readers plays a vital role in facilitating readers’ interpretation and conceptualization of the reading content. Third, the study aims to compare and contrast the degrees of coherence between ideational and interpersonal meaning to see which dimension of meaning tends to reach higher degree of intersemiotic coherence, whether textual and visual resources complement each other to a similar extent in projecting these two dimensions of meaning. As multimodal texts are layered rather than linear and sequential, they 6.

(18) provide readers with multiple entry points into reading passages (Jewitt, 2005). To obtain a more comprehensive account of the nature of text-image relations, different dimensions of meaning should be considered and accounted for as a whole. To that end, the study investigates the following three questions. The first question examines whether the visuals and verbal texts convey coherent ideational meaning to the readers. The second question explores whether the visuals and verbal texts address and involve the readers similarly. Followed from above, the third question compares whether ideational or interpersonal meaning has higher degree of complementarity between visual and textual parts in the selected texts. Research Questions 1. To what extent is ideational intersemiotic complementarity achieved in Book 5 of Joy and Kang-hsuan junior high school English textbooks in Taiwan? (1) How are the six sense relations- repetition, synonymy, antonymy, hyponymy, meronymy and collocate used? (2) How are the represented participants and process presented? 2. To what extent is interpersonal intersemiotic complementarity achieved in Book 5 of Joy and Kang-hsuan junior high school English textbooks in Taiwan? (1) How is the reinforcement of address between the use of gaze and Mood structures achieved? (2) How is the reinforcement of address between the use of horizontal angles and personal pronouns achieved? 3. To what extent does the degree of ideational and interpersonal complementarity differ? (1) Is there any genre-related pattern with respect to texts with different degree of ideational and interpersonal complementarity? 7.

(19) CHAPTER TWO LITERATURE REVIEW In this chapter, the first section provides an overview of the SFL theory and the two metafunctions the current study focuses on, the ideational and interpersonal. The second section reviews theories and frameworks of multimodal discourse analysis. The third section reviews the multimodal studies on non-textbook materials and textbooks. The fourth section describes the present situation of junior high school textbooks in Taiwan and summarizes studies on junior high school English textbooks. Systemic Functional Linguistics An Overview SFL is a linguistic school growing out of the research of John Rupert Firth and was mainly developed by Michael Halliday in the 1960s. As the term suggests, it contains two pivotal components: systemic linguistics and functional linguistics. Systemic linguistics refers to Halliday’s view that language is a complicated network of systems and subsystems. Each subsystem represents a set of choices for people to construct meaning under a specific environment (Eggins, 2004). Functional linguistics, on the other hand, accounts for language use in different occasions. Elements in linguistic system are best explained and investigated by reference to their functions in the system (Eggins, 2004; Fries, Gregory & Halliday, 1995). Rather than rules and grammatical forms, in SFL, more emphasis is on functions and the semantics of languages (Halliday& Webster, 2008). Halliday (1994) distinguished three metafunctions, or semantic functions of language: ideational, interpersonal and textual, each of which represents a strand of meaning. Ideational meaning is to represent experience and reality in language; interpersonal meaning is to establish and maintain 8.

(20) interpersonal relationship between individuals; textual meaning is to create and organize texts into logical and coherent semantic units (Halliday & Matthiessen, 2013). One of the major contributions of SFL to linguistics is its detailed examination of modern English with respect to both its structures and meaning of language use in social interaction. How language is structured and used in various contexts are the main research inquiries of this line of linguistists (Eggins, 2004). In SFL theory, language is perceived as part of broader socially-based semiotic systems, systems which imply that semiotic modes are distinctly unique but equivalently potential resources (Bezemer & Kress, 2010). Meanings are not only constituted linguistically but also socially. Semantic complexity permits ideational, interpersonal and textual metafunctions to be encoded together in linguistic units. Linguistic discourse therefore is a kind of social process realized simultaneously through the three metafunctions and constrained by social and cultural contexts (Eggins, 2004; Halliday, 1994). As a functional-semantic approach to language, SFL seeks to explore what language does and how it does it. It has shared ground with sociolinguistics, educational linguistics, cognitive linguistics, language acquisition, anthropology, critical discourse analysis and multiple discourse analysis (Eggins, 2004; Fairclough, 2013; Grumperz, 1971; Halliday, 1978; Kress & Van Leeuwen, 1996; Labov & Waletzky, 2003). SFL provides a theoretical foundation for numerous studies and offers an analytical methodology, which allows comprehensive and systematic analysis of language patterns. In the next two sections, the systems of ideational and interpersonal metafunctions which this study centers on will be explicated.. 9.

(21) Ideational Meaning and Transitivity System According to SFL theory, the three metafunctions work in unison to do simultaneous encoding and structuring of clauses. All these different layers of meaning are necessary to perform linguistic functions in social life (Eggins, 2004). Among the three metafunctions, ideational metafuction speaks to people’s experience of the external and internal world. People are able to depict outside environmental events and express inside feelings with languages. This strand of meaning encompasses two components: the experiential meaning of clause as representation and logical meaning of clause complexes (Eggins, 2004; Halliday, 1985). Due to the research focus, only the experiential component was reviewed here. Experiential meaning is realized through the system of Transitivity. Transitivity system construes the patterns of experience. It organizes the world into manageable sets of units, or process types, with configurations of associated participants and circumstances. Process type is encoded by verbal groups, dealing with what is happening. Participant is presented via nominal groups, referring to people and things included in the process. Circumstance is usually realized via adverbial or prepositional groups, signifying any kind of attendant facts or supplementary situations associated with the process such as time, space, manner, etc. (Halliday & Matthiessen, 2013). In Transitivity system, while participant is obligatory and inherent in the process, circumstance is optional and peripheral. These three elements are arranged in configurations that offer the model for illustrating people’s experience of what is going on and happening (see Figure 1). Underlying different process types, there is a distinct model for constructing a specific experience domain. This system explores the relationship between lexical choices and meanings, aiming to explicate how an action is taken, by whom and on what. Halliday (2013) stated that 10.

(22) “Transitivity system is the lexicogrammatical resources for construing a quantum of change in the flow of events as a figure – as a configuration of elements centered on a process” (p. 213). Process of outer world and process of inner consciousness are differentiated clearly in Transitivity system (Halliday & Matthiessen, 2013).. Figure 1.Central and Peripheral Elements in the Experiential Structure of the Clause (Halliday & Matthiessen, 2013, p.222). Figure 2.The Grammar of Experience: Types of Process in English (Halliday & Matthiessen, 2013, p. 216) 11.

(23) As illustrated in Figure 2, six basic process types construe the Transitivity system. Among them, material, mental and relational processes are main types of process, which cover the majority of English clauses. Behavioral, verbal and existential are subsidiary ones, placed at the boundaries between three main types: behavioral at the boundary between material and mental, verbal at the boundary between mental and relational, and existential at the boundary between relational and material. The six process types are positioned in order to form a circle with different areas symbolizing different process types. Central areas signify the prototypical members of the types while broader areas indicate that these types are actually continuous rather than clear-cut (Eggins, 2004; Halliday & Matthiessen, 2013). The definitions and examples of each process are provided in Table 1. Table 1. Definition and examples of each process Process Type. Definition. Example. Three Main Process Types Material. the process of doing and happening. run, sing, paint, send, bake, clean, write, burn. Mental. Relational. the process of how people feel and think. think, know, understand, feel, love, hate, fear, hear. the process of identifying and characterizing. be, have, become. Three Subsidiary Process Types Behavioral. Verbal Existential. the process of “physiological and psychological behaviors” (Halliday & Matthiessen, 2013, p. 301). laugh, dream, cough, stare, watch, cry, breathe. the process of saying. say, tell, ask, argue, speak. warn. the process conveys that something exists or happens. “there” followed by the verb be or exist, arise, occur, etc.. 12.

(24) Interpersonal Meaning and Mood System While ideational metafunction fulfills the purpose of constructing external and internal experience, interpersonal metafunction comprises the linguistic patterns for establishing social interaction between individuals (Norton & Christie, 1999). In interpersonal metafunction, a clause is perceived as an exchange between addressors and addressees (Halliday & Hasan, 1985). With the use of various grammatical structures and personal pronouns, writers are able to establish different social interaction with readers. Interpersonal meaning is embodied within several systems: Mood, Modality, key, personal pronoun and attitudinal modifier system. Only the Mood system and personal pronouns are explained here because the focus of the current study is on speech function and use of pronouns in textbooks. Mood System The Mood system indicates how clauses of speech and writing are structured into speech functions. According to Halliday (1985), giving and demanding are two main speech roles. During the interaction process, people give or demand two kinds of commodities as exchange: either goods and services or information. Combining speech roles and commodities renders four fundamental speech functions: (1) statements (giving information), (2) questions (demanding information), (3) offers (giving goods and services) and (4) commands (demanding goods and services) (Halliday & Matthiessen, 2013). These four speech functions correspond to a set of proper responses such as accepting the statement; answering the question; taking the offer and performing the command (Halliday 1994). Based on the addressor’s productions and the addressee’s responses, the degree of interaction can be determined (Royce, 1999). With different ordering of constituents, the four speech functions can be 13.

(25) constructed into three Mood structures (see Table 2). Statements are conveyed through declarative clauses. Questions are conveyed through interrogative clauses. Commands are conveyed through imperative clauses. Offers can be conveyed through various ways. The order and employment of different clause structures affect the intensity of social exchange (Halliday & Matthiessen, 2013). In written texts, writers can have various degrees of interaction with readers by forming declarative, interrogative, imperative clauses and sentences (Halliday & Matthiessen, 2004). Table 2. Speech Functions and Mood Structures (adapted from Eggins, 2004, p. 147) Commodity Speech function. Giving. Information. Goods and services. Statement (declarative). Offer (various realization). E.g. The book is written by J.K. Rowling. try?. Question (interrogative) Demanding. E.g. Would you like to give it a. E.g. Have you ever read Harry. Command (imperative) E.g. Don’t run in the museum.. Potter?. Personal Pronoun In SFL, the use of personal pronouns can influence the interpersonal relationship between the writer and the reader (Halliday, 1985). Readers are involved into the texts explicitly with the use of inclusive “we” and “our” and second personal pronoun “you” and “your”. With these personal pronouns, writers acknowledge their readers’ presence, show identification and expect readers’ reaction (Halliday, 1985; Hyland, 2001). In summary, language is a system of meaning potential in SFL. Users of a language make selections from this system to complete different functions in various contexts of situation. In the preceding sections, the Transitivity system of ideational 14.

(26) function, Mood system and personal pronouns of interpersonal function are introduced. Ideational function represents the content, ideas and sequences of events, which affects text comprehension. Interpersonal function represents the social relationship, which influences text explanation and evaluation (Martin & Wodak, 2003). Both functions are pertinent to reading instruction and analysis of junior high school textbooks. In the ensuing section, an overview of multimodal discourse analyses will be proffered and how SFL approach can be applied will be elucidated. Theories and Frameworks of Multimodal Discourse Analysis Multimodality is defined as the multiple ways to co-deploy and co-contextualize diverse semiotic resources in creating meaning of a text (Kress, 2010). As a social semiotic and human communication theory, multimodality conceptualizes communication practices in which the interplay of textual, visual, audio, spatial and gestural modes work conjointly to project meaning for the whole text (Kress, 2010; Yang & Zhang, 2014). Nowadays, due to the advance of multimedia and digital technology, it is well acknowledged that language is not the only modality to get meaning across and communication is inherently multimodal (Hyland, 2011). With increasing interdependency of words and images for meaning-making, there is an urgent need to understand how these diversified modalities separately and concurrently produce different dimensions of meaning (Unsworth, 2001). From 1950s onwards, there is a burgeoning body of multimodal research investigating the incorporation of language and other semiotic resources (e.g. Barthes, 1964, 1977; Bateman, 2008; Iedema, 2001a, 2001b, 2003; Kress and Van Leeuwen, 1996; Lemke 1998a, 1998b, 2002; Martin, 2002; Martinec, 2005; O’Halloran, 1999, 2005, 2011; O’Toole, 1994; Royce, 1998; Unsworth, 2001; Ventola et al., 2004). Barthes’s (1964) 15.

(27) pioneering essay, The Rhetoric of the Image was considered the first step toward multimodal discourse analysis. He defined three possible image-text relations: (1) anchorage, text supporting image, (2) illustration, image supporting text, and (3) relay, text and image being equal. He contended that “It is not the image which comes to elucidate or realize the text, but the latter which comes to sublimate, patheticize or rationalize the image” (p. 25). The meaning of images is largely contingent on how the verbal (written) text accompanies it. Later on, many researchers have made attempt to develop systemized theoretical frameworks to analyze verbal-visual intersemiotic relations created under specific contexts (e.g. Forceville, 1996; Kress & Van Leeuwen, 1990, 1996; O’Toole, 1994). One line of researchers has utilized linguistic theory to explain non-linguistic modes, including the Prague School, the Paris School and Social Semiotics. Among them, Social Semiotics, which adopts Halliday’s SFL to examine the visuals has been rather productive (Royce, 1999). More detailed explanation will be offered in the next subsection. SFL Approach to Visual Analysis In Halliday’s (1978) discussion on language and meaning, he claimed that “There are many other modes of meaning, in any culture, which are outside the realm of language” (p. 4). SFL, which highlights the transmissions and exchanges of meaning in a socio-cultural context, lends theoretical support to subsequent social semiotic models (Fei, 2007; Halliday, 1978; Lemke, 2002; O’Halloran, 1999; O’Toole, 1994; Unsworth & Wheeler, 2002). Some scholars have maintained that apart from languages, Halliday’s social semiotic theory can be expanded and applied to studies of other modes (Jewitt & Oyama, 2001; Kress & Van Leeuwen, 1996; Martinec, 1998, 2000, 2001; O’Toole, 1994). Among them, O’Toole (1994) and Kress and Van 16.

(28) Leeuwen (1996) have laid the foundations for later multimodal research (O’Halloran, 2011). O’Toole (1995) mentioned that SFL offers “a powerful and flexible model for the study of other semiotic codes" (p.159). Taking a social-functional approach, O’Toole adopted a rank-scale method to define the units of architecture and derived a framework to describe the spatial arrangement of visual arts and displayed arts (Unsworth, 2011). Three basic functions, representational, modal and compositional were proposed to analyze and evaluate art works (O’Toole 1994, 1995). Another influential endeavor to characterize images in a systematic manner is Kress and Van Leeuwen’s (1990, 1996) social semiotic approach to images. Also following Halliday’s (1985) metafunctional perspective of language, Kress and Van Leeuwen's (1990, 1996) visual grammar reported three corresponding functions for interpreting visual designs. First, representational function explores the represented participants, activities, attributes or the characteristics of the participants and circumstances involved. Second, interactional function delineates how the image-producer utilizes visual designs and techniques to interact with the viewers and what kind of visual address, level of involvement, social distance and power relation are built between what is viewed and the viewers. Third, compositional meaning deals with the information distribution along with issues pertinent to page layout. Three principles: information value, salience and framing were proposed to determine the degree of coherence between visual and verbal elements (Kress & Van Leeuwen, 1996; Unsworth & Wheeler, 2002). Royce’s Intersemiotic Complementarity Inspired by Halliday’s (1985) metafunctional approach to verbal communication and Kress and Van Leeuwen’s (1996) visual grammar, Royce (1998) 17.

(29) noted that the three metafunctions embodied in linguistic structures are similarly evident and useful in visual structures. He originated the concept of Intersemiotic Complementarity to describe “where visual and verbal modes semantically complement each other to produce a single textual phenomenon” (p. 26). These two modes do not simply co-occur on the page, forming a conjunction relationship; rather, the relationship is “synergistic” in nature. “Meaning created by a synergistic combination between two codes is “greater than the sum of individual elements or contributions” (p. 27). He presented a descriptive framework to measure intersemiotic complementarity by recognizing some metafunctionally-based principles under which visual and verbal resources intersemiotically operate three kinds of meaning. For intersemiotic ideational meaning, he extended Halliday (2004) and Halliday and Hasan’s (1985) lexical cohesion to categorize six semantic sense relations, including repetition, synonymy, antonymy, hyponymy, meronymy and collocation across visual and verbal modes. For intersemiotic interpersonal meaning, the intersemiotic relationship between the visuals and the viewers is examined with respect to reinforcement of address in Mood system and attitudinal congruence and attitudinal dissonance in Modality system. For intersemiotic textual meaning, the focus is on the layout, which permits features on the page to cohere as part of the whole multimodal text and to deliver a sense of consistency to viewers. The use of information value, visual salience, balancing centers, vectors, visual framing and reading paths are some principles of composition (O'Halloran, 2008; Royce, 1998). Royce’s framework has been widely applied to fields such as writing, graphology, electronic publication of newspapers, computational concordancing, cross-cultural and cross-linguistic issues (Bateman, Delin & Henschel, 2007; Bowcher, 2007, 2012; Bowcher & Liang, 2013; Mrowa-Hopkins, 201; Ramakrishnan, 18.

(30) 2012). Royce (2007) himself also stretched this framework to accommodate applications of language education. He emphasized the need to broaden traditional, linguistic-exclusive communicative competence to multimodal communicative competence, which enables learners to take advantages of an array of semiotic resources available in the course of meaning-making process (Mrowa-Hopkins, 2011). After introducing the major theories and theoretical frameworks in multimodal discourse analysis, below the empirical studies on non-textbook materials and textbooks are reviewed. Studies on Multimodal Discourse Analysis The last few decades have witnessed the growing popularity in using multimodal means to generate meaning (Fairclough, 2000). Books, magazines, websites, textbooks, just to name a few, usually employ two or more different semiotic modes to distribute information. Hence, multimodal discourse analyses of various kinds of materials have been gaining ground in recent research (Duncum, 2004; Serafini, 2012). Multimodal Studies on Non-textbook Materials Picture books have been the focus of several multimodal discourse analyses. For example, based on Kress and Van Leeuwen’s (1996) visual grammar and Halliday’s (1985) SFL, Guijarro and Sanz (2008) made effort to evaluate the degree of coherence created by visual and verbal resources in the picture book, Guess how much I love you. The textual analysis showed a clear thematic progression in accordance with the visual elements. Gazes, especially long shots, were the predominant visual techniques utilized to indicate objectivity and distance. On the other hand, to increase interaction with readers, the images in the picture books were placed frontally to form a parallel angle, suggesting involvement and equality. The 19.

(31) frequent uses of imperative Mood structure also augmented the interactive nature. By composing the story with two different semiotic resources, the plot became more accessible and comprehensible for children. Guijarro’s (2010) another study attempted to analyze the interpersonal meanings expressed by the verbal and the visual modes in the picture book, The Tale of Peter Rabbit. Halliday’s (1985) SFL and Kress and Leeuwen’s (1996) Visual Social Semiotics were adopted as the analytical tools in the study. The results showed that the frequent use of declarative clauses in the text meant low degree of interaction with the readers, but the eye contacts, distance, angles created by pictures did establish some degree of affinity. Readers’ identification with the main characters was created by the synchronization of verbal and the visual modes. Much the same as picture books, magazines are another type of material in which pictures receive much coverage on the page layout. Leonzini (2012) illustrated how the visual mode served as a device to introduce and remark on the verbal mode and then reached intersemiotic complementarity in The Economist magazine. Intersemiotic complementarity was proved to be achieved in the ideational, interpersonal, and textual metafunctions. The degree of interactivity among visual and verbal segments was also substantiated. In a similar vein, Yang and Zhang (2014) concentrated on the ideational aspect of English editorials in The Economist. The analysis manifested that editorial discourses assigned nearly equal weight to visual and verbal modes and that pictures took an essential part in consolidating ideas and attitudes that the written texts intended to explain, judge and comment on. The juxtaposition of two semiotic modes in editorials might help readers to comprehend the content more effectively. Some informational texts have been exceedingly multimodal as well. 20.

(32) Francesconi (2011) investigated the way visual and verbal devices promoted important values in tourist brochures, like heritage, hospitality and diversity. The analysis showed that both devices had a role to play in communicating the core ideas and values, first through visuals and then elaborated, clarified and substantiated by the accompanying verbal texts. Evidence of visual-verbal partnership can be also found in Bowcher and Liang’s (2013) research, which probed into how tourist site entry tickets and recipes make use of multimodal resources to convey messages. Adopting Royce’s intersemiotic complementarity framework of ideational meaning, their analysis of entry tickets revealed that internal intersemiotic complementarity was created semantically via sequences of sense relations between visual and verbal components on both front and back of the tickets. The predominantly used sense relation was repetition for represented participants, collocation, synonymy or antonymy for processes and attributes. Through visual-textual complementarity, tickets were not only related to target tourist site, but potential links to wider social network. Bowcher and Liang’s (2013) another study was concerned with culinary materials in Japanese and Chinese. Drawing on Halliday and Matthiessen’s (1999) semantic domain framework and Martinec and Salway’s (2005) system of image-text relations, they examined recipes on Chinese dumplings in both Chinese and Japanese versions. The data analysis showed that while there were obvious linguistic similarities between two versions, socio-cultural and contextual differences arose when looking at the combined effect entailed by multimodal semiotic resources. The distinctive roles played by languages and images could clearly be seen. In addition to paper-based materials, Knox (2007) scrutinized the visual, verbal and visual–verbal communication of three online English newspapers homepages from different countries. He observed the emergence of a new news genre in response 21.

(33) to demands of presenting news on the computer screens. Consistency between visual and verbal designs was prominent in electronic kind of news-specific visual grammar. Findings from past studies on non-textbook materials all pointed to the fact that visuals are not redundant with the textual mode in meaning-making. It has been a cross-genre and cross-medium phenomenon that information transfer is an intricate activity which involves multiple modes. In the ensuing section, multimodal studies on textbook are reviewed. Multimodal Studies on Textbooks Like many other materials, contemporary textbooks have seen the coexistence of visual and verbal communication (Royce, 2002). Multimodality has become the norm and changed the landscape of teaching and learning (Kress, 2000). The radical shift of textbook design has come from one where language was the dominant vehicle for all of the essential information to the current state where language is no longer dominant (Kress 2000). This situation has prompted researchers to conduct analyses of multimodal textbook discourses in different content areas. Lemke (1998a, 1998b, 2000a, 2000b, 2002b) investigated the coactions of verbal-visual representations in science textbooks in the U.S. and found advantages of images and graphs over language as sole carrier of all meanings in scientific texts. Images and written texts contributed to different aspects of meaning. Lemke’s (1998a) argued that, “It is the nature of scientific concepts that they are semiotically multimodal in this sense, and this may well be true in other systems of semiotic practices as well” (p.111). Images accompanied and helped to make the texts more concrete and understandable for readers. Royce (2002) demonstrated the procedures of using his intersemiotic complementarily framework to analyze intersemiotic ideational meaning in an 22.

(34) environmental science textbook, Science and Life: Work, Leisure, Technology and the Environment. An extract dealing with the topic of water cycle was analyzed. The first step was to analyze represented participants, processes, circumstances and attributes in the images. During the process, a set of descriptive glosses, termed the images’ visual message elements (VMEs) would be identified. The following step was to investigate how VEMs cohere systematically to the lexical choices in the textual part based on six sense relations: repetition, synonymy, antonymy, hyponymy, meronymy and collocation. It was found that occurrences of meronyms and repetitions were most frequent and nearly every sentence in the written texts comprised lexical choices that were linked to the water cycle VEMs. The finding reinforced that images were not put on the page randomly, but were there to work in tandem with verbal part to generate coherent multimodal texts in a specific context. Guo (2004)’s multimodal analysis of biology textbook, Essential Cell Biology, similarly revealed the specialty of visuals and language. Visuals supported and extended language in meaning-making and language “anchored and constrained the many possible meanings made in the visuals” (p.214). They were interdependent and co-contextualized mutually. Guo (2004) also observed that each type of visual image (e.g. schematic drawing to show procedures, statistical graphs to show numerical data) displayed distinct conventions of meaning-making, not only in the combinations of technique (e.g. dots, lines, etc.), but also in the relations with the verbal texts. The infusion of visual and verbal modes in second language learning textbooks has also been an active area of research (Baldry, 2000). For example, Giaschi (2000) carried out an analysis in English as second/foreign language (ESL/EFL) textbooks used in the U.K. to explore the affordance of visual and textual modes. It was found that a number of messages that were relayed through the images across the textbooks 23.

(35) did not seem to be relayed through the written text. For example, power, gender and social issues were embedded in gestures, eye contacts and activities illustrated in images, but not in texts. ESL/EFL teachers, therefore, should note the distinct affordance permitted by different semiotic vehicles. Tan (2012) also undertook an analysis of senior high school English textbooks, Advance with English in China. The results demonstrated that the co-existence of visuals and texts did facilitate students’ language learning. Nonetheless, the way visuals repeated the text content seems more intuitive; no principles could be found. Chen (2009, 2010) delved into the creation of interpersonal meaning in multimodal EFL textbooks. The 2009 study described how verbal and visual semiotic resources were coordinated to construct interpersonal meaning in 18 EFL primary and secondary school textbooks in China. The findings revealed that together with verbal texts, visuals featured to realize attitudinal and emotional meanings. For example, there were passages where the texts were statements guiding readers to do the practice exercise, committing no attitudinal meaning. It was via the visuals that express a mood of happiness and excitement by showing a smile on the faces of represented participants. Chen concluded that while the texts explained to readers how they should perform, the visuals implied the attitude the students should assume. Linguistic features together with visual properties should be considered when designing textbooks. Chen’s 2010 article discussed how multimodal resources were arranged to enable dialogic engagement with readers in 17 EFL textbooks for primary and secondary schools in China. It was found that one kind of engagement meaning may be achieved by sets of different multimodal tools and one kind of multimodal tool may render different engagement meanings. The calibration of engagement value was in close association with devices like dialogue balloons, texts, and illustrations. 24.

(36) Some researchers went beyond the scope of single set of textbooks to carry out comparative analyses. Bezemer and Kress (2010) offered a multimodal report of historical changes in English textbooks in England by comparing 1930s, 1980s and 2000s versions. The results showed that the use of multimodal modes has changed between 1930s and 2000s. Every mode functioning in the textbooks, including writing, image, layout and typography afforded meaning and potential for learning. Pedagogical relation was also altered in ways that authors, designers and learners equally share the responsibility for coherence. For present-day learners, thorough textual understanding necessitates fluency in comprehending not only writing, image, layout and typography mutually but also the overall learning environments. Knowledge construction and production has become a more participatory and interactive process. More recently, a comparative multimodality study on two series of EFL textbooks for college students, Experiencing English (EE) and New Century College English (NCCE), was carried out by Liu and Qu (2014). According to the findings, the two series of textbooks were visually-verbally coherent with respect to Royce’s intersemiotic complementarity and synonymy was the most often utilized sense relation. Differences between EE and NCCE resided in pictures and margin styles. In EE, pictures, most of which were photos, were organized to assist learners in anchoring new knowledge to prior learning experience and demonstrated stronger interpersonal intersemiotic complementarity. NCCE, designed for more advanced learners, the margin style was intended to help learners develop autonomous learning in that more space was left for students to use. Liu and Qu (2014) advised that textbook designers be cognizant of target learners’ English proficiency and appropriately employ various multimodal resources to achieve optimal intersemiotic 25.

(37) complementarity. In sum, these studies have addressed multimodal texts in diverse types of materials and genres in different contexts. Regarding the three metafunctional meaning, it is concurred that visual and verbal aspects are equally essential in making sense of texts. In order to deliver comprehensive meaning, the collective potentials of various modes need to cohere and complement each other. Also, it is now almost impossible for readers to understand language and its uses without referring to other modes of communication that are co-present in the text. In language learning context, learners have been exposed to a multitude of semiotic modes in learning materials and textbooks; therefore, a new set of reading strategies is needed to foster learners’ reading ability. Since different modes provide different affordances, the arrangement of these resources and their degree of coherence are of great importance to assist readers in accessing messages more easily. However, there is a dearth of research investigating the degree of how different metafunctional complementarity is achieved through textual and visual modes. The present study thus sets out to compare and contrast to what extent ideational meaning and interpersonal meaning reach complementarity in Taiwanese Junior high school English textbooks. The subsequent section will briefly describe the current state of junior high school English textbooks in Taiwan and review relevant literature. Junior High School English Textbooks in Taiwan In 1996, to correspond to the global trend and taking teaching expertise and teacher autonomy on board, the Ministry of Education (MOE) in Taiwan opened the market for private publishers to design elementary, junior high and senior high school English textbooks. From then on, the National Institute for Compilation and Translation was no longer the only authorized textbook publisher, but changed its role 26.

(38) to censor, regulating the qualification of textbooks. Major publishers of junior high school English textbooks are Kang-hsuan, Joy, Longman, Nan-I, and Hess (Yen, 2011). Despite the fact that these non-government publishers are devoted to textbook development based on MOE’s curriculum guidelines, that is, One Guide-Multiple Text policy. Researchers (e.g. He, 2004) have observed diversity among different editions of textbooks concerning the presentation of texts and images. Teachers have the freedom to select suitable textbooks for their students (MOE, 1998). After the privatization of English textbooks, textbook relevant issues have been an area of expanding interest (MOE, 1998; Peng, 2008). Earlier studies can be categorized into four groups: content analysis, textbook evaluation, textbook selection, and textbook compilation (Wang, 2005). Content analysis studies have often centered on particular learning components contained in textbooks: (1) Vocabulary and phrases, including vocabulary sizes, distributions, frequencies, recycling frequencies, overlapping, collocations and phrasal verb usages (e.g. Han, 2008; He, 2004; Huang, 2012; Lin, 2008; Wang, 2013), (2) Grammar, including instruction, grammatical structure recycling, practice activities (Chang, 2006; Chu, 2008; Hung, 2008), (3) Learning activity, including activity design, authenticity, comparison between textbooks (Hsu, 2003; Wu, 2008), (4) Pronunciation (Liao, 2004), and (5) Comics (Wang, 2014). Some have attended to the embedded issues and phenomena: (1) Gender roles (Tang, 2013; Wang, 2015), (2) Global education (Yu, 2014), (3) Important issues (Liu, 2014), (4) Cultural awareness (Chiang, 2013), and (5) Class ideology (Huang, 2010). Others have paid special attention to reading section: (1) Readability (Hsieh, 2015; Hsueh, 2012; Lin, 2007; Yen, 2011) and (2) Genre (Peng, 2008). While there has been an abundance of studies on content analysis of junior high 27.

(39) school textbooks, most of them focused on the texts only rather than other semiotic codes co-existing in textbooks. The present study therefore seeks to investigate how textual and visual modes functions together in textbooks. It is hoped that a more comprehensive understanding of textual-visual relation, an alternative multimodal approach to language instruction and suggestions for future textbook compilation can be offered.. 28.

(40) CHAPTER THREE METHOD The major goal of this present study is to investigate how ideational and interpersonal meaning is constructed respectively through the texts and images and conjointly through both modes in the 9th grade lessons of two editions of Taiwanese junior high school English textbooks, Joy and Kang-hsuan. This chapter details the overall research design including the materials, theoretical frameworks and procedures of analysis. To illustrate the process, an example is provided at the end of the chapter. Materials The texts under analysis in this study are from Book 5 of two major editions of junior high school English textbooks, Joy and Kang-hsuan. These two editions have better market share than other editions of textbooks in Taiwan (Han, 2008). Book 5 is analyzed since it features longer reading passages, which have higher syntactic and genre variation. Because the Comprehensive Assessment Program for junior high school students is held in May, Book 6 only contains six lessons. Book 5, which consists of nine lessons, was used. A total of 18 reading passages were collected. Table 3 shows an overview of the reading passages. Table 3. Lessons in Joy and Kang-hsuan Editions Joy Edition No.. Kang-hsuan Edition. Lesson 1 (L1). Title Have You Decided on the Gift? Fortune Cookies. Lesson 2 (L2). Origami Is Amazing, Isn't It?. Blue Cheese. Lesson 3 (L3). People Get Excited About Halloween The Butterflies Were Marked. World Ocean Day. Lesson 4 (L4). 29. Out with the Old, In with the New.

(41) Lesson 5 (L5). Can You Tell Me What to Do?. What Do People Do to Get Good Luck?. Lesson 6 (L6). He Asked Us If We Wanted to Visit Him Studying Is Important, and So Is Taking Up a Hobby She Is the Woman Who Wants the Public to Pay Attention to the Working-class People A Girl I Met Online Asked Me Out. A Letter from Emily. Lesson 7 (L7) Lesson 8 (L8). Lesson 9 (L9). An Open Letter to Nick Vujicic Stockholm- A “Green” City. Do You Ever Feel Like a “Volcano”?. Theoretical Frameworks The theoretical foundations of this present study were Halliday’s (1985, 1994) SFL, Kress and Van Leeuwen’s (1996) Visual Grammar and Royce’s (1999) intersemiotic complementarity. SFL conceptualizes texts as a system of meaning potential, which simultaneously encodes three metafunctional meaning in situational context: ideational, interpersonal and textual. In this study, SFL was utilized as a theoretical paradigm to analyze the ideational and interpersonal aspect of the texts. With the metafunctional view of communication, SFL provides an analytical lens to analyze non-linguistic semiotic systems (Kress & Van Leeuwen, 1996; Royce, 1999; Van Leeuwen, 2006). The commonality is that all systems deliver more than one kind of meaning in instances of communication (Kress & Van Leeuwen, 1996). Kress and Van Leeuwen (1996) expanded SFL’s three metafunctions to explain correspondent visual meanings, referring to them respectively as representational, interactional and compositional. This study adopted this framework to analyze ideational and interpersonal aspects of visual images. To reduce terminology redundancy, the metafunctional terms of the SFL model- ideational and interpersonal were used in this study instead of representational and interactional. 30.

(42) To evaluate the degree of coherence and regulate the connection between textual and visual modes, Royce (1999) brought forth the concept, “intersemiotic complementarity” to describe visual-verbal relationship. The concept elucidates how various semiotics operate along the metafunctional dimensions ideationally, interpersonally and textually to constitute a coherent text (Unsworth, 2011). Based on the principles of SFL and Visual Grammar, Royce’s analytical framework starts with visual analysis, text analysis, and then followed by analysis of complementarity between the two modes. The major reason why visual analysis precedes textual analysis is because of ease and efficiency of analysis. In contemporary education and communication system, textual component is considered to be the main carrier of meaning in page-based multimodal texts (Barthes 1977, Kress & Van Leeuwen, 1996). Textual component therefore imparts more profound, complicated and larger numbers of meanings, which may not be conveyed by the visuals. According to Royce (1999), “These meanings would in a sense be ‘superfluous’ to the analysis” given that they are not semantically related to the visuals in a systematic and clear manner (p.137). In this study, Royce’s (1999) framework was used to guide the analysis procedures and further relate the analysis of the visuals and the verbal texts. The detailed description of the visual, textual and intersemiotic analysis is outlined in the following sections. Visual Analysis Kress and Van Leeuwen’s (1990, 1996) visual analysis of ideational meaning commences with analyzing represented participants and processes. Represented participants are entities, who or what in the visuals. Processes are actions taking place. To be able to investigate intersemiotic ideational complementarity, during the process of analysis, lexical items that refer to the visuals are identified, which Royce (1999) 31.

(43) termed Visual Message Elements (VMEs). These VMEs are not isolated but greatly context-bound, bearing semantic properties that closely relate to each other and reasonably occur in a multimodal text. The VMEs will be the reference points for the textual analysis (Royce, 1999). For interpersonal meaning, Kress and Van Leeuwen (1996) proposed that interpersonal meaning in images could be identified through systems of address, involvement, social distance and power relations. Since the current study centers on interpersonal intersemiotic complementarity between the visuals and texts, the focus is on the system of address and involvement. The former corresponds to speech functions and the latter aligns with the use of personal pronoun. The system of address describes images through gaze of represented participants. Usually, the represented participants who gaze at the viewer are human or animals, but not all the time. For example, Figure 3 shows a creature on the ATM screen. Although it is depicted like a box-shaped machine, it looks directly at the viewer, demanding interaction with its viewers (Kress & Van Leeuwen, 1996). Kress and Van Leeuwen (1996) pointed out that “whether they are human or not, by being represented as looking at the viewer, they are represented as human, anthropomorphized to some degree” (p. 118).. Figure 3.The ATM Screen (Kress & Van Leeuwen, 1996, p.119) 32.

(44) Reinterpreting Halliday’s (1984) four speech functions in the textual mode into image acts in the visual mode, Kress and Van Leeuwen noted that visuals usually perform two kinds of interactive image acts: demand and offer. The configuration of a “demand” image is established through direct gaze, which is an indicator of explicitly acknowledging and involving the viewers. The absence of gaze, on the other hand, forms an “offer” image, denoting no requirement of viewer reaction or action to the presented information (Landert, 2014; Kress & Van Leeuwen, 1996, p.122; Van Leeuwen, 2006). For example, in Figure 4, the two pictures on the upper side illustrate three participants; all of them are looking directly at the viewer, showing a demand image act. The two pictures on the lower side, on the contrary, depict two participants; both of them are not looking at the viewer, suggesting an offer image act.. Figure 4. A Screenshot from Book 4 Lesson 4 of Kang-hsuan Edition The system of involvement interprets images through horizontal angle. Different degree of angle determines the level of involvement felt by the viewers (Kress & Van Leeuwen, 1996). The horizontal angle is categorized into frontal and oblique, representing a continuum from involvement to detachment. According to Kress and 33.