The major goal of this present study is to investigate how ideational and interpersonal meaning is constructed respectively through the texts and images and conjointly through both modes in the 9th grade lessons of two editions of Taiwanese junior high school English textbooks, Joy and Kang-hsuan. This chapter details the overall research design including the materials, theoretical frameworks and
procedures of analysis. To illustrate the process, an example is provided at the end of the chapter.
Materials
The texts under analysis in this study are from Book 5 of two major editions of junior high school English textbooks, Joy and Kang-hsuan. These two editions have better market share than other editions of textbooks in Taiwan (Han, 2008). Book 5 is analyzed since it features longer reading passages, which have higher syntactic and genre variation. Because the Comprehensive Assessment Program for junior high school students is held in May, Book 6 only contains six lessons. Book 5, which consists of nine lessons, was used. A total of 18 reading passages were collected.
Table 3 shows an overview of the reading passages.
Table 3. Lessons in Joy and Kang-hsuan Editions
Joy Edition Kang-hsuan Edition
No. Title
Lesson 1 (L1) Have You Decided on the Gift? Fortune Cookies Lesson 2 (L2) Origami Is Amazing, Isn't It? Blue Cheese Lesson 3 (L3) People Get Excited About
Halloween
World Ocean Day
Lesson 4 (L4) The Butterflies Were Marked Out with the Old, In with the New
30
Lesson 5 (L5) Can You Tell Me What to Do? What Do People Do to Get Good Luck?
Lesson 6 (L6) He Asked Us If We Wanted to Visit Him
A Letter from Emily Lesson 7 (L7) Studying Is Important, and So Is
Taking Up a Hobby
An Open Letter to Nick Vujicic Lesson 8 (L8) She Is the Woman Who Wants
the Public to Pay Attention to the Working-class People
Stockholm- A “Green” City
Lesson 9 (L9) A Girl I Met Online Asked Me Out
Do You Ever Feel Like a “Volcano”?
Theoretical Frameworks
The theoretical foundations of this present study were Halliday’s (1985, 1994) SFL, Kress and Van Leeuwen’s (1996) Visual Grammar and Royce’s (1999) intersemiotic complementarity. SFL conceptualizes texts as a system of meaning potential, which simultaneously encodes three metafunctional meaning in situational context: ideational, interpersonal and textual. In this study, SFL was utilized as a theoretical paradigm to analyze the ideational and interpersonal aspect of the texts.
With the metafunctional view of communication, SFL provides an analytical lens to analyze non-linguistic semiotic systems (Kress & Van Leeuwen, 1996; Royce, 1999; Van Leeuwen, 2006). The commonality is that all systems deliver more than one kind of meaning in instances of communication (Kress & Van Leeuwen, 1996).
Kress and Van Leeuwen (1996) expanded SFL’s three metafunctions to explain correspondent visual meanings, referring to them respectively as representational, interactional and compositional. This study adopted this framework to analyze ideational and interpersonal aspects of visual images. To reduce terminology
redundancy, the metafunctional terms of the SFL model- ideational and interpersonal were used in this study instead of representational and interactional.
31
To evaluate the degree of coherence and regulate the connection between textual and visual modes, Royce (1999) brought forth the concept, “intersemiotic
complementarity” to describe visual-verbal relationship. The concept elucidates how various semiotics operate along the metafunctional dimensions ideationally,
interpersonally and textually to constitute a coherent text (Unsworth, 2011). Based on the principles of SFL and Visual Grammar, Royce’s analytical framework starts with visual analysis, text analysis, and then followed by analysis of complementarity between the two modes. The major reason why visual analysis precedes textual analysis is because of ease and efficiency of analysis. In contemporary education and communication system, textual component is considered to be the main carrier of meaning in page-based multimodal texts (Barthes 1977, Kress & Van Leeuwen, 1996).
Textual component therefore imparts more profound, complicated and larger numbers of meanings, which may not be conveyed by the visuals. According to Royce (1999),
“These meanings would in a sense be ‘superfluous’ to the analysis” given that they are not semantically related to the visuals in a systematic and clear manner (p.137). In this study, Royce’s (1999) framework was used to guide the analysis procedures and further relate the analysis of the visuals and the verbal texts.
The detailed description of the visual, textual and intersemiotic analysis is outlined in the following sections.
Visual Analysis
Kress and Van Leeuwen’s (1990, 1996) visual analysis of ideational meaning commences with analyzing represented participants and processes. Represented
participants are entities, who or what in the visuals. Processes are actions taking place.
To be able to investigate intersemiotic ideational complementarity, during the process of analysis, lexical items that refer to the visuals are identified, which Royce (1999)
32
termed Visual Message Elements (VMEs). These VMEs are not isolated but greatly context-bound, bearing semantic properties that closely relate to each other and reasonably occur in a multimodal text. The VMEs will be the reference points for the textual analysis (Royce, 1999).
For interpersonal meaning, Kress and Van Leeuwen (1996) proposed that interpersonal meaning in images could be identified through systems of address, involvement, social distance and power relations. Since the current study centers on interpersonal intersemiotic complementarity between the visuals and texts, the focus is on the system of address and involvement. The former corresponds to speech functions and the latter aligns with the use of personal pronoun.
The system of address describes images through gaze of represented participants.
Usually, the represented participants who gaze at the viewer are human or animals, but not all the time. For example, Figure 3 shows a creature on the ATM screen.
Although it is depicted like a box-shaped machine, it looks directly at the viewer, demanding interaction with its viewers (Kress & Van Leeuwen, 1996). Kress and Van Leeuwen (1996) pointed out that “whether they are human or not, by being
represented as looking at the viewer, they are represented as human, anthropomorphized to some degree” (p. 118).
Figure 3.The ATM Screen (Kress & Van Leeuwen, 1996, p.119)
33
Reinterpreting Halliday’s (1984) four speech functions in the textual mode into image acts in the visual mode, Kress and Van Leeuwen noted that visuals usually perform two kinds of interactive image acts: demand and offer. The configuration of a
“demand” image is established through direct gaze, which is an indicator of explicitly acknowledging and involving the viewers. The absence of gaze, on the other hand, forms an “offer” image, denoting no requirement of viewer reaction or action to the presented information (Landert, 2014; Kress & Van Leeuwen, 1996, p.122; Van Leeuwen, 2006). For example, in Figure 4, the two pictures on the upper side illustrate three participants; all of them are looking directly at the viewer, showing a demand image act. The two pictures on the lower side, on the contrary, depict two participants; both of them are not looking at the viewer, suggesting an offer image act.
Figure 4. A Screenshot from Book 4 Lesson 4 of Kang-hsuan Edition
The system of involvement interprets images through horizontal angle. Different degree of angle determines the level of involvement felt by the viewers (Kress & Van Leeuwen, 1996). The horizontal angle is categorized into frontal and oblique,
representing a continuum from involvement to detachment. According to Kress and
34
Van Leeuwen (1996), a frontal angle means “what you see here is part of our world, something we are involved with” whereas an oblique angle denotes “what you see here is not part of our world, it is their world, something we are not involved with”
(p. 143). A frontal angle is formed when the frontal plane of the viewer parallels with the frontal plane of the represented participants in the image. In other words, the represented participants face the viewer “front on”, which maximizes the sense of involvement of the viewer. If the represented participants are illustrated at an oblique angle, that is, when the frontal plane of the viewer and that of the represented
participants are not parallel, the viewer is positioned more detached from the visuals.
For instance, Figure 5 displays a picture of five represented participants. The three holding a red menu form a parallel angle with the viewer in that they face the viewer front on, indicating reader involvement. The frontal planes of the waiter and the girl sitting on the right side form an oblique angle with the viewer, indicating no reader involvement.
Figure 5. A Picture from Book 4 Lesson 7 of Joy Edition
Textual Analysis
The textual analysis was informed by SFL (Halliday, 1985). Ideational meaning is the function of language to display “how phenomena of the real world are
represented in linguistic structures” (Halliday, 1985, p.102). Since the analysis in this
35
study began with the visuals and the primary focus was on spotting Transitivity features that were connected semantically to the VMEs, a thorough clause by clause Transitivity analysis was not carried out. Instead, every sentence was analyzed by identifying lexical items of participants and processes that can be closely related to already-identified VMEs (Royce, 1999).
As for interpersonal meaning, two aspects were investigated in the present study.
One was speech functions and the other was personal pronouns. The degree of
interaction and types of reader address are contingent on the four speech functions: (1) statements, (2) questions, (3) offers and (4) commands. They correspond to three Mood structures: (1) statement by means of declarative, (2) question by means of interrogative, (3) offer by means of various realizations and (4) command by means of imperative (Halliday & Matthiessen, 2004). In this study, the Mood structure was analyzed to decide whether the reader is addressed. If the sentence is declarative, it means no reader address, only providing information. If the sentence is interrogative or imperative, it should be further analyzed to determine whether the question or command is addressed to the reader.
Another aspect to examine was the use of personal pronouns. Personal pronouns can influence the writer-reader relationship. Through the choices of
personal pronouns, writers can express their attitude toward the readers and establish a sense of involvement with the readers (Halliday, 1985). There is a very strong social exchange between the writer and the reader through the use of the first person plural pronouns, “we”, “us” and “our” or second person pronouns, “you” and “your” (Carter, 2001; Royce, 1999; Stephan & Patzold, 2004; Walsh, 2006). In this study, the degree to which readers are being involved was ascertained by the use of first or second person pronouns. The degree of involvement was not further differentiated between
36
first and second person pronouns; a more straightforward and clear-cut method was used.
Intersemiotic Complementarity: Combining the Visuals and Texts
After the analysis of the visuals and texts, the next stage was to examine how the intersemiotic complementarity is realized to create a coherent multimodal text. For intersemiotic ideational complementarity, Royce (1999) extended the concept of sense relations introduced by Halliday (1994) and Halliday and Hasan (1976, 1985) to conceptualize how meaning is distributed across visual and textual mode in
multimodal texts. He outlined six types of intersemiotic sense relations: repetition, synonymy, antonymy, hyponymy, meronymy and collocation. Once the VMEs are identified from analyzing the images, how they complement in relation to the textual part can be described by these six intersemiotic sense relations. The definition of each intersemiotic sense relation is provided in Table 4.
Table 4. The Definition of Six Intersemiotic Sense Relations (adapted from Royce, 2002, p. 194)
Intersemiotic Sense Relations Definition
Repetition An image of a teacher can generate the VME,
“teacher”, which may be intersemiotically repeated by the lexical item “teacher” in the textual aspect.
Synonymy Similarity relation: An image of a teacher can generate the VME, “teacher”, which may be intersemiotically synonymized by the lexical item “instructor” in the verbal aspect.
Antonymy Opposition relation: A graph showing increased nuhmber of world population, generating the VME, “grows over the years”, may be intersemiotically related through antonymy to the lexical item “decline”.
37
Hyponymy Class-subclass relation: An image showing different types of animals in the zoo, generating the VME, “animals”, may be intersemiotically related through hyponymy to the lexical item “panda” (a type or sub-class of animals).
Meronymy Part-whole relation: A sketch showing a classroom, generating the VME, “classroom”, may be intersemiotically related through meronymy to the lexical item “blackboard”
(referring to specific parts of the classroom).
Collocation Expectancy relation: An image of a weather map, generating the VME “weather forecast”, may collocate with the lexical item “rainfall”
(words that commonly co-occur in this topic area).
While intersemiotic ideational complementarity tackles how the visual and textual modes together distribute ideas and content, intersemiotic interpersonal complementarity is concerned with whether these two modes coherently build the interactive relations with the readers. Royce (1999) proposed that interpersonal intersemiotic complementarity can be achieved semantically via the relation of reinforcement of address. Here, “address” is not restricted to the system of address, but means visual and textual modes complement each other to interact with the reader in the same way. As summarized in Table 5, in this study, reinforcement of address is realized in two ways: (1) between the use of gaze and Mood structures (GM) and (2) between the use of horizontal angles and personal pronouns (AP).
38
Table 5. Interpersonal Functions in the Visual and Textual Modes
Visual Mode Textual Mode
Reader Function Two Image acts
Demand/ Offer Function Involvement
Involvement / No Involvement
Involvement
Involvement / No Involvement
Take Figure 6, for example. In the picture, the four represented participants, Aladdin, Genie, Abu and Jasmine have no direct eye contact with the readers, forming an offer image act, requiring no reader reaction. The frontal planes of the four
represented participants are oblique to the readers, suggesting little reader
involvement. As for the textual part, the sentence is in declarative Mood structure, which functions to provide information, indicating no reader engagement.
Furthermore, there is no use of first or second personal pronouns to refer to the
readers, suggesting no reader involvement as well. In this case, both the visual and the textual modes do not address or involve the readers, achieving both ways of
reinforcement of address. Reinforcement of address is an optimal form of address, which would lead to a coherent multimodal text (Royce, 1999).
39
Figure 6. A Picture from Book 4 Lesson 5 of Joy Edition
In the foregoing section, the theoretical frameworks underpinning this research are expounded, including Kress and Van Leeuwen’s (1996) Visual Grammar for visual analysis, Halliday’s (1985) SFL for textual analysis and Royce’s (1999) framework for intersemiotic complementarity between the two modes. The overall multimodal analysis procedures are presented in the next section.
Procedures of Multimodal Analysis
Figure 7 outlines the procedures of analysis. Each text undergoes three phases of analysis: (A) intersemiotic ideational analysis, (B) intersemiotic interpersonal analysis and (C) comparing the degree of intersemiotic ideational complementarity and
intersemiotic interpersonal complementarity.
40
Phase A. Intersemiotic Ideational Analysis Stage I-Visual Analysis
(1) Analyze the following features in each picture:
Represented participants (people, animals, objects)
Processes (actions taking place) (2) Identify VMEs that refer to the pictures.
Stage II-Textual Analysis
(1) Identify lexical items in each sentence that can be linked to the VMEs.
Stage III- Intersemiotic Ideational Complementarity Analysis
(1) Interpret the semantic relationship between VMEs and lexical items in terms of six intersemiotic sense relations: repetition, synonymy, antonymy, hyponymy, meronymy and collocation.
(2) Count the tokens of sense relations in the reading passage.
Phase B. Intersemiotic Interpersonal Analysis Stage I-Visual Analysis
(1) Examine each picture with respect to:
Gaze: to determine the image acts
Horizontal angle: to determine the level of reader involvement.
Stage II-Textual Analysis
(1) Examine each sentence with respect to:
Mood structure: to determine the speech functions
Use of personal pronouns: to determine the level of reader involvement.
Stage III-Analysis of the Intersemiotic Interpersonal Complementarity (1) Determine the types of reinforcement of address:
between the use of gaze and Mood structures (GM)
between the use of horizontal angles and personal pronouns (AP) (2) Count the tokens of reinforcement of address in the reading passage.
Phase C. Comparing Intersemiotic Ideational Complementarity and Intersemiotic Interpersonal Complementarity
(1) Compare whether ideational or interpersonal meaning reaches higher degree of intersemiotic complementarity in all the 18 reading passages.
(2) Examine whether there are genre-related patterns in these texts with different degrees of complementarity.
Phase (A) began with visual analysis of represented participants and processes in each picture and identifying VMEs. In stage two, these VMEs then worked as
reference point to identify lexical items in the textual part that can be linked semantically to the pictures through the six intersemiotic sense relations. For each
Figure 7. Procedures of Multimodal Analysis
41
reading passage, (1) the tokens of sense relations, (2) the percentage of matched sentences, that is, among all the sentences, how many of them have matched sense relations occurring and (3) the density of token distribution, which was obtained by dividing the total tokens by the percentage of matched sentences, were calculated to get the degree of intersemiotic ideational complementarity in stage three.
Phase (B) began with examining whether there is direct gaze from the animated represented participants toward the reader in each picture to know the image acts.
Afterward, whether the horizontal angle of each picture is frontal or oblique was investigated to determine the level of reader involvement. In stage two, sentences depicting the pictures were analyzed in terms of Mood structure to know the speech functions, and use of personal pronouns to know the reader involvement. For each reading passage, (1) the tokens of reinforcement of address, (2) the percentage of matched sentences, that is, among all the sentences, how many of them have reinforcement of address occurring and (3) the density of token distribution, which was obtained by dividing the total tokens by the percentage of matched sentences, were counted to attain the degree of intersemiotic interpersonal complementarity in stage three.
Finally, the first stage of phase (C) was to determine whether ideational or interpersonal meaning reaches higher degree of intersemiotic complementarity in the 18 multimodal texts. The second stage was to categorize the texts based on
Macken-Horarik’s (2002) genre classification (see Table 6).
Table 6. Genre Classification (adapted from Macken-Horarik, 2002, p.23)
Genre Social Purposes Location
Recount (1) to retell events to inform or entertain (2) arranged in a temporal sequence.
(1) personal letters (2) written or oral
histories
42
Information Report
(1) to describe things in the social or natural environment
(1) encyclopedia (2) brochures (3) documents Explanation (1) to explain why or how things are
(2) It sets out the logical steps in a process (3) It includes exposition, discussion,
procedure and narrative
(1) essays
(2) public forums (3) manuals (4) novels News Story (1) to present recent events of public
importance
(1) newspapers (2) television
In the subsequent section, a multimodal analysis of an extract from Joy edition, Book 5, Lesson 7 is provided to exemplify the procedures.
Example
Taking Up a Hobby
As the saying goes, "All work and no play makes Jack a dull boy." Studying is important, and so is having fun.
A great way to have fun is to take up a hobby.Hobbies keep you from getting bored and give you a chance to learn new
skills.Some hobbies may even lead to a lifetime career.
Hobbies give people pleasure. Some people play sports:
basketball, tennis, badminton, and so on. Some do indoor activities, like cooking, playing chess, and collecting stamps. All of the above are good hobbies. ○10They
help people relax and forget about their worries.
The reading passage in the screenshot was used as an example to demonstrate the procedures of multimodal analysis. The passage contains10 sentences and 4 pictures
P1
P2
P3 P4
From Joy Edition, Lesson 7
43
as marked. Each sentence was numbered and each picture was marked P1, P2, P3 and P4 from the left-hand side. A step by step analysis of this reading passage is
as marked. Each sentence was numbered and each picture was marked P1, P2, P3 and P4 from the left-hand side. A step by step analysis of this reading passage is