Derived from Halliday’s SFL model (1994), “Visual Grammar” was developed by Kress and van Leeuwen (1996, 2006) to analyze visuals to account for other semiotic meanings than only from language. Expanded from Halliday’s (1994) metafunctional perspective of language, Visual Grammar is presented to explain three corresponding meaning of visuals, including representational, interactional and compositional. Representational meaning corresponds to the ideational metafunction, dealing with the participants, activities, attributes or the characteristics of the participants and circumstances involved. Interactional meaning reflects the interpersonal metafunction, examining the interaction (visual address, level of involvement, social distance and power relation) built between the viewers and what is viewed. Compositional meaning echoes the textual metafunction, exploring the page layout which determines the extent to which coherence in the text is achieved between the visual and verbal elements.
Figure 3. Correspondence between Halliday’s (1994) SFL and Kress & Van Leeuwen’s (1996) Visual Grammar (Attar, 2014)
18
Figure 3 represents the correspondence between Halliday’s (1994) SFL and Kress & Van Leeuwen’s (1996) Visual Grammar. The present study adopted the Visual Grammar framework to analyze representational perspective of visual components in the dialogue. Representation stands for the represented participants displayed in the image and how these participants associate with one another in meaningful ways. Drawing upon Kress & Van Leeuwen’s (1996) visual grammar, representational dimension of visual elements are realized through the represented participants, the processes and circumstances. People, things and places are identified as represented participants. There are two types of representational dimension of visual elements, Narrative and Conceptual (Figure 4).
Figure 4. Variables of Representational Analysis in Visuals
Narrative process is similar to the Material processes in the system of transitivity.
It possesses participants called “Actor” and “Goal”. Actor means “the participants from whom or which the vector departs and which may be fused with the vector to different degrees” (Kress and van Leeuwen, 2006) while Goal refers to the other participant, whom the vector is pointed. Narrative process is formed when the two
Representational meaning (Kress and Van Leeuwen,1996)
Narrative process
Action process Reaction process
Mental process Verbal process
Conceptual process
Analytical process Classificational process
Symbolic process
19
participants are “connected by a vector, represented as doing something to or for each other” (Kress and van Leeuwen, 2006). Narrative process can be differentiated based on the kinds of vector and the number of participants involved, including Action, Reaction, Mental, and Verbal process. The definition and example of each process is listed below.
Action Process
In the Action process, “Actor” refers to the active participant “from which the vector emanates or which itself, in whole or in part, forms the vector” (Kress and Van Leeuwen, 2006). The passive participant “at which the vector is directed” is labeled as
“Goal”.
Reaction Process
Reaction process is constituted when the vector is created by the eyeline of one or more participants. “Reacter” is the active participant whose direction of the glance creates the eyeline. “Phenomenon” is passive participant “at which the eyeline is directed” (Kress and Van Leeuwen, 2006).
Mental Process
When a vector is formed by a thought bubble that connects thinkers and their thought, Mental process is established. The participant who produces the thought bubble is
“Senser”. What enclosed in the thought bubble functions as “Phenomenon”.
Verbal Process
Verbal process refers to “a vector formed by the arrow-like protrusion of a ‘dialogue
20
balloon’ or similar device connects two participants, a Sayer and an Utterance” (Kress and Van Leeuwen, 2006). “Sayer” is the participant from whom the dialogue balloon originated, while “Utterance” is the participant (verbal) contained in the dialogue balloon.
The other representational process is Conceptual process, which represents
“participants in terms of class, structure or meaning, in other words in terms of their more generalized and more or less stable and timeless essence” (Kress and van Leeuwen, 2006). Conceptual process resembles to the Attributive process in the system of transitivity which displays “process of being” (Halliday, 2004; Kress and van Leeuwen, 2006). Different from the Narrative process, there is no vector which determines conceptual process of an image. Basically, conceptual process includes three types: Analytical, Classification and Symbolic. The following section will provide the definition of each process.
Analytical process
In Analytical process, participants are associated in terms of the part-whole structure.
Two kinds of participants involved, including “Carrier” and “Possessive Attributes”.
“Carrier” denotes the participant which symbolizes the whole, while “Possessive Attributes” signifies other participants which are the parts of the whole.
Classificational process
Classificational process connects participants with each other regarding a kind of relation, or taxonomy. In this process, one participant plays the role of “Superordinate”
and the other is “Subordinate” of that participant.
21
Symbolic process
Symbolic processes are about “what a participant means or is” (Kress and Van Leeuwen, 2006). In this structure, “Carrier” denotes the identified participant, while
“Symbolic Attribute” pertains to the identity of the participant.
Summary
To conclude, Halliday’s (1994) SFL has offered a comprehensive approach for multimodal discourse analysis (MDA), not only written text (Halliday, 2002, 2004;
Halliday and Matthiessen, 1999; Martin, 1992; Martin and Rose, 2003) but also extension to other semiotic resources in multimodal context (Halliday, 1994; Kress &
Van Leeuwen, 1996; Lemke, 2002; O’Halloran, 1999; O’Toole, 1994; Unsworth &
Wheeler, 2002). Following the paradigmatic shift from the study of language alone to
“intersemosis” concerning the integration of language with other semiotic resources, how different modes intertwine together to construct a multimodal texts received much attention (Lemke, 2000; Liu and O’ Halloran, 2009; O’ Halloran, 2005; Royce, 1998).
The theoretical foundation of the research is Liu and O’Halloran’s (2009) Intersemiotic Cohesive Device which investigating semantic relations among linguistic and pictorial components and thus integrate different modalities into a semantically coherent text. The present study targets on Intersemiotic Parallel Structure, one of Cohesive Devices which integrates textual and visual semiotic resources. Halliday (2004)’s Transitivity System is adopted to examine the ideational metafunction of textual modes, which includes six types of processes. Kress and Van Leeuwen (1996)’s Visual Grammar is utilized to analyze the representational aspect of visual components. Intersemiotic Parallel Structures will take effect when the two
22
semiotic modes possess analogous Transitivity form. After introducing the major theoretical frameworks of the present multimodal research (see Figure 5), empirical multimodal studies will be reviewed in the next part.
Figure 5. Theoretical Framework of the present study