Our Proposal: Multimodal Fusion Model

This study proposes the “multimodal fusion model” that includes the multimodal level (i.e. visual, aural, and verbal mode) and conceptual level to analyze the complex interaction between the multimodalities, the conceptual level, and interpretation. In other words, multimodal fusion model embraces the fact that the conceptual level, the verbal, visual, and aural modes are all encapsulated and integrated simultaneously in the real interpretation process of multimodal discourse thus we consider the conceptual level and the verbal mode, visual mode, or aural mode, naturally connected and overlapped with each other as shown in Figure 4, while Blending Theorists view different concepts or modes as in distinct input spaces connected with vital relations that are drawn with the blended lines. We acknowledge the significant integration of conceptual level as blending theorists suggested yet further assign a specifically important role of the conceptual level connecting with the verbal and visual/aural mode in multimodal fusion model.

Our analysis of the multimodal data indicates that multimodal cues including the verbal, visual, and aural cues each play significant parts in facilitating the interaction between different modes and the conceptual level, thus we stress that it is necessary to examine the multimodal cues specifically

79

in each mode in multimodal fusion model while interpreting the multimodal data.

In multimodal fusion model, we stress the central role of “genre” with the consideration on distinct affordances and constraints of each medium as Forceville (2012, 2014) advocates that genre is the most significant interface between text and context by emphasizing the significant role of “genre” as the

“key base” in our model (Figure 4). Through our analysis, we found the

“contrasting visual/aural features” within the visual/aural mode and in comparison with the “verbal mode” from the data are often the key elements that influence how certain topic or concept is conceptualized in different versions of work. Thus in our model, viewing the contrasting features in each mode and in comparison with the other mode is the main step to interpret multimodal data. We believe “genre” plays an important part in multimodal fusion model to predetermine and dominate how the conceptual level, the verbal mode, and the visual/aural mode will be fused and intertwined. In multimodal fusion model, we emphasize how different mediums and modalities, including the verbal mode, visual, and aural mode, are fused with the conceptual level, leading to various combinations and representations in specific genre, such as in political cartoon and poetry music.

80

In this model, the “metonymic-metaphoric network” is considered the fundamental basis of multimodal fusion model since we view multimodal fusion model as the cognitive mechanism that works at the interactive level of multimodal metaphor and metonymy, hoping to “recognize the diversity of forms and semantic content that make up different tropes” (Gibbs and Colston, 2012, p.342).

In view of interpretation, we believe that ‘people may not process all figurative meaning in the same way, precisely because the kinds and forms of different tropes are sufficiently varied as to resist classification within a single theoretical framework’ (Gibbs and Colston, 2012, p. 3-4), thus we do not agree that a single theory or model can be generalized and account for how all aspects of figurative language is understood. Under such view, this study incorporates “audience responses” with our analysis to extend the discourse-internal analysis of figurative meaning to an interactional level of interpretation analysis on multimodality and metaphor/metonymy from the perspectives of analyzer and audience (See Chapter 4 and 6).

This study focuses on the data collected from political cartoon and poetry music, which combine verbal, visual, and aural modes that reflect Mandarin language and culture, to investigate how certain topic or concept has been

81

represented in various representations of specific multimodal genre. With the primary focus on multimodal metaphor and metonymy, this study will demonstrate how the verbal mode, visual/aural mode, and the conceptual level are interactively fused and interrelated in the genre of political cartoons and poetry music. We will illustrate how certain topic and concept (i.e. politics, emotion, space, etc.) are conceptualized and construed differently in the multimodal genre accompanied with the use of prominent multimodal cues (visual/aural/verbal cues) that facilitate multimodal realization and interpretation.

In this study, we also examine how specific effects, such as humor and irony in political cartoons or musical effects in poetry music, result from multimodal fusion in political cartoon. Our findings indicate that the audience interpret the data and perceive/sense the effects through the comparison and contrast between multimodal cues and prominent multimodal features.

In the following chapters, we will provide detailed examination of the data (i.e. political cartoons and musical scores) and cross-comparison analysis under the scope of multimodal fusion model.

82

Chapter 3 Multimodal Fusion in Political Cartoon: Debates on U.S. Beef

Imports into Taiwan

In this chapter, this study proposes a "multimodal fusion" model to account for the cognitive mechanisms involving political cartoons with regard to U.S. beef import issues as reported in Taiwanese newspapers. In the following sections, data collection is provided in Section 3.1, data analysis with representative examples illustrating the process of multimodal fusion are presented in detail in Section 3.2, the analysis of conceptual scenario and conceptual metaphor is in Section 3.3, which is followed by further discussion in multimodal fusion and political stance in Section 3.4.