Organization of study

In the following chapter, we will review the theoretical bases for this study. In chapter 3, we propose a multimodal methodology for analyzing the textuality of paintings. In chapter 4 and 5, analyses and results of the examinations are displayed. A discussion of our overall findings is presented in chapter 6, before the thesis reaches its finale.

Chapter 2

Theoretical background

The current chapter outlines the theoretical bases for this study. We will review the notion of iconicity, which serves as a crucial principle directing the textuality of verbal art. Moreover, we will review Halliday’s (1990) definition of ‘textual reality’ based on the two facets of iconicity. We will additionally review the related models on

grammatical iconicity (Hiraga, 2005) and multimodal text compositions (Kress and van Leeuwen, 1996). Finally, we will briefly outline some basic features of modern Western painting.

2.1 Iconicity and grammar

Iconicity refers to the alignment of linguistic form and linguistic content, contrasting the notion of arbitrariness. In a Saussurian tradition, the relationship between signifiers (form) and signified (content) is considered arbitrary, in which there is not necessarily a motivation between the correspondence of form and meaning. However, as reviewed by Hiraga (1994; 2005), Jakobson (1971) considers the opposite on the basis of the

Peircean sign trichotomy (1955 [1902]). Signs are divided into icons, indices and symbols based on levels of resemblance to their corresponding referents. Icons are the

most object-resembling signs, and symbols being the most arbitrary.

Icons, the most iconic or non-arbitrary signs, are further divided into images, diagrams, and metaphors by Peirce. Images, diagrams and metaphors together

constitute a hierarchy of iconicity. From images to metaphors, there is an iconic continuity based on similarity principles of “mimicry (quality)”, “analogy (structure)”

and “parallelism (association)” (Hiraga 1994; 2005). In Peircian semiotics, the three are also termed as semiotic “firstness”, “secondness”, and “thirdness”, accordingly.

In Halliday (2002 [1990]), a functional analysis to scientific writings is presented.

In his writing, Halliday mentions the intersection of science and verbal art, in which scientific writings are “constituted out of the impact between scientific and poetic forces of meaning.” (2002 [1990]:177). It is therefore claimed that the grammar of verbal expressions is often iconic in nature, which constructs “the total reality in which we now live, a reality consisting of semiotic entities in a periodic flow of information”

(2002 [1990]:175).

The reality of textual grammar here consists of two kinds of iconicity: Iconicity in

‘semiotic entities’, and iconicity in the ‘periodic flow of information’. In the following,

we will outline both facets of textual iconicity, and their theoretical extension to the analysis of multimodal texts.

2.2 Iconicity and Grammatical Metaphors

In Halliday (2002 [1990]:175), the ‘iconicity in semiotic entities’ particularly refers to cases of ‘nominal packages’. It is the process of compressing a verbal event into the form of nouns, which is also called “nominalization”. In English, this is often put into practice by adding certain suffixes at the end of the verb stem, such as the

productive –tion. Therefore, as an example, to nominalize is an action, whereas

nominalization implies a noun-like ‘semiotic entity’. When a verbal event is packaged

as a noun, it could then be processed in speech and in our mind as moveable entities, or objects. By the change of grammatical forms, we think of the same concept in different ways-- from the more dynamic “action” to the more entity-like and compacted idea of an “object”.

From action to object, this particular process is defined as an instance of

grammatical metaphor in Halliday (1994a). Put simply, grammatical metaphors could

be identified when the grammatical form of an expression suggests its perceived

meaning. It is metaphorical, because there is an analogical alignment between form and meaning. The scope of grammatical metaphors could be instantiated at the level of phonology, or at the level of morphology and phrase. However, morpho-syntactic examples of nominalization, or nominal packages, remain the central discussion of grammatical metaphors in Halliday’s work.

In the work of Hiraga (2005), she explores the instantiation of grammatical metaphors in Japanese poetry, including verbal poems and pictorial poems, extending the analysis scope of grammatical metaphors to that of discourse. It is reported that grammatical metaphors plays a crucial part in the structuring of poetic texts. The intense interplay of poetic content and its grammatical arrangement co-creates the complete aesthetic meaning of the poetic discourse.

Based on her observation, Hiraga proposed a typology of grammatical metaphors (shortened as GM), a “systematic classification of grammatical metaphors” (2005:174).

The model consists of a list of non-exhausting GMs called ‘iconic mapping rules’ found in poems. The interplay of iconicity and metaphors has been highlighted, involving iconic instantiations found locally in sound-symbolism, word order, and even in the printing arrangements of pictorial poems. These iconic instances add to the

interpretations and meanings of the poems observed. Comparing to Halliday’s original idea on GM, Hiraga’s GI model has gone beyond the morpho-syntactic realizations to the level of discourse. The observed iconic instantiations “contribute to the interplay of form and meaning at the level of both text segments and text macro-structure.” (Hiraga 2005:52).

Hiraga’s GM typology consists of a set of non-exhausting iconic mapping rules

directing poetry compositions. These rules are proposed in the backbone of Lakoff and Johnson’s (1980) Conceptual Metaphors, in which they describe schematic and

productive form-content alignments across poetic discourse. There are two major categories in Hiraga (2005), namely relational diagrams and structural diagrams.

On the one hand, relational diagram refers to iconicity instantiated within one comparative dimension. For instance, if some forms display similarity, and their meanings indeed display similar semantic or pragmatic characteristics, this would then identified as applying to the GM of SIMILARITY IN MEANING IS SIMILARITY IN FORM. Relational diagrams still include iconic mapping rule DIFFERENCE OF MEANING IS DIFFERENCE IN FORM.

On the other hand, structural diagrams demand structural mappings between

‘structure of form and structure of content’ (Hiraga 2005:175), so it differs from cases of

relational diagrams. For example, if some repetitive phrasal structure intensified and enriched the meaning of a text, the quantity of forms then suggests the importance of their textual meaning. This would be an instantiation of MORE FORM IS MORE MEANING, which involves a structural alignment between form-quantity and meaning quality. Structural diagrams still include TEMPORALITY IS LINEARITY (e.g. when the linear placement of the expressed ideas aligns with their event order), and

SEMANTIC RELEVANCE IS CLOSENESS. Semantic relevance, in Hiraga

(2005:180)’s term, refers to expressions that are “semantically closer”. In her linguistic model, it is often the case that lexical items that are placed closer express stronger relevance, or stronger, more affirmative pragmatic effect.

Besides the mentioned rules, Hiraga also observes some iconic mappings that are linguistic or cultural-specific, such as ones found in linguistic frozen expressions, or in the unique grammar of politeness in Japanese culture. While the specific principles may seem distant from our multimodal exploration, other iconic rules, in fact, are found applicable in a pilot study on painting and iconicity. In Chen and Su (2014), 5 out of the 7 mentioned iconic mapping rules are found to be applied in a mixture of 120 matured paintings done by Spanish artist Miro.

It is noted in Hiraga (2005:226) that the issue of iconicity remains unexplored in the area of non-verbal communication. Based on the work of Hiraga (2005) and Chen and Su (2014), this model concerning the interplay of metaphor, iconicity and poetic expressions, serves as a promising starting point for the current exploration.

2.3 Iconicity and textual flow

Another line of “textual reality”, besides the iconic instantiation of grammatical metaphors, is defined by Halliday as the iconic flow of textual information, which is constructed by Information values, i.e. the relative idea of Given or New information of a text. Specifically, he states:

There is a movement from a given Theme (…) to a rhematic New (…) (T) his movement in time construes iconically the flow of

information.

(Halliday 2002 [1990]:175)

This definition of textual reality is based on Halliday’s theory on textual pattern.

As claimed, the clausal structure in English consists of two intertwined information lines, namely the couplet of Theme and Rheme, which is speaker-oriented, as well as Given and New, which is listener-oriented. The two information lines are simultaneous in nature: As the Theme and the Rheme realized via the “segment position” in a clause, the Given and the New accompany prosodic contrasts. Therefore, it is possible that the

two interrelated systems combine, resulting in an overlap of the Theme and the Given, together serving as the starting information or “ground” (as contrary to “figure” in

gestalt psychology) of an expression.

For instance, in the sentence I think it looks great, the front part I think serves as

expression. Meanwhile, the part it looks great remains more “unpredictable” and “news worthy”, which expresses the speaker’s evaluation. Without this part, the sentence fails to give out its essential “punch line”. It could thus be counted as the Rheme, or the New of the sentence. In this case, the overlap of Theme and Given, as well as that of Rheme and New, applies to Halliday’s definition of a default, or “unmarked” instance of textual information. The proceeding of a text, then, is made possible by the spatial as well as meaningful tension between the Given and the New. This is one of the core principles that constitute the grammar of natural languages (2002 [1990]:173). The default

arrangement, as noted, consists of a Theme or Given information at the sentence initial, and a follow-up of Rheme or New. Concerning the writing directionality of most Western languages, it is often the case that such “default” information flow of linguistic texts lines up from the left, to its right-branching sentential parts:

Halliday’s Information system is extended into the realm of the multimodality.

Specifically, social semioticians Kress and van Leeuwen (1996; 2006) studied the

compositional pattern of visual texts in general. They have proposed a general pattern, or ‘visual grammar’, across most western visual layouts. The proposed diagram for

visual compositions is as illustrated:

Figure 2.1 Kress and van Leeuwen (2006:179)

This diagram predicts the general textual composition of visual materials in the context of Western cultures: From the left to the right, there is a pattern of Given to New;

from the top to the bottom, there is a pattern of Ideal to Real. It is claimed that a

Given-New information pattern could be observed across most Western visuals, which is postulated to be influenced by the writing-directionality in most Western languages,

following Halliday’s view. Usually, there would be a left-placing of Given information, while the more “surprising”, unexpected content featured at the right, and labeled as the

New. The circle in the diagram stands for the sometimes centralized visual focus observed. Kress and van Leeuwen claimed that a centralized visual composition would

still follow the overall tendency suggested by this diagram, which is exclusively designed for the two-dimensional nature of pictoriality.

Besides the criteria of Ideal-Real and Given-New, which are based on Information values of in-text pictorial areas, Kress and van Leeuwen mentioned two alternative

principles which also play a role in influencing visual compositions. The two principles are Saliency and Framing, respectively. While Saliency refers to the perceptually salient qualities of a visual, which is often created by stronger contrast in color, tone, size or sharpness of pictorial compositions (Kress and van Leeuwen 1996:212), Framing refers to the compositional separation of textual areas, which is an important feature of applied visual layouts.

Kress and van Leeuwen’s (1996; 2006) model, while inspiring in nature, consists of a mixture of visual genres, ranging from composite pictorials such as magazine layouts, films, photography, picture books, and single pictorials like paintings, graphic designs, and advertisements. The realm of paintings has not been systematically focused on in this model. This gap prompts the current need to study modern paintings in a systematic way. In the following section, we will briefly introduce the basic

characteristics of the set of pictorials in wait for systematic discussion, and with the utmost essentialness to be involved in cognitive multimodal studies.

2.4 Modern Western nonrepresentational paintings as autonomous pictorials This study specifically focuses on a selection of nonrepresentational modern Western paintings. According to Luhmann (2000), after the Renaissance, painters in the Western world slowly gained their autonomous in art creation, shedding away the traditional burden, for instance to paint for religious duties. Since the late 19^th century to the early 20s, the outburst of artistic movements such as Impressionism, Fauvism and Cubism opened up a brand new extent of artistic freedom, leading to the heydays of Modern Art.

Art movements mushroomed, each striving to defy the traditional ways of depiction.

The paintings created in the Modern Art era, which strongly coincides with the

blooming of the mentioned artistic movements around 1900 till the late 70s (concerning the life spans of the featured artists), revolutionized the way we see and think visually.

Besides its expressive autonomy, modern Western paintings are also comparatively autonomous due to their default forms: They are almost always presented on the flatness of a rectangular canvas, as commented by Schapira (1969):

“We take for granted today as indispensable means the rectangular form of the sheet of paper and its clearly defined smooth surface on which one draws and writes (…) the smooth prepared field is an invention of a later stage of humanity.”

(Schapiro 1969:9)

In a sense, the somewhat conventionalized form of modern painting pieces strengthens their independence. They are considered single pieces, ready to be displayed or

purchased, rather than serving as an illustration for other communicative purposes. We could say that the autonomy of a modern painting is co-shaped by the “conventionalized”

use of canvases. Such materiality distinguishes “works of art” away from “crafts”, such as pictures painted on vases, walls or furniture. Summing the above, modern paintings, especially ones that originated in the mentioned contexts (20^th century Western modern Art movements), are ideal to be seen as independent “texts” conveyed in a pictorial mode. This stands as the basis for our assumption. In the next chapter, we will introduce how we examine these paintings via a cognitive multimodal approach.

Chapter 3

A cognitive multimodal model

This chapter introduces a methodology for textually analyzing paintings. Our model combines Hiraga’s (2005) grammatical metaphors (GM) with Kress and van Leeuwen’s (1996) visual compositional principles, serving to analyze paintings in a multimodal approach. We aim to find out whether non-representative paintings qualify the definition of a ‘reality of text’, and constitute a structurally-arranged text genre. The following illustrates the present methodological design, the adopted models, as well as details of data selection.

3.1 Adopted models

Iconicity remains one of the central characteristics of linguistic grammar, not only in the figurative case of poetic texts (Jakobson 1960; Hiraga 2005), but also in scientific writings (Halliday, 1990). This phenomenon is nonetheless never fully explored in modalities outside language. In this chapter, we adopt related models for the exploration of iconicity to a corpus of nonrepresentational modern Western paintings.

Two foci line up in our examination of paintings. First, the existence of iconic grammatical metaphors is investigated across a spectrum of nonrepresentational

Western modern paintings via Hiraga’s (2005) model. Second, the pattern of an iconic textual flow across painting-layouts is analyzed via the semiotic model of Kress and van Leeuwen (1996).

Hiraga’s linguistic model defines and lists out instances of iconic GMs, and Kress and van Leeuwen’s visual model helps provide a working definition for the visual counterparts of GI in painting-plates. The pictorial GM in this study, then, is defined when the visual forms and the multimodal semantic cues of a painting align with each other, creating a comprehensive meaning of the whole artwork.

Some of the GMs in Hiraga (2005) could directly be mapped to the examination of pictorials in Kress and van Leeuwen (1996), for instance the GMs of form quantity, distance and similarity. As noted in 2.2, there are iconic rules that only apply to

linguistic data, such as the iconic principle on linearity, and how linguistic politeness is instantiated through the in-text distance of lexical items. These iconic rules would be excluded in this study. While the politeness issue is clearly missing in the case of

paintings, the linearity issue too needs to be “hedged” in order to fit our visual data now, as implied in Chen and Su (2014).

Instead, in the two-dimensional case of paintings, Kress & van Leeuwen’s (1996) visual-oriented model could complement Hiraga’s (2005) linguistic model, for it focuses on an overall compositional flow of paintings via a four-part diagram (Fig. 2.1), rather

than a linear one. In the aspect of textual directionality, it is more flexible and visually-oriented than Hiraga’s model. In this study, both models are adopted and adjusted in generating a method suitable for multimodal exploration.

3.2 Data

Our data consists of 90 nonrepresentational modern Western paintings created in the 20^th century, including 30 figurative paintings, 30 semi-figurative paintings, and 30 abstract paintings, following the categorization in Arnheim (1969). The concepts of figurative and abstract are relative: they constitute two ends of a continuum, and could never be put in a clear-cut dichotomy. The inclusion of all styles of paintings ensures that a

comprehensive data pool is covered. Moreover, the three-group sampling expects to bring forth possible cross-group comparisons in compositional principles.

Each sampling group contains paintings created by one specifically representative painter. For the quality of the data, it is ensured that these representatives remain productive painters. The sampled paintings are selected out of three chronological painting albums of the representative artists, excluding practices of representational artwork, sketches, prints or other miscellaneous scripts. After the exclusion, the amount of samples is balanced to approximately thirty in each of the three painting groups.

The pure reality-simulative artworks are excluded. They would not constitute idealistic examples of expressive paintings since a representational intention, or restrictions bound to photography or real-world referents are involved. Instead,

nonrepresentational paintings are selected, with the pictorial contents generated via the painter’s free association. In this sense, paintings are autonomously expressive in its creation process and creative form. In the following, we will illustrate details of the sampled data.

1) Figurative painting samples: Dali

Thirty of Spanish artist Dali's paintings are representative as figurative, nonrepresentational modern Western paintings, for they are:

i. Figurative in style; the viewers can easily identify and name the painted elements;

ii. Nonrepresentational in intention, in which the painted events are mainly created via the process of free association, so they may lean closer to expressive texts rather than to real-world snapshots.

There are no abstract parts in Dali’s paintings-- every line and pile of paint constructs part of a figurative object. We can always name his painted items, even though his art does not rely on the tradition of real-life painting. His paintings thus serve ideally as our figurative samples.

2) Semi-figurative painting samples: Miro

Thirty of Spanish artist Miro's paintings are representative as being semi-figurative

Thirty of Spanish artist Miro's paintings are representative as being semi-figurative