Summary of results

An obvious grammatically iconic applicability can be seen throughout our painting corpus. As demonstrated, the sum in the last columns of Table 4.1 to 4.3 represents the extent of GI in each sample. The higher the number, the more grammatically iconic a painting is. So far, the highest GM sum is 10, and the lowest is 3 in our data (cf.

Appendix 2).

An overall statistics could be seen in the table below (Table 4.4). The numbers stand for rounded up percentages of the applicable instances within each painting category.

As suggested by the statistics, the average applicability of GI is highly positive in the selected nonrepresentational modern paintings: There is an average extent of 92.6%

of GI in the figurative group, an average of 85% in the semi-figurative, and an average of 85% in the abstract group, with a total average of 87.4% in the current sample pool.

Painting groups GI principles Ratio

Table 4.4 Applicability of GI in painting groups

Specifically, a slight declining of GI applicability could be observed from the figurative to the abstract. Within the listed examination aspect of GI, the principles of SIMILARITY and DIFFERENCE are sometimes “flouted” in the first two painting groups, e.g. in cases of figure-ground reversion in Dali’s artwork. Moreover, it could not apply in the abstract group due to lack of semantic contrast. Similarly, the iconic

mapping of SEMANTIC RELEVANCE IS CLOSENESS fails to be applied in the abstract group.

The most applicable rules are the ones concerning “quantity and importance”. It has achieved an average of 92.3% applicability across all three groups. Even in the abstract cases, in which often no concrete reference could be found in their titles nor in the pictorials, the up-front eye-catchiness of the figure zones still foregrounding itself as the attention center of the pictorial text.

Also notable is the highly-featured “markedness” GM principle termed as MARKED FORM IS MARKED MEAING in Hiraga (2005). Put simply, the more marked a form, the more marked its textual function. Even in abstract paintings, which are usually referent-free, an especially marked visual focus could be found. The MARKED zones are usually humble in size, but are always marked in visual form and textual placement.

As we reach the end of the first-stage examination on iconicity, the next chapter continues with the second-stage examination.

Chapter 5

Textual flow in paintings

In this chapter, the second research question of this thesis is addressed. We examine whether the sampled nonrepresentational paintings display a general textual flow. The criteria are adopted from Kress and van Leeuwen’s (1996) semiotic model for visual composition. Our investigation to the selected paintings is presented in 5.1(figuratve samples), 5.2 (semi-figurative samples) and 5.3 (abstract samples). In 5.4, the overall results are summarized.

5.1 Textual flow in figurative paintings

Among the three interrelated factors determining visual text compositions, it is often the case that Information value and Saliency play a role in the figurative group, with

Framing mildly hinted (rather than obviously instantiated) in these non-commercial visual texts.

In The persistence of memory (Fig.5.1), the left side of the painting consists of three clocks placed side-by-side, while the right part featuring a horse-like face (or a face-like horse), with a clock on it. The outline of the “horse” extends to the right of the picture, which matches the concept of the title of the painting.

As the left side of the painting leads the flow of the pictorial text to the more conceptual area of the right, this is an instance of the progressing of Given to the New, with the extending image functioning as the “punch line” of this painting, the dynamic, continuing concept of personified time. This arrangement thus attunes with the “left is Given, right is New” information structure reported (Kress and van Leeuwen, 1996).

Thus, we identity the New information zone to be mainly at the right half of this painting, as circled in Fig.5.2.

In this compositional design, the placement of the pictorial items follows the Information value (of Given and New), but also accords with their visual-based Saliency and Framing characteristics: The left and the right zones could be seen as separated by their surrounding background, fulfilling the definition of framing in Kress and van Leeuwen (1996). That is to say, in this painting, and several others in the figurative category, the three reported factors (Information value, Saliency, Framing) co-influence the composition of the whole visual text. This painting, like many figurative instances, applies all three compositional principles listed in Kress and van Leeuwen (1996), as listed in Table 5.1

Figure 5.1 The persistence of memory (Dali, 1931)

Figure 5.2 New info zone for The persistence of memory

Title Compositional principle

The persistence of memory

Info value +

Saliency¹ +

Framing +

Sum 3

Table 5.1 Textual composition of The persistence of memory

5.2 Textual flow in semi-figurative paintings

In semi-figurative samples, however, rarely can we find all three compositional principles applied. In the following example entitled Characters in the night (Miro,

1950), the title aid us to identify three characters, a bird, and three pictorial signs for

“stars” in Miro’s invented art language (Corbella, 1993). However, no obvious action is

seen in this painting: It expresses a state, rather than an event. Hence, no clear “punch line” or New information zone can be identified in this painting. Put simply, the mostly discussed compositional principle in the Grammar of Visual Design (Kress and van Leeuwen, 1996), Information value, fails to apply in this semi-figurative painting.

Furthermore, there is also a lack of separation between the figure zones, or even between the figure zones and the ground layer. The Framing principle thus fails to apply in this Miro painting as well.

What stands as the hint on reading this picture, instead, is the bright whiteness in the painting-plate. The white zones form a rhythmic placement, changing in size and angles, creating a visual path within the canvas. Our eyes could more easily follow the white zones while reading the painting, though not necessarily with a certain

directionality. In the category of Miro’s semi-abstract art, the principle of Saliency starts to outshine the other two reported principles in previous studies (Kress and van

Leeuwen 1996; 2006).

Figure 5.3 Characters in the night (Miro)

Painting title Compositional principle

Characters in the night

Info value - Saliency +

Framing -

Sum 1

Table 5.2 Textual composition of Characters in the night

5.3 Textual flow in abstract paintings

In abstract paintings, similar phenomena are observed: Information value and framing do not serve as useful criteria for the textual compositions of the paintings. The remaining principle that hints the possible reading paths of an abstract painting is the compositional principle of Saliency in Kress and van Leeuwen (1996).

In the example entitled Convergence, the verbal cue adds more emphasis to such a

process, in which the dynamics of colors and lines direct our eyes. Again, we may freely

“travel” along the u-shaped visual-perceptual rhythm defined as compositional Saliency

in Kress and Leeuwen (1996).

Figure 5.4 Convergence (Pollock, 1952)

Painting title Compositional principle

Convergence

Info value -

Saliency +

Framing -

Sum 1

Table 5.3Textual composition of Convergence

5.4 Summary of results

After our analyses from 5.1 to 5.3, we are here to outline a summary of results.

In the aspect of textual directionality, what is found in this thesis differs from that reported in Kress and van Leeuwen (1996). The main difference lies in the

low-applicability of the textual principle of Information value and Framing in nonrepresentational modern paintings. In our data, the samples do not rely on

Information value as the determining factor of compositional directionality. We in fact found a continuum of distribution in the three data groups in the diagram below:

Figurative paintings distinguish themselves in terms of Information value, while this feature is almost non-applicable to paintings in the semi-figurative and abstract group.

Similarly, Framing is not applicable to all three styles of Modern paintings. Only the figurative paintings utilized Framing to a great extent in structuring the pictorial elements, while the other two groups declined greatly in, or even abandoned the feature.

Therefore, the most applicable group to Framing is the figurative; the semi-figurative barely applied, and the abstract remains totally non-applicable to Framing.

Contrary to the declining applicability from the figurative to the abstract in Information value and Framing, all the painting styles involve Saliency as a

directionality-related principle. As illustrated, however, a general directionality textual pattern could not be determined by Saliency alone yet.

Figure 5.5 Applicability of compositional principles (Kress et al. 1996) in the current corpus

Figure 5.5 shows how the compositional principles proposed by Kress and van Leeuwen (1996) are applied in our data. The underlying principle of our selected Modern Western paintings appear different from the previous studies, in which Information value and Framing stop to function effectively in determining the textual flow of Modern paintings. Saliency, though helpful, does not reveal a certain textual flow directionality, such as the left-to-right tendency suggested in previous studies. In short, our results revealed that two-thirds of the proposed compositional principles ceased to function in semi-figurative and abstract paintings. In the next chapter, we will address more to this phenomenon.

Chapter 6

“Textuality” in nonrepresentational modern Western paintings

“Is it conceivable that art, as a kind of “writing”, builds a bridge between perception and communication, that it compensates for the

communication system’s inability to perceive?”

-- Luhmann (2000:17)

Our analyses revealed two important findings. First, the selected paintings greatly reflect a pattern of grammatical iconicity. Second, compositions of these

nonrepresentational modern paintings are governed by principles alternative to Information value and Framing proposed by the previous studies.

Our findings uncover the unique structuring characteristics of nonrepresentational modern Western paintings. As the results unfold, the current sampling brings out what is unseen in the Visual Grammar by Kress and van Leeuwen (1996). In their model, as reviewed, there is a mixture of visual genres, and a methodology involving mostly commercial graphics. This could explain why the textuality of nonrepresentational modern Western paintings does not entirely follow the Visual Grammar, and that our data set is a special case comparing to the majority of multimodal studies.

6.1 Modern paintings and multimodality

Among our data, the figurative and the semi-figurative groups show similar extents of title descriptiveness, for their painting titles are about as semantically informing; the abstract paintings, on the other hand, are not always entitled descriptively or featuring semantically concrete titles.

Additionally, the interplay of textual meaning and pictorial content implies the multimodal nature of modern painting pieces: They constitute an independent genre of visual text. Verbal cues are linked to modern paintings, however minor they may seem, co-expressing the meaning of the painting as a complete multimodal text. Despite the seemingly uneven verbal-nonverbal ratio in a modern painting piece (the depicted part usually outshining the verbal cue of the title), the visual part of modern paintings still appear more independent and autonomous comparing to other applied visuals.

Kress and van Leeuwen (1996) have not systematically mentioned paintings, let

alone a spectrum of nonrepresentational modern paintings, which cline more to the system of “writing” under the guideline of modern arts. As noted, these paintings are

created to defy the traditional expressions of visual mimicry. On the basis of the current findings, we believe that linguistic cues can be viewed cognitively via Conceptual Blending (Fauconnier and Turner, 1995), which generates a richer and more comprehensive account of the art piece as a whole:

Figure 6.1 Modern paintings as multimodal texts

In a sense, painting titles escalate the semanticity for the painted content (as in the examples of painting samples featuring semantically descriptive titles), ensuring a smoother emergence of a holistic comprehension of the piece. Furthermore, the existence of painting titles implies the social reality of the paintings contextualized as

“modern artwork” in wait for exhibition, purchase, or collection.

6.2 Textual patterns and iconicity in nonrepresentational modern paintings Fusing the visual aspects and the cognitive linguistic notions in investigating Grammatical Iconicity in modern paintings, we have come up with the following finding: The previous semiotic textuality predictions do not fully describe the textual patterns of nonrepresentational modern paintings in general. For the most part, only the compositional principle of Saliency stands as an applicable principle affecting the reading paths of paintings. That is to say, the directionality principle of modern paintings is greatly involved with perceptual transitions. In this respect, modern paintings rely on similar resources of content-progressing to music pieces (cf. Donald 2006): The body of art is greatly shaped by the cumulating and transitions of perceptual cues, alternative to the dominance of Information value, as in the case of most linguistic grammar (Halliday 1990; 1994a) and commercial visual layouts (Kress and van

Leeuwen, 1996). It is thus not the case that all visual genres progress directionally exactly like verbal texts or highly commercial multimodal texts, in which Information value functions as a promising textual flow determent factor.

To better illustrate the inapplicability of Information value in our data, a visualization of the recorded instances of New information zones among all 90 paintings is shown in Figure 6.2 to 6.4:

Figure 6.2 Patterns of New Information zones in the figurative group

Figure 6.3 Patterns of New Information zones in the semi-figurative group

Figure 6.4 Patterns of New Information zones in the abstract group

It is obvious that in the figurative group (Figure 6.2), the circled instances of New remain bountiful. In the semi-figurative case (Figure 6.3), the applicability of Information value drops. Finally, in the abstract group (Figure 6.4), no sign of Information value could be found. Therefore, the commonality of Information value does not suit the visual genre like ours. There must be an alternative principle governing the compositional flow of paintings.

Let us return to the research questions of this study. It is clear that after our examination, a textual reality dictated by iconicity is indeed validated in the case of nonrepresentational modern Western paintings, from the figurative to the abstract. These paintings:

1) Stably involve the interplay of various kinds of iconic grammatical metaphors, fulfilling the criterion of “iconic semiotic entity” in Halliday (1990);

2) Vaguely display a textual flow, in which:

(1) In figurative cases, the compositional principles of Information value, Saliency, and Framing (Kress and van Leeuwen, 1996), are all actively applied;

(2) In semi-figurative cases, the applicability of Information value and Framing starts to sharply decline in determining the textual flows of the paintings;

(3) In abstract cases, both Information value and Framing could not be applied.

The remaining clues for textual flow lie in the titles, and the Saliency zones involved.

As shown in (1) to (3) above, Information structure ceases to function as the dominating principle of Modern painting textuality. The most promising principle appears to be Saliency, as it is the only factor applied in all three styles of painting groups in our corpus. Nevertheless, we propose here that Saliency (Kress and van Leeuwen, 1996) serves as the basis for the structural grammar of paintings, rather than serving as the grammar itself, since it does not provide further clue in determining textual

directionality, as illustrated in Chapter 5.

Given 1), it is clear that paintings are doubtlessly validated in the aspect of iconic GM. The aspect of the criteria for an iconic textual flow in 2), however, requires a re-interpretation from the previously commented.

The following figures displays our manually analyzed process, or records, of Examination 1, the investigation on the textual reality of Grammatical Iconicity (GI).

Each circle represents an instance of analysis, fitting in the approximate location a GM is found in its painting-plate, which is represented by a square (following Kress and van Leeuwen, 1996). There are a total of 30 circles in each of the three squares standing for the figurative to the abstract grouping:

Figure 6.5 Patterns of GI in the figurative group

Figure 6.6 Patterns of GI in the semi-figurative group

Figure 6.7 Patterns of GI in the abstract group

Specifically, in Figure 6.5 to 6.7, the red circles stand for the recorded GI of SALIENCY. The blue circles, on the other hand, are the cases for the GI of

MARKEDNESS. According to this visualization of analysis, we have found that the GI zones of SALIENCY almost always fall at the left of the paintings in all styles. In addition, the GI zones of MARKEDNESS almost always fall at the right of the paintings.

If we consider the psychological impact of Saliency, which is already hinted in Kress and van Leeuwen (1996), it is exactly what attracts human vision easily. It is thus reasonable that the GI zones of SALIENCY serves as the starting points, or textual on-sets, in nonrepresentational modern paintings. Meanwhile, we have found a lack of SIZE Grammatical Iconicity application in the MARKED zones. However, they almost always possess a unique textual status, and regularly arranged at the right.

The observation based on our sampling further imply a holistic pattern of

nonrepresentational modern painting textuality: The SALIENCY zones often serve as a start of a painting-text, and the MARKED zones mark, or help direct viewer attention, to the finale of a painting, ending the general flow of the text.

The textual reality of the currently discussed paintings, therefore, not only contains the regularity of GI, but is in fact structurally directed by these form-meaning

alignments. As illustrated, the structurally-arranged GI principles, instead of a mainly semantic-based Information value or a purely perceptual-based visual Saliency (Kress and van Leeuwen, 1996), functions as the determining textual principle of the current visual corpus. The GI patterns outshine the principle of Information value, and direct the textual flow of the paintings from the left (SALIENT ZONES) to the right

(MARKED ZONES), forming a generally left-to-right directionality (Fig. 6.8 to 6.10):

Figure 6.8 Textual flow formed by GI patterns in the figurative group

Figure 6.9 Textual flow formed by GI patterns in the semi-figurative group

Figure 6.10 Textual flow formed by GI patterns in the abstract group

In the model of Grammatical Iconicity, Hiraga (2005:44-45) has put it:

“The relationship of form and meaning in grammatical metaphorical mapping is diagrammatical, because what is preserved is an analogical relationship mediated indirectly by grammatical metaphors. This contrasts with a direct attributive connection such as pure imagic iconicity between the linguistic form and meaning, e.g. a case of onomatopoeia and visual language such as logographs.”

The structural iconicity in Hiraga’s (2005) model is categorized as Diagrammatic

Iconicity in Nanny and Fischer’s Taxonomy of Iconicity (1999), which is dictated by the law of analogy. It is diagrammatic in nature, as it captures structural mappings between poetic textual features and poetic aesthetics. Diagrammatic Iconicity contrasts Imagistic Iconicity, which merely follows the “firstness” (Pierce 1995, see Chapter 2) principle

based on principles of mimicry. Freeman (2009:172-173) once comments that diagram is ‘the abstraction of the structure of image, serving to symbolize the mental processes of creating concepts in the mind’, and also: “When diagram takes on the characteristics

of image, then iconicity at the more abstract level happens”. Through our examination, we have highlighted the existence of Diagrammatic Iconicity, aside from the reported

of image, then iconicity at the more abstract level happens”. Through our examination, we have highlighted the existence of Diagrammatic Iconicity, aside from the reported