CHAPTER 3 A COGNITIVE MULTIMODAL METHODLOGY
3.3.3 Examination 2: Iconicity in Information Flow
For the second part of examining whether paintings constitute ‘a reality of text’, the existence of a flow of textuality is tested among the sampled. In Hiraga’s
Grammatical Iconicity model, the particular rule demands a linear modality trait, and is found non-applicable in two-dimensional pictorials (Chen and Su, 2014). We will compensate this non-applicability by adopting Kress and van Leeuwen’s (1996) visual compositional diagram outlined in 2.4, examining an underlying textual flow in our selected paintings.
We specifically focus on the contrast of Left and Right, because it is where a comparison of linguistic grammar and visual pattern (both in Western languages) could be conducted. Hence, we will mainly observe the horizontal textual flow dictated by Given and New, following Kress and van Leeuwen’s (1996) four-part diagram, aided by the compositions of Saliency and Framing. The overall aspects are shown inTable 3.1:
Steps Source Target Alignment Visual forms Textual meaning Iconicity
Preliminary Figures Semantics/ title --
Examination 1
Similarity Shape Similarity
Iconicity in Grammatical
Metaphors Difference Shape Difference
Closeness Distance Semantic relevance
To recap, in the current multimodal methodology, we combine Hiraga (2005) and Kress and van Leenwaun (1996). The steps include a pinning down of painting semanticity via the help of panting titles, an identification of main figures, an analysis of grammatical iconicity, and finally an examination of the textual flow, or
directionality, of the painting. Our analyses are presented in the following chapters.
Chapter 4
Grammatical iconicity in paintings
“I believe that perceptual form is the strongest, most indispensable means of communicating through works of art. Why should form exist, were it not for making a content readable?”
--Arnheim (1982)
In this chapter, we focus on answering this question: Are nonrepresentational modern paintings in general structured by grammatical metaphors? To find out, we combined the cognitive linguistic model of grammatical iconicity (Hiraga, 2005) as well as criteria for visual saliency (Kress and van Leenwen, 1996).
A total of five grammatical iconic mappings are examined in each painting.
Specifically, the titles of the paintings are analyzed in order to ensure a multimodal meaning of the pieces; also, as explained in Chapter 3, the figure zones serve as the major visual forms analyzed in this study. In the following, the examination of all three painting groups is shown separately in 4.1 to 4.3.
4.1 Grammatical iconicity in figurative paintings
In the Figurative group, the five aspects of iconic grammatical metaphors are generally found highly applicable.
For instance, in Dali’s The lugubrious game (1929) (Fig. 4.1), the theme
suggested by the title could be mapped to the major “figure zone (similarly conveyed in an obvious human shape). The happening of the “game”, as entitled, matches the gestalt area in the middle, which is salient in size, sharpness effect created by depicted details, color and total richness. The combination of these visual factors together creates a highly grammatical iconic status of this painting, in which the saliency of visual forms enhances the textual meaning (Fig. 4.2).
It is also noticeable how the iconic rule MARKED MEANING IS MARKED FORM applied in this painting (Fig. 4.3). Comparing to the other parts of the picture, the blue area shaped like branches (at the centralized bottom) appear protruding—it is painted with bright blue, with is particularly high in color saturation in this picture. It may not be as eye-catching as the SALIENT zone, for it shares a similar color tone to the “ground” layer (the blue sky); also, it is not huge comparing to the main figures.
In the aspect of visual unusualness, even if it is not “news worthy” in semantics, it is perceptually marked. In fact, we could more effectively follow the “strangely
brightened” blueness and further notice to the right of the painting, in which the figure
zone consists of two people with blood. The redness of the blood attunes with the neighboring part, a depiction of a man’s buttock, which is also tainted with blood. The blood is only a detailed part of this painting, yet its visual-perceptual trait as well as its textual symbolicity remains special. Given these reasons, we will mark the zone circling the bright, branch-shaped blue as MARKED MEANING IS MARKED FORM.
Finally, the iconic rule SEMANTIC RELEVANCE IS CLOSENESS is also applied in this painting. The areas with (complete) human characters, specifically, are placed in the same line, rather than arranged randomly amid the picture (Fig. 4.4).
Thus, their similar meaning aligns with their closer placement.
The total number of applied GMs in this painting adds up to10 (Table 4.1), a high applicability of GMs in all 11 grammatical iconicity aspects (cf. Table 3.1). This is a common iconic extent across the figurative samples, with the average
applicability being 9.13. The analysis results of all 30 figurative paintings could be found in Appendix2.
Figure 4.1
The lugubrious game (Dali, 1929)
Figure 4.2 SALIENT zone
Figure 4.3 MARKED zone
Figure 4.4
SEMANTIC RELECVANCE
Visual elements Textual meaning Iconic alignment
Similarity Shape Similarity +
Difference Shape Difference +
Closeness Distance Semantic relevance +
Saliency
Table 4.1 Grammatical iconicity for The lugubrious game
4.2 Grammatical iconicity in semi-figurative paintings
In the Semi-figurative group, Miro’s paintings are always composed of figure zones that are large or more in quantity, with visual contrasts of sharpness, tones or color saturation. It is therefore the case that the Semi-figurative paintings sampled are bountiful in iconic GMs.
Below is a canonical case: In Characters and birds rejoicing at the coming of night (Fig. 4.5), the figure zones stand out, as hinted by both the title and the contrast
created by the dark “ground”. Moreover, the figure zones are greater in size and featured with more details, which intensify the visual complexity or sharpness of
these zones, aiding their textual importance (hinted by the title) in the whole
multimodal text. This fulfillment of SALIENT CONENT IS SALIENT FORM could be easily found throughout all of Miro’s sampled paintings (Fig. 4.6).
The characters again are arranged closely, following the iconic rule SEMANTIC RELEVANCE IS CLOSENESS (Fig. 4.7). The characters are portrayed in a
semi-figurative fashion, but still they display a similarity, so that the elements
representing “humans” are distinguished from his usual sign-like depictions of “stars”
and “birds” (cf. Corbella, 1993). This follows the principle SIMILARITY IN FORM IS SIMILARITY OF MEANING and DIFFERENCE IN FORM IS DIFFERENCE OF MEANING.
Finally, it is reasonable that the SALIENT zones display the brightest color saturation (red with black stripes for the left character, lush green figure in the middle, and the blue at the right). They are also circled by a layer of misty white, which further strengthens the tonal contrast. It is noticeable, however, that the abstracted line-like signs, which are deciphered as the “bird” element in Miro’s art language (Corbella,1993), all point to the right. Moreover, the dynamics of the lines is
accompanied by a specific spot of bright red at the lower-right corner of this picture.
It is a rather unusual combination, given that the same redness is featured at the SALIENT zones, but now reprised at the vectoring end of the lines, as if intended to
“stop” our attention right at the redness, before we finish viewing this picture. We
mark the small zone of redness at the right as MARKED (Fig. 4.8), for it is special in visual trait and also in textual arrangement. The applied GMs in this painting is a bountiful10 in total (Table 4.2), similar to many of our semi-figurative samples. The average number of GMs in our 11-aspect chart is a 9.2 in the current painting group.
Figure 4.5 Characters and birds rejoicing at the coming of night
(Miro, 1968)
Figure 4.6 SALIENT zone
Figure 4.7 SEMANTIC RELEVANCE Figure 4.8 MARKED zone
Similarity Shape Similarity +
Difference Shape Difference +
Closeness Distance Semantic relevance +
Saliency
Table 4.2 Grammatical iconicity of
Characters and birds rejoicing at the coming of night
4.3 Grammatical iconicity in abstract paintings
Even in Abstract paintings, an extent of pictorial grammatical iconicity could be found. The form-content alignment here, however, is not created on a strict semantic basis, but on a perceptual one. Therefore, as noted before, the aspects concerning semanticity fail to apply in abstract paintings, including
SIMILARITY/DIFFERENCE IN FORM IS SIMILARITY/DIFFERENCE OF MEANING, and SEMANTIC RELEVANCE IS CLOSENESS.
In Gray (Fig. 4.9), the title helps guide our reading and interpreting towards the dense mixture formed by the entanglement of wild black lines. With the verbal clue, we might more be effectively drawn to the focus zone of this painting, which is salient in size and perceptual vividness, even if there is hardly any concrete semanticity or real-world reference linked to it (Fig. 4.10).
While SALIENT zones are common in Pollock’s paintings, it is also the case of MARKED zones. Again, it is mainly created on a contrast of visual-perceptual basis, which in turn marks a “textual twist” of a picture. For instance, the MARKED zone of Gray lands at the lower-right, as the rhythms of the black lines shift in the reversed
direction, forming a stronger right-leaning circle (Fig. 4.11). It is linked to the larger gestalt of the SALIENT zone, and constructed in the same color and complexity. The change in arrangement suggests its MARKEDNESS in this picture. The applied GM of Gray, like many of our abstract samples, is 3 in total (Table 4.3), which is lower than the applicability of the figurative and semi-figurative paintings sampled.
Figure 4.9 Gray (Pollock, 1948)
Figure 4.10 SALIENT zone Figure 4.11 MARKED zone
Table 4.3 Grammatical Iconicity for Gray
Visual elements Textual meaning Iconic alignment
Similarity Shape Similarity /
Difference Shape Difference /
Closeness Distance Semantic
relevance /
4.4 Summary of results
An obvious grammatically iconic applicability can be seen throughout our painting corpus. As demonstrated, the sum in the last columns of Table 4.1 to 4.3 represents the extent of GI in each sample. The higher the number, the more grammatically iconic a painting is. So far, the highest GM sum is 10, and the lowest is 3 in our data (cf.
Appendix 2).
An overall statistics could be seen in the table below (Table 4.4). The numbers stand for rounded up percentages of the applicable instances within each painting category.
As suggested by the statistics, the average applicability of GI is highly positive in the selected nonrepresentational modern paintings: There is an average extent of 92.6%
of GI in the figurative group, an average of 85% in the semi-figurative, and an average of 85% in the abstract group, with a total average of 87.4% in the current sample pool.
Painting groups GI principles Ratio
Table 4.4 Applicability of GI in painting groups
Specifically, a slight declining of GI applicability could be observed from the figurative to the abstract. Within the listed examination aspect of GI, the principles of SIMILARITY and DIFFERENCE are sometimes “flouted” in the first two painting groups, e.g. in cases of figure-ground reversion in Dali’s artwork. Moreover, it could not apply in the abstract group due to lack of semantic contrast. Similarly, the iconic
mapping of SEMANTIC RELEVANCE IS CLOSENESS fails to be applied in the abstract group.
The most applicable rules are the ones concerning “quantity and importance”. It has achieved an average of 92.3% applicability across all three groups. Even in the abstract cases, in which often no concrete reference could be found in their titles nor in the pictorials, the up-front eye-catchiness of the figure zones still foregrounding itself as the attention center of the pictorial text.
Also notable is the highly-featured “markedness” GM principle termed as MARKED FORM IS MARKED MEAING in Hiraga (2005). Put simply, the more marked a form, the more marked its textual function. Even in abstract paintings, which are usually referent-free, an especially marked visual focus could be found. The MARKED zones are usually humble in size, but are always marked in visual form and textual placement.
As we reach the end of the first-stage examination on iconicity, the next chapter continues with the second-stage examination.
Chapter 5
Textual flow in paintings
In this chapter, the second research question of this thesis is addressed. We examine whether the sampled nonrepresentational paintings display a general textual flow. The criteria are adopted from Kress and van Leeuwen’s (1996) semiotic model for visual composition. Our investigation to the selected paintings is presented in 5.1(figuratve samples), 5.2 (semi-figurative samples) and 5.3 (abstract samples). In 5.4, the overall results are summarized.
5.1 Textual flow in figurative paintings
Among the three interrelated factors determining visual text compositions, it is often the case that Information value and Saliency play a role in the figurative group, with
Framing mildly hinted (rather than obviously instantiated) in these non-commercial visual texts.
In The persistence of memory (Fig.5.1), the left side of the painting consists of three clocks placed side-by-side, while the right part featuring a horse-like face (or a face-like horse), with a clock on it. The outline of the “horse” extends to the right of the picture, which matches the concept of the title of the painting.
As the left side of the painting leads the flow of the pictorial text to the more conceptual area of the right, this is an instance of the progressing of Given to the New, with the extending image functioning as the “punch line” of this painting, the dynamic, continuing concept of personified time. This arrangement thus attunes with the “left is Given, right is New” information structure reported (Kress and van Leeuwen, 1996).
Thus, we identity the New information zone to be mainly at the right half of this painting, as circled in Fig.5.2.
In this compositional design, the placement of the pictorial items follows the Information value (of Given and New), but also accords with their visual-based Saliency and Framing characteristics: The left and the right zones could be seen as separated by their surrounding background, fulfilling the definition of framing in Kress and van Leeuwen (1996). That is to say, in this painting, and several others in the figurative category, the three reported factors (Information value, Saliency, Framing) co-influence the composition of the whole visual text. This painting, like many figurative instances, applies all three compositional principles listed in Kress and van Leeuwen (1996), as listed in Table 5.1
Figure 5.1 The persistence of memory (Dali, 1931)
Figure 5.2 New info zone for The persistence of memory
Title Compositional principle
The persistence of memory
Info value +
Saliency1 +
Framing +
Sum 3
Table 5.1 Textual composition of The persistence of memory
5.2 Textual flow in semi-figurative paintings
In semi-figurative samples, however, rarely can we find all three compositional principles applied. In the following example entitled Characters in the night (Miro,
1950), the title aid us to identify three characters, a bird, and three pictorial signs for
“stars” in Miro’s invented art language (Corbella, 1993). However, no obvious action is
seen in this painting: It expresses a state, rather than an event. Hence, no clear “punch line” or New information zone can be identified in this painting. Put simply, the mostly discussed compositional principle in the Grammar of Visual Design (Kress and van Leeuwen, 1996), Information value, fails to apply in this semi-figurative painting.
Furthermore, there is also a lack of separation between the figure zones, or even between the figure zones and the ground layer. The Framing principle thus fails to apply in this Miro painting as well.
What stands as the hint on reading this picture, instead, is the bright whiteness in the painting-plate. The white zones form a rhythmic placement, changing in size and angles, creating a visual path within the canvas. Our eyes could more easily follow the white zones while reading the painting, though not necessarily with a certain
directionality. In the category of Miro’s semi-abstract art, the principle of Saliency starts to outshine the other two reported principles in previous studies (Kress and van
Leeuwen 1996; 2006).
Figure 5.3 Characters in the night (Miro)
Painting title Compositional principle
Characters in the night
Info value - Saliency +
Framing -
Sum 1
Table 5.2 Textual composition of Characters in the night
5.3 Textual flow in abstract paintings
In abstract paintings, similar phenomena are observed: Information value and framing do not serve as useful criteria for the textual compositions of the paintings. The remaining principle that hints the possible reading paths of an abstract painting is the compositional principle of Saliency in Kress and van Leeuwen (1996).
In the example entitled Convergence, the verbal cue adds more emphasis to such a
process, in which the dynamics of colors and lines direct our eyes. Again, we may freely
“travel” along the u-shaped visual-perceptual rhythm defined as compositional Saliency
in Kress and Leeuwen (1996).
Figure 5.4 Convergence (Pollock, 1952)
Painting title Compositional principle
Convergence
Info value -
Saliency +
Framing -
Sum 1
Table 5.3Textual composition of Convergence
5.4 Summary of results
After our analyses from 5.1 to 5.3, we are here to outline a summary of results.
In the aspect of textual directionality, what is found in this thesis differs from that reported in Kress and van Leeuwen (1996). The main difference lies in the
low-applicability of the textual principle of Information value and Framing in nonrepresentational modern paintings. In our data, the samples do not rely on
Information value as the determining factor of compositional directionality. We in fact found a continuum of distribution in the three data groups in the diagram below:
Figurative paintings distinguish themselves in terms of Information value, while this feature is almost non-applicable to paintings in the semi-figurative and abstract group.
Similarly, Framing is not applicable to all three styles of Modern paintings. Only the figurative paintings utilized Framing to a great extent in structuring the pictorial elements, while the other two groups declined greatly in, or even abandoned the feature.
Therefore, the most applicable group to Framing is the figurative; the semi-figurative barely applied, and the abstract remains totally non-applicable to Framing.
Contrary to the declining applicability from the figurative to the abstract in Information value and Framing, all the painting styles involve Saliency as a
directionality-related principle. As illustrated, however, a general directionality textual pattern could not be determined by Saliency alone yet.
Figure 5.5 Applicability of compositional principles (Kress et al. 1996) in the current corpus
Figure 5.5 shows how the compositional principles proposed by Kress and van Leeuwen (1996) are applied in our data. The underlying principle of our selected Modern Western paintings appear different from the previous studies, in which Information value and Framing stop to function effectively in determining the textual flow of Modern paintings. Saliency, though helpful, does not reveal a certain textual flow directionality, such as the left-to-right tendency suggested in previous studies. In short, our results revealed that two-thirds of the proposed compositional principles ceased to function in semi-figurative and abstract paintings. In the next chapter, we will address more to this phenomenon.
Chapter 6
“Textuality” in nonrepresentational modern Western paintings
“Is it conceivable that art, as a kind of “writing”, builds a bridge between perception and communication, that it compensates for the
communication system’s inability to perceive?”
-- Luhmann (2000:17)
Our analyses revealed two important findings. First, the selected paintings greatly reflect a pattern of grammatical iconicity. Second, compositions of these
nonrepresentational modern paintings are governed by principles alternative to
nonrepresentational modern paintings are governed by principles alternative to