Chapter 2. Literature Review
2.3 The Difference between Composed and Compiled Soundtracks
Despite the negligence of soundtrack design in educational multimedia, the value of composed soundtracks should not be underestimated. A study by Lipscomb and Kendall (1994) proved that the professionally designed soundtracks do match the film better. The participants viewed 5 different segments in a feature film, Star Trek IV: The Voyage Home and asked them to choose the best fitting soundtrack for each segment. In all the segments, the soundtrack that most frequently selected was the score originally composed for the excerpt. This experiment showed that a soundtrack specifically designed for a film differs from any other music.
However, what element in music makes the composed soundtrack unique remains implicit. We may try to analyze the difference based on the 6 functions of music that induce emotion reaction in a film proposed by Cohen (2001).
Function of Soundtracks Composed Compiled 1. Provision of Continuity v △
2. Direction of Attention v △
3. Mood Induction v △
4. Communication of Meaning v △
5. A Cue of Memory v △
6. Immersion v △
v: function fulfilled △: function fulfilled incompletely Table 2 The function of composed and compiled soundtracks
doi:10.6342/NTU201801179 As shown in Table 2, for the provision of continuity, compiled music could only signals
separation but not continuity. Since all the canned music in one multimedia are with different orchestration, style, mixing, it is hard for canned music to make connections in 2 different scenes or events. For the guidance of attention, a music excerpt must either match the dynamic aspect or the implied meaning of an object. It is difficult for a piece of compiled music to meet this requirement while at the same time fits the plot and emotion of the visual part. For mood induction, Baumgartner et al. (2006) have proved that emotional congruence is a critical factor for music excerpts to support the emotional expression in a film. For communication of meaning, Bullerjahn and Güldenring (1994), Thompson et al. (1994), Cohen (2001), and Boltz (2001) have shown that emotional congruent soundtracks had the power to lead viewers’
interpretation of a film. For memory enforcement, as Boltz (2004) indicated, when the audio and visual dimension in multimedia is integrated, the viewer would remember the film better.
For immersion, since music contributes to the reality of the narrative, the more the music fits the film’s emotion, plot, historical setting, and pacing, the more the viewer can immerse in a film.
One may argue that for function 3, 4, and 5, emotion congruence plays a critical role on whether a piece of compiled music could function could function as well as composed music. This is indeed a question need to be further investigated. Although many of the educational multimedia uses emotional incongruent music as backgrounds, there are still some multimedia in which the musical emotion matches the visual information. In what way the emotion congruence compiled music differs from emotion congruence composed music thus become an issue worth investigating.
doi:10.6342/NTU201801179 2.4 Summary
To conclude, the soundtrack serves as a potentially substantial source of emotional expression in multimedia context. When combined with visual information, music could influence the emotional reaction, the interpretation, and the memory of visual events. However, it seems not all soundtracks can achieve these functions. Composed soundtracks have been proved to match film segments better and could be recognized by the viewer. However, what element in composed music that makes it unique remains implicit; in addition, if the compiled music is emotional congruence, what is the difference between composed and compiled music could be a question need further investigation. The present project strived to contribute to these unknown aspects of soundtrack design in the following chapters.
doi:10.6342/NTU201801179
Chapter 3. Soundtrack Design
In this chapter, the researcher first introduces the soundtrack elements that induce emotions. The audio-visual counterpoint, the relationship between characters and music (Leitmotiv), and the mode of music excerpts that are considered important in soundtrack design, and were further discussed. Following a practice of soundtrack design, including the story characters, theme, as well as the soundtrack design of WaterBob.
3.1 Soundtrack Elements Inducing Emotion
As Lalo Schifrin (2011), a professional score composer stated, the ultimate objective of the soundtrack is to psychologically support the film, no matter through what composing method.
“All the spectrum of emotions and mental states can be expressed by music.”, said Schifrin (2011, page). Many other composers and musicologists have also proposed numerous composing theories to validate how music induce emotion. Many score writters stressed the audio-visual relationship. The symbolic connection between characters and music (Leitmotiv) is as well emphasized. Other subtle elements including the orchestration and the harmony also need to be carefully designed through the composers’ keen intuition, rtistic techniques, and a wealth of experience.
Among all the subtle details, composer Aaron Copland (1945) concluded four elements to listen for in music: rhythm, melody, harmony, and tone color. Other musicologists categorized the “music cues” with a more complex manner (Juslin and Lindström, 2010) (e.g., pitch, mode, melodic progression, rhythm, tempo, sound level, articulation, and timbre). How the music elements associated with the emotion has also been studied. Since all the elements are intercorrelated, it is challenging to determine the emotional induction of an individual music element (Eerola et al., 2013). Nonetheless, most of the studies agree with that mode and tempo
doi:10.6342/NTU201801179 are the most critical factors for emotional reaction (Juslin and Lindström, 2010; Eerola et al.,
2013), although the influence varies among different discrete emotions. As a result, in this paper researcher focus on analyzing the mode, particularly the melody and harmonic structure, in soundtrack design. As for the design of tempo and timbre, although were also carefully dealt with, are not the central issues in this paper due to the limited context.
3.2 The Audio-Visual Counterpoint
As stated by Schifrin (2011), the music composing counterpoint theory could be applied on the audio-visual relationship. The audio-visual relationship could be either parallel or contrary (Figure 1). Parallel counterpoint relationship is waterproof and more conservative, while contrary counterpoint relationship is rather excited and is often used in an ironic or reflective situation. Since most educational media propose critical questions iming to raise environmental awareness, the latter matches more in the case.
doi:10.6342/NTU201801179 Figure 1 The audio-visual counterpoint
doi:10.6342/NTU201801179 3.3 Leitmotiv
Leitmotiv (German, meaning leading motive) is a musical theme that accompanies a specific element in which it appears. A character, a location, or a unique situation in the plot could be accompanied by a leitmotiv to signal its unique meaning in a story. Richard Wargner is typically be regarded as the first one to use this idea. In his opera Ring Cycle, hundreds of leitmotivs were identified. Among all the themes, the most essential leitmotiv that represents the critical concept Rhinegold, repeats in the opera many times throughout the 4 mammoths.
The theme appears in many different variations since the Rhinegold concept changes in the opera. In film industry, Star Wars is considered an indicator for applying leitmotiv. Each character, situation, and location all has unique themes. The theme for the lead role, Luke Skywalker, transformed from noble to sorrow, helps the audience to undergo emotion changes with Luke.
3.4 The Analysis of Mode
There are different ways to analyze the mode in music since the aesthetic values changes all the time. However, the mathematical analysis method used in the 20th century could be easily used to explain most of the music pieces existed. In the following section, Allen Forte’s analyzing method for atonal music (1976) would be applied to elaborate the soundtrack design in the educational digital storytelling the researcher and her team created.
The color of Intervals
All melodies and harmonic structures in a mode, even those most complex ones, could be destructed to intervals. The distance between every 2 notes forms an interval. Since there are 12 different pitches in a sale, there could be 12 different intervals, from 0 semitone to 12 semitones. Trough transformation, the 12 intervals could then be reduced to 7 interval classes (0=0, 1=11, 2=10, 3=9, 4=8, 5=7, 6=6). The class 0 is omitted for practical purpose, leaving 6
doi:10.6342/NTU201801179 intervals classes. As listed below, each interval class has its unique feature and emotional
influences:
Class 1 (1 semitone), the minor second: It is the most disconsonant interval.
Harmonically, it provides a distinctive characteristic and could induce a terrifying feeling. One classic example was the theme song of Jaws composed by John Williams. The melody hovers around the E and F note, providing a feeling that "grinding away at you, just as a shark would do, instinctual, relentless, unstoppable." as described by Williams. In another example, “The Crisis” from The Legend of 1900, Ennio Morricone put a semitone below every mediant in triad. In this case, the minor second provides an anguished feeling symbolizing the tragic romance.
Class 2 (2 semitones), the major second: It is a slightly dissonant, emotionally neutral interval. If serving as the transformation of the minor seventh, it conveys a sense of hope and longing (Schiffrin, 2011). In Leonard Bernstein’s “Somewhere” from West Side Story, the repeating major second expresses the desire for a better future for the lovers.
Class 3 (3 semitones), the minor third: All the third intervals are imperfect consonant.
With a flowing quality, the thirds are often used as the motive power in music. The minor trird is the foundation interval in the minor triad and could express sadness, nostalgia, or the feeling of something lost (Schiffrin, 2011). A great number of examples from Beethoven’s Symphony No.5 to “Summertime” from Porgy and Bess composed by George Gershwin would illustrate the emotion effect brought out minor third. The interval was also used in most of the musicology experiments as an association to sadness.
Class 4 (4 semitones), the major third: It is imperfect consonant, following the unison,
the octave, and the perfect fifth in the order from consonant to disconsonant. The major third is the foundation interval of the major triad and is commonly used to express positive emotions, from delightfulness to ecstasy. Examples including the major third (for example, Johann
doi:10.6342/NTU201801179 Strauss II’s Op.314 “The Blue Danube”) were commonly used in musicology experiments as
an association with happiness.
Class 5 (5 semitones), the perfect fourth: It is once considered consonant in the
medieval polyphony stage since it formed the basis for music composition. Nonetheless, when the major and minor triads become the core element in music composition, the perfect fourth is then be considered disconsonant. Nowadays, if the perfect fourth serves as the transformation of the perfect fifth, it is considered consonant; otherwise, it is disconsonant. It provides a neutral, steady feeling with sometimes a crystal clear texture. It was used at the beginning of the "Bridal Chorus" from Wagner's Lohengrin.
Class 6 (6 semitones), the augmented fourth: Containing a tritone, the augmented fourth
is the most intensive dissonant interval. Called “diabolus in musica” (the devil in music) in the Middle Ages, this interval contains overwhelming emotion that cannot be suppressed. It also has a strong tendency that leads to the perfect fifth. “Maria” from West Side Story (Leonard Bernstein) illustrated the burning desire provided by the augmented fourth.
The “Interval Vectors”
Introduced by Martino in 1961, the description of all the intervals in one combination of pitches is called an interval vector. Any pitch combination could be denoted as [ic1ic2ic3ic4ic5ic6] (ic is the abbreviation for the total number of a specific interval class). For example, the interval vector for the major triad is [001110] since there is 1 minor third (class 3), 1 major third (class 4), and 1 perfect (class 5).
The interval vector is like the color palette for a musician. With the interval vector, the color and emotion of a pitch combination could be further analyzed. For example, if the proportion of dissonant intervals is relatively more substantial, the emotion expression could also be more intensive.
doi:10.6342/NTU201801179 3.5 Practices of Soundtrack Design
Our team joined On-line to On-site Water Education (O2O), a project of Ministry of Education. In O2O, audio books and games were created to deliver environmental concept about water resource preservation. Among them, the audiobook WaterBob was chosen to be used as a practice in this paper.
The theme of WaterBob is about limited water resource (full content in Appendix). The lead character, WaterBob, is the representation of either physical water, water storage facility, or water resource agency. Three other supporting characters relatively represent different water consumption parties: the Mouse (civil livelihood water), the Cat (industrial water), and the Rabbit (agricultural water).
WaterBob’s duty was to collect water resource and distribute water resource to the 3 water consumers. In return, WaterBob got energy candy from the consumers. However, water demand increased. The priority of water distribution among different consumer parties became an issue. Moreover, water supply decreased due to droughts and pollution. WaterBob tried to negotiate with water consumers, yet none of them accepted getting less water or paying more energy candy. WaterBob was exhausted and could not supply water anymore. Not until then did the water consumers felt regret, but it was too late. WaterBob would never come back again.
As shown in Table 3, the structure of the story could be divided into 4 main sections and additional opening and ending section. In section 1, WaterBob distributed water to the consumers; in section 2, water demand increased and water supply decreased; in section 3, WaterBob tried to negotiate but was refused; In section 4, the consumers were regretful, but WaterBob would never come back. There is no plot in the opening and ending section. Only the soundtrack and a few pictures leads the viewers into the story in the beginning and last longer after the end of the story.
doi:10.6342/NTU201801179 Opening Section 1 Section 2 Section 3 Section 4 Ending
/ WaterBob
distributed water to the consumers
Water demand increased and water supply decreased
WaterBob tried to negotiate but was refused
WaterBob would never come back
/
Table 3 Sections of WaterBob
3.6 Soundtrack Design of WaterBob
Music theories were applied to design the soundtracks of WaterBob. Although there were many aspects about soundtrack design, in this research we focus on the audio-visual relationship of the video and the main theme of each character.
The audio-visual counterpoint
The full audio-visual counterpoint relationship is shown in Figure 2. In the opening section, there is no plot but a picture of WaterBob bathing in the water. Music is the dominant element leading the viewers to immerse into the story, so the music is expressive and with an unclosed ending chord that arouses curiosity. Section 1 is the introduction of the story and the main characters, so there is not much tension in the music. Section 2 is where the main crisis arises, so the music twist dramatically to alert the audience. In section 3, the crisis keeps worsening. The tension in visual presentation and the audio narrative is high enough, so the sense of the presence of music decreases, leaving space for the visual and narrative information.
In section 4, a critical issue is proposed. There is no music at all so the viewers will not be distracted. In the ending section, again there is no visual information. Nonetheless, that is when the viewer reflects what they have learned from the story. So the expressive music in the opening is used again to make the audience last longer in the immersion and think deeper.
doi:10.6342/NTU201801179 Figure 2 The audio-visual counterpoint of WaterBob
The leitmotivs and mode analysis
Leitmotiv could be applied to various elements: the characters, the environment, and even the meaning of the story. In WaterBob, however, Objective View Story (OVS) was applied, and personified objects are created to describe environmental issues from the object’s view. In this case, the characters are the most essential element that should be emphasized.
Therefore, the leitmotivs for each character were thus designed.
The instrument is designed according to the characteristic of each character, and the melody and harmony vary according to the plot and emotion expression in each section (full score in the appendix).
doi:10.6342/NTU201801179 WaterBob
The lead character WaterBoB is a tender figure with curvy and crystal appearance. The soft and round timbre of the vibraphone is thus used to represent WaterBob. The leitmotiv for WaterBob is a 4-note descending scale, illustrating the flow of the river. The semitone between G and F sharp provides a little swaying feeling. As the story develops, the mode changes from major to minor and the number of interval classes 1 and 6 (minor second and augmented fourth) increases to reflect the stressful situation faced by WaterBob.
Figure 3 WaterBob and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Interval Vectors
Soft Tender
Vibraphone Electric piano
Water replenish Peaceful [101220](major form)
Insufficient
water Worried [101220](minor
form) [122230]
[212221](minor form)
WaterBob dying Pitiful / Table 4 The design of leitmotiv of WaterBob
doi:10.6342/NTU201801179 The Mouse
The Mouse is a tiny bathing lover. A melody played by flute is just suitable to show the agility of the mouse and the association with water. The number of interval classes 1 and 6 (minor second and augmented fourth) increases to show the anger of the Mouse.
Figure 4 The Mouse and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression Interval Vector Small
Bathing lover Flute solo Clarinet Strings
Water delivery Delightful [001110](major form)
Water delivery Delightful [001110](major form) Refusing more
payment Helpless [202222]
[001110](minor form)
Table 5 The design of leitmotiv of the Mouse
doi:10.6342/NTU201801179 The Cat
Being the first customer asking for more water supply, the Cat plays the role of antagonist in the story. The bassoon is thus used to reflect the unctuous, conning characteristic of the Cat. The leitmotiv of the Cat has more disconsonant intervals than the other characters at the beginning of the story. The most dramatic harmony (with the most disconsonant intervals) appears in the second section where it is the turning point of the story.
Figure 5 The Cat and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Water delivery Delightful [114112]
[111111]
Demanding more water Refusing more
payment
Helpless [114112]
[323331]
[111111](minor form)
Table 6 The design of leitmotiv of the Cat
doi:10.6342/NTU201801179 The Rabbit
The guitar was chosen to show the pastoral feature of the Rabbit. The six-eight time rhythm illustrate the bouncy motion of the Rabbit. The most intensive plot happens in the second section where the Rabbit takes the lead to shout at WaterBob, so the interval vector contains more disconsonant intervals.
Figure 6 The Rabbit and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Interval Vector
Pastoral Bouncy
Guitar Strings Vibraphone
Water delivery Delightful [001110](major form) [011110](major form) Refusing less
water Anger [001110](minor form)
[211132]
[012120]
[402122]
Refusing more
payment Helpless [111120](minor form)
[010101]
Table 7 The design of leitmotiv of the Rabbit
doi:10.6342/NTU201801179 The Main Theme
The video addressed a critical question: “How can we reach a balance between water supply and different demand customers?”. This question was not pointed out by any character, scene, nor plot, yet the audience could sense it within the story. A theme song was composed to emphasize the question and was placed at the beginning and the ending sections of the video.
The theme song provided a sad atmosphere, with an open chord that gives the story a reflective feeling.
Instrument Section Plot Emotion Expression
Interval Vector
Electric piano The Beginning The Ending
Reflective Sad
[001110] (minor form) [001110] (major form) [001110] (major form) [122230]
Table 8 The design of leitmotiv of the main theme
doi:10.6342/NTU201801179
Chapter 4. Methodology
The present research investigates the effects of soundtracks inducing emotion on environmentally responsible behavioral intentions. Thirty-six college students were randomly assigned into two groups. While watching the video, one group listened to composed soundtracks, whereas another group listened to compiled soundtracks downloaded from online
The present research investigates the effects of soundtracks inducing emotion on environmentally responsible behavioral intentions. Thirty-six college students were randomly assigned into two groups. While watching the video, one group listened to composed soundtracks, whereas another group listened to compiled soundtracks downloaded from online