Chapter 3. Soundtrack Design
3.2 The Audio-Visual Counterpoint
As stated by Schifrin (2011), the music composing counterpoint theory could be applied on the audio-visual relationship. The audio-visual relationship could be either parallel or contrary (Figure 1). Parallel counterpoint relationship is waterproof and more conservative, while contrary counterpoint relationship is rather excited and is often used in an ironic or reflective situation. Since most educational media propose critical questions iming to raise environmental awareness, the latter matches more in the case.
doi:10.6342/NTU201801179 Figure 1 The audio-visual counterpoint
doi:10.6342/NTU201801179 3.3 Leitmotiv
Leitmotiv (German, meaning leading motive) is a musical theme that accompanies a specific element in which it appears. A character, a location, or a unique situation in the plot could be accompanied by a leitmotiv to signal its unique meaning in a story. Richard Wargner is typically be regarded as the first one to use this idea. In his opera Ring Cycle, hundreds of leitmotivs were identified. Among all the themes, the most essential leitmotiv that represents the critical concept Rhinegold, repeats in the opera many times throughout the 4 mammoths.
The theme appears in many different variations since the Rhinegold concept changes in the opera. In film industry, Star Wars is considered an indicator for applying leitmotiv. Each character, situation, and location all has unique themes. The theme for the lead role, Luke Skywalker, transformed from noble to sorrow, helps the audience to undergo emotion changes with Luke.
3.4 The Analysis of Mode
There are different ways to analyze the mode in music since the aesthetic values changes all the time. However, the mathematical analysis method used in the 20th century could be easily used to explain most of the music pieces existed. In the following section, Allen Forte’s analyzing method for atonal music (1976) would be applied to elaborate the soundtrack design in the educational digital storytelling the researcher and her team created.
The color of Intervals
All melodies and harmonic structures in a mode, even those most complex ones, could be destructed to intervals. The distance between every 2 notes forms an interval. Since there are 12 different pitches in a sale, there could be 12 different intervals, from 0 semitone to 12 semitones. Trough transformation, the 12 intervals could then be reduced to 7 interval classes (0=0, 1=11, 2=10, 3=9, 4=8, 5=7, 6=6). The class 0 is omitted for practical purpose, leaving 6
doi:10.6342/NTU201801179 intervals classes. As listed below, each interval class has its unique feature and emotional
influences:
Class 1 (1 semitone), the minor second: It is the most disconsonant interval.
Harmonically, it provides a distinctive characteristic and could induce a terrifying feeling. One classic example was the theme song of Jaws composed by John Williams. The melody hovers around the E and F note, providing a feeling that "grinding away at you, just as a shark would do, instinctual, relentless, unstoppable." as described by Williams. In another example, “The Crisis” from The Legend of 1900, Ennio Morricone put a semitone below every mediant in triad. In this case, the minor second provides an anguished feeling symbolizing the tragic romance.
Class 2 (2 semitones), the major second: It is a slightly dissonant, emotionally neutral interval. If serving as the transformation of the minor seventh, it conveys a sense of hope and longing (Schiffrin, 2011). In Leonard Bernstein’s “Somewhere” from West Side Story, the repeating major second expresses the desire for a better future for the lovers.
Class 3 (3 semitones), the minor third: All the third intervals are imperfect consonant.
With a flowing quality, the thirds are often used as the motive power in music. The minor trird is the foundation interval in the minor triad and could express sadness, nostalgia, or the feeling of something lost (Schiffrin, 2011). A great number of examples from Beethoven’s Symphony No.5 to “Summertime” from Porgy and Bess composed by George Gershwin would illustrate the emotion effect brought out minor third. The interval was also used in most of the musicology experiments as an association to sadness.
Class 4 (4 semitones), the major third: It is imperfect consonant, following the unison,
the octave, and the perfect fifth in the order from consonant to disconsonant. The major third is the foundation interval of the major triad and is commonly used to express positive emotions, from delightfulness to ecstasy. Examples including the major third (for example, Johann
doi:10.6342/NTU201801179 Strauss II’s Op.314 “The Blue Danube”) were commonly used in musicology experiments as
an association with happiness.
Class 5 (5 semitones), the perfect fourth: It is once considered consonant in the
medieval polyphony stage since it formed the basis for music composition. Nonetheless, when the major and minor triads become the core element in music composition, the perfect fourth is then be considered disconsonant. Nowadays, if the perfect fourth serves as the transformation of the perfect fifth, it is considered consonant; otherwise, it is disconsonant. It provides a neutral, steady feeling with sometimes a crystal clear texture. It was used at the beginning of the "Bridal Chorus" from Wagner's Lohengrin.
Class 6 (6 semitones), the augmented fourth: Containing a tritone, the augmented fourth
is the most intensive dissonant interval. Called “diabolus in musica” (the devil in music) in the Middle Ages, this interval contains overwhelming emotion that cannot be suppressed. It also has a strong tendency that leads to the perfect fifth. “Maria” from West Side Story (Leonard Bernstein) illustrated the burning desire provided by the augmented fourth.
The “Interval Vectors”
Introduced by Martino in 1961, the description of all the intervals in one combination of pitches is called an interval vector. Any pitch combination could be denoted as [ic1ic2ic3ic4ic5ic6] (ic is the abbreviation for the total number of a specific interval class). For example, the interval vector for the major triad is [001110] since there is 1 minor third (class 3), 1 major third (class 4), and 1 perfect (class 5).
The interval vector is like the color palette for a musician. With the interval vector, the color and emotion of a pitch combination could be further analyzed. For example, if the proportion of dissonant intervals is relatively more substantial, the emotion expression could also be more intensive.
doi:10.6342/NTU201801179 3.5 Practices of Soundtrack Design
Our team joined On-line to On-site Water Education (O2O), a project of Ministry of Education. In O2O, audio books and games were created to deliver environmental concept about water resource preservation. Among them, the audiobook WaterBob was chosen to be used as a practice in this paper.
The theme of WaterBob is about limited water resource (full content in Appendix). The lead character, WaterBob, is the representation of either physical water, water storage facility, or water resource agency. Three other supporting characters relatively represent different water consumption parties: the Mouse (civil livelihood water), the Cat (industrial water), and the Rabbit (agricultural water).
WaterBob’s duty was to collect water resource and distribute water resource to the 3 water consumers. In return, WaterBob got energy candy from the consumers. However, water demand increased. The priority of water distribution among different consumer parties became an issue. Moreover, water supply decreased due to droughts and pollution. WaterBob tried to negotiate with water consumers, yet none of them accepted getting less water or paying more energy candy. WaterBob was exhausted and could not supply water anymore. Not until then did the water consumers felt regret, but it was too late. WaterBob would never come back again.
As shown in Table 3, the structure of the story could be divided into 4 main sections and additional opening and ending section. In section 1, WaterBob distributed water to the consumers; in section 2, water demand increased and water supply decreased; in section 3, WaterBob tried to negotiate but was refused; In section 4, the consumers were regretful, but WaterBob would never come back. There is no plot in the opening and ending section. Only the soundtrack and a few pictures leads the viewers into the story in the beginning and last longer after the end of the story.
doi:10.6342/NTU201801179 Opening Section 1 Section 2 Section 3 Section 4 Ending
/ WaterBob
distributed water to the consumers
Water demand increased and water supply decreased
WaterBob tried to negotiate but was refused
WaterBob would never come back
/
Table 3 Sections of WaterBob
3.6 Soundtrack Design of WaterBob
Music theories were applied to design the soundtracks of WaterBob. Although there were many aspects about soundtrack design, in this research we focus on the audio-visual relationship of the video and the main theme of each character.
The audio-visual counterpoint
The full audio-visual counterpoint relationship is shown in Figure 2. In the opening section, there is no plot but a picture of WaterBob bathing in the water. Music is the dominant element leading the viewers to immerse into the story, so the music is expressive and with an unclosed ending chord that arouses curiosity. Section 1 is the introduction of the story and the main characters, so there is not much tension in the music. Section 2 is where the main crisis arises, so the music twist dramatically to alert the audience. In section 3, the crisis keeps worsening. The tension in visual presentation and the audio narrative is high enough, so the sense of the presence of music decreases, leaving space for the visual and narrative information.
In section 4, a critical issue is proposed. There is no music at all so the viewers will not be distracted. In the ending section, again there is no visual information. Nonetheless, that is when the viewer reflects what they have learned from the story. So the expressive music in the opening is used again to make the audience last longer in the immersion and think deeper.
doi:10.6342/NTU201801179 Figure 2 The audio-visual counterpoint of WaterBob
The leitmotivs and mode analysis
Leitmotiv could be applied to various elements: the characters, the environment, and even the meaning of the story. In WaterBob, however, Objective View Story (OVS) was applied, and personified objects are created to describe environmental issues from the object’s view. In this case, the characters are the most essential element that should be emphasized.
Therefore, the leitmotivs for each character were thus designed.
The instrument is designed according to the characteristic of each character, and the melody and harmony vary according to the plot and emotion expression in each section (full score in the appendix).
doi:10.6342/NTU201801179 WaterBob
The lead character WaterBoB is a tender figure with curvy and crystal appearance. The soft and round timbre of the vibraphone is thus used to represent WaterBob. The leitmotiv for WaterBob is a 4-note descending scale, illustrating the flow of the river. The semitone between G and F sharp provides a little swaying feeling. As the story develops, the mode changes from major to minor and the number of interval classes 1 and 6 (minor second and augmented fourth) increases to reflect the stressful situation faced by WaterBob.
Figure 3 WaterBob and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Interval Vectors
Soft Tender
Vibraphone Electric piano
Water replenish Peaceful [101220](major form)
Insufficient
water Worried [101220](minor
form) [122230]
[212221](minor form)
WaterBob dying Pitiful / Table 4 The design of leitmotiv of WaterBob
doi:10.6342/NTU201801179 The Mouse
The Mouse is a tiny bathing lover. A melody played by flute is just suitable to show the agility of the mouse and the association with water. The number of interval classes 1 and 6 (minor second and augmented fourth) increases to show the anger of the Mouse.
Figure 4 The Mouse and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression Interval Vector Small
Bathing lover Flute solo Clarinet Strings
Water delivery Delightful [001110](major form)
Water delivery Delightful [001110](major form) Refusing more
payment Helpless [202222]
[001110](minor form)
Table 5 The design of leitmotiv of the Mouse
doi:10.6342/NTU201801179 The Cat
Being the first customer asking for more water supply, the Cat plays the role of antagonist in the story. The bassoon is thus used to reflect the unctuous, conning characteristic of the Cat. The leitmotiv of the Cat has more disconsonant intervals than the other characters at the beginning of the story. The most dramatic harmony (with the most disconsonant intervals) appears in the second section where it is the turning point of the story.
Figure 5 The Cat and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Water delivery Delightful [114112]
[111111]
Demanding more water Refusing more
payment
Helpless [114112]
[323331]
[111111](minor form)
Table 6 The design of leitmotiv of the Cat
doi:10.6342/NTU201801179 The Rabbit
The guitar was chosen to show the pastoral feature of the Rabbit. The six-eight time rhythm illustrate the bouncy motion of the Rabbit. The most intensive plot happens in the second section where the Rabbit takes the lead to shout at WaterBob, so the interval vector contains more disconsonant intervals.
Figure 6 The Rabbit and the leitmotiv
Characteristic Instrument Section Plot Emotion Expression
Interval Vector
Pastoral Bouncy
Guitar Strings Vibraphone
Water delivery Delightful [001110](major form) [011110](major form) Refusing less
water Anger [001110](minor form)
[211132]
[012120]
[402122]
Refusing more
payment Helpless [111120](minor form)
[010101]
Table 7 The design of leitmotiv of the Rabbit
doi:10.6342/NTU201801179 The Main Theme
The video addressed a critical question: “How can we reach a balance between water supply and different demand customers?”. This question was not pointed out by any character, scene, nor plot, yet the audience could sense it within the story. A theme song was composed to emphasize the question and was placed at the beginning and the ending sections of the video.
The theme song provided a sad atmosphere, with an open chord that gives the story a reflective feeling.
Instrument Section Plot Emotion Expression
Interval Vector
Electric piano The Beginning The Ending
Reflective Sad
[001110] (minor form) [001110] (major form) [001110] (major form) [122230]
Table 8 The design of leitmotiv of the main theme
doi:10.6342/NTU201801179
Chapter 4. Methodology
The present research investigates the effects of soundtracks inducing emotion on environmentally responsible behavioral intentions. Thirty-six college students were randomly assigned into two groups. While watching the video, one group listened to composed soundtracks, whereas another group listened to compiled soundtracks downloaded from online musical service platform, and simultaneously they were asked to provide their emotional reactions (positive, negative, or mixture) by pressing buttons. The data were collected by a researcher self-developed documenting tool and further analyzed by visualization tool.
4.1 The Educational Multimedia - WaterBob
Two sets of soundtracks, one is researcher self-composed while another is compiled one downloaded from online musical service platform, were added onto WaterBob, a digital storybook previous created by Yang (2017), from the same research team. WaterBob, a 10-minute digital storybook, was created in hopes of increasing learners’ awareness of water conservation. In the composed group, the accompanying soundtracks were composed by the researcher following composing theories as described in chapter three; in the other group, the soundtracks were downloaded from online Creative Common music resources (music credit to Purple Plant
http://www.purple-planet.com and Youtube Audio Library https://www.youtube.com/audiolibrary/music).
Since emotional congruence has been demonstrated to be an influential factor, the emotional expression of each download music excerpt was close to the corresponding composed soundtracks. Although there was plenty music in the CC resources, it is still challenging to find music excerpts expressing exactly the same emotion. Nonetheless, the
doi:10.6342/NTU201801179 instance, two composed music segment for Cat triggered complex emotion (shown in Table 9),
so the researcher could only use download music that was as close to the composed music as possible.
Section Subsection Composed music Compiled music
Opening / Reflective Reflective
Section 1 Mouse 1 Delightful Delightful
Cat 1 Delightful
Queer
Delightful Magical
Rabbit 1 Delightful Delightful
WaterBob 1 Peaceful Peaceful
Mouse 2 Delightful Delightful
Section 2 Cat 2 Queer
WaterBob2 Worried Worried
Section 3 Rabbit 3 Helpless Helpless
Cat 3 Helpless Helpless
Mouse 3 Helpless Helpless
WaterBob 3 / /
Section 4 Reflection / /
Ending / Reflective Reflective
Table 9 The emotion expression of WaterBob
doi:10.6342/NTU201801179 Having dealt with the emotion congruence issue, the remaining differences between the
two music groups were the music style, the orchestration, and the space of mixing, and whether leitmotivs were designed. These difference further influenced the cohesion of music segments and the coherence of the story between the two groups.
4.2 Participants
In the first stage of the experiment, recruiting advertisements was posted on PTT, indicating the purpose, procedure and potential participants of the study with compensation of NTD 250 per test. The screening process was then conducted to select two similar groups to examine the intervention of composed soundtracks vs. compiled ones. In doing so, a self-developed online questionnaire using Google Form was released to select proper participants.
The questionnaire includes 2 parts corresponding to the purpose of the present study, that is, to examine participants’ emotional induction of soundtracks in educational multimedia toward water conservation issues. The first part of the questionnaire contains 4 questions.
Every 2 questions measured Environmental Awareness and Music Awareness with 4 points grading scale (From 1 point, extremely disagree, to 4 points, extremely agree).
The Questions are listed below:
Environment Awareness
1. In general, I am aware of environmental issues.
2. I practice water conservation in daily life.
Music Awareness
1. Music influences my mood and feeling.
2. I am aware of the soundtracks while watching a video.
The average grade of questions 1 and 2 is then be defined as “Environment awareness”, while the average of the grade of questions 3 and 4 be defined as “Music awareness”.
doi:10.6342/NTU201801179 The second part of the questionnaire asked the participants to provide emotional
reactions towards video and music materials. The video material was The Beans and the Water Fairies, a three-minute water environment educational video developed by our team with
composed music. The music material was an audio file composed of four music section download online (similar to the compiled soundtracks used in quantitative data collection) (music credit to Purple Plant
http://www.purple-planet.com and Youtube Audio Library https://www.youtube.com/audiolibrary/music).
From the result of the screening questions, 36 subjects out of 100 respondents were selected and randomly divided into two groups, and the grade of Environment Awareness, Music Awareness, their emotions, and the sex ratio were balanced in both groups (Table 10).
Group Group Table 10 Grouping Result
doi:10.6342/NTU201801179 4.3 Data Collection
To serve the purpose of research, documenting tool as well as the following measurement tools were adopted to collect data. A visualization tool build up with Tableau was also utilized to collect and further analyze data.
Interactive Response
Since there are many sections in WaterBob inducing different emotional reactions, it is necessary to document the interactive responses from the viewers while watching the digital story.
Emotion Categoriess
Within literature, emotions are often classified into two categories: positive or negative emotions (Hinton et al., 2008; Schutz et al., 2006; Boltz, 2004). Although the emotional expressions in the segments of WaterBob are rich, in the experiment the researcher conclude all the expressions into 3 emotion types: positive, negative, and mixture (Figure 7). Although many musicology experiments use measurements that categorizes emotion to more than 3 categories (for example, the Geneva Emotion Wheel contains 20 emotion types), they are too complicated for this research. Since the interactive response is asked to be provided while watching the film simultaneously, the viewer may be distracted from the film when responding their emotion if there are too many emotion types to choose. With just three categoriess to choose, the distraction would not be slight and could be neglected.
doi:10.6342/NTU201801179 Figure 7 The emotion categories of WaterBob
Documenting Tool
A website (Figure 8) produced by HTML and JavaScript was designed for the respondents to record their emotional reactions while watching the material simultaneously.
As shown below, 3 buttons (positive, negative, or mixture) were placed below a video player screen. Users could easily record their emotional reactions and the triggered times.
Figure 8 The documenting tool
doi:10.6342/NTU201801179 Visualization Tool
An Excel file was specifically designed for the experiment. With the format prepared
An Excel file was specifically designed for the experiment. With the format prepared