Chapter 4. Methodology
4.2 Participants
In the first stage of the experiment, recruiting advertisements was posted on PTT, indicating the purpose, procedure and potential participants of the study with compensation of NTD 250 per test. The screening process was then conducted to select two similar groups to examine the intervention of composed soundtracks vs. compiled ones. In doing so, a self-developed online questionnaire using Google Form was released to select proper participants.
The questionnaire includes 2 parts corresponding to the purpose of the present study, that is, to examine participants’ emotional induction of soundtracks in educational multimedia toward water conservation issues. The first part of the questionnaire contains 4 questions.
Every 2 questions measured Environmental Awareness and Music Awareness with 4 points grading scale (From 1 point, extremely disagree, to 4 points, extremely agree).
The Questions are listed below:
Environment Awareness
1. In general, I am aware of environmental issues.
2. I practice water conservation in daily life.
Music Awareness
1. Music influences my mood and feeling.
2. I am aware of the soundtracks while watching a video.
The average grade of questions 1 and 2 is then be defined as “Environment awareness”, while the average of the grade of questions 3 and 4 be defined as “Music awareness”.
doi:10.6342/NTU201801179 The second part of the questionnaire asked the participants to provide emotional
reactions towards video and music materials. The video material was The Beans and the Water Fairies, a three-minute water environment educational video developed by our team with
composed music. The music material was an audio file composed of four music section download online (similar to the compiled soundtracks used in quantitative data collection) (music credit to Purple Plant
http://www.purple-planet.com and Youtube Audio Library https://www.youtube.com/audiolibrary/music).
From the result of the screening questions, 36 subjects out of 100 respondents were selected and randomly divided into two groups, and the grade of Environment Awareness, Music Awareness, their emotions, and the sex ratio were balanced in both groups (Table 10).
Group Group Table 10 Grouping Result
doi:10.6342/NTU201801179 4.3 Data Collection
To serve the purpose of research, documenting tool as well as the following measurement tools were adopted to collect data. A visualization tool build up with Tableau was also utilized to collect and further analyze data.
Interactive Response
Since there are many sections in WaterBob inducing different emotional reactions, it is necessary to document the interactive responses from the viewers while watching the digital story.
Emotion Categoriess
Within literature, emotions are often classified into two categories: positive or negative emotions (Hinton et al., 2008; Schutz et al., 2006; Boltz, 2004). Although the emotional expressions in the segments of WaterBob are rich, in the experiment the researcher conclude all the expressions into 3 emotion types: positive, negative, and mixture (Figure 7). Although many musicology experiments use measurements that categorizes emotion to more than 3 categories (for example, the Geneva Emotion Wheel contains 20 emotion types), they are too complicated for this research. Since the interactive response is asked to be provided while watching the film simultaneously, the viewer may be distracted from the film when responding their emotion if there are too many emotion types to choose. With just three categoriess to choose, the distraction would not be slight and could be neglected.
doi:10.6342/NTU201801179 Figure 7 The emotion categories of WaterBob
Documenting Tool
A website (Figure 8) produced by HTML and JavaScript was designed for the respondents to record their emotional reactions while watching the material simultaneously.
As shown below, 3 buttons (positive, negative, or mixture) were placed below a video player screen. Users could easily record their emotional reactions and the triggered times.
Figure 8 The documenting tool
doi:10.6342/NTU201801179 Visualization Tool
An Excel file was specifically designed for the experiment. With the format prepared in advance, the data recorded from the documentary tool could be presented in figure immediately. The researcher could recognize the unexpected emotional responses right after the participants finishing watching the material, so the indication of those unexpected emotional responses could then be clarified during the interview.
For further analysis of the data, Tableau Public was used as the visualization tool. Tableau Public allows the researcher to analyze multidimensional data with various calculation functions and to present the data.
4.4 Experiment Design and Data Analysis
The purpose of the study is to exam whether composed soundtrack could significantly induce expected emotional responses. To answer the research question one, the simple t-test, also known as the independent samples t-test, was used for comparisons with a continuous dependent variable. In our case, participants were invited to watch two videos, composed vs.
compiled ones. Simple t-test in SPSS 23 was used to provide full details of the t-test calculation, including sample mean, the sum of squares, and standard deviation.
4.5 Procedure
Upon arriving at the lab, the participants watched WaterBob with either composed or compiled soundtracks. Participants’ emotional responses to soundtracks triggered by educational multimedia, the WaterBob, were collected. That is, interactive emotional reactions (positive, negative, or mixture) toward the material were asked to be provided simultaneously.
After watching WaterBob, follow-up interviews were held to understand the details behind their responses. A set of questions were asked to know what the participants feel about the
doi:10.6342/NTU201801179 visual and audio information. The meaning of the emotional responses gathered in the
qualitative investigation was also specified in the interview.
Interactive Emotional Responses
The participants viewed the material individually. A Macbook and a mac earphone were used as the media. Before the investigation, the participants received the following instructions explaining task requirement:
Please press the buttons (Positive, Negative, or Mixture) which represent your emotional reaction while watching the video.
Positive: including but not limited to exciting, entertaining, delightful, passionate, encouraging...
Negative: including but not limited to nervous, angry, sad, fearful, guilty...
Mixture: all feelings that cannot be categorized into Positive nor Negative Follow-up interviews
Since the emotion responses could only be categorized into 3 types, the exact meaning is vague. Thus, an interview is needed for the researchers to further explain the quantitative data. A self-designed question set was used in the interview. The question set can be divided into 3 parts: questions about the video and music, questions about the specific meaning of particular emotion reactions, and general questions. The first and second part of the questions were used to explain the quantitative result. The general questions were designed to get a clear view about what people think about the educational multimedia nowadays. Since in the experiment only one material was used, the researcher was interested in the participants’
opinion and feelings toward other educational multimedia.
doi:10.6342/NTU201801179 Question set
Questions about WaterBob
Do you remember any particular plot or music?
Is there any particular plot or music that you like/ dislike? Why?
What do you think about the plot design?
What do you think about the music design?
Questions about emotion reactions
Unexpected emotion reactions will be identified. The participants would be asked to explain what they feel that emotion at that time.
General questions
Have you ever watched education multimedia before?
If yes, could you briefly describe the theme and presentation of the multimedia?
How many detail could you still remember?
What’s your feeling and reflection after watching that multimedia?
Is the multimedia knowledge driven or emotion driven? with real person or in animation?
Do you remember the background music in the multimedia?
Do you think the visual and audio matched each other?
doi:10.6342/NTU201801179
Chapter 5. Results and Discussion
The researchers analyzed the emotion induction between the two groups and found a significant difference. Through a simple t-test, a significantly higher environmentally responsible behavioral intent was suggested in the intervention group than in the control group.
The follow-up interview also showed a positive relation between emotional induce and environmentally responsible behavioral intentions.
5.1 The Heat Map of Emotional Induction
The emotional responses from individual participants are shown in Figure 9. Each strip represents one button hit from the participants. All the button hits were further gathered and summarized in Figure 10, which illustrates a more explicit emotion flow while watching the video.
Since the emotional settings for music in both groups, composed vs. compiled soundtracks, are the same, we can see that the overall emotional response pattern from the 2 groups is similar. However, with closer examination, it is found that there are some regions in which the participants answered unexpected emotional responses.
To further explain the details and specific phenomenon, the participants were asked to explain what is the meaning of those unexpected emotional. These follow-up interview data were used to further analyze the click data exhibit in the heat map. First, researcher recognized some irrelevant data; for example, two participants pressed “negative” when Mouse, one of the main characters in WaterBob, first came out. They reported that the reason they did that was because they were afraid of mice. This emotional response is not related to music design nor storyline. Some other participants reported negative responses about the narration or the breaking time were also identified and removed.
doi:10.6342/NTU201801179 Composed Soundtrack Group
Compiled Soundtrack Group
Figure 9 The heat map of emotional induction
doi:10.6342/NTU201801179 Composed Soundtrack Group
Compiled Soundtrack Group
Figure 10 The emotional flow
doi:10.6342/NTU201801179 After removing all the irrelevant data, all the unexpected emotional responses were
summarized. Three particular sections in which the experiment group hits were identified as unexpected emotion (Figure 11). To be specific, the plot of the three sections are: (1) the Cat asked for more water supplement, (2) WaterBob cannot find water, and (3) the consumers felt regret about what they had done.
Figure 11 The unexpected emotion responses
doi:10.6342/NTU201801179
41
From Figure 11 we can see that the hitting frequency varies in participants. To eliminate the influence caused by this difference, we use the below formula to normalize the data.
C = ∑ (&'× )*
+ × )') (Equation 1) P = .
)* (Equation 2) Where:
C = count of unexpected emotion P = Percentage of unexpected emotion
ui = unexpected emotion response in the region (individual) et = emotion response in the region (total)
ei = emotion response in the region (individual) N = group size
After eliminating the influence of differences on the count of emotion, the result is shown in Table 11. In the composed music group, the count of unexpected emotion in the three sections are 4.82, 0.00, and 0.00, all lower than those in the compiled music group which are 14.08, 9.95, and 7.30, with t=3.02, 2.89.2.53, respectively, all of p<.001. For the percentage of unexpected emotion, in the composed music group are 3.30%, 0.00%, and 0.00%, all lower than those in the compiled music group who are 9.58%, 5.38%, and 5.29%. Both the count and percentage showed that the accuracy of emotional reaction is higher in the composed music group.
doi:10.6342/NTU201801179 for more water supplement
Table 11 The count and percentage of unexpected emotion (t-test)
Discussion
There are three sections in which the experiment group hits more unexpected emotion.
From the interview, we found that in these sections the participants interpreted the story falsely;
for example, in section one, participant no.4 reported “I felt happy for the Cat since it got more orders”; in section two, participant no.89 reported “I thought the polluted river is a hope for WaterBob”; in section three, participant no.19 and no.89 reported “I thought the animals were self-reflecting so maybe good things could happen”.
doi:10.6342/NTU201801179 From the interview we may conclude that in the three sections, the “accuracy” of
emotion triggered by the download soundtrack is lower than the composed soundtrack. Instead of following our designed storyline, the participants may interpret the story in their own way and ended up have false understandings of the story. To find the reason why unexpected happened mostly in these particular three regions, we further analyze the plot. In region one, Cat asked for more water supply, which is a turning point in the whole story; in region two, WaterBob could only find polluted water supply; in region three, the animals were self-reflecting, so it’s when the viewer could think about what they learned in the story.
All of the plots were meaningful, with complex emotion, and could be interpreted variously. In regions like these three, composed music could guide the viewer’s emotion, ensuring they interpret the story the same way we want them to; in contrast, the download music could not support the plot as well as the composed music, thus could not guide the viewer’s emotion as good as in the other group.
Another feature these regions share in common is that they all contain complex emotion.
Emotion is one of the most concerned factors in video production. In the previous section we have also mentioned that emotion setting of music is one of the control factors; however, even the website has already categorized the music into 20 types (upbeat, dance, cool, relaxing, reflective, sneaky, funky, cute, easy, creepy, rock, retro, jazz international, drama, tense, dark horror, and mysterious), far more than the three emotion type in our experiment design, we still could not find music that precisely fits the complex emotion combination in some of the plots.
5.2 Soundtrack Design Issues and Principles
Most participants considered the soundtracks as an essential factor in digital stories or educational multimedia. Participant no.4 and no.46 stated: “With the support of soundtracks, I felt easier to immerse in stories and thus understand the content better.” Participant no.25
doi:10.6342/NTU201801179 provided an even stronger statement: “Soundtracks are like catalyst that support the multimedia
even when the viewer is not aware of the audio information.” Participant no.45 also indicated:
“Without soundtracks, I would not be willing to watch multimedia with long duration or complex content.”
However, corresponding to the problems stated in chapter one, participants show negative attitude toward the soundtracks in multimedia that they have viewed. As participants no.17 and no.89 stated, “No soundtrack would be better than low-quality soundtracks.” From the participants’ feedback on the general questions in the interview, the researcher concludes the following three issues in soundtrack design nowadays which are emotional incongruence, modal incongruence, and lack of originality.
Emotion incongruence
Participant no. 89 stated, “I had watched one multimedia in which the soundtrack provided a happy feeling when the characters were fighting with each other. I felt that soundtrack is very wired and distracting.” Participants no.18, no.37, no.46, and no.18 also indicated: “There are hardly any soundtracks in educational multimedia that provide negative emotion expression.” In other words, the soundtracks used in multimedia generally provide positive emotion expression, which could not fit in with negative plots.
Modal incongruence
Some other participants indicated that the style of visual and audio information do not match each other. Participant no. 17 and no.65 stated: “Some of the soundtracks I’ve heard are very outdated, making the film seemed like produced 30 years ago.” Participants no.64 stated:
“It feels awkward to me when educational multimedia use soundtracks with vocal and lyrics.”
doi:10.6342/NTU201801179 Lack of originality
Almost none of the participants could provide a clear description about soundtracks in educational multimedia. Although some of the participants could try to describe the soundtracks when encouraged by the researcher, their statements are “the soundtracks are similar to those in movies.” (no. 64), “It was just some soft music.” (no.18 and no.72), and “I think I have heard similar music in TV shows.” (no.99). Rather than their feeling or emotion, the participants use soundtracks in other multimedia to describe the educational soundtracks, indicating the soundtracks are lack of originality and infection.
Elements for Soundtrack Design
From the participants’ feedback, five essential factors were thus concluded for soundtrack design (Table 12). According to participants no.45 and no.64, the viewers are more likely to distract from the multimedia when the duration is long, more transitions in the soundtrack are thus needed to assist audience better immersed in the plot. Transitions are also important for multimedia with complex content as stated by participants no.65 and no.66.
Soundtracks with positive emotional expression may also help understand complex content in educational multimedia (participants no.4, no.37, and no.65). Participants no.42 and no.45 stated that the texture of music should be more complicated if the plot is predictable, while the soundtrack(s) should be simple when important questions are provided by the multimedia to allow reflection and critical thinking. These statements correspond to the assumption that contrary audio-visual counterpoint should be applied on educational multimedia. Another audio element, the audio narration, should also be considered in soundtrack design. That is, the more emotion and words in the narration, the less should the soundtrack be. Finally, the viewing occasion would also influence soundtrack design. When viewing the multimedia alone or online, the emotional influence of the soundtracks is more essential than viewing in class.
doi:10.6342/NTU201801179 Elements of
multimedia Soundtrack Design Principles
Duration More transitions are needed in the soundtracks when the duration of the film is longer.
The difficulty of the
content (a)When the content of the film is challenging, more transitions are needed in the soundtracks.
(b)Positive emotional expression in the soundtracks helps the viewers understand challenging content.
Predictability of the plot
The texture of the music should be more complex if the plot is easy to predict.
Amount of the audio narration
The more the audio narration is, the less should the soundtracks be.
Viewing occasion When viewing the multimedia alone or online, the emotional influence of the soundtracks is more critical than viewing in class.
Table 12 Elements of soundtrack design
doi:10.6342/NTU201801179
Chapter 6. Conclusion
In the present study, the researcher developed a novel experiential intervention with a purpose to extend our understanding of the emotional induction effects of soundtracks in educational multimedia. Specifically, the researcher developed soundtracks based on composing theories. In addition, one of the unique contributions is the introduction and interpretation of audio-visual counterpoint, the relationship between characters and music (Leitmotiv), and the mode of music.
Based on our preliminary analysis exhibit on heat maps, two results converged. First, compared to compiled soundtracks, intentionally design ones significantly induced emotions in most sections and plots. Second, intentionally design soundtracks are meaningfully going along with digital stories, especially when they are designed based on composing theories. In addition, the follow-up interview also indicated that most of the participants recognize the value of adding soundtracks to educational multimedia, and further proved to be an effective strategy for promoting intention to actively engage in pro-environmental activities.
A lingering question is the connection between each response to different types of educational multimedia. Specifically, if the scenarios and plots changed, the soundtrack design method remain unclear. Limitations of the experiment design were noted in two aspects: the sample selected and the frequency of intervention. The sample, consisting of 67% female college students, could limit the generalizability of the results. Gender may cause differences in their emotional responses. Indeed, the measures process and calculating of interactive emotional responses were designed arbitrarily by the researcher and research team, that could be further elaborated. To yield further insight into the effectiveness of the water conservation in the context of environmental education, future research should recruit a broader range of participants from different populations. In addition, the outcomes of repeated interventions, e.g.
composed soundtracks going along with different types of digital stories, and measurements of