EFL Students’ Views on Writing
with Pictures, Sound, and Music
Yu-Hsiu Huang
Southern Taiwan University of Science and Technology yuhsiu@mail.stust.edu.tw
Abstract
The purpose of this research is to explore college English as foreign language (EFL) students’ views on using texts, images, sound, and music to create multimodal projects in writing. The important factors, as well as the strategies students use in completing their multimodal writing projects are also investigated. The subjects consisted of 45 university EFL students from two writing classes in a university in southern Taiwan. They were requested to write stories and essays using a multimodal software program called Photostory. Students’ views on their experiences using the media were collected through interviews and reflection sheets. In general, the students considered multimodal composition to be interesting and engaging, and reported that it gave them a sense of achievement. Elements students need to consider in completing the project include: sentence production, story structure, text/picture design (position and semantic relation), and display speed. The better works of students include the following features: more complete sentences; well-organized story structures; clear text/picture color contrast; coherence in text font, color, size, position; sentence number; appropriate display speed; and good picture quality throughout the films. Students used a variety of text, picture, and music strategies to convey meanings and create the appropriate effects for their stories. Factors that may affect their composition experiences are computer access, technical problems, topics, and multimodal effects. Future study could focus on the contribution of multimodal composition on students of different levels.
Key Words: computer-assisted language learning (CALL), English as foreign language (EFL), multimodal writing
INTRODUCTION
In the 21st century, communication and language use has come to be closely connected to information technology (Chapelle, 2001). As computer technology advances, nonlinguistic modes become more important in human communication. A large number of new forms of communication media, thus, have come to join the previously existing computer-mediated communication (CMC) and World Wide Web (WWW) in human communication and so have created new requirements at work. Non-linguistic modes such as pictures, videos, and sound effects, in blogs, websites, emails, and presentations, have become essential in daily personal and work related communication. To succeed in the modern world, workers need to develop both English proficiency and multimedia communication skills, and familiarities with non-linguistic modes of expression to accomplish their tasks. In response to the emerging trend mentioned above, many English as foreign language (EFL) educators and learners have developed creative ways of using information technology in English teaching and learning. In order to help students cope with their future job requirements, it is important for language educators to understand the nature and the possibilities of multimodal communications and its applications for English language teaching.
One common problem faced by many EFL writing teachers is students’ low motivation in English writing. Some students write creatively, but others write briefly and hastily, or just copy texts they find on the Internet. The level of transformation, and therefore, the level of learning is limited. On the other hand, most students seem to enjoy the web-based, interactive, multimedia, task-oriented reading
and writing activities (online games, blogging, Facebook, and so on). Nowadays, many teachers have found that as long as students have access to computers and the Internet in the classroom, they look at texts and photos on various blogs and Facebook pages, or play online games. Teachers and traditional paper-and-ink texts are no competition for the colorful, multimodal, and interactive online texts and games. Therefore, rather than fighting a losing battle against these socio-technological trends, English as second language (ESL) teachers would be wise to develop new multimedia tasks that incorporate multiple communication elements to transfer writing into a more interesting and engaging task.
The present study aims to explore (1) the positive and negative experiences students encounter using Photostory in English multimodal composition, (2) the knowledge and skills of multimodal composition students develop through the multimodal writing project, and (3) the strategies students use to create multimodal effects. The findings and results from this study will provide EFL teachers with insights into the uses of multimedia and technology in English composition.
LITERATURE REVIEW
The rapid advancement of computer technology, the new possibilities of computer-mediated communication, the World Wide Web, and the multimedia technology have all transformed the notions and practices of reading and writing (Warschauer, 1996). In developed countries, the computer screen has joined print as the medium of literacy as there is an increasing amount of writing and
reading taking place with computers. Much of the writing in academia, business, and government is practiced at computer screens and sent to others via networked computers (Warschauer, 2000). An increasing amount of reading is also done at the screen, including email and web browsing.
In recent years, visual and sound effects have emerged alongside the verbal expressions in communication. Non-linguistic forms of communication have become increasingly more popular, with many people choosing to or even being required to create multimedia texts for work and for personal communication purposes (Andrews, 2001; Nelson, 2006). Computers provide new possibilities for people working with knowledge and information; at the same time, the fact that so much on-line communication and screen reading/writing is in English also requires new skills of English learners and users (Slaouti, 2002). Warschauer (2000) suggests that it is no longer sufficient for writers to put grammatically correct sentences on paper. Hence, aside from basic computer typing and operating skills, new types of writing/authoring skills are required. Learners now have to develop competencies in integrating texts, images, and audiovisual materials into a multimedia presentation and use effective strategies to communicate a message in various circumstances of computer-mediated communication (Warschauer, 2000).
Multimodal composition includes not only writing, but a “design,” a new form of composition that involves the combination of text, images, and sound (Kress, 1998, 2003). Kress (1998) refers to a tendency termed synaesthetic semiosis, a process through which meanings are presented by multiple co-present semiotic modes that
interact with each other to create new forms of meaning. Kress (2003) also identifies the co-operation concepts: transformation and transduction, which refer to the reshaping of semiotic resources within and across modes, respectively. Learning is often undertaken through the transformation of one “text” into another, for instance, hearing words and writing them down. When students rewrite the plot of a novel in a diary entry, the text is processed by the brain in the transformation. On the other hand, transduction takes place when meanings are transferred through different media. Andrews (2001) points out that the value of technology lies in the new modes of “text production” as learning occurs via the information processing through the transformation and transduction of “texts” (Andrews, 2001). What multimedia technology allows students and teachers to do is to make the transitions between many more different channels of communication and between different genres and forms (Andrews, 2001). Since different modes have different organizing logics and different affordances, they also demand certain epistemological commitments of the user (Kress, 2008). English learners need to be exposed to them and helped to take advantage of the resources provided by the computer technology so that they can enter new discourse communities and conduct their practice of work, which are increasingly computerized.
Studies looking at ESL/EFL multimodal composition suggest a number of affective and cognitive advantages. Multimodal composition provides ESL writers with tools for sharing knowledge, displaying self expression, projecting cultural identities, and expressing emotional connections with their written topics (Shin & Cimasko, 2008). Research on students’ use of PowerPoint also shows
that writers display their disciplinary and individual identities through image, page layout, background design and color (Tardy, 2005). Research by Nelson (2006) on ESL digital story writing suggests that multimodal communication helps amplify authorship, and as students recreate a text across modal boundaries, their stories tend to convey a deeper, fuller quality of meaning through the synaesthetic process, that is, transduction. In addition, multimodal composition may help students notice the invisible or inaudible semiotic features of language. Nelson (2006) notes that multimodal composition can serve to increase the possibilities of learners’ emergent knowledge and positively affect their intellectual and affective development. Thus, multimedia tools make it easier to create a vivid text and represent self with texts, images, and sound (Nelson, Hull, & Roche-Smith, 2008).
Findings from school multimedia projects also suggested that using visual images can engage children in learning English (Walton, 1995). In addition, computer aided projects support learners’ collaboration, creativity, independent learning, reflection, working towards real and authentic purposes, pupils acting as experts, and talk for learning (Tweddle, 1997). However, even if students are familiar with website design, they do not automatically become productive designers. It has been suggested that students should write using a variety of tools and find real audiences to promote authentic interaction and exchange so writing in school can appear like a dialogue and become more engaging for students (Andrews, 2001). Moreover, when writing multimodal texts, students have been found to be affected by the disciplinary traditions and follow the rules of “supposed to be,” thus, hindering their creativity. In addition,
students’ past experiences and their ideas about writing may also affect their multimodal composition (Shin & Cimasko, 2008). Linguistic modes may dominate the discourse as students maintain a traditional approach and hesitate to incorporate visual and sound elements because of their views of academic discourse (Shin & Cimasko, 2008).
Verbal expressions are no longer the only mode English teachers have to deal with. Of course, combining communication elements and resources for self expression is not a small task. Multimodal composition should not only be viewed as simply a matter of displaying technical skills (Sheppard, 2009). The complex choices regarding audience, technology, and content have certain effect on the presentation of text and the reception by readers (Sheppard, 2009). Hence, collaboration of art or music with language and multimedia design requires training (Andrews, 2001). However, as of present, educators are not likely to have deep enough skills and understanding of the use and instruction in new technological forms, and they need to develop and understand the affordances of language and other modes in order to help students appreciate both the constraints and possibilities of multimodal composition.
As previous studies have pointed out, multimodal composition helps to amplify authorship and allows writers to present disciplinary and individual selves, convey deeper and more engaging forms of meaning through multiple modes, helps EFL students notice the semiotic properties of language, and, finally, positively affects individuals’ intellectual and affective development. More practical guidelines for both teachers and students for doing multimodal projects, especially on the text/picture, music, sound design, are thus
needed. Many studies on multimedia and EFL writing (as mentioned above) also concentrate on the interaction between verbal and visual modes. Other non-linguistic modes such as music and sound are often overlooked. It is important that both teachers and students understand the advantages and disadvantages of new technologies and multimodal composition in order to reach more meaningful education outcomes. Therefore, the following research questions will guide this study.
1. What are the positive and negative experiences students encounter using Photostory in English multimodal composition?
2. What knowledge and skills of multimodal composition do students develop through the multimodal writing project? 3. What are the strategies students use to create multimodal
effects?
METHOD
In order to have a thorough understanding of students’ perceptions of multimodal writing, qualitative research methods were adopted in this study. The data sources of the study include interviews with students, students’ reflection sheets, and the multimodal texts created by students.
Participants
The participants include 24 first-year and 21 second-year writing students in a university in southern Taiwan. Among the 45 students, four are male and 41 are female. The students were
conveniently chosen as they agreed to take part in the research by writing with picture and sound editing software. All of them reported they had never used the picture and sound editing software to compose picture stories before.
Procedure
The procedure of the writing process and research is as follows. First, the first-year students were taught the basic English sentence and paragraph writing skills that include: sentence structures, sentence combination skills, and paragraph structures. The second-year students were taught the essay writing skills and were introduced to different writing genres. Both groups of students were required to hand in assignments each week. Topics ranged from descriptions of a person, an event, a memorable trip, a place they like, comparisons of people, cultures, cities, and festivals, to discussions of the advantages and disadvantages of modern technology or policies.
Second, students were introduced to the multimedia software “Photostory.” Photostory is a software program made by Microsoft. Photostory was chosen because users can conveniently type in text, insert images, add music, and record voices1
1 The voice recording function is not offered in PowerPoint software. It can be
downloaded from: http://www.microsoft.com/windowsxp/using/digitalphotography/ PhotoStory/default.mspx
to create their own digital stories. After downloading the software, students can start a new story by inserting pictures into new pages. Then, corresponding texts, music, and voice recordings are added into each page. Users can choose special effects for their images; they can also choose fonts, sizes, colors, and positions for texts as well as music from their own
collections. Stories are built with pages consisting of images, transcripts, and voice recordings. Users can also click and drag a page to adjust the order of pages. After finishing the story, students can preview to see if the story is satisfactory or save the story for later viewing.
Third, students were told that their work would be evaluated in terms of content, coherence, language (fluency and accuracy), the quality of overall multimedia design, and consideration of audience, purpose, and appropriateness. The nonlinguistic modes were to be assessed based on the students’ ability to complement and enhance the text.
Fourth, instruction was given in both basic grammar and in elements of visual and sound design, such as, line, color, shape, size, distance, perspective, texture, tone, tempo, and artistic style.
Fifth, students chose their best assignments and presented their work with Photostory twice a semester (in weeks 8 and 17). Sixth, interviews were conducted after students completed their works. Finally, students filled out the reflection sheets.
Research Instruments
The present research used semi-structured interviews, reflection sheets, and document analysis for data collection. Students were interviewed in the midterm and final week. At the beginning of the interview, the researcher described the purpose of the study. The interviews were conducted in Chinese so that interviewees would be able to understand the directions of the interviewer. The quotations were later translated from Chinese to English. Students were encouraged to give their own views and each student was given ten to
20 minutes to talk about their writing experiences. The following questions are the interview prompts used to elicit students’ opinions about wiring.
1. How do you feel about using texts, images, sound, and music in your composition project?
2. What do you think about the writing software?
3. What are the advantages and disadvantages of multimodal writing?
4. What are the elements that affect your multimodal writing? 5. What strategies do you use with integrating texts, pictures,
sound, and music?
6. Do you have problems completing the project?
Besides interviews, students were also invited to fill out the reflection sheets in the final week. Prompt questions on reflection sheets included: (1) What do you think about making Photostory films? (2) Were there any challenges or interesting experiences? (3) Did you make any progress in the second project with the content, text, image, music, or structure? (4) Did you notice any changes to your works?
Data Analysis
The researcher transcribed the resulting recorded interviews. After the transcription of each interview, the researcher categorized and analyzed the content of the transcription and the reflection sheets. Participants’ viewpoints and suggestions were chosen based on their relevancy to the research questions. The interview transcripts and the reflection sheets were read carefully to find out themes and categories related to the positive and negative experiences, advantages,
disadvantages, elements, and strategies of multimodal writing. Each student was given a unique code: S1, S2, S3, and so on. Then, students’ multimedia films were watched and analyzed by the researcher to examine their strategies of dealing with texts, sound, and music. Furthermore, the multimodal effects students created were also analyzed.
RESULTS
The results are presented in three parts: (1) positive and negative experiences students encounter using Photostory in English multimodal composition; (2) knowledge and skills of multimodal composition students develop through the multimodal writing project; and (3) strategies students use to create multimodal effects.
Positive and Negative Experiences Students Encounter Using Photostory in English Multimodal Composition
Developing a sense of achievement. Students had positive
responses to the experiences of Photostory composition and many of them report that doing the project and watching their completed films gives them a sense of achievement. The experience of finding suitable pictures and music for their projects is reported as being fun and interesting.
It may take me more time to do it, but generally speaking, it is like my own project. I had great fun doing it. (S 064, interview)
Being able to play with sound and images also gave me a sense of self development and accomplishment. (S 110, reflection sheet)
It is a new program. After I finished my work, I felt a great sense of achievement. (S 112, interview)
I think I have made some improvements with the content and sound effects. Learning to use the software may be very helpful for me in the future. (S110, interview)
I am satisfied with my work and happy to share it with my classmates. (S 074, interview)
I am satisfied with my work because I put effort into creating it. (S 097, interview)
Multimodal effects. The multimodal composition was praised
by students as multimedia software allowed them to create multimodal effect by transforming the cold and plain texts into something lively and different, which was a new writing experience for most of them.
The combination of text, pictures, and music turned the boring text alive and made it interesting. It just takes you a little bit more time to find the suitable picture, images, and music, then your text becomes very different!! (S 101, reflection sheet)
Multimedia composition also creates special effects and arouses emotions, which affect both the creator and the viewers. Students also reported that using Photostory allowed them to introduce themselves and express their creativity as other modes enhance the author’s sense of self-expression. Some students recorded their life events and then shared the films as presents for their friends. The multimodal presentation was reported as more appealing than plain text, and the viewing experiences can be affective, as students reported they themselves were “touched and moved when watching the films.”
It is fun making my own essay into a film. With the music and pictures, I can show the meaning of my essay better and more easily. (S 103, interview) Photostory is quite useful. I made one with pictures taken with my friends, and then sent it to my friend as a present. (S 094, interview)
My topic is college life, so I added many pictures I have taken since I came to Tainan. It is like a record of my life ... when I saw my work, I was touched. I prefer Photostory. Other media only show texts, with Photostory, you can add pictures and music. (S 123, reflection sheet)
I felt a sense of achievement after I finished my project. When I saw my last picture, I just felt “it’s over? So fast!”(S 061, interview)
Word is plain and boring, it just shows text. With Photostory you can show pictures, texts, and music. The feeling is different. (S 061, reflection sheet)
I feel my Photostory work is full of emothions .... I used red as my background color. My topic is New Years. Red symbolizes happiness. I also chose that kind of music. (S 109, interview)
It is fun using Photostory because you can create something and introduce something about yourself. (S 118, reflection sheet)
Transformation and transduction. Learning takes place
through transformation, while new forms of meanings are created through transduction. In the present study, students first had to transfer paragraphs and essays into subtitles, then search for and/or design pictures to fit the texts. The act of finding relevant pictures to match the texts also seems to help students develop a better understanding of the topic. For example, in a story about the similarities and differences between the United Kingdom and Taiwan, a student (S 115) compared the locations, foods, religions, cultures, habits, and means of transportation of the two countries. The act of searching and designing corresponding images, for example, finding two maps indicating the locations of the two countries in relation to the rest of the world (see Appendix Pictures 1 and 2), enriched the student’s understanding of the topic and promoted meaningful learning and intellectual development. Students writing about scenic spots or tours in foreign countries also made efforts to find related picture showing cultural information, for instance, the royal weddings held in Saint’s Cathedral, the special interior design of historical monuments, the history of the Tate museum, and so on (Appendix, Picture 3).
It was also reported that deciding on images and music can also stimulate brainstorming and motivate students to write more. Students reported that when they can look at the pictures, they have more ideas to write about. As they had to describe the scenes and details in the picture, they were prompted to elaborate upon the images and ideas represented with more sentences and words. Listening to the music also helped them to recall memories and find inspiration.
It is like telling the story from the picture and I like it. (S 101, reflection sheet)
It was interesting. I can listen to the music and compose my essay. It is more effective. With the pictures, I am clearer about what I will write. (S 080, interview)
I see the picture and then I know what I am going to write about. I take the pictures out and arrange them according to time order, then I think about what to write. (S 073, interview)
When I see the pictures, I can think of more sentences. Pictures give me more ideas. For example, cats usually eat their food on the ground. I took a picture of my cat drinking milk from a milk bottle. People do not know that cats can drink from a milk bottle. So, with a picture like that, I can describe the scene. (S 126, interview)
Pictures provided concrete ideas and details for students to write about. Similarly, listening to music helped them to become more emotionally engaged with the topic. According to some students,
listening to music during the composition process was inspirational. Music also helped to arouse emotions when students watched the completed films. Moreover, transformation took place when students had to transfer their essays into subtitles. Transduction took place when they had to transfer the meanings among pictures, sound, music, and texts. Both transformation and transduction occurred during the process of multimodal writing and were found to be effective in enhancing attitude, stimulating ideas, organizing writing, and inspiring students to produce works of greater length.
Difficulties in multimodal writing. On the other hand, some
students complained that finding the relevant pictures and modifying them for the story was difficult and time consuming. They had to spend time searching for appropriate images and music; editing images and text; and arranging the order.
It takes me three to four, or four to five hours to make the project, and I also spent one to two hours trying to improve and edit the film. (S 058, interview)
Computer and technical problems. Most students considered
Photostory useful and user-friendly. However, technical problems such as problems with downloading, project saving, dealing with computer breakdowns, and lack of computer access were reported by students. Computer and equipment access was reported by some as a problem as well.
I do not have microphone at the computer, so I can not record my own voice. I only added the background music. It is a pity. (S 123, interview)
Though most students considered the software useful and were satisfied with the functions offered, some students compared Photostory to PowerPoint and thought PowerPoint offers better functions and special effects. For instance, the variety of fonts available in the software was also considered insufficient by some students (e.g., S 098). Some students also reported that, without careful design, the texts tended to block the images (Picture 4 in Appendix), and thus damaged the overall aesthetic effect. Students were also reluctant to record their voices, either because of their lack of confidence or because of their consideration of artistic effect.
Students’ computer skills and familiarity with the software.
Their unfamiliarity with the functions of the software was problematic for some students. Students often reported that they spent a lot of time figuring out the functions of the software the first time they used it, but the operation process became easier when they did it the second time. Students who consider themselves less competent in computer skills tend to report they experience more difficulties in Photostory composition.
It took a lot of time and efforts to make a Photostory. In the beginning I would plan and imagine something nice, but in the end my work did not show the result .... It is my first time using the software .... I am actually a computer idiot. (S 058, interview)
However, most students reported that as they became more familiar with the software, they were able to better exploit its various functions and manipulate it appropriately in order to develop the style, design, and fonts necessary to suit their needs.
The software was actually rather easy. I learned to use the functions and made some progress. What I did the second time I used it is better than what I did the first time. (S 097, interview)
The styles of my two projects are very different. The second time I was more familiar with the process, and could do things faster. I knew what I needed to do and made my design accordingly. (S 109, reflection sheet) The second time, I knew how to use it, so the design became more sophisticated. (S 118, reflection sheet)
Knowledge and Skills of Multimodal Writing Students Develop Through the Multimodal Writing Project
Elements students needed to consider when producing multimodal projects include text design, picture design, picture order, display speed, and topics. Through writing with Photostory, students developed awareness of multimodal elements and skills of multimodal composition.
Text. Elements students must consider during text design
include: deciding the number and length of the sentences shown on the page, breaking up sentences, deciding the color, size, font, and position of the texts. An important step in making the project is to cast paragraphs and essays into films subtitles. Most students based their videos on the essays or paragraphs they had written for their weekly assignments. The essays (by the second year class) are four to five paragraphs long, on average, and the paragraph (by the first year class) usually consists of four to five sentences. It is generally rather
difficult to transfer a whole essay into a Photostory film. Some students presented the whole essay on one single page, but some students adopted a different approach, segmenting their essay to various degrees, even going so far as to break a sentence down over several different pages, as one sentence may contain several different ideas that can be presented with different images.
I spent a lot of time doing the project. I had to consider how to break up the sentence so that they go well with the pictures. (S 058, interview)
A common problem found when analyzing students’ films is that students sometimes inserted too many sentences on one page, which made the page look crowded and messy. Viewers also find it difficult to read so much linguistic information on a page.
On the other hand, while most students typed in complete sentences on the page, some students preferred to type in only key words instead of complete sentences. For instance, a film by one student contained either no words at all or just one key word on each page. The page looked empty and awkward as the student left out the linguistic element and relied on the picture to tell the story. However, the student explained that she intended to create an artistic style with only pictures and music.
Aside from writing with incomplete and run-on sentences, some students tend to ignore capitalization and punctuation marks. Even though the instructor read and corrected students’ essays before they started the project, spelling and grammar errors can still be found in some students’ texts.
Pictures. Pictures play an important role in multimodal
composition. Elements of image design include: size, color, sequence, lightness, and style. Aside from writing English texts, students also had a chance to display their understanding of visual arts, and some made very creative choices in image design when creating multimodal texts. Finding relevant pictures is considered an important process in the composition experience and can affect students’ emotion. Nonetheless, finding suitable pictures to match the content was reported by some as difficult. The results showed mixed situations.
The most important point in making Photostory is to find relevant pictures. It is sometimes difficult to find them. (S 096, interview)
When I found the picture to match my essay, I felt really glad about it. (S080, interview)
Finding pictures is more troublesome. You have to find the picture with the correct size so the screen will not go blurry. (S 098, reflection sheet)
It is more troublesome to find pictures and I don't know what pictures go well with my texts. (S 075, reflection sheet)
Students also found that abstract and narrow topics cannot be easily expressed with appropriate pictures.
Maybe my topic (my favorite food) is more difficult. I only have one kind of favorite food. It is not that easy to express it with pictures. (S 118, interview)
On the other hand, some students preferred writing with plain text and consider finding pictures troublesome. One of the problems found is that there is sometimes little relevance between pictures and texts. A student might conveniently find pictures on the Internet and quickly copy and paste them on the story without carefully considering the relevance to their texts. For instance, one student wrote that the weather in Taipei is colder and it rains more often than in Tainan (where the students attend school). However, the picture she showed was a snowing scene taken in a western country. Another student wrote about going to a supermarket to shop for food, but the picture she used showed a supermarket in the US. One other student wrote about the Confucian temple in Tainan, but the picture she used shows the nine turtles in Chihkan Tower, a different historical site in the city (Pictures 5 and 6 in Appendix).
Picture order. Picture order and coherence are elements
students need to consider in a multimodal project so that the subjects, styles, and qualities of pictures connect or follow in a natural and sensible way.
The problem I faced was I did not know how to design the page and how to make the story go smoothly. The second time I tried, I became better when deciding the order of the pictures. I was still troubled with finding pictures but the process became smoother. (S 053, interview)
With the topic of narrative, students find it easier to arrange pictures according to the time order.
My topic is the summer camp. I will arrange the pictures from the beginning to the end, chronologically. It is like telling a story. (S 106, interview)
Text and picture design. Text and picture relation is considered
one of the important elements in Photostory composition. Students have to consider how to place the sentences and words in each page to achieve the overall multimodal effect.
The text and picture position of the multimodal composition is also an important issue. Aside from the sentence number, students have to consider the position, size, color, and font of the text so as to best match the background picture. Arranging text and picture position is considered a challenge as students report that the texts may block the picture and the picture quality may also affect the clarity of the text. Some students might have too many sentences and the whole page becomes cluttered with words. Some might just quickly type in the text without considering the aesthetic effects, so the text blocked the faces of the people or main objects in the pictures.
I will think about text fonts—how big should they be so they will not block the picture. Do the color of my texts and the background picture go well together? And also colors and lightness—is the picture too bright or too dark? (S 073, interview)
Coherence in position, size, color, font of the text, and text/picture design was essential to achieving the overall effect, and when well executed enhanced the film quality. Hence, coherence of the text position was another issue that should be considered. In many students’ works, the sentences might be at the bottom of the first page
but may appear in the middle of the second page and then finally showing up at the top of the third page. It appears that there was no real purpose or consideration for design.
It was found that works reached a higher level of quality when the students paid attention to each individual element as well as the interrelation of the elements. The characteristic of good works include: complete and correct sentences, clear color contrast between the picture and text, as well as coherent text/picture design with appropriate text position. What generally works the best is one to two sentences on a page, with each sentence placed at the bottom of the page or in the space without blocking the main objects in the picture, with the color, font, and style of the text remaining the same throughout the film. The videos are also greatly enhanced by having good quality of voice recording and matching music.
The three pictures (Pictures 7, 8, and 9 in Appendix) were taken from the work about the differences between scooters and motorcycles by a student whose work displayed the features of good sentence and image design. Only one or two sentences appear on each page, without blocking the main object of the page. Texts and images supplemented each other to achieve the goals of meaning making and pleasant aesthetic effect
Display speed. The control of the display speed is also another
element students had to learn. Some of the films ran too fast, so the audience did not have enough time to read the text, but sometimes a page with only one key word paused for a long time before turning to the next page.
Sometimes the pages go too fast and I cannot catch up with the speed. (S 098, interview)
Projects of better quality also show coherence in the display speed with each page showing up for five to seven seconds without awkwardly long pauses or, conversely, flipping by too fast.
The second time, I learned how to control the speed of display. I will use it in other reports and projects. (S 123, interview)
Topic. The ability to select a good topic also affects students’
multimodal composition experiences. Students also report that choosing a topic which they are enthusiastic about (e.g., my pet, my favorite singer) can motivate them to write “from the bottom of their heart.” Some students also prefer topics that are relevant to their life experiences (e.g., my university life, the Chinese New Year) to abstract argumentative topics.
I can display my pictures one after another. My last topic was “University life.” I feel it is like my memoir, something worth collecting and treasuring. (S 061, interview)
If you have the experiences of taking part in the activities, then you can write about them. You can also show the pictures of yourself taking part in the games. (S 106, interview)
It is more difficult to write about abstract topics like the death penalty, abortion, and so on. (S 115, interview)
It is easier to find relevant pictures for certain topics. You can also write more about them, like my topic, Chinese New Year. (S 076, interview) The Strategies Students Use to Create Multimodal Effects
Students use a number of strategies to create multimodal effects. Elements students consider regarding the text/picture’s semantic relations include concrete and simple ideas, abstract ideas, time and tense, action and process, similar objects, and humor.
Concrete and abstract ideas. Concrete ideas, such as food,
ingredients, cities, scenic spots, train stations, and so forth, are often represented with matching object images. Further, abstract ideas are often represented with real and concrete images. For example, one student wrote about the benefits of using Facebook to make friends. To represent the idea of making friends, she showed a picture of one of her friends. Another student used a picture of a couple of twin sisters for the sentence “there are also many similarities between the two cities” (Appendix, Picture 10). One student wrote about the steps to study for exams. Pictures she used are books, scenes in the library, in the classroom, and students taking exams. When writing about a country she wanted to visit, a student used the national flag of the United Kingdom, and pictures of Big Ben and London Bridge. The idea of Buddhism was represented with a Buddhist temple and Christianity represented with the images of Jesus Christ, the Canterbury Cathedral and the interior design of a church. With the topic “the benefits of learning a foreign language,” a student showed the advantages of “I will have the chance to study abroad” with a picture of students taking part in EFL classroom activities. With the sentence “I can travel to other countries I like,” a student showed
pictures of a person standing on the top of a mountain looking down, and a picture of a shining Eiffel tower and street scene of Paris at night.
Quality and color of the pictures. Quality, color, resolution,
and brightness of the pictures can also be used to represent the past, present, and future tense in time. For example, when writing about modern technologies, students often use pictures with brighter colors and higher resolution (Appendix, Pictures 11, 12, 13). When writing about an event which already happened or an invention in history, students found pictures with fading colors, lower resolution, or black-and-white pictures (Pictures 14 to 18 in Appendix). Events for the present and the future are presented with colorful and brighter pictures (Appendix, Pictures 19 to 22).
My topic is about my cat. I chose black-and-white pictures for things which happened in the past, pictures with ordinary color and lightness for the present. Pictures of the future are lighter. (S 126, interview)
Action and process. When writing about actions and processes,
many students try to find pictures of the exact action related in the text. For example, in a process story about how to make chocolate cakes, the student made efforts to find the picture of exact actions: “pour some milk” with a picture of milk pouring down and splashing; “put it into the oven” with a picture of an oven door opening (Appendix, Pictures 23 and 24). Action and process stories are also often presented with pictures showing objects before the action and then pictures showing the results of the action. For example, one of the students made the sentence “cut the cucumber,” and in her film,
she used one picture with a complete cucumber, and the next picture showing a sliced cucumber on the plate.
When students were unable to find exact object images to represent their written ideas, they most often adopted similar visual object representations. For example, one student used a picture with a “heart” to match the word “a heart-shaped mould.” Another wrote about the idea of “having a conversation with foreigners” by showing the picture of two people facing each other in a conversation.
Humorous effect. There are several strategies students use to
create humorous effects: using modified images of art and pop culture, drawing directly onto a picture, and/or using funny and comic pictures. A student showed the idea of information anxiety, Internet addiction and emotion crisis associated computer breakdown using a modified comic picture of “Scream” by Munch (Picture 25 in Appendix). Another strategy commonly used to create humorous effects is to use pictures from pop culture, such as pictures of famous television programs and soap operas with easily recognizable characters and celebrities. Writing about the custom for the married daughters to return to their original families on the second day of the Chinese New Year, a student used a picture of a popular soap opera “Mon’s House,” which immediately caught the audience’s attention and created another level of meaning for the story. Another student created humorous effects by adding a picture of a potato with a human face sitting on a couch watching television in his story about the danger of watching too much television. Adding one’s own drawings, like arrows or a mustache, also creates a humorous effect (Picture 26 in Appendix).
It can be seen that students have a sophisticated knowledge of visual literacy and are able to use them to add deeper meanings to their works. Thus, multimodal composition offers them a chance to create meaning at a different level and make intertextual references and connections between texts/images.
Using music. The present study deals with an assignment in
which students had to choose and edit music to create a multimodal effect. Designing music for the film was considered challenging but fun.
Editing music is also challenging and we have to learn it. (S 062, interview) I explored the software functions and found that I can use the editing function to edit the music so the timing of the film display can go along with the playing music. (S 094, interview)
I also designed the music to go with the timing of the film display. I had to edit and cut the songs with some programs. It was actually fun to do it. It gave me a sense of accomplishment. (S 118, reflection sheet)
Students also developed awareness that music is an effective tool to express and evoke emotions. It was an important element students used to convey the emotion and the atmosphere they wanted for the topic. An overall examination of their works shows that lively and cheerful music was often chosen for topics like activities and festivals. Peaceful music tended to be used for explanation or process essays. Sad or melancholic music was used to indicate the end of a party or an activity. Rock music was often used when the topics are
about cars, motorcycles, or technology. Chinese style music was chosen for topics about historical sites. One of the students used songs by her favorite singer as the story was an introduction to the singer. One student decided not to use loud and noisy music with drums and gongs for “Chinese New Year,” as would be traditional. Instead, she chose something peaceful and pleasant as she explained that she preferred “an easygoing and light” atmosphere.
In the beginning, the music is lively and exciting. I used guitar in the middle to show activities and games are going on, so it is happy and exciting. Then, in the end, I used sad music because the camp is going to finish. (S 106, interview)
Multimodal composition provides ESL writers with tools for sharing knowledge and for self expression. Non-linguistic modes allow writers to present knowledge, project cultural identities, and express emotions. In the present study, students used music to display their individual identities. As they made choices of music styles, they also presented themselves as unique individuals. The chance to actively play with musical element, design, and style, was empowering as they were able to exert control over every facet of their production, and make more personal choices in the process of creating their pieces of “artwork.”
Students who made greater efforts than average, and gave more consideration in designing both individual elements and the interactional effect among different modes presented works of better overall quality. However, some of the students seemed to have hastily chosen a piece of music without carefully considering the styles and
the compatibility of such music with their written topic. For instance, one student used romantic music for her story about food. One student chose sentimental music for the topic “Chinese New Year,” and one student played sad music when the background picture was actually something comic and funny. Apparently they merely used the music they had at hand without careful consideration of its suitability.
In addition to the features of adding background music and pictures, students can also record their voice using the Photostory software. However, most of the students were not confident about their pronunciation and some of them reported that adding their own voice might negatively affect the overall effect; the students viewed the films to be more artistic without their own voice.
DISCUSSION
The results of this study indicate that the multimodal writing practice helped to promote a positive cycle of writing experience. Students were offered opportunities to create and design texts and images; these students as artists displayed awareness of multiple art forms and made a mark by self representation. The multimodal project also facilitated development of creativity and independent learning.
A Positive Learning Cycle
A positive cycle of experience and composition can be deduced from students’ views of multimodal composition. The composition process started as students used significant experiences in their life, and their feelings, as source materials for their topic (a person they
liked, a pet they missed, an activity they took part in, a festival they looked forward to, and so on) and developed that into a project with texts, pictures, music, and sound. Creating and viewing the multimodal project with multimedia helped them recall memories and arouse emotions. The combination of appealing aesthetic results, intellectual development, affective engagement, as well as their successful experiences, in turn enhanced students’ sense of achievement, which then motivated them to make more efforts to improve or complete more similar projects.
Multimodal composition also offers a number of affective advantages and features. For example, students use non-linguistic modes to express emotional connections with the topic, which, as it grows and expands as they develop it, can, through their choices of certain music and images, develop into a work that conveys a sense of “atmosphere” or a “feeling.” Emotions can be aroused by the multimodal effect, which may not be so easily achieved through a traditional essay employing a single linguistic mode.
Students as Artists
Non-linguistic modes enable writers to present knowledge in different ways (Shin & Cimasko, 2008). One of the reasons students find the composing experience thrilling is the capability to show their work, share the memories, express their opinions, tell stories, and present themselves through a combination of music, images, and texts—like creating a film or a real art project. Many students find image manipulating and music editing (and for some, voice recording) an engaging experience. In the process, students also became more confident about themselves as multimodal artists. These findings are
in accordance with those of Andrews (2001) who also finds that students can make meaning though a wide variety of forms and media products. Students feel empowered as they are able to choose, edit, and experiment with both linguistic and non-linguistic modes to achieve the intended effect while expressing emotions and individual uniqueness.
Students Display Awareness of Multiple Art Forms
Visual art and music are key elements students use to convey their feelings about the topic. For example, lively and cheerful music was often chosen for topics like activities and festivals. Peaceful music is for explanation or process essays. Sad or melancholic music was used to indicate the end of a party or an activity. “Masculine” or “rock” music was often used when the topics were about cars, motorcycles, and technology. Most students displayed their awareness of the styles of image and music and the compatibility with the topics in creating the overall artistic and affective effect.
Representations of Self: Make a Mark
Using Photostory helps students to see the act of writing from a different perspective. Multimodal writing allows them to try and experiment with various tools to recreate memories and experiences for self-expression, which gives them a sense of achievement as they are able to enjoy “their own capacity to make a mark in the world” (Arnold, 1991) through multiple forms of human expressiveness (Andrews, 2001). In Arnold’s (1991) recursive model of writing, the core self moves up and away from the starting point, “with the expressive self always at the center of the spiral. The outer edges of
the spiral touch different kinds of writing as the writer increasingly widens the range of types of writing while at the same time holding on to the centrality of the self and the energy focused therein” (Arnold, 1991, p. 43). With multimodal composition, students widened not only the range of types of writing, but also the variety of media, as they increasingly became skillful at exploiting different modes to express the self. Self-reflection and reflexiveness, the fundamentals of self-development and the personalization of knowledge, are conducted not only with texts, but also with music, pictures, and voice. Creativity and Independent Leaning
In the present research, students also report that they had to learn and push themselves through their use of media functions, look for resources, think about different and creative ways to arrange texts, pictures, and music, create styles and manipulate the modes to achieve the effects they wanted. Students also reported that they learned from their first experience, and from watching each other’s films. As a result, these students became more competent with their second multimodal assignment. A comparison of students’ works at the same stages of their first and second production reveals that many of their works for the final project include more sentences, more pictures, and more special effects. Many of them also made significant progress in overall multimodal design and an improvement in the quality of their videos. Being able to share their works with the class also made the composition process more meaningful for these students. These findings are in accordance with the discoveries of research into other forms of computer aided projects, that multimodal
project support creativity, independent learning, reflection, as well as real and authentic purposes (Tweddle, 1997).
CONCLUSION
The study aimed at understanding EFL students’ views on multimodal composition and how they used texts, images, music, and sound to complete the project. Multimodal composition is considered interesting and fun by most students in the study. Using Photostory helps students to see the act of writing from a different perspective. Multimodal composition allows them to exert control, manipulate multi-elements, and experiment with the media for self-expression. Even though students reported they had to spend more time and make extra efforts, completing the project gave them a sense of achievement.
Elements student need to consider in completing the project include: sentence production, story structure, text/picture design (position and semantic relation), and display speed. Even though text is not the only element students have to deal with to complete a successful project, it is important for teachers to emphasize the importance of writing with good English. Better works of the students display the features of complete sentences, well-organized story structure, clear text/picture color contrast, coherence in text font, text color, text size, text position, sentence number, and display speed throughout the videos, with one to two sentences at the bottom of the page, each page displaying for five to seven seconds.
Transformation and transduction enrich students’ understanding of the topic, promote intellectual development, and foster independent
and meaningful learning. Students were offered tools for creative self-expression. However, without careful designing, the texts may often block the pictures. Students need to be instructed about how to modify the size of the font and arrange the text position so that it will not affect the overall aesthetic effect. Students can also be informed of the guidelines regarding sentence structure, story order, text/picture design, coherence, and time control, to achieve better multimodal composition quality.
In addition, students prefer writing about topics that are real and relevant to their life experiences, like summer camp, university life, favorite singers, pets, festivals, and so on. Music is recognized as an effective tool to create and express emotions about the topic. Choosing the styles of music and considering their compatibility with the topic is important in creating the overall artistic and affective effect. The recording function was not fully exploited as some students did not have the recording equipment, and some were not confident with their pronunciation, and some preferred the artistic effect shown with only music and pictures.
Multimodal composition offers a number of affective advantages and chances for intellectual development. Students might be motivated to make efforts to complete the project as they were themselves affectively engaged with the story they were creating. The multimodal, transformation, and transduction effect may be effective in enhancing students’ attitude and motivation, as well as organization and length in writing.
REFERENCES
Andrews, R. (2001). Teaching and learning English: A guide to recent
research and its applications. London: Continuum.
Arnold, R. (1991). Writing development. Milton Keynes, UK: Open University Press.
Chapelle, C. A. (2001). Computer applications in second language
acquisition: Foundations for teaching, testing and research.
Cambridge, UK: Cambridge University Press.
Kress, G. (1998). Visual and verbal modes of representation in electronically mediated communication: The potential of new forms of text. In I. Snyder (Ed.), Page to screen: Taking literacy
into the electronic era (pp. 53-79). London: Routledge.
Kress, G. (2003). Literacy in the new media age. London: Routledge. Kress, G. (2008). Writing in multimodal texts––A social semiotic
account of designs for learning. Written Communication, 25, 166-195.
Nelson, M. E. (2006). Mode, meaning, and synaesthesia in multimedia L2 writing. Language Learning and Technology,
10(2), 56-76.
Nelson, M. E., Hull, G. A., & Roche-Smith, J. (2008). Challenges of multimedia self-presentation: Taking, and mistaking, the show on the road. Written Communication, 25, 415-440.
Sheppard, J. (2009). The rhetorical work of multimedia production practices: It’s more than just technical skill. Computers and
Shin, D. S., & Cimasko, T. (2008). Multimodal composition in a college ESL class: New tools, traditional norms. Computers
and Composition, 25, 376-395.
Slaouti, D. (2002). The World Wide Web for academic purposes: Old study skills for new? English for Specific Purposes, 21, 105-124.
Tardy, C. M. (2005). Expressions of disciplinarity and individuality in a multimodal genre. Computers and Composition, 22, 319-336. Tweddle, S. (1997). A retrospective: Fifteen years of computers in
English. English in Education, 31(2), 5-13.
Walton, K. (1995). Picture my world: Photography in the primary
school. London: The Arts Council.
Warschauer, M. (Ed.). (1996). Telecollaboration in foreign language
learning. Honolulu, HI: University of Hawai’i.
Warschauer, M. (2000). The changing global economy and the future of English teaching. TESOL Quarterly, 34, 511-535.
ABOUT THE AUTHOR
Yu-Hsiu Huang is an assistant professor of Southern Taiwan University of Science and Technology. She has been teaching English at different levels for six years. Her research interests include computer assisted English writing, extensive reading, and English for specific purposes.
APPENDIX
Picture 1. An image taken from a student’s writing comparing Taiwan and the United Kingdom.
Picture 2. An image taken from a student’s writing comparing Taiwan and the United Kingdom.
Picture 3. An image from a student’s writing about scenic spots or tours in foreign countries.
Picture 4. An example of texts blocking the images.
Picture 5. An example of a student conveniently finding pictures on the Internet without carefully considering the relevance to their texts.
Picture 6. An example of a student misusing pictures taken from the Internet. The student wrote about the Confucian temple in Tainan, but the picture she used shows the nine turtles in Chihkan Tower, a different historical site in the city
Picture 7. An example of a student’s work displaying the features of good sentence and image design.
Picture 8. An example of a student’s work displaying the features of good sentence and image design.
Picture 9. An example of a student’ work displaying the features of good sentence and image design.
Picture 10. Abstract ideas are often represented with real and concrete pictures.
Picture 11. An example of writing about
modern technology. Picture 12. An example of writing about modern technology.
Picture 13. An example of writing about
modern technology. Picture 14. An example of a student’s writing about an event which already happened with pictures of lower resolution, or in black and white.
Picture 15. An example of a student’s writing about an event which already happened with pictures of lower resolution, or in black and white.
Picture 16. An example of a student’s writing about an event which already happened with pictures of lower resolution, or in black and white.
Picture 17. An example of a student’s writing about an event which already happened with pictures of lower resolution, or in black and white.
Picture 18. An example of a student’s writing about an event which already happened with pictures of lower resolution, or in black and white.
Picture 19. Events for the present and the future are presented with colorful and brighter pictures.
Picture 20. Events for the present and the future are presented with colorful and brighter pictures.
Picture 21. Events for the present and the future are presented with colorful and brighter pictures.
Picture 22. Events for the present and the future are presented with colorful and brighter pictures.
Picture 25. Using modified images of art and pop culture to create humorous effects.
Picture 26. Adding one’s own drawings also creates a humorous effect.
