Chapter 3 Methodology
This chapter presents the method of the research. The design, participants, procedure of the study, data collection (including instruments), and data analysis of the study are elaborated in order.
The current study aimed to investigate the effect of vocabulary learning via online reading and reading-writing activities in Moodle (Modular Object-Oriented Dynamic Learning Environment) as compared with via traditional vocabulary instruction in EFL class.
To explore the research questions, posttests were applied to measure students’ vocabulary gains and word retention after the online activities. Participants were divided into two groups:
the experimental group learned vocabulary through online activities in Moodle in the lab while the control group received vocabulary instruction in the classroom one hour per week during the 16-week span (from 1/26/2006 to 6/07/2006). Both groups were required to read the same materials and take the same posttests.
Aside from the data analysis obtained from the posttests, complementary measures including a questionnaire and interviews were employed to reinforce the validity and reliability of the investigation.
3.1 Participants
The participants for this research were 34 ninth graders drawn from two classes at Shijian Junior High School: 17 from one class who took the experimental online course in the lab and 17 from the other class who took the traditional EFL course in classroom. 19 of the participants were female and 15 were male. To ensure that the experimental group and the control group were homogenous in terms of English language proficiency, participants were
selected based on their scores on three screening tests: a vocabulary test retrieved from Levels Tests Online, a reading comprehension test adopted from a set of GEPT pretest available on the Internet, and a 50-item grammar test randomly selected from the CD-formatted English @ Hand Test Bank (see Appendix 1, 2, and 3). Table 2 shows the descriptive statistics for the
participants’ scores on these tests. Results revealed that the experimental group and the control group were homogenous at the outset of the treatments.
Table 2
Descriptive Statistics for Group Comparisons
Experimental group Control group
Test type n M SD n M SD
Vocabulary test 17 59.18 14.36 17 57.12 22.51
Reading comprehension 17 51.65 24.97 17 53.41 23.22
Grammar test 17 59.24 22.86 17 58.18 21.26
Overall 17 56.68 19.36 17 56.23 20.42
To explore the impact of the effectiveness of learning in Moodle on students at different language proficiency levels, the participants in each group were further categorized: six participants in each group were classified as low level, five as intermediate level, and six as high level according to their overall scores on the screening tests. The low level participants had overall scores under 45, the intermediate level participants within 48-69, and the high level participants, over 70.
3.2 The Procedure of the Study
At the beginning of the study, students were given a two-hour orientation at the Digital Learning Web at Shijian Junior High School (http://elearning.sjjh.tp.edu.tw). The web
employed Moodle (Modular Object-Oriented Dynamic Learning Environment) as a course management system (CMS). The administrator managed the site (including site colors, fonts, layout, etc.), user, course and modular construction to suit local needs. For teachers, building a course through this system means adding course activity modules to the main page. In the Digital Learning Web at Shijian Junior High School, it is easy to add a new activity by merely clicking “Turn on editing” under Administration and select the type of activity from the popup menu (see Figure1). Twelve activity modules listed in the menu were available, namely, workshop, assignment/uploading, forum, the ready-made questionnaire, chat room, quiz/questionnaire, survey, topic interpretation, online questionnaire, online resources, learning journal, and course journal. Other functions like tracking, email integration, time restriction, excel, also benefited teachers in giving an EFL course.
Figure 1. The Popup Menu of Course Activities (Modules)
After the orientation in Moodle, participants started their online course. They were required to complete one section of online learning tasks within two hours, one hour per week, under the schedule. In other words, participants were required to be engaged in an online reading activity or an online reading-writing activity each week throughout the sixteen weeks.
As an adjunct to the regular EFL course (see Robb, 2004), the proposed online course in Moodle was sequenced in accordance with the schedule of the regular EFL course, had the same topics in the textbook, and its online activities were used in place of some activities involved in the textbook. As shown in Figure 2, two topics of the textbook English @ Hand Book Six, i.e., Hobbies and The Family Outing, were exhibited in the online course.
Figure 2. The Topic-Formatted Online Course
The syllabus of the proposed online course is shown in Table 3.
Table 3. The Syllabus of the Online Course
Week Date Topic
1st 1/26 Orientation in Moodle 2nd 2/10 Orientation in Moodle
3rd 2/22 Unit 1 Graduation Trip (online reading-writing) 4th 3/1 Unit 1 Graduation Trip (online reading-writing) 5th 3/8 Unit 1 Graduation Trip (online reading)
6th 3/15 Unit 2 Television (online reading) 7th 3/22 Unit 2 Television (online reading) 8th 3/29 Unit 3 Good Study Habits (online reading) 9th 4/12 Unit 3 Good Study Habits (online reading) 10th 4/19 Unit 4 Career Plans (online reading-writing) 11th 5/3 Unit 4 Career Plans (online reading-writing) 12th 5/10 Unit 4 Career Plans (online reading)
13th 5/17 Unit 5 Hobbies (online reading-writing) 14th 5/24 Unit 5 Hobbies (online reading-writing) 15th 5/31 Unit 5 Hobbies (online reading)
16th 6/7 Unit 5 Hobbies (online reading)
Two activities in the textbook English @ Hand Book six, namely, Warm Up and Keep Reading, were implemented in the online course. The former was conducted via online reading-writing activity while the latter was conducted via online reading activity. While participants of the experimental group were engaged in the online learning activities in the lab, participants of the control group were instructed in the classroom, reading the same materials.
Comparisons of the treatments received by the experimental and control groups are briefly shown in Table 4.
Table 4
Comparisons of the Treatments Received by the Experimental Group and the Control Group Group
Activity
Control group Experimental group
Warm Up 1. The teacher asked students a question pertaining to the topic of the unit, and listed options (i.e., the target words or phrases) for students to vote for.
The meaning of each option, whether a word or a phrase, was explicitly explained by the teacher.
2. The teacher taught students the printed comments retrieved from CBBC Newsround. The meanings of the new words were well interpreted. Useful expressions were pointed out for students to conduct the following discussion activity.
3. Students discussed their likes and dislikes about the comments in pairs or in groups. While discussing, they were encouraged to use the useful expressions pointed out by the teacher.
1. Through Online Survey:
Students read a question posted as an online survey. They answered the question by checking an option among others (i.e., the target words or phrases). They had to look up the options in online dictionaries if they did not understand their meanings.
2. Through Online Reading &
Assignments:
Students read online comments retrieved from CBBC on the online learning journal. They were asked to look up new words in the online dictionaries and listed useful expressions or words by themselves for the succeeding writing task. They were also required to upload the useful expressions or words as online assignments via the same online journal.
3. Through Online Forum:
Students presented their likes and dislikes about the online comments by writing on the forum. While writing, they were encouraged to use the expressions or words they listed in their online assignments.
Keep Reading
1. The teacher read aloud the selected reading, translated it and interpreted the assumed unknown words for students; the students took notes between lines.
2. Students were given an oral comprehension check about the reading.
1. Through Online reading & Assignments:
The students read the selected reading on the online learning journal and they were encouraged to use online dictionaries while encountering new words. In this online learning journal, they were asked to list all the new words they found in the reading and make a word list with equivalent Chinese meanings with the help of online dictionaries.
2. Through Online comprehension testing:
Students took a four-item online comprehension test and were given immediate feedback. They were allowed to use online dictionaries to review the reading or read the items.
Although it was hard to control the treatments precisely, we expected the comparisons of the treatments could be focused on (1) using online dictionaries to learn vocabulary vs.
acquiring vocabulary through receiving explicit instructions, and (2) learning vocabulary through writing vs. oral presentation.
While designing the online course, limitations of ninth graders’ English language ability were always the major concern. For this sake, Brandl’s (2002) teacher-centered approach was adopted by the present study. That is, the teacher guided the learning paths, selected online tools, prescreened and pre-selected the reading materials. Based on this approach, the online tool WordChamp was introduced to the students (see 2.2.3 in this thesis), and CBBC Newsround (available at http://news.bbc.co.uk/cbbcnews/default.stm.) was chosen for its richness and provision of simplified news stories and online comments fit for young EFL learners. For online reading activities, five news stories pertaining to the topics of the textbook were chosen (see Appendix 4); while for online reading-writing activities, three
online comments pertaining to the topics of the textbook were chosen (see Appendix 5).
The online reading and online reading-writing activities were conducted alternately. For each of them, posttests were given to assess participants’ performance. Then after the online course ended, a questionnaire and interviews were given to obtain participants’ perceptions of the activities. The procedure of the study is presented by a flowchart as in Figure 3. Details about each activity, including reading materials, selection of target words, online tasks, posttests, scoring, a questionnaire, and interviews are elaborated in order in the following sections.
Figure 3. Flowchart of the Procedure
3.2.1 Online Reading Activity
This activity consisted of a series of tasks: online reading and assignments, and online reading comprehension test. Prior to the beginning of the online reading activity, the reading materials used in this activity were analyzed to ensure they were not far beyond the participants’ English language proficiency, and target words were determined in order to measure the vocabulary gains.
Reading Materials
The reading materials used in the online reading activity were five news stories retrieved from CBBC Newsround. As shown in Table 5, they were analyzed in terms their readability.
Table 5
Readability Estimates for Readings Used in Online Reading Activity Reading
Readability
R1.
Exam Tips
R2.
Dream Trip
R3.
News &
Nightmares
R4.
Dream Job
R5.
Unusual Hobbies
Number of words 120 85 117 205 85
Number of paragraphs 10 5 9 14 6
Number of sentences 5 3 6 15 7
Flesch reading ease 73.9 52.2 76.7 80.3 84
Flesch-Kincaid Grade Level 7.1 11.6 6.7 5.4 4.5
Among the five news stories listed in the table, “Dream Trip” was the most difficult in terms of its reading ease and Grade Level. However, it had the least words. Besides, it was adopted for the reason that many of the words had already appeared in learners’ textbook.
Similarly, the article “Unusual Hobbies” was selected for containing more unknown words though it was easier to read.
Vocabulary Selection
Since participants in the present study varied greatly in terms of their English language proficiency, it was difficult to find mutual target vocabulary for them. Therefore, in the online reading activity, individual participant set his or her own individual target words for each reading. In order to determine each individual’s target words, each participant was asked to read the printed news stories retrieved from CBBC Newsround and list words unknown or unfamiliar to him/her (see Appendix 6). In case that participants might mistakenly received some new words or unfamiliar words as known, they were further asked to translate the same news stories into Chinese (see Appendix 7). Then the teacher as the researcher double checked each individual’s translation work, picked out the words that were translated incorrectly, and then added these words in the individual’s word list. The final word list was thus the target words set for each news story. For each participant, there were five sets of target words (word lists) in total for there were five readings assigned in the online reading activity.
Online Reading and Online Assignments
The tasks were created with a Moodle module named learning journal. At first, students were required to click on the learning journal wherein they were asked to read the same news story mentioned in vocabulary selection. Participants were required to make a list of unknown words with their equivalent Chinese meanings (see Figure 4) on the learning journal and uploaded the assignments. While doing the online reading and the online assignments, students were encouraged to use WordChamp (available at http://www.wordchamp.com/lingua2/Reader.do) or other online dictionaries to find the meanings of unknown or unfamiliar words. Finally, the teacher gave marks and feedback after receiving the students’ online assignments.
The purpose of asking students to make a word list online was to encourage students to notice the unknown words and try to learn them through lookups (see Al-Seghayer, 2003).
Figure 4. Online Reading and Assignments
Online Reading Comprehension Testing
The task was created with a testing module. Students were asked to take a four-item online comprehension test on the same news story assigned in the previous tasks. Again, they were reminded to use online dictionaries when they encountered unknown or unfamiliar words in reading these items or previewing the reading. The choices of each item were selected from most of the participants’ target words--words which were assumed unknown or unfamiliar to most of them. As soon as students completed an item, they would get an automatically immediate response to their answers.
Since comprehension testing is a formal classroom activity where vocabulary is not the main focus, it has thus been widely adopted by research into incidental vocabulary learning.
In this respect, the task was designed in a fashion of implicit (or incidental) vocabulary learning whereas the choices of each item were composed to encourage noticing and add exposure to target words. Through each round of online reading activity (i.e., from vocabulary
selection, online reading and assignments, to comprehension test), students were assumed to repeatedly read the same material for at least five times and receive a minimum of five encounters of the target words.
The Immediate Vocabulary Tests
When each round of online reading was finished, each participant was given a paper-and-pencil immediate test to assess their vocabulary gains. The same process of online reading activity was repeated for five different news stories assigned in the online course. The five immediate vocabulary tests were word level translation. Participants were required to translate the listed English words (the list of each individual’s target words) into Chinese.
The immediate tests were composed of each individual’s target words set for each news story. They were given immediately after each round of online reading activity was finished, and each immediate test took approximately ten minutes. The raw score on each individual’s vocabulary gains was calculated according to the following formula:
the sum of the correct items Vocabulary Gains
the sum of the individual's target words set for one reading
=
The Delayed Vocabulary Test
After the five rounds of online reading activities and the five immediate tests were accomplished, a delayed vocabulary test was administered to assess each participant’s vocabulary retention. Since each participant had different target words set for each reading, they had their own individual delayed tests. Each participant’s delayed test was composed of his or her own correctly-answered items on the five immediate vocabulary tests (see Appendix 8 for an example). The raw score on each individual’s vocabulary retention was calculated according to the following formula:
the sum of the correct items Vocabulary Gains=
the sum of individual's target words set for the five readings
3.2.2 Online Reading-Writing Activity
This activity consisted of the following tasks to be completed in order: online survey, online reading and assignments, and online writing. Before participants began these tasks, the reading materials used in this activity were also analyzed, and target words were set. After the online tasks were completed, posttests including a word level translation test and a cloze test were given to measure the participants’ vocabulary gains.
Reading Materials
The reading materials used in the online reading-writing activity were also retrieved from CBBC Newsround. They were online comments on three different topics (see Appendix 5). Written and posted by kids (under 18) all over the world, the online comments were considered to be close to the participants’ experience and interests. Here they were also analyzed to ensure that their readability fit for students’ language proficiency (see Table 6).
Table 6
Readability Estimates for Readings Used in Online Reading-Writing Activity Reading
Readability
What’re their dream trips?
What’re their dream jobs?
Unusual hobbies
Study Habits (Unit Three of the textbook)
Number of words 469 211 314 175
Number of sentences 31 12 27 15
Flesch reading ease 70.1 77.7 78.5 67.7
Flesch-Kincaid Grade Level 7.2 6.2 4.8 6.3
Compared with the reading text of the textbook, e.g. Unit Three, the online comments selected for the online reading-writing activity were not far beyond the students’ reading ability-- if we assume that the readability of the readings in the textbook was in line with most of the ninth grader’s English language proficiency.
Vocabulary Selection
Unlike the reading materials used in online reading activity (i.e., the news stories retrieved from CBBC Newsround), the reading materials used in the online reading-writing activity were a set of discrete sentences with similar patterns. They were shorter in length and most of the words unknown to the participants were nouns. These factors to some extent helped to reduce the burden of reading. Therefore, to set a list of mutual target words for all the participants was feasible.
The mutual target words were drawn through a 50-item word level translation test given to the participants. As a result, 18 words or phrases unknown to all of the students were selected.
Online Survey
This task was created with a survey module. In this task, students were asked to answer a question by checking one option (see Figure 5). These options were also the target words or phrases of the online reading-writing activity. After the survey was conducted, the teacher would show them the results of the survey through broadcast system in the lab (see Figure 6).
Figure 5. Online Survey
Figure 6. A Result on the Online Survey
The purpose of this task was threefold: (1) since the options were composed of target words or phrases, learners’ opportunities of exposure to these target items were added; (2) in
an attempt to answer the question by checking the right option, learners were assumed to use online dictionaries to find the meanings of each option and thus vocabulary learning was likely to occur; and (3) the task was designed to function as pre-teaching (see Nation, 1990), warming-up, previewing, or pre-reading strategies which could be employed by learners to partially realize the schema-based concepts. That is to say, by taking the online survey, learners were expected to draw inference prior to the succeeding reading.
Online Reading & Assignments
These tasks were created with a learning journal module. After clicking on the learning journal, students were asked to read the online comments retrieved from CBBC Newsround by the teacher. Next, they were asked to find and list useful expressions or words for the following writing (presenting) task. A link to WordChamp was again offered as the reading aid.
Like in all the other online tasks, students were asked to hand in their assignments through the online journal within time restriction. Then the teacher would check their assignments, gave marks and feedback. The tasks are shown in figure 7.
Figure 7. Online Reading and Assignment
Through the assignment of listing useful expressions, we expected that participants would notice the patterns, expressions or even discrete words. Hopefully, the target words were also noticed.
Online Writing
This task was created with a forum module. While writing on the online forum--a task entitled“Likes and Dislikes”, participants were required to reflect “their own comment” on the online comments they read in the preceding task. Since they were encouraged to write by using the words or expressions they listed earlier in the learning journal, they were thus likely to avoid making too many errors in writing.
A sample of a student’s online writing is shown in figure 8.
Figure 8. A Sample of Student’s Online Writing
The process of the online reading-writing activity, i.e., from online survey, online reading and assignments, to online writing, was repeated for three different readings (online comments on three different topics). Through each round of online reading-writing activity, students were assumed to receive at least five encounters with the target words.
Posttests for Online Reading-Writing Activity
After the three rounds of online reading-writing activities were finished, two types of vocabulary tests were given: a word level translation test and a cloze test. They took the participants about 30 minutes to finish. The word level translation test was composed of the 18 target words selected at the beginning of the online reading-writing activity (see Appendix 9). The 18 items of the cloze test were adapted or retrieved from Collins Cobuild Dictionary or Oxford Dictionary (see Appendix 10). The target words were also given in the cloze test where it functioned as a word bank. For both of the vocabulary tests, each correctly-answered item scored 5.5 points.
3.3 A Questionnaire and Interviews
To elicit information concerning students’ perceptions of the online course, students who conducted the online activities were asked to fill out a questionnaire when all the online tasks were finished. The questionnaire included seven issues, i.e. the accessibility of the learning web, the online dictionaries, the reading materials, the favored tasks, the effectiveness of each task, the participants’ preference, and the advantages and disadvantages of the online course (see Appendix 11).
Finally, interviews with participants at different language proficiency levels were conducted to see their reflections of the learning experience in Moodle.
3.4 Data Collection and Data Analysis
For online reading activity, data were collected through participants’ scores on immediate and delayed vocabulary tests. For online reading-writing activity, data were collected through participants’ scores on a word level translation test and a cloze test. The raw scores on the immediate vocabulary tests and the delayed vocabulary test were presented as fraction
numbers originally. They were then converted into integers or a whole number with decimals.
For instance, 3/4 was converted into 75, and 7/8, 87.5 (see Appendix 12 for the raw scores and Appendix 13 for the converted scores).
In the current study, data collected from the vocabulary tests mentioned above was analyzed by means of four tools. Descriptive statistics were computed to compare the effect of different treatments. Independent samples t-test was used to detect if there were statistically significant differences between groups. Effect sizes (ES) were further used to assess the magnitude of a difference between means of two groups1. Besides, a two-way ANOVA (analysis of variance) was employedto detect if there were significant group-by-proficiency level interaction. Data collected from learners’ response to the questionnaire were analyzed by means of percentage and frequency.
Except effect sizes, the analysis used the SPSS statistical software package, version 10.0. Data collection instruments and tools used for data analysis corresponding to each research question are listed in Table 7.
1 ES = (means of experimental group－means of comparison group) ÷ standard deviation of comparison group. According to Fraenkel & Wallen (1993), most researchers consider that any ES of .50 or larger is an important finding.
Table 7
Data Collection and Analysis
Research question Data collection instruments
Tools used for data analysis 1. Do online reading activities in Moodle
enhance vocabulary learning?
Five immediate vocabulary tests
A delayed vocabulary test
Mean
Independent samples t-test
Effect sizes
2. Do online reading activities in Moodle help students at different language proficiency levels learn vocabulary?
Five immediate vocabulary tests
A delayed vocabulary test
Mean ANOVA
Independent samples t-test
Effect sizes 3. Do online reading-writing activities in
Moodle enhance vocabulary learning?
A word level translation test
A cloze test
Mean
Independent samples t-test
Effect sizes 4. Do online reading-writing activities in
Moodle help students at different language proficiency levels learn vocabulary?
A word level translation test
A cloze test
Mean ANOVA
Independent samples t-test
Effect sizes 5.What are learners’ perceptions of
vocabulary learning through online reading and online reading-writing activities in Moodle?
Participants’
responses to the questionnaire
Frequency Percentage
