Chapter Three Method
This chapter is concerned with the method of the present study and is comprised
of four parts. The participants and their background are introduced in section 3.1.
Instruments are addressed in section 3.2. The procedures are described in section 3.3,
and finally section 3.4 is dedicated to the description of some adopted statistical tools
and data analysis.
3.1 Participants
Thirty-four non-English-major college freshmen, mainly from the college of
education and liberal arts in National Taiwan Normal University, participated in this
study. They were categorized into intermediate-high level based on the sum of their
scores in the listening and reading placement tests administered in the beginning of
this semester. Their total scores of these two tests ranged from 128 points to 134
points and these students were later assigned to the same class.
3.2 Instruments
I adopted three instruments to collect data in the present study. A topic familiarity
questionnaire, which was distributed to the participants to determine their perceived
degree of cultural familiarity of the five topics, was introduced in section 3.2.1.
Reading comprehension tests, which were conducted to assess students’ general
understanding of the selected articles, were discussed in section 3.2.2. The pretest and
posttest of twenty target words, which were employed and compared to measure
students’ incidental vocabulary learning, will be addressed in section 3.2.3.
3.2.1 Topic Familiarity Questionnaire
A topic familiarity questionnaire was administered by the instructor of the class
to measure the participants’ prior background knowledge and to decide their cultural
familiarity of each text (see Appendix A). It comprised five culturally-related articles
(see Appendix B for one sample) selected from the Sinorama magazine based on their
length and difficulty level (see Table 3-1) assessed by the Dale-Chall Readability
Formula
1, with the title of each article as well as a brief introduction stated. The
researcher chose these texts from Sinorama magazine for the reason that it was
authentic reading material and its difficulty level suited the participants of the present
study, who were non-English-major college freshman at intermediate-high level. To
complete this questionnaire, the participants assessed their perceived cultural
familiarity with these five topics on a scale of 1 to 5. The number “5” denoted the
most familiar article and the number “1” signified the least familiar one.
1 The Dale-Chall Readability Formula (Dale, 1948) was designed to run Readability Calculations. It is originally a vocabulary-based formula. The formula uses its own word list as well as some other factors in the sample’s total number of words and sentences to assess upper elementary through secondary materials.
Table 3-1: Readability Calculations of the Texts
Title Aboriginal
Culture
Hakka Homes
Puppet Theater
Courtyard Compound
Math Anxiety
Words of the text 942 685 800 945 829
Syllables 1,411 1,125 1,276 1,419 1,349
Monosyllabic words 638 405 497 629 513
Words of 3 or more syllables 106 95 120 112 130
Difficult words 173 183 200 176 216
Sentence 44 42 43 33 44
Difficulty index 7.6 8.7 8.5 8.0 8.7
Note. 1. Texts marked were selected based on both difficulty index (7.6 – 8.7). 2. The 5 texts were for the familiarity rating questionnaire. 3. Raw Score of Readability (RE) = .0.0496*Avg. Sentence Length + 0.1579*Percent Difficult Words + 3.6365. 4. For further reference on Dale-Chall Readability Formula, see Dale (1948). 5. The title of each text is shortened. The full title for each text is as follows: Aboriginal Culture (I Can Only Cheer Them On!), Hakka Homes (Old Hakka Homes of Southern Taiwan), Puppet Theater (Taking the World by Storm: High Energy Puppet Theater Sweeps Taiwan), Courtyard Compound (The End of the Courtyard Compound), and Math Anxiety (Take the Anxiety out of Learning Math).
3.2.2 Reading Comprehension Test
After reading each culturally-related article selected from the Sinorama
magazine, the participants took an immediate reading comprehension test (see one
sample in Appendix B), which was administered to measure subjects’ comprehension
of each text. There were two global questions and three local questions in each test. In
total, there were five questions for each test. Each question was worth 1 point, and the
total possible score for each text were 5 points.
3.2.3 Pretest and Posttest of Twenty Target Words
Four words from each article, which amounted to twenty target lexical items (see
Table 3-2) in the pretest and posttest, were selected based on the suggestions of one
professor and one graduate student, these words’ levels (above 3) in a word list
published by College Entrance Examination Center (CEEC), and their frequency of
occurrence (at least twice) in the text. The above two experienced experts considered
these target lexical items to be closely related to each article and to be helpful in
enhancing topic understanding but their precise meanings might possibly be foreign to
the participants. These words were randomly arranged in the format of Vocabulary
Knowledge Scale (Paribakht& Wesche, 1993, p5) as a vocabulary test, which
measured the participants’ knowledge of these target words. Among these target
words, there were ten nouns, five adjectives and five verbs. The verb type of target
lexical items was presented in their infinitival forms, the nouns in their singular forms,
and adjectives in their original forms. The order of presentation of these twenty target
words in the pretest was different from that in the posttest. In this way, any possible
effects of episodic memory for the order of presentation of the target words could be
diminished.
Table 3-2: The Target Words of Five Texts
Title Target word Occurrences Parts of speech CEEC level
aboriginal 14 adj. 6
refine 2 vb. 6
rite 3 n. 6
Aboriginal Culture
tribe 4 n. 3
architecture 18 n. 5
altar 8 n. 0
unique 2 adj. 4
Hakka Homes
preserve 2 vb. 4
dialect 4 n. 5
innovation 2 n. 6
alienate 3 vb. 6
Puppet Theater
martial 3 adj. 5
compound 9 n. 5
extend 5 vb. 4
ancestral 4 adj. 0
Courtyard Compound
clan 4 n. 5
anxiety 6 n. 4
simplify 2 vb. 6
abstract 2 adj. 4
Math Anxiety
curriculum 12 n. 5
Note. 1. The number “zero” in this CEEC level represents that the level of the target word is higher than level 6. 2.
The title of each text is shortened. The full title for each text is as follows: Aboriginal Culture (I Can Only Cheer Them On!), Hakka Homes (Old Hakka Homes of Southern Taiwan), Puppet Theater (Taking the World by Storm:
High Energy Puppet Theater Sweeps Taiwan), Courtyard Compound (The End of the Courtyard Compound), and Math Anxiety (Take the Anxiety out of Learning Math).
3.3 Procedures
Six phases were involved in the data-collection procedures. In phase 1, the
listening and reading placement tests were administered on September 16
thto assess
the participants’ listening ability and reading proficiency. In phase 2, a survey
concerning their cultural familiarity of the five topics of the five selected articles was
conducted in the form of a questionnaire also on September 23
rd. In phase 3, the
participants’ prior knowledge about twenty target items was tested in the form of
Vocabulary Knowledge Scale as the pretest for half an hour on September 30
th. In
phase 4, subjects read each article mainly for the gist in each meeting in every other
week starting from October 7
thand answered five comprehension questions in a
comprehension test right after each assigned passage. Their knowledge of four target
words in each article in the Vocabulary Knowledge Scale was also measured. In phase
5, the originally random presentation of word order in the pretest was modified
alphabetically and then the revised version was employed as the posttest to assess the
participants’ knowledge of the twenty target words four weeks after the fifth test. In
phase 6, the researcher conducted an interview with students to investigate whether
they had noticed the word during the planned reading and how they deciphered the
meaning of the target words. Throughout the whole procedure, the participants were
not informed in advance about the researcher’s focus on their vocabulary learning and
about the immediate vocabulary test. They mainly read each article for global
comprehension. The aforementioned phases are summarized in Table 3-3 and will be
covered in the following seven sections respectively.
Table 3-3: The Six Phases of the Present Study
Purpose Task Date
listening and reading placement tests 09/16/2005 background knowledge
topic familiarity questionnaire (Ranking) 09/23/2005
pretest pretest of twenty target words 09/30/2005
immediate test experimental reading, immediate test on comprehension and vocabulary gain
10/07/2005 10/21/2005 11/04/2005 11/25/2005 12/09/2005 posttest posttest of twenty target words 01/06/2006
interview interview Spring, 2006
background knowledge topic familiarity questionnaire (Likert Scale:
Rating) Spring, 2006
3.3.1 Listening and Reading Placement Tests
All freshmen first took the listening and reading placement tests in the very
beginning of the semester on September 16
th. Fifty minutes were allotted for each test.
After that, some students were assigned to the intermediate-high proficiency group
based on their total scores of these two tests and became the participants of the
present study. The reliability for the listening and reading placement tests were 0.85
and 0.86 respectively, which reached the required standard 0.85. Different types of
test items were incorporated in these two tests to assess students’ discrepant aspects of
listening and reading ability. Besides, these two tests could distinguish good learners
from poor learners quite well. Consequently, they were proven to be highly reliable
and valid and a detailed description of them would be discussed in the following
section. However, these two sets of tests are still utilized as a means of measuring
non-English-major college freshmen’s reading and listening abilities and assigning
them to classes of different proficiency level in NTNU, they are kept confidential and
thus are not attached in the appendix section.
Listening Placement Tests. The listening placement test was composed of three
major parts, fifteen multiple choice questions for each, forty-five test items in total. It
was designed by professors of the English Department in National Taiwan Normal
University. In part I, students would hear a dialogue between two interlocutors. They
heard speaker A utter a statement and then selected the best responses to make the
whole dialogue reasonable and appropriate. In part II, students listened to fifteen
conversations between a man and a woman. After each conversation, they would hear
a question about the content of the conversation and had to choose the best answer. In
part III, they heard seven short talks first and answered the following two to three
relevant questions.
Reading Placement Test. The reading placement test consisted of three main
sections along with fifty multiple choice questions in total. It was also designed by
professors of the English Department in National Taiwan Normal University. Part I,
which included fifteen test items with three distracters and one correct answer for
each, aimed to evaluate the participants’ knowledge of words and phrases. Students
needed to choose the most appropriate answer which could best fit into the sentence
semantically and grammatically. In part II, students read three passages with five
places blanked out for each. They had to choose the answer which could best fit the
space and make the whole passage coherent. In part III, students read five texts
ensued by two to four relevant questions for each. These comprehension questions
ranged from local ones assessing students’ abilities in scanning for details, deriving
the word’s meaning from the context, judging the truth value of the statements based
on the text to global ones measuring their abilities to grasp the main idea, make
predictions and inferences, guess the attitude of the writer and the like.
3.3.2 A Survey on Cultural Familiarity
To investigate the participants’ perceived degree of cultural familiarity for five
topics of the selected five articles, the researcher undertook a survey by adopting a
questionnaire on September 23
rd. Subjects completed the questionnaires, on which
were the topics of these five chosen texts presented with brief sentence descriptions
about their content, by giving a ranking to each topic in terms of cultural familiarity
from the number “5”, meaning the most familiar, to the number “1”, the least familiar.
The distributions of frequency from 1 to 5 for each topic were later described and
compared to determine which topic was more familiar than the others, as indicated in
Table 3-4.
Table 3-4: Participants’ Assigned Ranking for Each Text
Ranking 1 2 3 4 5
Test Fr % Fr % Fr % Fr % Fr % Test1 8 23.53 7 20.59 5 14.71 12 35.29 2 5.88 Test2 3 8.82 10 29.41 11 32.35 6 17.65 4 11.76 Test3 10 29.41 6 17.65 5 14.71 8 23.53 5 14.71 Test4 5 14.71 10 29.41 10 29.41 7 20.59 2 5.88 Test5 8 23.53 1 2.94 3 8.82 1 2.94 21 61.76
Note. 1. Fr = Frequency. 2. Topic familiarity rankngs were collected from a survey questionnaire of participants.
The participants are asked to give ranking to these 5 topics in terms of cultural familiarity from 1 (very unfamiliar with the topic) to 5 (very familiar with the topic); N = 34. 3. The title of each text in each test is illustrated as follows: Test 1 = “I Can Only Cheer Them On!” Test 2 = “Old Hakka Homes of Southern Taiwan,” Test 3 =
“Taking the World by Storm: High Energy Puppet Theater Sweeps Taiwan,” Test 4 = “The End of the Courtyard Compound,” and Test 5 = “Take the Anxiety out of Learning Math”.
3.3.3 Pretest
The pretest, which was made up of twenty target words selected from five texts,
was administered to the participants for half an hour on September 30
thto assess their
prior knowledge about them. These words were randomized in the form of Vocabulary
Knowledge Scale (Paribakht & Wesche, 1993, p5). In this test, students had to
evaluate their knowledge of each word by means of Vocabulary Knowledge Scale,
which was a 5-point scale ranging from total innocence of the words, recognition of
the word, knowing its meaning to the ability to use the word appropriately from the
perspectives of semantics and grammar (See Figure 3-1). This instrument required the
participants to self-evaluate and display their knowledge of certain lexical items in
written form. It was not designed for measuring students’ ability beyond using words
in initial contextualized production, such as knowing additional word meanings, or
derivational, paradigmatic, semantic and other relationship and networks. Concerning
the reliability of this scale, it was estimated through test-retest administration of a
word list to 93 students at 6 different proficiency levels in the 1992 ESL summer
school program. A Pearson correlation of .89 for scores on the 24 content words,
and .82 for scores on the 8 discourse connectives were reported, which proved that
reliable responses from the participants could be collected through this instrument.
None of the participants were taught about any of these target words during the
interval between the pretest and the immediate test.
Figure 3- 1: VKS Elicitation Scale --Self Report Categories
(Designed by Paribakht& Wesche, 1993)
Self-report categories
I. I don’t remember having seen this word before.
II. I have seen this word before, but I don’t know what it means.
III. I have seen this word before, and I think it means __________.
(synonym or translation)
IV. I know this word. It means ___________. (synonym or translation) V. I can use this word in a sentence:
_________________________________________. (Write a sentence.) (If you do this section, please also do Section IV.)
Speaking of its scoring criteria (See Figure 3-2), one point was allotted if the
participants didn’t remember having seen the target word before. Two points were
given if the participants had seen the target word but did not know what it meant, or
when they made mistakes in the third, fourth, and fifth level. The participants would
get three points if they could provide a proper synonym or L1 translation for the target
word. Four points were allotted if the participants could use words in right semantic
context but in wrong grammar. Only when the participants used the target word with
both semantic and grammatical appropriateness could they win five points for that
lexical item. The whole Pretest is illustrated in Appendix C.
Figure 3- 2: VKS Scoring Categories-- Meaning of Scores
(Designed by Paribakht& Wesche, 1993)
Self-report Categories
I. 1. The word is not familiar at all.
II. 2. The word is familiar but its meaning is not known.
III. 3. A correct synonym or translation is given.
IV. 4. The word is used with semantic appropriateness in a sentence.
V. 5. The word is used with semantic appropriateness and grammatical accuracy in a sentence.
3.3.4 Experimental Reading and Immediate Tests on Comprehension and Vocabulary Gain
Subjects read five articles one by one in different weeks mainly to get the gist
and answer questions in the immediate comprehension test. They were later given
immediate vocabulary test on those four target words from each text unexpectedly in
each meeting in every other week. They read the article “I Can Only Cheer Them
On!” on October 7
th, “Old Hakka Homes of Southern Taiwan” on October 21
th,
“Taking the World by Storm: High Energy Puppet Theater Sweeps Taiwan” on
November 4
th, “The End of the Courtyard Compound” on November 25
th, and “Take
the Anxiety out of Learning Math” on December 9
th. Originally, the participants were
scheduled to read the fourth article “The End of the Courtyard Compound” on
November 18
th. It was postponed to one week later because they had to take their
midterm on November 18
th. After reading each text, subjects were required to answer
five comprehension questions (two global questions and three local questions) in a
comprehension test. After that, an immediate vocabulary test in the form of
Vocabulary Knowledge Scale was administered to assess their lexical gain of these
four target words in each article. One of five experimental readings with their
comprehension and vocabulary tests is served as a sample and is attached in Appendix
B.
3.3.5 Posttest on Retention
A posttest in the form of Vocabulary Knowledge Scale was conducted to assess
the participants’ retention of the twenty target words from the previous five texts four
weeks after the fifth test (January 6
th). Though the intervals between the previous five
tests and the posttest were different, they were over twenty eight days. According to
the results of German philosopher Hermann Ebbinghaus’ experiment, the rate of
forgetting leveled off over time (see the accompanying figure 3-3 entitled “Forgetting
Curve
2”). In other words, subjects’ retention of certain target items would be very
similar twenty eight days after their learning. In order to avoid the effect of different
intervals on students’ performance in vocabulary retention, the posttest was arranged
twenty eight days after the fifth test. The order of occurrence of all the target lexical
items in the posttest was arranged differently from that in pretest. The posttest is
attached in Appendix E.
2 German philosopher Hermann Ebbinghaus conducted the scientific study of human memory in experiments with a view to measuring forgetting. He initiated the experiment in 1879 and his findings were published in 1885 in his book, On Memory. Ebbinghaus made up numerous lists of nonsense syllables, which were comprised of pronounceable but meaningless three-letter combinations like XAK or CUV. He, the sole participant, tried to memorize a list by repeating those target items over and over, until he could recite the list once without error. His retention of the list was assessed after an interval ranging from 20 minutes to 31 days. The results revealed that the rate of forgetting was relatively consistent. The curve dropped relatively sharp at first and then it eventually appeared to flatten out, showing no additional forgetting over time. Other psychologists have since verified that the shape of the forgetting curve generally remains the same for many different types of material.
Figure 3- 3: Hermann Ebbinghaus' Forgetting Curve
Note. The X axile represents the time the retention is measured after learning. The Y axile signifies the percentage of memory Ebbinghaus still possesses for those target items.
