METHODOLOGY
This chapter gives an account of the design of this study. The characteristics of the subjects are introduced first. Second, the instruments, including the vocabulary test and the reading comprehension test, are described in detail. Finally, the procedures for conducting the tests and the method for analyzing the collected data are presented.
Participants
Participants in this study included eight intact classes of 2nd-year senior high school students, totally 313 students from two senior high schools in Keelung City, but after careful scrutiny, the final sample consisted of 306 subjects. The subjects, aged between 17 and 18, are all native speakers of Mandarin Chinese and have received formal English education for at least seven years, since they were 5th graders in the elementary school. The reasons to recruit students from different schools are to achieve a more comprehensive comparison among students with different English proficiency. The admitted students’ scores on the English test of the Basic Competence Test in the two schools range broadly, from the full scores of 60 to 30.
Moreover, since both schools adopt ability grouping in accordance with students’
academic grades, this study carefully selected from high- and low-achieving classes to make the samples as much correspondent as possible with the average of students in these schools. The samples in this study were controlled to fairly reflect the reality of bi-polarity English proficiency in senior high schools in Taiwan. Thus, the research findings might be more applicable to Taiwanese EFL learners at the high school stage
in general.
Instruments
Data for this study were collected through paper and pencil testing. Two major instruments were applied in this study: a vocabulary test and a reading test.
Vocabulary Test
The vocabulary test in this study was a bilingual version of Nation’s newly developed Vocabulary Size Test (VST). The reason for using the VST is that it is the corresponding test to measure ESL learners’ vocabulary size on the BNC Word Family List. The corpus-based BNC Word Family List, as Nation (2006) argued, may be a good representation of modern-time native speakers’ vocabulary, sequenced in a reasonable frequency order and thus provides more accurate estimates of the vocabulary size needed for English reading. As for the VST, Nation claimed that it could be used as a ‘rough guide’ for estimating a learner’s vocabulary size.
As introduced earlier, the Vocabulary Size Test aims at measuring non-native speakers’ receptive vocabulary size and the levels of the test are organized in accordance with the word frequency levels of the BNC Word Family List. Nation (2007) suggested that it is not necessary to make learners sit all the levels of the tests when the test is used with elementary or intermediate learners, but they should sit a few levels beyond their present level. Taiwanese university students’ vocabulary size was found to be below the level of 3,000 word families (Huang, 2000). In consideration of Taiwanese students’ proficiency level, this study used the 5th 1,000 level as the upper limit and administered the first five levels of VST to subjects.
The VST consists of ten items in each 1,000 words level, sampling one tested
word from every 100 words. Presented in a multiple-choice test format, each test item contains the word form of the tested word, put in a simple non-defining context.
The context is arranged to indicate the part of speech of the word, orient the test-taker to view it as an item of language use, and to provide a little extra associational help in accessing the meaning. The part of speech and the contexts are chosen to reflect the most frequent environments for the item. For example, with the word “instance”, there is very high frequency of using it within the phrase “for instance”, so this is used as the context. The distracters are carefully selected to share elements of meaning with the correct answer so that they are genuine choices. Test-takers need to have a moderately developed idea of the meaning of the tested word to be able to choose it from the four options.
As VST is primarily based on frequency-driven word lists, the levels of the VST are organized in a way that the test begins with the items more likely to be known. It is expected that for most test-takers, the scores for each level on the test descend from the higher frequency level to the lower one. The reason behind the expectation is that frequency level is strongly related to the likelihood of a word being known (scores at each level drop as a learner progresses through the test), and in general, learners acquire more frequently used words before they acquire less frequently used ones (Nation, 1990, cited in Schmitt, N., Schmitt, D., & Clapham, C., 2001). It has also been proved that learners do better on the higher frequency levels than on the lower frequency ones (Schmitt, 2001).
The VST in this study was made bilingually, with English words in the stem and Chinese in the options. The reason to use the learners’ first language in the test was because that this study targeted at EFL learners, whose English learning is usually bilingual-oriented in the form of L2-L1 word pairs. Laufer and Goldstein (2004) gave support to the use of bilingual vocabulary tests for learners in the foreign language
context. Laufer suggested that choosing the correct L1 translation of a word in the bilingual version was easier than choosing the correct L2 definition in a monolingual version of vocabulary test. In Taiwan’s teaching and learning context, vocabulary acquisition before the college stage is usually done through a systematic procedure and approach. Students in Taiwan usually learn new English words based on word lists in their English textbooks, compiled according to the English Curriculum set for schools (Shen, 2003). The target words are learned by memorizing their forms and meanings, which are most of the time presented in their Chinese translation equivalents and accompanied with their English interpretations. As the knowledge of a new English word is stored by means of its Chinese translation equivalent, a vocabulary test with options in Chinese is more valid in testing whether a learner recognizes the target word. Moreover, Read (2000) suggested that the proficiency level of the learners should be taken into consideration to decide whether to use a bilingual format or not. He noted, “L1 is likely to play a greater role in assessing the vocabulary knowledge of lower proficiency learners than those who are more advanced.” Since the vocabulary size of most senior high school students in Taiwan is under the 2,000 words level (Huang, 2000), the bilingual vocabulary test is likely to achieve higher validation than the monolingual one in assessing the recognition vocabulary size of these learners. The monolingual version uses synonymous English words or short defining phrases to express the meanings of the targeted words whereas the bilingual version uses their Chinese translation equivalents. Since Chinese translation is so straightforward that if a Chinese native speaker has a rough idea of the tested word, he should be able to choose or match the answer immediately and correctly without being hampered by a lack of knowledge of the synonymous English words and phrases. On the contrary, in a monolingual version, the meaning of a tested word is expressed by its broad definition in phrases in English, which
requires the test takers to read more English words and to have basic understanding of syntax structure before choosing the answers. As Nation (2001) stated, the greatest value of the first language in vocabulary testing was that it allowed learners to respond to vocabulary items in a way that did not draw on the irrelevant second language knowledge. Using first language translations for the meanings makes the test much more sensitive to partial knowledge (Nation, 2001, p.350). As a result, a vocabulary test in bilingual version requires a minimum of reading, allowing more items to be taken within a given period of time and is always expected to lead to higher test scores. Bilingual version can make the result of the test better reflect the learners’ real recognition vocabulary ability than monolingual one and thus bilingual version is more suitable for assessing the vocabulary size of second language learners with lower proficiency. As Nation (2001) argued, first language translations provided a very useful means of testing vocabulary.
The number of items in the Vocabulary Size Test in this study was 50 (10 items
× 5 levels). Because of its straightforward type of items, multiple choices in bilingual version, the test could be done very quickly. A session of 20 minutes was enough for sitting the test.
Reading Comprehension Test
The reading comprehension test in this study modeled the multiple-choice format of SAET reading comprehension section and was made of six passages and 24 questions, with four test items attached to each passage for measuring comprehension.
The SAET reading comprehension section normally contains four short passages, covering a variety of styles and topics and followed by comprehension questions. The reading passages, within the number of words ranging from 150 to 300 per text are short enough to encourage students to read them word-by-word. It could be to check
the students’ global comprehension of the passage and their grasp of particular reading skills or strategies (e.g., finding the main idea, making inferences). For the purpose of making the selection of texts more representative for the SAET in general, twenty passages in the SAET reading comprehension section of the years 2005 through 2009 were inspected with four major criterions: topic classification, the number of words, text coverage distribution and the readability of passages (Appendix C). A readability formula in Microsoft Word: the Flesch Reading Ease Formula was used to assess their grade-levels. Among the twenty reading passages, twelve had a length between 200 and 250 words. In terms of text coverage, about two thirds of the passages were covered more than 85% and 90% by the first 2,000 and 3,000 word families of the BNC Word Family List, respectively. Their readability indices rated by the Flesch Reading Ease formula range from the lowest 38 to the highest 75, with ten out of the twenty texts scoring between 60 and 70, and seven scoring between 50 and 59. The higher the score, the easier the text is to read. A readability index between 60 and 70 is considered a standard in difficulty and is estimated to be understandable by readers finishing the seventh or eighth grade education in America while an index between 50-60 is regarded as fairly difficult (from http://www.utexas.edu/disability/ai/resource/readability/manual/forcast-versus-flesch-English.html). Six texts, including the intact reading comprehension section of 2007 and two other texts extracted from that of 2008, whose criterions matched best with the average, were then chosen.
Table 10 Characteristics of the chosen reading samples from SAET Chosen Texts
Passage number 1 2 3 4 5 6
Sources 07-1 07-2 07-3 07-4 08-2 08-3
Number of
words 246 235 214 246 229 266
Flesch Reading
Ease 69.6 44.0 59.2 57.9 52.8 58.0
BNC 2,000
coverage (%) 87.40 88.23 82.33 92.30 90.26 86.94 BNC 3,000
coverage (%) 89.84 90.75 88.84 93.92 92.80 89.55 Topics Health Media Fashion Science Travel Art Note: in the row of sources, 07-1 means the first reading text in the SAET reading
comprehension section of the year of 2007.
As shown in Table 10, the chosen passages covered a variety of topics and the numbers of words ranged around 210-270 words. Except text number three, almost over 87% and 90% of the texts’ running words were covered by the first 2,000 and 3,000 word families of BNC Word Family List, respectively, close to the average text coverage in the past. The readability assessed with the Flesch Reading Ease Formula was mostly rated close to 60, fitting with the average readability score. The readability score of the first text was the highest 69.6 and the second was the lowest 44.0. These two passages were chosen to reflect the reality of SAET, because SAET reading comprehension section normally contains passages with distinct levels of readability. To prevent the possibility that some participants might have practiced the past SAET reading comprehension tests, at the end of each passage there was a question asking if the participant had read the passage before. If it was checked yes,
the subject was to be eliminated from the sampling population.
Procedures
The purpose of this study was to investigate the correlation between vocabulary size and reading performance in the college entrance examination context. First, the first five levels of the Vocabulary Size Test were adopted to measure the subjects’
receptive vocabulary size. Second, six passages and 24 test items were sampled from the SAET reading comprehension section to test subjects’ reading ability. In selecting the passages, the topic, the readability, the number of words and the text coverage of each passage were checked to ensure representability of the samples.
Then, three hundred and thirteen subjects from two senior high schools were recruited to take both the vocabulary size test and the reading test. The two types of tests were administrated at two separate sessions within two weeks. The testing time was arranged to prevent the subjects from fatigue, which might negatively affect their performance on the tests. The VST given at the first session lasted for 20 minutes.
The reading test alone was administrated at the second session and every subject was restricted to complete the test in 50 minutes. The administration of the SAET consisted of the reading of seven passages and writing a short composition and it required about one hundred minutes to complete. After repeated piloting, a span of 50 minutes was determined to be adequate for taking the six-passage reading test in this study. Since the subjects were from different schools and different classes, they did not take the tests at the same time. Finally, the tests were scored for further analysis.
Data Analysis
This study set out to investigate the correlation between vocabulary size and
reading performance and to examine what vocabulary size was needed for comprehending SAET reading texts. To analyze the data, the subjects’ scores on the tests were calculated. In the vocabulary test, a subject's score was calculated by counting the number of correct responses in each level, one point for each correct answer. When scoring the vocabulary test, the scores for each level and the total five levels were recorded. This provided a profile of a subject’s vocabulary in a number of distinct levels, rather than just a single-figure estimate of overall vocabulary size.
With ten items at each level, the highest score for each level was 10 and the possible highest score for the vocabulary test in this study was 50. As the scores were used to estimate a subject’s passive receptive vocabulary size, a score at each level represented the proportion of all the words known at that level. For example, if a subject scored 7 out of 10 at the 2nd 1,000 word frequency level, that meant 70% of items were correctly answered or 700 out of 1,000 words were known at this level.
As for the reading test, the scoring was based on the number of items correctly answered and the reading success was indicated by the percentage of correct answers on the reading test. Consequently, each subject had a set of scores containing a series of variables, as illustrated below:
Vocabulary Size Test Subject
L1 L2 L3 L4 L5 Total
Reading Test Score S1
After the marking and calculation of scores was done, the data were examined for anomalies. Firstly, three subjects checked having done part of the reading test were eliminated from the sample. Besides, four more sets of data were considered anomalous because they displayed obvious signs of pure guesswork or of performance incongruity between the two tests. For example, two students answered B, or C throughout the 24 items in the reading test. And two others were suspected of
dishonesty because they got low scores in the reading test but abnormal high scores on the vocabulary test with the same wrong answers at completely the same items.
Totally, there were seven sets of invalid scores discarded. Then the left 306 sets of scores were used for analysis.
The estimate of an individual subject’s vocabulary size on the vocabulary test in this study was computed in proportion, according to how many items answered correctly in each level. Because there were ten items at each 1000 word frequency level, each item in the test representing 100 word families, a test-taker’s score was multiplied by 100 to get his vocabulary size at the specific word family level in this study (Nation and Beglar, 2007). A subject’s accumulated vocabulary size at the five distinct levels: the 1,000, 2,000, 3,000, 4,000 and 5,000 words levels was calculated by adding his/her vocabulary size at each 1000-word frequency level. For example, if a subject scored 9 at the 1st 1,000 level, 8 at the 2nd level, 4 at the 3rd level, 6 at the 4th level and 4 at the 5th level, the vocabulary size of such a subject was estimated at 3,100 words: 900+800+400+600+400=3,100. In other words, the subject had a receptive vocabulary size close to 3,100 word families at the 5,000 words level, 2,700 word families at the 4,000 words level, 2,100 word families at the 3,000 words level, 1,700 word families at the 2,000 words level, and 900 word families at the 1,000 words level. To obtain deeper insights into the correlation across distinct levels of vocabulary size, the accumulated vocabulary sizes at each of the frequency level were correlated with the reading performance and were compared in next chapter.