To answer the three research questions posed in Chapter One, the qualitative data presented in Chapter Four will be presented and discussed.
The research questions are as follows:
1. How do the test items based on the multiple choice vocabulary test format fit the 3PL Latent Trait Theory model?
2. What are the reliability and validity of the newly developed vocabulary test based on the new word frequency list from COCA and tested by Item Response Theory?
3. How did the selected margin words of the test items show differences in item difficulty in their word frequency level?
There are two parts to this chapter. First, the empirical evidence will be
summarized to provide supporting evidence for the format, reliability, and validity of the vocabulary size test in question. The second section will discuss how the item difficulty (parameter b) interacts with the frequency bands. In addition, through content analysis, particular items will be closely examined to gain more information on the test items.
Test Quality
The preceding chapter mentioned that the overall model fit of the vocabulary test is 1.17, indicating that the multiple choice test format is an accurate and ideal estimate of vocabulary size since the value of the model fit is within 3. Furthermore,
45
there is only one misfit item out of 180 items. Good-fit items account for almost 99 percent of the test, confirming that this multiple choice test format is appropriate. As a consequence, the multiple choice test format constitutes a valid measurement format for vocabulary size. As for the validity of the test in question, it can be shown to be valid by examining the good model fit and the hierarchical mean difficulty of each frequency level. Moreover, it satisfies the basic assumption of Latent Trait Theory, which suggests that the test measures only one single ability. In other words, how the subjects perform on the test is only determined by the subjects’ vocabulary size, which ensures the validity of the whole test. Moreover, the mean difficulty values for each frequency level, as shown in Figure 14 in the preceding chapter, increase nonlinearly, which means that as the difficulty rises, frequency bands go upward. This corresponds with the findings of previous research that frequency plays an important role in word difficulty: If a word is more frequent, the simpler it is for learners to acquire it (Aizawa, 2006; Meara, 1992; Milton, 2006). Additionally, the relationship between frequency bands and test item difficulty is firm evidence that the test items can demonstrate the subjects’ vocabulary size. The reliability value of the test is 0.97, which indicates that there is no doubt that this vocabulary test is both reliable and valid.
Misfit Item
From the total of 180 test items, only one misfit item was identified. The misfit item showed low point biserial values, indicating that this test item could not show subjects’ true ability. Those who got the right answer were low achievers; however, high achievers could not do well on this item. The target words of the four items are receiver, laughter, hostage, and container, which are all in Level 4. They are comparatively more difficult words for senior high school freshmen and sophomores. Three of the four target
46
words contained distractors, which misled some higher achievers because of their partial knowledge of word parts. Laufer (1997) indicated that L2 learners face three kinds of lexical plight, which are words you don’t know, words you think you know, and words that you can’t guess. In the present study, it is found that the second type, “words you think you know,” had the nature of powerful distractors. The distracting word parts in these words make the comprehension of the target word more difficult. This can be seen in the stem, which describes someone who is kept as a prisoner by an enemy. As a consequence, since the other three options—receiver, laughter, and container—end with –er and the correct answer, hostage, does not, this might have resulted in misleading subjects into a false choice on the test because of the morphological structure of the distractors. To sum up, the cause of the misfit item might be the word difficulty and the distracting parts of the target words.
Item Difficulty and Frequency Bands
In this section, how the Item Difficulty (parameter b) interacts with frequency bands will be discussed. Furthermore, through content analysis, particular items will be closely examined to gain more information on test items.
Among the 179 good-fit items, the difficulty parameter fell in the proper range.
Any items with the highest or lowest difficulty values should be included. Since the present study aims to measure if subjects pass a vocabulary band of a particular level, it is necessary to include items that fall at both ends of the range of difficulty values.
Consequently, as long as they are good-fit items, they should be kept on the test.
In order to see whether the subjects pass a given frequency band, the test is split into two parts: Form A (from Item 1to 90) and Form B (from Item 91 to 180). Form A consists of target words chosen randomly from 950–1000, 1950–2000, 2950–3000,
47
3950–4000, 4950–5000, and 5950–6000, while Form B is made up of target words randomly chosen from 1001–1050, 2001–2050, 3001–3050, 4001–4050, 5001–5050, and 6001–6050. Table 3 clearly shows the average item difficulty in each level, and the clear line charts for Form A and Form B are shown in Figure 14 and Figure 15.
Table 3.The Descriptive Statistics for the Item Difficulty in Each Frequency Level
Form A LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 LEVEL 6
RANGE
ITEM 950-1000 ITEM 1950-2000 ITEM 2950-3000 ITEM 3950-4000 ITEM 4950-5000 ITEM 5950-6000
1 -2.6993 16 -1.8087 31 -1.6783 46 -0.7067 61 1.8365 76 0.2248
2 -2.6341 17 -2.182 32 0.179 47 2.082 62 0.1825 77 -0.7579
3 -2.2226 18 -2.6405 33 -0.2626 48 -1.6368 63 -0.2003 78 0.9703
4 -0.0415 19 -2.5133 34 0.5692 49 -0.913 64 -1.3657 79 -0.3827
5 -1.0698 20 -1.7892 35 -0.8647 50 1.5665 65 1.6961 80 1.0767
6 -1.1794 21 -0.373 36 -1.1668 51 -0.2429 66 0.5066 81 -0.0931
7 -2.5006 22 0.2975 37 -0.9331 52 -0.0672 67 0.1047 82 -1.0906
8 -1.0042 23 -1.4577 38 -1.6391 53 0.1388 68 0.1192 83 0.1131
9 -2.3933 24 -0.7488 39 0.026 54 -0.2867 69 -0.2731 84 2.1997
10 -2.556 25 0.3129 40 -0.4345 55 -0.7316 70 1.074 85 2.0754
11 -2.5304 26 0.7774 41 -0.3464 56 -0.4482 71 0.5943 86 2.8399
12 -1.4759 27 -1.4034 42 -0.7104 57 0.9043 72 -1.0475 87 -0.0193
13 -1.8137 28 -2.1502 43 -1.6777 58 -0.7366 73 0.082 88 -0.4582
14 -2.0063 29 -1.6904 44 -0.3062 59 -0.135 74 -0.5151 89 1.9099
15 -0.9834 30 -0.6581 45 1.0552 60 0.3823 75 1.5954 90 0.5531
Item Difficulty
Average -1.807367 -1.2018333 -0.546027 -0.0553867 0.29264 0.61074
Form B LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 LEVEL 6
RANGE
ITEM 1001-1050 ITEM 2001-2050 ITEM 3001-3050 ITEM 4001-4050 ITEM 5001-5050 ITEM 6001-6050
91 -1.653 106 -1.0927 121 0.0008 1.2576 151 0.5773 166 0.2182
92 -2.1937 107 -0.6758 122 1.2556 137 -0.4296 152 1.1982 167 -0.8751
93 -1.8266 108 -0.5444 123 -0.4192 138 -0.5253 153 0.4585 168 0.298
94 -1.1873 109 2.2162 124 -0.7613 139 -1.027 154 0.0695 169 0.3667
95 -0.8307 110 0.1107 125 -0.0139 140 0.6161 155 -0.926 170 -0.0326
96 -1.8602 111 -1.4432 126 -0.4449 141 -0.5197 156 0.7907 171 -0.1719
97 -0.062 112 -1.6895 127 -0.0828 142 -1.0324 157 -0.3718 172 0.3504
98 -0.4802 113 -0.8202 128 -1.2933 143 0.1438 158 -0.5411 173 0.3281
99 -1.9839 114 -0.4592 129 -0.9956 144 -1.4213 159 1.6764 174 0.9979
100 -0.908 115 -1.3432 130 1.9139 145 -0.6936 160 0.703 175 -1.0921
101 -0.5506 116 -0.9838 131 -0.1172 146 0.5397 161 -0.4582 176 -0.0704
102 -0.5935 117 0.1733 132 -0.2174 147 0.6915 162 2.0926 177 0.5864
103 -1.7989 118 -1.1227 133 -0.6872 148 -0.8904 163 1.5516 178 1.3376
104 -0.3609 119 0.2737 134 -0.8735 149 -0.229 164 0.6774 179 0.4758
105 -1.3145 120 -1.3644 135 -0.8442 150 0.6491 165 0.3429 180 -0.5029
Item Difficulty
Average -1.1736 -0.5843467 -0.23868 -0.1913667 0.5227333 0.1476067
48
Figure 14. The Relationship between Word Frequency Level and Item Difficulty (for Items 1–90)
Figure 15. The Relationship between Word Frequency Level and Item Difficulty (for Items 91–180)
The mean difficulty value of Form A across Levels One to Six is about −0.45, whereas the mean difficulty value of Form B across Levels One to Six falls at −0.25.
That is, Form B is slightly more difficult than Form A. It may be observed that because the range of margin words in each frequency band in Form B is higher than in Form A, the average item difficulty is thus higher than for Form A. In the past, researchers who designed vocabulary size tests picked target words distributed equally through their frequency band (Beglar, 2010; Beglar& Hunt, 1999; Cheng,
49
2014). However, there must be a fuzzy area at the edge of each frequency band;
therefore, the present study provides a tool for researchers to determine more carefully whether subjects passed a particular frequency level.
When analyzing the mean difficulty value of each frequency band, it is found that the word frequency is inversely proportional to item difficulty. That is, the higher the word frequency level, the lower the item difficulty. It is worth knowing that the mean difficulty values show a nonlinear upward slope. In fact, the increase of difficulty between Levels 1 and2 and Levels 2 and3 is much steeper than that
between Levels 3 and4, Levels 4 and5, and Levels 5 and6. Chen (2011) pointed out that this phenomenon further explained how the words from Levels One to Three, which are close to the 2000-word families, “usher in the relatively easy attainment of the learning goals of 4000 words.” In other words, it would be comparatively easy for second language learners to acquire words beyond Level Three if they have
constructed a mental lexicon of the first three thousand words. In addition, this situation further corresponds with two ideas related to vocabulary learning. First, vocabulary size plays an important role in reading comprehension. Extensive reading has long been associated with vocabulary gain. Second language learners can read extensively without help in L1 after they attain a certain vocabulary size
(Laufer,1982; Lee & Schallert,1997; Nagy et al., 1985; Nation, 1992, 2006). Hence, it is essential for learners to have a fairly large vocabulary to apprehend the material they read or acquire new words from the text.
Second is the idea of word schemata, which involve metacognitive,
metalinguistic, and morphological knowledge as well as typical patterns of word meaning (Nagy& Scott, 1990). Whether consciously or not, learners are able to recognize the elusive regularities and patterns of words and then take the further step
50
of applying them to acquire new words. Generative knowledge, which is the metalinguistic and morphological knowledge of words, should be given more emphasis, as this is knowledge that can be used in acquiring other new words (Baumann, Kame’enui, & Ash, 2003; Blachowicz& Fisher, 2004; Graves & Watts-Taffe, 2002; Scott & Nagy, 2009). It is known that word parts, word associations, collocations, and usage are useful techniques when L2 learners try to gain new words. Carrell and Eisterhold (1983) indicated that when teaching vocabulary in reading, the instructor has to keep target words connected with learners’ prior knowledge of vocabulary instead of only demonstrating the meanings of target words. It is paramount for instructors to combine pre-existing vocabulary and word consciousness for learners to gain new vocabulary, either through extensive reading or with the transfer of word schemata.
Form B showed a slight drop in mean difficulty from Level Five to Level Six, which might suggest some items in Level Six were too easy. Supposedly, items in Level Six should be more difficult than the mean item difficulty of Level Five.
Therefore, the top six easiest items in Level Six, whose mean item difficulty value is below zero, are listed in Table 4.
Table 4.Descriptive Statistics for the Three Parameters of the Items with Low Mean Item Difficulty
Although having low mean item difficulty, these items show satisfactory discrimination and guessing value. The test items are listed below in Table 5: