This chapter reports the results of the vocabulary test inspected with the three-parameter logistic Item Response Theory model (3PL IRT model). The first section
36
presents the comprehensive fit statistics and then reports item characteristic curves and the values of the three parameters.
Fit Statistics
The overall model fit of the test was calculated as 1.17, which demonstrates a good fit to the 3PL LTT model. That is, the entire vocabulary test presented
unidimensionality. This study was thus evaluated as evaluating a single latent ability. In general, most of the test items showed good fit. Only Item 59 (No.74 on the test) in Level Four failed to fit the LTT model. Since the levels of the words were rearranged on the test to prevent test fatigue, the item number was not the same as for the test item.
The clear coding was in Appendix C.
The misfit item is as follows:
59.someone who is kept as a prisoner by an enemy (A)receiver
(B)laughter (C)hostage (D)container
Figure 6. Test Information Function
37
Figure 7. Conditional Standard Error of Measurement (CSEM) Function In Latent Trait Theory, our interest is in estimating the value of the ability parameter for an examinee. The Test Information Function is used to calculate the amount of information that can be obtained from the test. The ability parameter is denoted by θ. Inspection of Figure 6 shows that the amount of information has a maximum at an ability level of 39.361 at θ = −0.300.Apart from that, as shown in Figure 7, the minimum CSEM was 0.159 at θ = −0.300, which was within the range−2<θ<2.This indicates that abilities were estimated with precision. Outside this range, the amount of information decreased rapidly, and the corresponding ability levels were not estimated very well. It can be observed that the test in question provided useful and valid information on learners whose abilities ranged from θ =−2 to θ = 2 and that it provided the most information when the value of θ fell between 0 and 2. Based on the unimodal distribution of test responses, the feature of
unidimensionality was confirmed, indicating that the test measured only one latent
38
ability. This demonstrates without a doubt the validity and reliability of this vocabulary size test.
Item Characteristic Curves (ICCs)
In Latent Trait Theory, the item characteristic curve describes the relationship between latent ability and performance on a test item. The shape of the ICC is
determined by the parameters a, b, and c: a indexes the discrimination of the item, while b is the item difficulty parameter, equal to the value of θ where the probability of a correct response equals .50 + (c/2). It follows that multiple-choice items with more positive b parameters are more difficult for examinees, as a higher trait level is required to choose the keyed response 50% of the time. Finally, c equals the probability of an examinee with an infinitely low θ obtaining a correct response due to guessing.
In order to examine the difficulty of an item more closely, let us discuss the ICCs for a parameter b. The further the curve is to the right end of the ability scale, the more difficult the item is. On the other hand, if the curve is closer to the left end, the item is easier. The most difficult items on this test were Items 86, 109, and 84, which lie on the right end of the scale. In contrast, the easiest items were Items 1,2, and 18, which lie on the left end of the scale, as shown in Figure 8.
39
Figure 8. Item Characteristic Curves for Items 1 and 86
Figure 9. Item Characteristic Curves for Items 26 and 116
Figure 10. Item Characteristic Curves for Items 47 and 86
40
The parameter a shows the discriminating power indicated by the slope of curves. If the slope of the curve is steeper, then the item has greater discriminating power. From Figure 9, we see that Item 116, with the steepest slope, had the most discriminating power. On the other hand, the milder curve of Item 26 manifested the least discriminating power. As for c, it is shown by the lowest level of the ICC. More details are shown in Figure 10. The guessing value of Item 47 was 0.2659; on the other hand, the guessing value of Item 86 was 0.189. This shows that Item 86 had the least probability of being influenced by guessing. The complete ICC graphs are provided in Appendix D.
Values of Item Parameters
A complete summary of parameter statistics is given in Table 2. The entire 180 items of parameter statistics are given in Appendix E. As shown in Table 2, the minimum difficulty statistics of the test was at −2.699, whereas the maximum was at 2.840. The range of the difficulty value was within the range of−3 to 3, which meant that the performance of items fit the 3PL LTT model. The distribution of difficulty across items is presented in Figure 12.
Parameter Items Mean SD Min Max
a (Discrimination) 180 0.888 0.185 0.410 1.401
b (Difficulty) 180 −0.352 1.127 −2.699 2.840
c (Guessing) 180 0.241 0.009 0.185 0.266
Table 2. The Descriptive Statistics for the Parameters a, b and c
Figure 11. The Frequency Distribution of the a Parameter
41
Figure 12. The Frequency Distribution of the b Parameter
The frequency distribution of a is shown in Figure 11. The acceptable range of discrimination was from 0.6 to 2.5 in practice, and values of discrimination lower than zero should lead to deletion of the test items. Negative discrimination value implies that the more excellent the subject, the lower the chance of answering the question correctly (Brown & Hudson, 2002). The discriminating power of the questions given in Table 2 show values ranging from 0.410 to 1.401, which means that all of the items on the test were in the acceptable range. The following items are the top five items—those with the greatest discriminating power. On the other hand, the five test items with the lowest discriminating power could also be listed as a comparison group.
Top five most discriminating items:
116.a natural ability for being good at a particular activity (A)extent (B)talent (C)contrast (D)airport 20. extremely big
(A)vast (B)warm (C)soft (D)afraid 137.a complete change from one thing to another
(A)portray (B)switch (C)persuade (D)function 115. an area of ground
(A)land (B)root (C)nose (D)cash 180. large, cow-like animal, with long, curved horns
(A)twilight (B)buffalo (C)sweep (D)demise
42 Top five least discriminating items:
26. a special ability to do a particular thing
(A)content (B)faculty (C)setting (D)outcome 25. small in amount or number
(A)obvious (B)unique (C)ethnic (D)limited 74. to move slowly from side to side
(A)alert (B)uphold (C)sway (D)grip 110. a slope in a road
(A)bowl (B)hill (C)hall (D)host
79. to pick up something quickly with your fingers
(A)insure (B)shiver (C)mobilize (D)pluck
Figure 13. The Frequency Distribution of the c Parameter
The last parameter to be discussed is the pseudo-guessing parameter. For the multiple choice format with four options, the value of the pseudo-guessing parameter should be lower than 0.25, which means that although the subjects guessed randomly, their probability of getting the correct answer should be below that value. In this study, most items had values of c between 0.24 and 0.28. The test items with the lowest guessing values were Items 86, 130, and 84, with c values of 0.1849,0.1977, and 0.202, respectively. On the other hand, the items with the highest guessing values were Items 47, 150, and 160, with c values of 0.2659, 0.2575, and 0.2566, respectively. As shown in Table 2, the mean guessing value of all the items is 0.241, which is slightly lower than the ideal value.
43 Top five items with the lowest guessing values:
86.to avoid activity, especially work
(A)stalk (B)bond (C)undo (D)loaf 130.knowing about a fact
(A)minor (B)given (C)armed (D)flat
84. to try to persuade people to buy goods or services you are offering (A)tout (B)tour (C)usher (D)surge
85.the use of something in order to get an advantage from it (A)brochure (B)exploitation(C)manifestation (D)workforce 149.the physical and electronic parts of a computer
(A)hardware (B)nerve (C)barrel (D)narrative
Top five items with the highest guessing values:
47.a part that you make or join to another part
(A)transaction (B)addition (C)determination (D)pension
150.to close an entrance or container so that nothing can enter or leave it (A)tent (B)myth (C)seal (D)cow
160.to suddenly reduce something greatly
(A)strive (B)neglect (C)slash (D)combat 26. a special ability to do a particular thing
(A)content (B)faculty (C)setting (D)outcome 51.the part of a person’s face below their mouth
(A)slide (B)chin (C)strain (D)horn
By focusing on the five items with the lowest and highest guessing values, it can be seen that when subjects guess the correct answer, the position of options might have an effect on their guess. The answers of the five items with the lowest guessing values were mostly Option A or D, while those of the five items with the highest guessing values were Options B and C. However, even the item with the highest guessing value had a guessing value of 0.2659, which is only slightly over 0.25, the ideal guessing value.
Therefore, this vocabulary test showed that it was less likely for the subjects to guess successfully on the test items.
44