In this section, the passing rates of the high achievers (Ph) and low achievers (Pl), and the discrimination indexes (i.e., D=Ph-Pl) on each reading level and item type were examined to explore the differences between the high and low achievers on
performances of different test items. Since the discrimination index (D) represents the discrepancy between the passing rates of the high achievers and low achievers, one-way ANOVA was run to examine how different reading levels and item types discriminated the high achievers from the low achievers. The following sections first report the descriptive statistics for the high and low achievers’ passing rates and the discrimination index on each reading level on the SAET and the DRET and the results of one-way ANOVA both tests, and then proceed to those of each item type on both exams. Table 12 showed the passing rates of the high achievers and low achievers and the discrimination index in both exams.
Results of the SAET
As illustrated in Table 15, in terms of the two reading levels in the SAET, high achievers performed quite well on both bottom-up skills and top-down skills, with passing rates of more than 80 for both levels. Low achievers performed poorly on either bottom-up or top-down skills, with passing rates of around 30 for each level.
One thing worth noting is that the passing rates of low achievers on top-down skills were slightly below the standard set by Jeng et al. (1999). Jeng et al. (1999) suggested that the idea item facility index for national examination such as college entrance examination should be 30 to 80.To look at the performance of high and low achievers on different types of reading skills, the passing rates of high and low achievers on each type of skill were examined. The results the passing rates of high and low achievers and the discrimination indexes on different reading levels in the SAET and DRET were presented in the following table.
Table 14. Passing Rates of High & Low Achievers and the Discrimination Indexes on Different Reading Levels in the SAET & DRET
Category N Mean SD
SAET Ph 1. Reading for Plain Sense
(Bottom-Up Skills) 58 85.0172 13.08205 2. Reading into Discourse
(Top-Down Skills) 35 80.1429 12.74672
Total 93 83.1828 13.10405
Pl 1. Reading for Plain Sense
(Bottom-Up Skills) 58 31.2586 12.74247 2. Reading into Discourse
(Top-Down Skills) 35 29.6286 10.28158
Total 93 30.6452 11.84469
D 1. Reading for Plain Sense
(Bottom-Up Skills) 58 53.7586 11.10604 2. Reading into Discourse
(Top-Down Skills) 35 50.5143 10.70561
Total 93 52.5376 11.01240
DRET Ph 1. Reading for Plain Sense
(Bottom-Up Skills) 45 72.2000 15.71566 2. Reading into Discourse
(Top-Down Skills) 29 68.5172 18.48478
Total 74 70.7568 16.82856
Pl 1. Reading for Plain Sense
(Bottom-Up Skills) 45 22.2222 7.47082 2. Reading into Discourse
(Top-Down Skills) 29 24.1724 8.76303
Total 74 22.9865 8.00084
D 1. Reading for Plain Sense
(Bottom-Up Skills) 45 49.9778 13.60396 2. Reading into Discourse
(Top-Down Skills) 29 44.3448 15.55698
Total 74 47.7703 14.56168
Note. Ph refers to passing rates generated by high achievers.
Pl refers to passing rates generated by low achievers.
D refers to the discrimination index (i.e., Ph-Pl).
The high and low achievers refer to the examinees who reach the top and the bottom 33% in the population of the SAET and DRET.
Table 15. Passing Rates of the High & Low Achievers and the Discrimination Indexes on Different Item Types in 2002-2007 SAET
Year Item Type Ph Pl D
Mean SD Mean SD Mean SD N
2002 1. Word Inference from Context 88.0000 .00000 28.5000 4.94975 59.5000 4.94975 2
2. Recognizing Cohesive Devices 99.0000 . 33.0000 . 66.0000 . 1
3. Recognizing and Interpreting Details 89.0000 7.44983 32.6667 11.96871 56.3333 13.53699 9 7. Recognizing Implications and Making Inferences 64.0000 2.82843 25.0000 8.48528 39.0000 5.65685 2 8. Recognizing and Understanding the Main Idea 97.0000 . 54.0000 . 43.0000 . 1
Total 86.7333 11.31034 32.5333 11.49451 54.2000 12.94604 15
2003 1. Word Inference from Context 76.0000 . 17.0000 . 59.0000 . 1
3. Recognizing and Interpreting Details 79.7500 14.25031 24.8750 10.34322 54.8750 10.50765 8 6. Recognizing Presuppositions Underlying the Text 87.0000 . 18.0000 . 69.0000 . 1 7. Recognizing Implications and Making Inferences 59.5000 6.36396 27.5000 .70711 32.0000 7.07107 2 8. Recognizing and Understanding the Main Idea 88.0000 11.13553 31.3333 8.73689 56.6667 4.50925 3
Total 78.9333 14.11922 25.5333 9.03854 53.4000 12.25794 15
2004 1. Word Inference from Context 91.5000 6.36396 27.5000 .70711 64.0000 7.07107 2 3. Recognizing and Interpreting Details
83.2727 14.02206 33.3636 13.42589 49.9091 12.01212 11 8. Recognizing and Understanding the Main Idea 75.0000 14.14214 37.5000 4.94975 37.5000 9.19239 2
Total 83.2667 13.30664 33.1333 11.74045 50.1333 12.76640 15
2005 1. Word Inference from Context 85.0000 . 27.0000 . 58.0000 . 1 3. Recognizing and Interpreting Details 80.0000 17.24980 31.1000 12.70564 48.9000 10.54567 10 4. Recognizing Functional Value 81.5000 9.18332 28.5000 7.76745 53.0000 6.83130 4 7. Recognizing Implications and Making Inferences 91.0000 . 37.0000 . 54.0000 . 1
Total 81.3750 14.27293 30.5625 10.67064 50.8125 9.15947 16
2006 1. Word Inference from Context 62.0000 14.14214 24.0000 9.89949 38.0000 4.24264 2 3. Recognizing and Interpreting Details 88.0000 10.54093 31.8000 16.73187 56.2000 11.38029 10 4. Recognizing Functional Value 71.5000 17.67767 19.5000 10.60660 52.0000 7.07107 2 7. Recognizing Implications and Making Inferences 97.0000 . 44.0000 . 53.0000 . 1
8. Recognizing and Understanding the Main Idea 91.0000 . 32.0000 . 59.0000 . 1
Total 83.4375 14.52799 30.0625 14.73078 53.3750 11.02044 16
2007 1. Word Inference from Context 71.5000 7.77817 23.0000 1.41421 48.5000 6.36396 2 3. Recognizing and Interpreting Details 88.5556 12.23837 32.3333 12.12436 56.2222 7.49630 9
4. Recognizing Functional Value 83.0000 . 45.0000 . 38.0000 . 1
6. Recognizing Presuppositions Underlying the Text 84.5000 6.36396 31.0000 11.31371 53.5000 4.94975 2 7. Recognizing Implications and Making Inferences 78.0000 . 14.0000 . 64.0000 . 1
8. Recognizing and Understanding the Main Idea 95.0000 . 54.0000 . 41.0000 . 1
Total 85.3125 11.34148 32.0000 12.77498 53.3125 8.70799 16
Note. Table 13 continued.
Ph refers to passing rates generated by high achievers.
Pl refers to passing rates generated by low achievers.
D refers to the discrimination index. (i.e., Ph-Pl).
As shown in Table 15, in the 2002 SAET, the high achievers performed well on all types of item types while low achievers performed quite well on “Recognizing and Understanding the Main Idea” items only. The high achievers performed extremely well on items measuring “Recognizing Cohesive Devices” items (Mean=99), followed closely by items on “Recognizing and Understanding the Main Idea”
(Mean=97), while the low achievers performed best on items measuring “Recognizing and Understanding the Main Idea” (Mean=54). As to items that were worst performed, both groups performed worst on items measuring “Recognizing Implications and Making Inferences” (Mean=64 vs. Mean=25 respectively for high and low achievers).
In addition, items of this type also produced the worst discriminatory power in 2002 SAET (D=39). For the discrepancy of these two groups, items on “Recognizing Cohesive Devices” (D=66) had the highest discrimination indexes. Thus items of this type best discriminated higher achiever from low achievers. As illustrated in Table 15, in the 2003 SAET, both high achievers and low achievers performed best on items measuring “Presuppositions Underlying the Text” (Mean=88 vs. Mean=31.33 respectively for high and low achievers). The high achievers performed worst on items measuring “Recognizing Implications and Making Inferences” (Mean=59) whereas the low achievers performed worst on items measuring “Word Inference from Context (Mean=19), which is far lower below the standard index set by Jeng et al (1999). In terms of the discrepancy of discrimination index, items on
“Presuppositions Underlying the Text” discriminated the examinees the best (D=69).
Similar to the results in 2002 SAET, items on “Recognizing Implications and Making Inferences” (D=32) had the weakest discriminatory power.
As shown in Table 15, in the 2004 SAET, it can be seen that the high achievers
performed best on items measuring “Word Inference from Context” (Mean=91.5) while the low achievers performed best on items measuring “Recognizing and Understanding the Main Idea” (Mean=37). As to the worst performed items between these two groups, the high achievers performed worst on items measuring
“Recognizing and Understanding the Main Idea” (Mean=75) whereas the low achievers performed worst on items measuring “Word Inference from Context”
(Mean=27.5), which is slightly below the ideal index set by Jeng et al. (1999). In 2004 SAET, items on “Word Inference from Context” (D=64) had the highest discrimination index in 2004 while items on “Recognizing and Understanding the Main Idea” had the lowest discrimination index.
As to the 2005 SAET, both high and low achievers performed best on items measuring “Recognizing Implications and Making Inferences” (Mean=91 and Mean=37 respectively). The items worst performed by the high achievers were items on “Recognizing and Interpreting Details” (Mean=80) whereas the items worst performed by the low achievers were items on “Word Inference from Context”
(Mean=27), which was below the reasonable discrimination index of 305 suggested by Jeng et al (1999). For the discrepancy between the two groups of examinees, the examinees, the high achievers and low achievers performed most differently on items measuring “Word Inference from Context” (D=58). That is to say, global reading skills best distinguished high and low achievers in 2005 SAET.
5 Jeng (1999) suggested that the minimum discrimination index for an English multiple-choice item with four options should be .30, which is slightly higher than Kuo’s (1989:271) .25 in order to have a higher standard of items.
As shown in Table 15, in 2006 SAET, both high and low achievers were good at items on “Recognizing Implications and Making Inferences” (Mean=97 vs. Mean=44 for the high and low achievers respectively). The high achievers performed worst on items measuring “Word Inference from Context” (Mean=62) while the low achievers performed worst on two types of item that had passing rates far below the reasonable discrimination index suggested by Jen et al. (1999)—“Word Inference from Context”
(Mean=24) and “Recognizing Functional Value” (Mean=19.5). Both high achievers and low achievers performed poorly on global skills. The two groups of examinees performed most differently on items measuring “Recognizing and Understanding the Main Idea,” with a discrimination index of 59.
Lastly, in 2007 SAET, both high achievers and low achievers performed the best on items measuring “Recognizing and Understanding the Main Idea” (Mean=95 vs.
54 for the high and low achievers respectively). Similar to the results in 2002 SAET, low achievers had relatively high passing rates on this item, with a mean of 54. The high achievers performed the worst on items measuring “Word Inference from Context” (Mean=71.5) whiles the low achievers performed the worst on items measuring “Recognizing Implications and Making Inferences” (Mean14). Items on
“Recognizing and Making Inferences” (Mean=64) had the best discriminatory power.
Again, global reading skills best discriminated high and low achievers in the SAET.
In general, in the SAET from 2002 to 2007, the discrimination indexes of most item types each year were far above the ideal discrimination index of 30 established by Jeng et al. (1999), with the average of discrimination index around 50. To find out whether the differences on the discrimination indexes among item types measuring different reading skills each year were significant, a two-way ANOVA test was run to analyze the data. The fixed factors are the year and item types while the dependent
variable is the discrimination indexes. As shown in Table 15, the results of the ANOVA revealed that no significant effect was found either in the item types on the discrimination index alone or on the factor of year alone. However, a significant effect was found on the interaction of item types and year on the discrimination indexes (F=1.842, p<.05), which suggested that the there were significant differences among the discrimination indexes of different item types. That is to say, the discrimination indexes varied according to different types of reading skills measured.
However, not all types of reading skills were tested each year and thus it was difficult to determine which type of reading skill best discriminated high achievers from low achievers.
Table 16. The ANOVA Analysis of Discrimination Indexes in the SAET Dependent Variable: D Making Inferences” since the passing rate for the high achievers was only 45. On the other hand, the low achievers performed poorly on all items and passing rates of all item types were lower than the minimum desirable discrimination index of .30
suggested by Jeng et al (1999), which means that the low achievers performed poorly on both local and global skills in 2002 DRET. Items that the high achievers performed the best are items on “Recognizing Text Organization” (Mean=86) while the low achievers performed the worst on this type of question, with a mean of only 14. Items of this type also has the highest discrimination indexes (D=72). That is to say, global skill “Recognizing Text Organization” best discriminated the high and low achievers.
From the results of 2003 DRET, it can be seen that high achievers in general performed well on all items since the passing rates of all items were more than 50 and most of them reached above 74. The high achievers performed best on items measuring “Recognizing Cohesive Devices” (Mean=83) whereas the low achievers performed best on items measuring “Recognizing Functional Value” (Mean=31.5).
Both groups of examinees performed worst on items measuring “Recognizing Implications and Making Inferences” (Mean=57 vs. Mean=14 for high and low achievers respectively). That is to say, items of this type were the most difficult for both high and low achievers. High and low achievers performed most differently on local reading skill “Recognizing Cohesive Devices” (D=64) in 2003.
As illustrated in 2004 DRET, both high achievers performed best on items measuring “Recognizing Functional Value” (Mean=73.5 vs. Mean=25 for high and low achievers respectively). Besides, the second best performed items for both groups of learners are items on “Recognizing Cohesive Devices” (Mean =73 vs. Mean=22).
These two types of questions, one is a local skill and the other is a global skill) best discriminated the examinees, with a discriminatory power of 48.5 for each skill.
For the 2005 DRET, the high achievers obtained high passing rates on all items and they performed best on items measuring “Recognizing and Understanding the Main Idea” (Mean=84.25). The low achievers performed best on items measuring
Table 17. Passing Rates of the High & Low Achievers and the Discrimination Indexes on Different Item Types in 2002-2007 DRET
Year Item Type Ph Pl D
Mean SD Mean SD Mean SD N
2002 1. Word Inference from Context 79.0000 . 29.0000 . 50.0000 . 1
2. Recognizing Cohesive Devices 85.0000 . 29.0000 . 56.0000 . 1
3. Recognizing and Interpreting Details 72.3750 12.02304 23.8750 8.30555 48.5000 8.88015 8
4. Recognizing Functional Value 85.0000 . 20.0000 . 65.0000 . 1
5. Recognizing Text Organization 86.0000 . 14.0000 . 72.0000 . 1
6. Recognizing Presuppositions Underlying the
Text 50.0000 . 16.0000 . 34.0000 . 1
7. Recognizing Implications and Making
Inferences 45.0000 . 23.0000 . 22.0000 . 1
8. Recognizing and Understanding the Main Idea 66.0000 . 18.0000 . 48.0000 . 1
Total 71.6667 14.25115 22.6667 7.18795 49.0000 12.93942 15
2003 1. Word Inference from Context 82.0000 . 23.0000 . 59.0000 . 1
2. Recognizing Cohesive Devices 83.0000 19.0000 . 64.0000 . 1
3. Recognizing and Interpreting Details 76.3750 18.14180 23.3750 10.35012 53.0000 12.80625 8 4. Recognizing Functional Value 77.0000 26.87006 31.5000 13.43503 45.5000 13.43503 2 7. Recognizing Implications and Making
Inferences 57.0000 1.41421 14.0000 5.65685 43.0000 4.24264 2
8. Recognizing and Understanding the Main Idea 74.0000 . 22.0000 . 52.0000 . 1
Total 74.5333 16.49618 22.6667 9.58570 51.7333 11.31665 15
2004 2. Recognizing Cohesive Devices 73.0000 3.53553 22.0000 11.31371 48.5000 7.77817 2 3. Recognizing and Interpreting Details 61.1667 10.34247 20.6667 3.44480 40.5000 8.59651 6 4. Recognizing Functional Value 73.5000 20.50610 25.0000 16.97056 48.5000 3.53553 2 7. Recognizing Implications and Making
Inferences 42.0000 . 20.0000 . 22.0000 . 1
Total 63.3636 13.26855 21.6364 7.11720 41.7273 10.12019 11
2005 1. Word Inference from Context 75.0000 16.97056 30.0000 2.82843 45.0000 14.14214 2 3. Recognizing and Interpreting Details 77.2000 14.41180 19.6000 5.72713 57.6000 14.65606 5 8. Recognizing and Understanding the Main Idea 84.2500 8.50000 27.7500 4.92443 56.5000 10.50397 4
Total 79.3636 12.21698 24.4545 6.59339 54.9091 12.77853 11
2006 1. Word Inference from Context 59.0000 . 17.0000 . 42.0000 . 1
3. Recognizing and Interpreting Details 68.6667 22.86190 17.0000 4.77493 51.6667 19.99667 6
4. Recognizing Functional Value 86.0000 . 48.0000 . 38.0000 . 1
6. Recognizing Presuppositions Underlying the
Text 35.0000 . 23.0000 . 12.0000 . 1
7. Recognizing Implications and Making
Inferences 62.5000 38.89087 30.0000 8.48528 32.5000 30.40559 2
Total 65.1818 23.65510 22.7273 10.77117 42.4545 21.39796 11
2007 3. Recognizing and Interpreting Details 73.1250 17.71551 25.7500 7.45941 47.3750 16.58689 8 6. Recognizing Presuppositions Underlying the
Text 45.0000 . 17.0000 . 28.0000 . 1
7. Recognizing Implications and Making
Inferences 63.0000 12.72792 19.5000 2.12132 43.5000 14.84924 2
Total 68.7273 17.72620 23.8182 7.12486 44.9091 15.76359 11
Note. Table 14 continued.
Ph refers to passing rates generated by the high achievers.
Pl refers to passing rates generated by the low achievers.
D refers to the discrimination index (i.e., Ph-Pl).
“Word Inference from Context” (Mean=30). However, local items on “Recognizing and Interpreting Details” (D=57.6) discriminated the examinees best, followed by global items on “Recognizing and Understanding the Main Idea” (D=56.5), and global items on “Word Inference from Context” (D=45).
As revealed in 2006 DRET, the high achievers performed well on all items except items on “Recognizing Presuppositions Underlying the Text,” with an average passing rate of 35 only. The low achievers performed worst on two types of items—“Word Inference from Context” (Mean=17) and “Recognizing and Interpreting Details” (Mena=17). Both high achievers and low achievers performed the worst on global items this year. Both high achievers and low achievers performed best on global items measuring “Recognizing Functional Value,” with a mean of 86 for the high achievers and a mean of 48 for the low achievers. For the discrepancy between the two groups of examinees, items on “Recognizing and Interpreting Details” (D=51.67) had the highest discrimination indexes, followed by items on
“Word Inference from Context” (D=42), items on “Recognizing Functional Value”
(D=38), items on “Recognizing Implications and Making Inferences” (D=32.5), and items on “Recognizing Presuppositions Underlying the Text” (D=12), which had the lowest discrimination indexes.
Finally, as shown in Table 17, in the 2007 DRET, both high achievers and low achievers performed best on items measuring “Recognizing and Interpreting Details,”
with an average passing rate of 73.13 for high achievers and an average passing rate of 25.75 for low achievers. Both high achievers and low achievers performed worst on items measuring “Recognizing Presuppositions Underlying the Text” (Mean=45 vs.
Mean=17) for high achievers and low achievers respectively). Notice that the average passing rates of low achievers on all items this year were lower than the minimum
desirable discrimination indexes suggested by Jeng et al (1999).
In general, the results in Table 17 revealed similar findings to those of the SAET.
Most of the item types reached the minimum reasonable discrimination index of 30 except for items on “Recognizing Implications and Making Inferences” (D=22) in 2002 and 2004, and items on “Recognizing Presuppositions Underlying the Text”
(D=12) in 2006 and (D=28) in 2007. This suggested that most of the item types reached a satisfactory discriminatory power of at least 30 but these two item types may be somewhat ineffective in discriminating high achievers from low achievers. As a result, it was difficult to determine which type of question had the highest or lowest discriminatory power from 2002 to 2007.
Table 18. The ANOVA Analysis of Discrimination Indexes in the DRET Dependent Variable: Discrimination Indexes
Table 18 presents the results of the ANOVA analysis of item types, the year, and the interaction of item types and year on the discrimination indexes. The fixed factors are the year and the item types. The dependent variable is the discrimination indexes.
The results of the ANOVA showed that there was significant effect on the reading skills on the discrimination index (F=2.534, p<.05), which indicated that the
discrimination indexes varied according to the types of reading skills measured.
However, no significant effect can be found on the factor of year alone (F=2.534, p<.05) or on the interaction of both factors (F=1.087, p>.05). That is to say, there were no statistical differences among the discrimination indexes of different types of reading skill each year. In sum, although a ranking difference among the discrimination indexes on various item types existed, it was hard to determine which types of items had the most and least discriminatory power in both SAET and DRET.
Summary
In this chapter, both qualitative and quantitative analyses of the test items were presented. The qualitative analyses showed what reading skills were measured in the SAET and DRET whereas the quantitative analyses illustrated the statistical results of the passing rates (i.e., all examinees, high achievers, and low achievers) and the discrimination index on each item.
A preliminary analysis of the 2002 SAET reading comprehension test items was done to help the three raters familiarize themselves with the coding procedure and to see what skills were measured on the test. Results are reported in frequencies, descriptive statistics, as well as qualitative descriptions.
The findings of the formal analysis revealed that in the SAET and DRET from 2002 to 2007, eight types of reading skills were identified (1) Word Inference from Context, (2) Recognizing Cohesive Devices, (3) Recognizing and Interpreting Details, (4) Recognizing Functional Value, (5) Recognizing Text Organization (only in DRET), (6) Recognizing Presuppositions Underlying the Text, (7) Recognizing Implications and Making Inferences, and (8) Recognizing and Understanding the Main Idea. Throughout the six years, items on “Recognizing and Interpreting Details”
were the most tested item type in both tests (61.30% on the SAET 55.40% on the
DRET). On both tests, items on “Recognizing Test Organization” were the least tested (0% on the SAET and 1.4% on the DRET). When the results of both tests are compared, the findings revealed that different types of skills tested in the SAET and DRET throughout the six years. Different reading skills appeared in both exams in different years.
Regarding the reading level which best discriminated the high achievers and low achievers, in 2002 to 2007 SAET, 5 years out of 6 were global level reading skills; for example, “Recognizing Presuppositions Underlying the Text” in 2003 (D=69), “Word Inference from Context” in 2004 (D=64) and 2005 (D=58), “Recognizing and Understanding the Main Idea” in 2006 (D=59) and 2007 (D=64). Only the reading skill that best discriminated the high achievers and low achievers in 2002 SAET was a local skill “Recognizing Cohesive Devices” (D=66). However, in the DRET, most of the skills that best discriminated the high achievers and low achievers are local skills;
for example, “Recognizing Cohesive Devices” in 2003 (D=64), “Recognizing and Interpreting Details” in 2005, 2006, and 2007. In 2002 DRET, the skill that best discriminated the high achievers and low achievers was a global skill “Recognizing Text Organization” (D=72). In 2004 DRET, one local skill and one global skill both best discriminated the high achievers and low achievers, i.e., “Recognizing Cohesive Devices” (D=48.5) and “Recognizing Functional Value” (D=48.5).
In general, the overall performances of all examinees differed on various types of reading comprehension questions in the SAET and DRET. In the SAET administered from 2002 to 2007, no general pattern of the discrimination indexes of the average passing rates on each item type was found. The pattern varies each year. However, it was worth noting that items on “Recognizing Implications and Making Inferences”
and items on “Recognizing and Understanding the Main Idea” had the lowest
discriminatory power in the SAET for two years. As to the DRET, items on
discriminatory power in the SAET for two years. As to the DRET, items on