This chapter presents the results of the research in three main sections. The first section presents the coding results of the English reading comprehension test items on BCTs and CAPs from 2011 to 2016. The second section discusses the analysis of cognitive levels appeared in BCT and CAP. The third section illustrates the results of examinees’ performances on different question types. The summary of the chapter is presented in the last part.
Analysis of Cognitive Types in BCT and CAP The overall coding results are presented in Table 9.
Table 9. Cognitive Type in BCT and CAP
Type BCT CAP
Count % Ranking Count % Ranking
Remember 5 8.5 % 3 1 1.6% 4
Understand 44 74.6 % 1 44 71% 1
Apply 3 5% 4 10 16.1% 2
Analyze 7 11.9% 2 7 11.3% 3
Evaluate 0 0 0 0 0 0
Create 0 0 0 0 0 0
Total 59 100% 62 100%
According to Table 9, Understand was found to be the most frequently appeared items of the BCT and the CAP from 2011 to 2016 (74.6% in BCT and 71%
in CAP). However, rankings of other cognitive levels in BCT and CAP were slightly different. In BCT, the order from the highest frequency to the lowest was Understand, Analyze, Remember, and Apply while in CAP the order was Understand, Apply, Analyze, and Remember. As for Evaluate and Create, they didn’t appear on either of
35
the exams.
From Table 9, it is obvious that four types of cognitive levels surfaced from 2011 to 2013 BCT and from 2014 to 2016 CAP. They are Remember, Understand, Apply and Analyze. In the following section, details from BCT will be discussed first and then the data from CAP will follow.
Cognitive Types Measured in BCT from 2011 to 2013
Cognitive types measured in BCT from 2011 to 2013 is shown in Table 10.
Table 10. Cognitive Types Measured in BCT from 2011 to 2013
Type Remember Understand Apply Analyze Evaluate Create Total
2011 3 (16%) 12 (63%) 1 (5%) 3 (16%) 0 0 19
2012 2 (10.5%) 15 (79%) 0 2 (10.5%) 0 0 19
2013 0 17 (80%) 2 (10%) 2 (10%) 0 0 21
From Table 10, we can see in 2011 BCT, Remember accounted for 16%, Understand accounted for 63%, Apply accounted for 5%, and Analyze accounted for 16%. In 2012 BCT, Remember accounted for 10.5%, Understand accounted for 79%, and Analyze accounted for 10.5%. In 2013 BCT, Understand accounted for 80%, Apply accounted for 10%, and Analyze accounted for 10%.
As for the item changes during the three years, items of Understand had accounted for the most and continued to grow, from 63%, 79% to 80%. Moreover, items of Apply increased from 5% to 10%. The tendency of the increasing importance of Understand and Apply became clear. As for items of Analyze, they started from 16%, decreased to 10.5%, and dropped to 10% in 2013. Similarly, items of Remember dropped from 16% in 2011 to 10.5% in 2012 and to 0% in 2013. This indicated the decreasing importance of these two categories in BCT.
Besides, items of Remember and Apply didn’t appear every year. The former
36
disappeared in 2013 while the latter disappeared in 2012. This showed they were easily replaced in the two years. As for higher cognitive levels such as Apply and Analyze, when items of these two categories were combined, they accounted for 21 % in 2011 and 20% in 2013, which indicated their steady frequency and importance.
Cognitive Types Measured in CAP from 2014 to 2016
Cognitive types measured in CAP from 2014 to 2016 are shown in Table 11.
Table 11. Cognitive Types Measured in CAP from 2014 to 2016
Type Remember Understand Apply Analyze Evaluate Create Total
2014 0 17 (80%) 2 (10%) 2 (10%) 0 0 21
2015 0 14 (64%) 4 (18%) 4 (18%) 0 0 22
2016 1 (5%) 13 (69%) 4 (21%) 1 (5%) 0 0 19
From Table 11, it is obvious that Remember, Understand, Apply, and Analyze also appeared in CAP from 2014 to 2016. Overall, items of Understand were the most common (80%, 64%, 69%) during the past three years. Items of Apply had been on the rise (from 10%, 18%, to 21%) while those of Analyze had increased (10% to 18%) until 2015 and then dropped to 5% in 2016. Besides, items of Remember increased from 0 % to 5% in 2016. Hence, the tendency of the increasing importance of Apply and the continual dominance of Understand were shown.
Another point was that higher cognitive levels such as Apply and Analyze appeared in CAP more often than those in BCT. We know that higher cognitive levels, Apply and Analyze, when combined, accounted for 20%, 36 %, and 26% in three-year CAP, higher than those in BCT (21%, 10.5%, 20%). This fact indicated that CAP tended to include test items of higher cognitive levels.
In 2016, four cognitive levels appeared together for the first time in CAP.
This might indicate that there existed a balance between items of high and low
37
cognitive levels. They also implied that four cognitive levels were important, only with slightly different proportions.
Similarities and Differences between the Cognitive Types Measured in the BCT and CAP
Similarities. According to Table 10 and Table 11, both exams shared some similar points as follows. First, items of Understand accounted for the most, and items of Evaluate and Create didn’t appear. From 2011 BCT to 2016 CAP, items of
Understand were the majority, accounting for 63%, 79%, 80%, 80%, 64% and 69%.
As for Evaluate and Create, they weren’t easily measured in multiple choices, so they didn’t appear in either exams.
Second, items of Apply increased, which showed its increasing importance in BCT and CAP. In BCT, items of Apply, though disappeared in 2012 BCT, accounted for 5% in 2011 BCT, 10% in both 2013 BCT and 2014 CAP, 18% in 2015 CAP, and 21% in 2016 CAP. Overall, they fluctuated from 0 to 21% in the six-year time frame.
In CAP, items of Apply increased at a faster speed. Their rising importance, especially in CAP, matched CAP’s goal of relating to students’ living experiences, and activating their learning.4
Differences. As for differences between BCT and CAP, they are listed as follows. First, CAP included more items of Apply and Analyze, which indicated that CAP contained more items of higher cognitive levels than BCT did. For instance, when Apply and Analyze were combined, they accounted for 20%, 26%, and 36%
during the three-year time frame in CAP. However, in BCT, they together accounted for 10.5%, 20%, and 21%, which were lower than CAP. This fact showed that CAP tended to include higher cognitive level test items.
4 These were goals of CAP, cited from CAP’s official website (http://cap.ntnu.edu.tw/background.html).
38
Second, items of Apply were increasing faster in CAP than in BCT, which indicated that Apply needed our attention and would be discussed later. In BCT, items of Apply accounted from 0, 5% to 10%, but in CAP, they accounted from 10%, 18%
to 21%. A faster growing trend in CAP was revealed, which pointed out CAP
emphasized students’ abilities of applying learned information to different situations.
Third, items of Remember and Analyze would be limited in certain
proportions. For instance, in 2011 BCT, Remember and Analyze each accounted for 16%, in 2012 BCT, 10.5% each, and in 2013, they accounted for 0 and 10%
respectively. In 2014 CAP and 2015 CAP, Remember didn’t appear and Analyze accounted for 10% and 18%. In 2016 CAP, both accounted for 5%. Overall, items of Remember and Analyze were under 20%, and the former sometimes didn’t appear.
This pointed out that Remember, a low cognitive level, played a basic role in both tests. Items for Analyze, a high cognitive level, tended to be more challenging and often fluctuated between 5% and 18%, and reached its lowest point of 5% in 2016 CAP.
Examinees’ Performance on Each Cognitive Category
As indicated before, four cognitive types were measured on BCT and CAP.
Test takers’ performances were shown through their passing rates. Since data of 2011 BCT wasn’t available5, only data from 2012 BCT to 2016 CAP provided by RCPET were adopted in the following analysis.
According to RCPET, the difficulty levels in 2012 and 2013 BCT were
categorized into four based on the passing rate of each item: extremely easy (a passing rate of 82% and above), easy ( a passing rate between 82% and 62%), intermediate (a passing rate between 62% and 38%), and extremely difficult (a passing rate between
5 In 2011, Taipei, New Taipei, and Keelung City had their own local BCT. The national 2011 BCT didn’t precisely indicate all students’ performances in Taiwan and therefore the overall passing rate didn’t represent the national trend.
39
38% and 13%). In 2014 and 2015 CAP, with the first three ranges of percentage, the fourth one was further divided into two: difficult (a passing rate between 38% and 20%), and extremely difficult (a passing rate of 20% and below). In 2016 CAP, their difficulty level ranges were different. They were extremely easy (a passing rate of 69% and above), easy (a passing rate between 69% and 48%), intermediate (a passing rate between 48% and 38%), difficult (a passing rate between 38% and 18%), and extremely difficult (a passing rate of 18% and below).
From the difficulty levels, we may know how examinees performed on these test items. In the following analyses, test items’ difficulty levels will be matched with the Bloom’s cognitive categories to show their relationship.
The analysis of 2012 BCT was shown in Table 12.
Table 12. Analysis of Cognitive Types and Items’ Difficulty Levels in 2012 BCT Remember Understand Apply Analyze Evaluate Create Total %
6EE 1 1 5.2%
E 1 7 1 9 47.4%
I 8 1 9 47.4%
ED 0 0
Total 2 15 0 2 0 0 19 100%
% 10.5% 79% 0 10.5% 0 0 100%
From Table 12, we know 79% of test items belonged to Understand. Among them, seven out of fifteen were easy, and eight out of fifteen were intermediate. This meant that most of the Understand items were easy for most test takers. As for Analyze, 10.5% of test items belonged to it. Among the two items, one was easy and the other was intermediate. In terms of Remember, it accounted for 10.5% of the test
6 Extremely Easy was presented as EE, easy as E, intermediate as I, extremely difficult as ED.
40
items, and most examinees did well on it.
The analysis of cognitive types and items’ difficulty levels in 2013 BCT is shown in Table 13.
Table 13. Analysis of Cognitive Types and Items’ Difficulty Levels in 2013 BCT Remember Understand Apply Analyze Evaluate Create Total %
EE 0 0
E 9 1 10 48%
I 8 1 2 11 52%
ED 0 0
Total 0 17 2 2 0 0 21 100%
% 0 81% 9.5% 9.5% 0 0 100%
From 2013 BCT, we can see 81% of test items belonged to Understand. Nine out of seventeen items were easy, and the remaining eight were intermediate, which showed that examinees performed well on more than half of the items of Understand.
Then, 9.5% of items belonged to Apply. One was easy, and the other was intermediate.
As for Analyze, 9.5% of items also belonged to this category, and both of the two items were intermediate. This indicated that items of Analyze belonged to higher cognitive level and were more challenging for most examinees in 2013 BCT. Overall, all intermediate items were of Understand, Apply, and Analyze. They accounted for more than half of the test items in 2013.
41
The analysis of the cognitive types and items’ difficulty levels in 2014 CAP is shown in Table 14.
Table 14. Analysis of Cognitive Types and Items’ Difficulty Levels in 2014 CAP Remember Understand Apply Analyze Evaluate Create Total %
EE 0 0
E 4 1 5 24%
I 11 1 1 13 62%
D7 2 1 3 14%
ED 0 0
Total 0 17 2 2 21 100%
% 0 81% 9.5% 9.5% 0 0 100%
Table 14 shows that 81% of the items were of Understand. Among them, four out of seventeen were easy, eleven were intermediate, and two were difficult. As for items of Apply, they accounted for 9.5% of items, which were either easy or
intermediate. This indicated that some items of Understand could be difficult for some test takers, and that items of Analyze, which belonged to higher cognitive level, tended to be hard for test takers.
7 Difficult was presented as D.
42
The analysis of cognitive types and items’ difficulty levels in 2015 CAP is shown in Table 15.
Table 15. Analysis of Cognitive Types and Items’ Difficulty Levels in 2015 CAP Remember Understand Apply Analyze Evaluate Create Total %
EE 0 0
E 8 4 1 13 59%
I 5 2 7 32%
D 1 1 2 9%
ED 0 0
Total 0 14 4 4 0 0 22 100%
% 0 63.6% 18.2% 18.2% 0 0 100%
From Table 15, we can see that 63.6% of the items were of Understand. This indicated that most of the items of Understand tended to be easy, but some could also be difficult. 18.2% of items were of Apply, and all of them were easy for most
examinees. As for items of Analyze, one of them was easy, two of them were
intermediate, and one was difficult. This showed that items of Analyze tended to be of moderate difficulty level. Overall, most examinees could answer items of Understand, Apply, and Analyze, but sometimes, items of Understand and Analyze could be challenging for some test takers.
43
The analysis of cognitive types and items’ difficulty levels in 2016 CAP is shown in Table 16.
Table 16. Analysis of Cognitive Types and Items’ Difficulty Levels in 2016 CAP Remember Understand Apply Analyze Evaluate Create Total %
EE 0 0
E 1 12 3 1 17 89%
I 0 0
D 1 1 2 11%
ED 0 0
Total 1 13 4 1 0 0 19 100%
% 5.3% 68.4% 21% 5.3% 0 0 100%
From Table 16, we can see that 68.4% of test items belonged to Understand.
Twelve of the thirteen items were easy. But one item of them was hard. 21% of the items belonged to Apply. Among them, three out of four were easy and one of them was difficult. For the rest, they were items of Remember and Analyze, which were easy for most test takers. This showed that Remember tended to be easy in most cases.
Another point was that Analyze, though a higher cognitive level, could be easy items for most examinees.
From the above analyses, we can see that most items were Understand and they tended to be easy. However, there were exceptions. In some cases, there were difficult items of Understand, easy items of Apply and Analyze. Since CAP is the current senior high school entrance exam, further analysis of CAP will be presented in the next section to explore what makes some lower cognitive level items different and higher cognitive level items easy.
44
Difficult Items of Understand 1. Question 52 in 2014 CAP.
What can we learn about the cities in the report?
(A) One city in this year’s top five is in the east.
(B) Few people come to Oxtown to do business.
(C) No city in the north entered this year’s top ten.
(D) Goosetown is Animal Island’s second biggest city.
Table 17. Item Analysis of Question 52 in 2014 CAP
Choice A B C D
% 28.12% 34.08% 15.52% 21.85%
Passing rate: 0.28 Item Discrimination: 0.23
Explanation: This question tested examinees’ inferring ability, and the correct answer was (A). From Sharkville’s description, we knew Sharville had the good winter sports like Oxtown, and we could infer that Sharville and Oxtown were both in the east.
Therefore, we inferred that Oxtown, which ranked the fourth, was in the east, and we got (A) as the right answer. However, from Table 17, about 34.08% of examinees chose (B). From Oxtown’s description, it said it was not only a business city, which meant that many people went there to do business. This contradicted with (B)’s description of few people going there for business purposes. From students’ incorrect choice, we knew that some tended to neglect the key word ‘‘few’’or the statement that
Here is this year’s report on the Top Ten Cities of Animal Island by Best Living.Com.
(1) Goosetown: Climbing up from last year’s second place, Goosetown comes in first for its lovely parks, cultural centers, and comfortable living space.
(2) Tigerville: Losing its top place to Goosetown, Tigerville is still a beautiful city, and as green as ever.
(3) Oxtown: Not just a famous business city, Oxtown has turned itself into a garden city.
(6) Sharkville: With winter sports as good as Oxtown’s, this exciting city is our second best pick in the east.
＊
45
‘‘not just a famous business city.’’ As a result, examinees should pay more attention to details of text description. Overall, its passing rate was 0.28, which meant that only about 28% test takers answered it correctly. If examinees could attend to more details, it might have higher passing rate. From the above information, it was shown that items of Understand like this one could be difficult. Test takers must read carefully, find out the key words, and infer from the text to get the right answer.
2. Question 54 in 2014 CAP.
What do we know about the six people in the interview?
(A) They all said good things about the roof garden.
(B) Some of them were paid to help build the roof garden.
(C) They all talked about the roof garden before it was built.
＊(D) Some of them worked on the plan to build the roof garden.
Table 18. Item Analysis of Question 54 in 2014 CAP
Choice A B C D
% 29.68% 27.33% 17.69% 24.57%
Passing rate: 0.24 Item Discrimination: 0.26
Explanation: This question tested examinees’ inferring ability, and the correct answer was (D). From Wilber’s statement, we knew Jasmine was the one who helped collect
Six people are being interviewed about the garden on the roof for their building.
Jasmines: I loved the idea when Wilber first told me about it. We had lots of
meeting with our neighbors, trying to make them understand why it’s good to build a garden on the roof.
Wilber: The whole thing wasn’t easy at first. But Jasmine helped a lot. And she was really good at making people happy to give money for the roof garden.
David: My kids love going up there. They sit there watching butterflies and birds.
Samuel: You want something green? Visit the park! It’s only one block away!
After the roof garden was built, bugs started flying into my apartment! And the kids leave mud on the stairs when they come down from the roof.
Rosie: Our building is now cooler in the summer. My baby sleeps well even on hot summer days!
Flora: Guess where these tomatoes are from! They’re from our roof!
46
funding for roof repairing, which was done before the repairing project. Hence, we knew (D) was the right answer. From Table 18, we knew nearly 30% of examinees chose (A). Their reason might be that most of the residents gave good responses, but they neglected Samuel’s assertion, which pointed out that bugs began to annoy him and kids made dirty stairs everywhere. From Samuel negative feedback, we knew not all of the residents had positive feedbacks, so (A) was wrong. Overall, its passing rate was 0.24. From this difficult item of Understand, we know if examinees should be taught to mark each opinion positive or negative, pay attention to details, and find the relevant and correct clues.
Easy Items of Apply 1. Question 34 in 2016
If store’s sales keep slumping, what would the sales chart look like?
Table 19. Item Analysis of Question 34 in 2016 CAP
Choice A B C D
% 51% 30% 11% 8%
Passing rate: 0.51 Item Discrimination: 0.61
From the statement that now it is less than 500,000, test takers should point out By July, almost every part of the country had been attacked by Cow Cold. In only two months, the number of dead cows has risen to 5,000.
The sale of milk in Osteen State has slumped because of Cow Cold. Before Cow Cold, the sale of milk in Osteen State was 2.5 million each week; now it is less than 500,000.
＊
47
the last figure of less than 500,000. They found the correct answer of (A). Though it was transformed from written text to statistical figures, examinees could still find the key words and relate them to the correct statistical information. Hence, we know that test item of Apply like this can be designed as an easy item. If teachers can let
students start from easy items of Apply, students’ confidence and interest can be built, which may help them try harder items of Apply next time.
2. Question 26 in 2015
Explanation: The item was designed in the form of a game. If test takers could distinguish the steps of each card and apply the game rules to each participant, they
＊
48
could tell who the final winner was. Hence, the key of answering the item was to know and follow its rules. Since most examinees had similar experiences of playing games before, most of them could answer the item correctly.
Table 20. Item Analysis of Question 26 in 2015 CAP
Choice A B C D
% 16% 14% 64% 6%
Passing rate: 0.64 Item Discrimination: 0.48 Easy Items of Analyze
1. Question 35 in 2015
＊
What lesson can we learn from Froggie’s story?
(A) Life is short; one should live it to the fullest.
What do we know about Froggie?
(A) Life is short; one should live it to the fullest.
(A) Life is short; one should live it to the fullest.