In this study, reading comprehension items of the BCT from 2011 to 2013 and those of the CAP from 2014 to 2016 are collected and analyzed, using the revised Bloom’s taxonomy. In this section, data analysis, coding procedure, coding examples, coding results, and interrater reliability are discussed in detail.
Data Source
The reading passages included for analysis in this study were be from those of the BCT from 2011 to 2013 and those of the CAP from 2014 to 2016. In BCT, a total of 10 passages with either 19 or 21 test items on the tests from 2011 to 2013 were collected. In CAP, a total of 9 passages, 21 test items on the test in 2014, 8 passages, 22 test items on the test in 2015, 7 passages, 19 test items on the test in 2016 were collected (See Table 7). Generally speaking, in the BCT, each test contains 10 reading passages, and each passage also included 2 to 4 test items while in the CAP, each test contained 7 to 9 reading passages, and each passage contained 2 to 4 test items. All reading test items were analyzed according to the cognitive levels in the revised Bloom’s taxonomy. Examinees’ passing rates were collected from the Research Center for Psychological and Educational Testing (RCPET).
Table 7. Reading Passages and Comprehension Test Items Analyzed
Exam Year Number of
Reading Passages
Number of Test Items
BCT 2011 10 19
2012 10 19
2013 10 21
CAP 2014 9 21
2015 8 22
2016 7 19
29
Data Analysis and Coding Procedure
The purposes of the analysis are two-folded: First, it aims to categorize all reading comprehension test items based on the revised Bloom’s taxonomy and calculate test questions that match Bloom’s six levels in Bloom’s revised taxonomy.
Second, it shows the difficulty levels of different items in English tests in the BCT and the CAP.
There are two phases of the data analysis. One is the coding process, and the other is the analysis of the difficulty levels and passing rates. First, in the coding process, two experienced junior high school English teachers and the researcher were the raters. The revised Bloom’s taxonomy, the sample coding sheet (Appendix 1), and the reading comprehension passages from 2011 to 2016 were given to raters for coding. A meeting was held for raters to see if they all agreed on the revised Bloom’s classification of test items. After the coding was done, interrater reliability was checked by comparing the coding of these raters. If their coding results weren’t consistently categorized, the coders would read the principles of classification again and had a discussion until they reached a consensus on this issue. Afterward, the coded data would be compared with test items’ difficulty level data from the RCPET, and items’ passing rates would be provided to indicate test takers’ performance.
Coding Examples
Coding examples were taken from Lan (2009)’s and Li (2009)’s studies3. 1. Example of “Remember” (From Lan, Q42 of 2004 SAET)
3 Examples of Evaluate and Create were provided from Li (2009)’s study: A content analysis of the ability of high-order-thinking in social studies workbooks in primary school: taking the cognitive process dimension in a revision of bloom's taxonomy of educational objectives as analysis framework.
A sense of humor is just one of the many things shared by Alfred and Anthony Melillo, 64-year-old twin brothers from East Haven who made history in February 2002. On Christmas Eve, 1992, Anthony had a heart transplant from a 21-year-old donor. Two days before Valentine’s Day in 2002, Alfred received a 19-year-old heart, marking the first time on record that twin adults each received heart transplants.
“I’m 15 minutes older than him, but now I’m younger because of my heart and I’m not going to respect him,” Alfred said with a grin, pointing to his brother while talking to a roomful of reporters, who laughed frequently at their jokes.
While the twins knew that genetics might have played a role in their condition, they recognized that their eating habits might have also contributed to their heart problems.
“We’d put half a pound of butter on a steak. I overdid it on all the food that tasted good,
30
What did Alfred and Anthony have in common?
(A) Lifespan.
(B) Career goals.
＊(C) A sense of humor.
(D) Love for bicycling.
Explanation: In the first line, it is said that“A sense of humor is just one of the many things shared by Alfred and Anthony. Because a sense of humor is stated directly, without any changes or word transformation, it belongs to Remember.
2. Example of “Understand” (From Lan, Q47 of 2004 SAET)
What does “the world” mean in “show the world to the world?
(A) The global village.
(B) The leading museums.
(C) The ancient civilization.
(D)The international public.
Explanation: In this paragraph, we know that the British Museum was under the pressure to send its sculptures to countries of their origins. Yet, it asserted that putting these sculptures in the British Museum was the best choice so that the public could see these old cultures and civilizations. Therefore, it said that it showed the world, which meant the ancient civilization, to the world.
3. Example of “Apply’’ (From Lan, Q41 of 2002 SAET)
How long had Tim and Dom been at sea when their boat was hit by a fishing The British Museum has not signed the declaration, but says it fully supports it. Over the recent years, it has faced growing pressure to hand back the Elgin Marbles, sculptures taken from the Parthenon in Athens, Greece, in the 19th century. But the British Museum has said that the Museum is the best possible place for them. “They must remain here if the museum is to continue to achieve its aim, which is to show the world to the world,” said the director of the museum.
They set out from Japan on May 17, 2001. They had rowed nearly 5,500 miles when their boat was hit by a fishing ship on September 17, 2001.
＊
a 19-year-old heart, marking the first time on record that twin adults each received heart transplants.
31
Explanation: From the statement, it says that they started their journey on May 17 and finished it on September 17. If students apply the written information and do the calculation, they’ll know that Tim and Dom had been at sea for four months.
4.Example of “Analyze” (From Lan, Q50 of 2006 DRET)
This passage was most likely written by someone who _____________.
(A) liked hunting wild animals (B) made laws against the gray wolf
(C) advocated the protection of the gray wolf
Native Americans could not understand the white man’s war on the wolf. The Lakota, Blackfeet, and Shoshone, among other tribes, considered the wolf their spiritual brother. They respected the animals’ endurance and hunting ability, and warriors prayed to hunt like them. They draped themselves in wolf skins and paws, hoping they could acquire the wolf’s hunting skills of stealth, courage, and stamina.
Plains Indians wore wolf-skin disguises on raiding parties. Elite Comanche warriors were called wolves.
The white settlers’ war on the wolf raged on. Western ranchers continued to claim that thousands of cattle were killed every year by wolves. In 1884, Montana created its first wolf bounty—one dollar for every dead wolf, which increased to eight dollars in 1893. Over a period of thirty-five years, more than eighty thousand wolf carcasses were submitted for bounty payments in Montana. Moreover, the government even provided free poison. Finally, in 1914, ranchers persuaded the United States Congress to provide funds to exterminate wolves on public lands.
The last wolves in the American West died hard. No place was safe, not even the nation’s first national park, Yellowstone. The park was created in 1872, and from its very beginning, poisoned carcasses were set out to kill wolves. Nearly 140 wolves were killed by park rangers in Yellowstone from 1914 to 1926. In October 1926, two wolves cubs were trapped near a bison carcass. They were the last animals killed in the park’s wolf control programs.
Ranchers had won the war against the wolf. Only in the northern woods of Wisconsin, Minnesota, and Michigan could the howl of native gray wolves be heard.
The vast lands of American West fell silent. The country had lost its greatest predator.
＊
＊
32
(D) appreciated the gray wolf’s hunting skills
Explanation: Items of Analyze refer to the analysis of author’s viewpoints or values. For instance, readers must read beyond the lines to get the author’s hidden meaning to know his or her true viewpoints and stands. Such viewpoints or stands are abstact and implicit. From the above underlined sentences, we know the wolves in America are going to become extinct and they can only be seen in few places. Mother Nature’s silence seems to mourn for the losing of the wolves. Hence, human beings must stand up to prevent the wolves from extinction, which is the hidden meaning that the author is trying to convey. The example question here requires readers to realize his or her viewpoint and value in the passage, so it belongs to the Analyze category.
5.Example of “Evaluate” (From Li, 2009)
Explanation: From the statement, we know that students are required to evaluate the pros and cons of the set-up of landfills. This is an example of high cognitive thinking process. Students need to think and judge the advantages and disadvantages of building up landfills, and even come up with some possible and doable suggestions.
6.Example of “Create” (From Li, 2009)
Explanation: This is an example of Create. Students need to choose a perspective and come up with some current environmental problems of the Earth, like plastic bags, air pollution, global warming, acid rain, the decreasing biodiversity, and so on. They may
If you were against the set-up of landfills, under what circumstances would you change your opinions, or not?
A Medical Record of the Earth
From the perspective of the ______, I’ve diagnosed the Earth’s environmental problems.
(1) Possible reasons: ________________________________________________
(2) Potential impacts: _______________________________________________
(3) Solutions to the problems:_________________________________________
33
brainstorm and create their own answers based on the issue.
Coding Results
In this section, the coding results and interrater reliability are presented and discussed. The researcher and two other raters coded 121 English reading
comprehension test items, 59 on the BCT and 62 on the CAP, by using the revised Bloom’s taxonomy as the coding rubric.
Discussion of Interrater Reliability
After the first coding results, the interrater reliability was shown in Table 8.
Table 8. Interrater Reliability (Total Number: 121)
Coder 1 vs. Coder 2 Coder 1 vs. Coder 3 Coder 2 vs. Coder 3
Table 8 showed that Coder 1 and Coder 2 had the highest interrater reliability (98%). They only had two items in disagreement. Coder 1 and Coder 3 had the lowest interrater reliability (84 %). They had 19 items in disagreement. The interrater
reliability between Coder 2 and Coder 3 was 86%. They had 17 items in disagreement.
After the first coding meeting, 85% of the items were coded consistently by the three raters. When raters disagreed with one another on certain items, the majority would decide the results. However, another expert was invited to code the inconsistent ones.
The three raters then had the second coding meeting to classify some item definition before finalizing the results and coding rubrics (See Appendixes 2 & 3).
Summary
In this chapter, the method adopted in this study is reported. The participants, data source, data analysis, coding procedures, coding examples, and interrater reliability are shown. The study results will be introduced in the next chapter.
34