This chapter included the conclusions of the present study, discussions of the results, and suggestions for future studies.
Conclusions
The purpose of the present study is to validate an English reading comprehension test with ordered multiple-choice items for EFL senior high school students in Taiwan. The test was modified from the sample items of the North Carolina Testing Program, based on the reading framework for 2013 NAEP and the rationale of ordered multiple-choice items. The coefficient alpha was used to measure the reliability of the English reading comprehension test, and according to acceptable values for describing the internal consistency of Cronbach α, .844 (original scoring) and .814 (ordered multiple-choice scoring) represent good
reliability of the modified English reading comprehension test.
As for the validity, the items of the English reading comprehension test contained three cognitive targets listed in the reading framework for 2013 NAEP: locate and recall, integrate and interpret, and critique and evaluate. The distractors of the items were further designed into ordered choices by the researcher. According to the results of item analysis, the distractor levels and the frequency distribution of different groups of test performance showed a
hierarchical pattern that helps the researcher identify students’ level of understanding in
reading comprehension. In addition, five items showed low discriminating functions because of ambiguous distractors and unclear stem statement. The other 34 items showed good discriminating functions and could be representative samples of the English reading comprehension test.
As for the difficulty, the results showed that literary texts (D̅=.74) were easier than informational texts (D̅=.65). Item difficulty of both text types shared the same pattern that the third cognitive target, critique and evaluate, is the much difficult one of the three cognitive targets, followed by integrate and interpret (the second target), and locate and recall (the first target).
On the other hand, the researcher investigated gender differences on the English reading comprehension performance on various cognitive targets, and the results showed that there were no significant differences between males and females. That is, the performances on various cognitive targets of boys and girls were not different in the present study.
Discussions Item Difficulty
According to the analysis of the results, it is found that items with higher cognitive targets are much difficult for students. The English reading comprehension test was conducted based on the reading framework for 2013 NAEP, and contained three cognitive targets: locate and recall, integrate and interpret, critique and evaluate. These cognitive targets were related to the mental processes that underlie reading comprehension. Items assessing the first category of cognitive targets usually focus on essential elements of text, such as main ideas, supporting ideas, characters or setting. Students only needed to match the response with what was given in the text. Therefore, the items assessing the first cognitive target was easier for students; Items assessing the second category of cognitive targets usually ask students to form generalizations about the information given from the text or ask for interpretations of an implied messages from the text. In this case, items assessing the second cognitive target was much difficult than the first ones; Items assessing the third category of cognitive targets requires students to evaluate the overall quality of a text or to determine the most significant information of a text. Students needed to assess the text from different perspectives and synthesize the information with other experiences. Therefore, items assessing the third cognitive target was the most difficult for students.
Since the English reading comprehension test was designed for EFL students, English proficiency was a part of factors that affected item difficulty. Several empirical studies
suggested that second language proficiency was a better predictor of second language reading comprehension (Alderson, et al., 1977; Perkins, et al., 1991). Therefore, the researcher
compared the reading texts with English vocabulary for senior high school students,
published by College Entrance Examination Center, and added Chinese translation next to the words that were not in the lists to make sure the readability of the texts. However, several items contained ambiguous information. It is suggested that during the process of validation, expert review, which the present study was missing, played an important role in revising items since experts of the field could help point out the blind spots of the revision. Also, English teachers could be invited for items or texts revising, since they understand what proficiency level their students are and what language is accessible to their students.
Constructed-response item
In the reading framework for 2013 NAEP, besides multiple-choice items, the assessment using constructed-response items to measure the extent of students’ understanding of texts.
Since multiple-choice items in NAEP assessment were scored dichotomously, ordered multiple-choice items could enhance the function of diagnosing students’ extent of
understanding. In Taiwan’s educational system, multiple-choice items were commonly used for assessing students’ performances, because of the convenience of scoring. As students are
answering multiple-choice items, they may guess the answer since the possible answers are listed in front of them, and therefore, it will be meaningless for administrating the
assessment. To avoid this kind of problem, the researchers could use different kinds of item types in the assessment. In this case, the assessment will provide stronger validity and reliability evidence for analyzing the results.
However, under EFL situation, answering constructed-response items requires students’
achieving a certain level of proficiency of English language. That was in accordance with Cummins’ (1979) theory that reaching “a certain minimum or threshold level of competence in a second language” (p.229) might allow students to engage the higher mental operations involved in comprehension. Therefore, the researchers need to make sure that target participants are capable of answering this kind of items. Otherwise, the difficulty of the assessment will be higher, and causing the results that were unexpected.
Gender differences
The results of the present study showed that there were no significant differences
between boys and girls on reading comprehension performance. As mentioned earlier, female students had better reading comprehension performances than male students (OECD, 2013;
Ko, et al., 2013; NAGB, 2014). However, in L2 context, the results showed an inconsistent pattern when gender differences were considered in L2 reading comprehension performances.
A possilble reason for the results was that participants of the present study were seniors in high school. Since they were about to take General Scholastic Ability Test for entering a college, they were extensively prepared for this important test. Therefore, the differences between these students were smaller because they were equiped with certain level of language proficiency. In this case, the variety of the group was smaller and therefore the results produced no significant gender differences on English reading comprehension.
The results of the present study was similar to Brantmeier’s (2003) study. The difference was that gender-oriented content was considered in her study. She used two passages (one was male-oriented topic while the other was female-oriented) to explore gender differences on L2 reading comprehension, and the her analysis showed that the topic familiarity balanced out differences. However, in Bügel & Buunk’s (1996) study, they also included gender-based text for students’ reading comprehension, and their results suggested that male students generally had better reading comprehension performances than female students on the gender-neutral text. This finding contrasted the results in L1 context that females had better reading comprehension performances than males (OECD, 2013; Ko, et al., 2013; NAGB, 2014). Studies for EFL reading comprehension in the future might consider including topic familiarity as a variable for exploring gender differences on reading performances.
The Application of the Present Research
The purpose of the present study was to validate an English reading comprehension test for EFL senior high school students. Teachers could follow the rationale presented from the present study to construct their own English reading comprehension tests, and use them to diagnose the problems that students have trouble comprehending, such as which cognitive targets that students have not reached, or which text type/item type students are not familiar with and need more practices. Besides, the results of the present study could be further investigated on related reading comprehension issues, such as exploring different possible factors that influence students reading comprehension performance, or as a tool for checking teachers’ reading instruction strategies, etc.
Suggestions for Future Studies Participants
The participants of the present research were EFL senior students in high school from Kaohsiung City. To make the samples much more representative for the design, it is suggested that the participants could be acquired from different regions or learning stages.
Therefore, the variability of the participants could provide a more complete information for the researcher to understand the overall reading comprehension performance of target students.
Instruments
The instruments of the present study contained literary texts and informational texts.
However, only fiction (literary) and exposition (informational) were used for data collection.
It is suggested that other categories of literary texts, such as speeches, autobiography or poetry, and informational texts, such as persuasive or procedural text, could be included to diversify the assessment. In this way, the assessment not only provided a sufficient amount of each cognitive targets but also balanced the difficulty of reading texts and items. The
diversity of the assessment may provide an intact result of the analysis and classification of students’ performances for teachers.
On the other hand, the instruments were adapted using only multiple-choice items for data collection. However, in the reading framework for 2013 NAEP, constructed-response
items are one of the item types for the assessment. Therefore, this kind of items can be designed into ordered rationale and applied to the test. With the help of constructed-response items, the researcher may further investigate the reading comprehension performance of the target participants.
Factors that influence reading comprehension performance
According to the research design mentioned earlier, gender issue was one of the variables examined by the researcher. During the process of developing the English reading comprehension test, the researcher found that besides variables included in the test like text types and item types, other factors such as students’ prior knowledge, English proficiency, personal reading habits, topic familiarity, or even reading policies of a country may be factors that influence reading comprehension performance. It is suggested that these possible
variables could be further discussed in future studies.
References
吳清基（民99）。推動臺灣的閱讀教育─全民來閱讀。研考雙月刊，34（1），62-66。
Abdorahimzadeh, S. (2014). Gender differences and EFL reading comprehension: Revisiting topic interest and test performance. System, 42, 70-80.
Alderson, J. C., & Urquhart, A. H. (1984). Reading in a foreign language. New York:
Longman.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (U.S.). (2014). Standards for educational and psychological
testing. Washington, DC: AERA.
Arellano, M. D. C. (2013). Gender differences in reading comprehension achievement in English as a foreign language in Compulsory Secondary Education. Tejuelo, 17, 67-84.
Ary, D., Jacobs, L. C., Sorensen, C. K., & Walker, D. A. (2014). Introduction to research in
education (9th ed.). Wadsworth: Cengage Learning.
Ashcraft, M. H. (2004). Cognition (3rd ed.) (認知心理學，陳學志、邱發忠、劉政宏、林 宜利、陳萩卿、林奕宏、陳淑蓉譯)。台北：學富。(original work published 2002)
Brantmeier, C. (2003). Beyond linguistic knowledge: Individual differences in second language reading. Foreign Language Annals, 36(1), 33-43.
Brantmeier, C. (2003). Does gender make a difference? Passage content and comprehension in second language reading. Reading in a Foreign Language, 15(1), 1-27.
Brantmeier, C. (2005). Effects of reader's knowledge, text type, and test type on L1 and L2 reading comprehension in Spanish. The Modern Language Journal, 89(1), 37-53.
Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11(1), 33-63.
Bügel, K., & Buunk, B. P. (1996). Sex differences in foreign language text comprehension:
The role of interests and prior knowledge. The Modern Language Journal, 80(1), 15-31.
Carrell, P., Devine, J., & Eskey, D. (1988). Interactive approaches to second language
reading. New York: Cambridge University Press.
Christine, N. (2005). Teaching reading skills in a foreign language. Oxford: Macmillan.
Ina, V.S. M. & Michael, O. M. (2013). PIRLS 2016 assessment framework. Boston:
International Association for the Evaluation of Educational Achievement.
Cummins, J. (1979). Linguistic interdependence and the educational development of bilingual children. Review of Educational Research, 49(2), 222-251.
Dole, J. A., Duffy, G. G., Roehler, L. R., & Pearson, P. D. (1991). Moving from the old to the new: Research on reading comprehension instruction. Review of Educational Research,
61(2), 239-264.
García, G. E. (1991). Factors influencing the English reading test performance of Spanish-speaking Hispanic children. Reading Research Quarterly, 26(4), 371-392.
Gagné, E. D., Yekovich, C. W. & Yekovich, F. R. (1998). The cognitive psychology of school
learning (2nd ed.) (教學心理學-學習的認知基礎，岳修平譯)。台北：遠流。(original
work published 1993).
Gorjian, B., & Javadifar, M. (2013). Effects of gender and passage content on multiple-choice reading comprehension test. Procedia - Social and Behavioral Sciences, 84, 723-727.
Grabe, W. (2009). Reading in a second language: moving from theory to practice. New York:
Cambridge University Press.
Grabe, W., & Stoller, F. L. (2013). Teaching and Researching: Reading (2nd ed.). UK:
Routledge. (Original work published 2002).
Johnson, P. (1981). Effects on reading comprehension of language complexity and cultural background of a text. TESOL Quarterly, 15(2), 169-181.
Jung, J. (2009). Second language reading and the role of grammar. Teachers College,
Columbia University Working Papers in TESOL & Applied Linguistics, 9(2), 29-48.
Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329-354.
King, K. V., Gardner, D. A., Zucker, S., & Jorgensen, M. A. (2004). The distractor rationale
taxonomy: Enhancing multiple-choice items in reading and mathematics. Retrieved from
http://images.pearsonassessments.com/images/tmrs/tmrs_rg/Distractor_Rationales.pdf.
Kintsch, W. (1988). The role of knowledge in discourse comprehension: a construction-integration model. Psychology Review, 95(2), 163-182.
Ko, H. W., Chan, Y. L., Chang, C. Y. & Yu, T. Y. (2009). PIRLS 2006 National Report:
Reading literacy study of fourth grade Taiwanese students (2nd ed.). Retrieved from
http://lrn.ncu.edu.tw/Teacher%20web/hwawei/PIRLS%202006%20National%20Report
%EF%BC%882nd%20Edition%EF%BC%89.pdf
Ko, H. W., Chan, Y. L. & Ciou, J. H. (2013). PIRLS 2011 National Report: Reading literacy
study of fourth grade Taiwanese students. Retrieved from
http://lrn.ncu.edu.tw/Teacher%20Web/hwawei/Project/PIRLS%202011%E5%AE%8C%
E6%95%B4%E5%A0%B1%E5%91%8A.pdf
Lalas, J. W. (1984). Comprehension questions in ESL textbooks. Retrieved from https://ojs.lib.byu.edu/spc/index.php/TESL/article/viewFile/3163/2937.
Li, C. C. (2014). Analysis of reading comprehension levels of 4th to 6th grade students in
Pingtung County using the Rasch Model (Master’s thesis). Retrieved from
http://140.127.82.166/retrieve/20874/102NPTT0328013-001.pdf
Lin, I. C. (2010). The development of a reading comprehension test for 5th and 6th grade
students using NAEP reading framework (Master’s thesis). Retrieved from
ndltd.ncl.edu.tw/r/23909285826307337658
Lin, I. C., Lily, C. & Lu, I. C. (2013). The development of reading comprehension test for 5th and 6th graders using the Rasch Model. Bulletin of Educational Psychology, 45(1), 39-61.
Lin, J., Chu, K., & Meng, Y. (2010). Distractor rationale taxonomy: Diagnostic assessment
of reading with ordered multiple-choice items. Retrieved from
http://images.pearsonassessments.com/images/tmrs/tmrs_rg/LinJie_DistractorrationaleA ERA_Paper_51010.pdf?WT.mc_id=TMRS_Distractor_Rationale_Taxonomy_Diagnostic _Assessment.
NAGB. (2012). Reading assessment and item specifications for the 2009 national assessment
of educational progress. Washington, DC: the American Institutes for Research.
NAGB. (2012). Reading framework for the 2013 national assessment of educational
progress. Washington, DC: the American Institutes for Research.
OECD. (2013). PISA 2012 assessment and analytical framework: mathematics, reading,
science, problem solving and financial literacy. OECD Publishing. Retrieved from
http://dx.doi.org/10.1787/9789264190511-en
OECD. (2013). PISA 2015 draft reading literacy framework. OECD Publishing. Retrieved from
http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Frame work%20.pdf
OECD. (2014). PISA 2012 Results: What Students Know and Can Do – Student Performance
in Mathematics, Reading and Science (Volume I, Revised edition). OECD Publishing.
Retrieved from http://dx.doi.org/10.1787/9789264201118-en
Paris, S. G., & Stahl, S. A. (2005). Children's reading comprehension and assessment. New Jersey: Lawrence Erlbaum Associates, Inc.
Peng, S. (2019). A Study of the Differences between EFL and ESL for English Classroom Teaching in China. IRA International Journal of Education and Multidisciplinary Studies (ISSN 2455-2526), 15(1), 32-35.doi: http://dx.doi.org/10.21013/jems.v15.n1.p4
Perkins, K., Brutten, S. R., & Pohlmann, J. T. (1991). The effect of second language
proficiency on second language reading comprehension as measured by questions from different cognitive levels. Per Linguam, 7(1), 49-57. Retrieved from
http://perlinguam.journals.ac.za/pub/article/viewFile/345/382.
RAND Reading Study Group (2002). Reading for understanding: toward an R&D program
in reading comprehension. Santa Monica, CA: RAND.
Taiwan PISA National Research Center. (2011). TAIWAN PISA 2009 national report. ROC, Taipei: Psychological Publishing Co., Ltd.
Yoshida, M. (2012). The interplay of processing task, text type, and proficiency in L2 reading. Reading in a Foreign Language, 24(1), 1-29. Retrieved from
http://nflrc.hawaii.edu/rfl/April2012/articles/yoshida.pdf.
Zhou, L., & Siriyothin, P. (2011). Effects of text types on advanced efl learners' reading comprehension. Journal of Language and Culture, 30(2), 45-66. Retrieved from http://www.lc.mahidol.ac.th/lcjournal/FullPaper/JLC30-2-LinZhou.pdf.
Appendix 1: The English reading comprehension texts & questions (original version)
Appendixes
Appendix 2: The English reading comprehension texts & questions (modified version)