高中英文閱讀理解次序選擇題測驗之效度檢核

全文

(1)國立屏東大學英語學系碩士班碩士論文 . 指導教授：陳庸博士 . 高中英文閱讀理解次序選擇題測驗之效度檢核 The Validation of an English Reading Comprehension Test with Ordered Multiple-Choice Items for EFL Senior High School Students 研究生：楊季維撰中華民國一百零九年七月.

(2)

(3)

(4) 摘要本研究的目的在於檢核一份適合高中生使用之英文閱讀理解測驗的信效度。本測驗取自美國 North Carolina Testing Program，參考 NAEP2013 閱讀評量架構，依照閱讀理解之三個認知目標(尋找與回憶、整合與解釋、批判與評鑑)及二種文本類型(文學、訊息文本)形成雙向細目表，搭配次序選擇題的改寫原理，依此兩項基準檢核測驗的信效度；本測驗包含二篇文學文本及四篇訊息文本，共 39 題次序選擇題。檢核過程包含一次預試及正式施測，共有 109 位高雄市的高中生參與研究施測。本研究經過試題分析，檢核之各項信效度結果如下：一、信度：本測驗在傳統二元計分及次序計分上，具備良好的α信度。二、內容效度：測驗內容經研究者修改檢核後，大致符合閱讀理解構念之代表性樣本。三、構念效度：（1）試題難度因閱讀理解認知目標提升而增加；（2）訊息文本難易度較文學文本高；（3）分析結果顯示男性受試者與女性受試者在英語閱讀理解表現上並未達到顯著差異。最後，研究者根據分析結果提出相關議題之討論及未來研究之建議。關鍵字：NAEP、EFL、閱讀理解測驗、次序選擇題. I.

(5) Abstract The major purpose of this study is to validate an English reading comprehension test for English as a Foreign Language (EFL) senior high school students. The English reading comprehension test was modified from the sample items of the North Carolina Testing Program, based on the reading framework for 2013 NAEP, while its item distractors were revised, based on the rationale of ordered multiple-choice items. The English reading comprehension test was made up of three cognitive targets (locate/recall, integrate/interpret, and critique/evaluate) and two text types (literary and informational text), while its item distractors were scored hierarchically to measure students’ levels of understanding. A total of 109 EFL senior high students from Kaohsiung Municipal Jhongjheng Senior High School participated in this study. Reliability and validity evidences of the test were examined, and the results were summarized as follows: 1. Reliability: the English reading comprehension test has satisfactory reliability for original scoring procedure (0/1) and ordered scoring procedure. 2. Content-based validity: Researcher judgements and empirical evidence supported the adequacy and representativeness of the content of the English reading comprehension test to that defined by the NAEP reading framework.. II.

(6) 3. Construct-based validity: (1) As the cognitive targets of the English reading comprehension test increased, the difficulty of items/distractors increased as well; (2) Informational items were much difficult than literary items; (3) Male and female students performed no differently on various cognitive targets of the English reading comprehension test. According to the results of this study, discussion and suggestions for future studies were provided in the last section. Keywords: NAEP, EFL, reading comprehension test, ordered multiple-choice item. III.

(7) Table of Contents Chinese Abstract ................................................................................................. I English Abstract .................................................................................................II Table of Contents.............................................................................................. IV List of Tables .................................................................................................... VII List of Figures ................................................................................................... IX Chapter One Introduction.................................................................................. 1 Background and Significance of the Study ........................................................................ 1 The Purpose of the Study and Research Questions ........................................................... 7 Definitions of Terms .......................................................................................................... 8. Chapter Two Literature Review ...................................................................... 10 Reading Comprehension .................................................................................................. 10 Definitions of Reading Comprehension................................................................... 10 Models of Reading Comprehension......................................................................... 14 Reading Comprehension Processes ......................................................................... 16 Levels of Reading Comprehension .......................................................................... 22 Developmental Stages of Reading Comprehension ................................................. 28 The Assessments of Reading Comprehension ................................................................. 30 Methods of Measuring Reading Comprehension .................................................... 30. IV.

(8) Ordered Multiple-Choice Items ............................................................................... 34 International Reading Comprehension Assessments ............................................... 38 Factors that Influence L2 Reading Comprehension......................................................... 51 Proficiency and Reading Comprehension ................................................................ 51 Prior Knowledge and Reading Comprehension ....................................................... 52 Gender and Reading Comprehension ...................................................................... 53 Text Type and Reading Comprehension .................................................................. 54. Chapter Three Methods ................................................................................... 55 Participants ....................................................................................................................... 55 Instruments....................................................................................................................... 56 Research Procedure .......................................................................................................... 59 Data Analysis ................................................................................................................... 61 Research Limitation ......................................................................................................... 63. Chapter Four Results ........................................................................................ 65 The Validity of the English Reading Comprehension Test .............................................. 65 The Reliability of the English Reading Comprehension Test .......................................... 78. Chapter Five Discussions and Conclusions .................................................... 80 Conclusions ...................................................................................................................... 80 Discussions ...................................................................................................................... 82. V.

(9) Suggestions for Future Studies ........................................................................................ 87. References .......................................................................................................... 89 Appendixes ......................................................................................................... 96 . VI.

(10) List of Tables Table 1 The comparison of processes of reading comprehension from different sources ........ 28 Table 2 Chall’s stages of reading development ........................................................................ 29 Table 3 Levels of understanding and associated student errors for reading comprehension .. 35 Table 4 Comparison of an Ordered Multiple-Choice item and a Traditional Multiple-Choice item (Example one: main idea question) ......................................................................... 36 Table 5 Comparison of an Ordered Multiple-Choice item and a Traditional Multiple-Choice item (Example two: vocabulary-in-context question) ...................................................... 37 Table 6 Cognitive targets for the NAEP reading assessment................................................... 39 Table 7 Passage lengths and percentage distribution of literary and informational passages .......................................................................................................................................... 41 Table 8 Percentage distribution of time to be spent on specific item types ............................. 42 Table 9 Percentage distribution of cognitive targets by grades............................................... 43 Table 10 Definitions of achievement levels .............................................................................. 43 Table 11 Percentages of the PIRLS reading assessment devoted to each reading purpose and comprehension process .................................................................................................... 45 Table 12 Approximate distribution of score points in the PISA reading assessment ............... 47 Table 13 Characteristics of the participants ............................................................................ 56 Table 14 The frequency distribution of the cognitive targets of the English reading. VII.

(11) comprehension questions ................................................................................................. 57 Table 15 Example of Ordered Multiple-Choice item used for scoring procedure ................... 58 Table 16 The results of item analysis of literary items and the frequency distribution of different performance group ............................................................................................ 72 Table 17 The results of item analysis of informational items and the frequency distribution of different performance group ............................................................................................ 73 Table 18 Independent sample T-test results of students’ genders and reading comprehension scores ............................................................................................................................... 77 Table 19 The Reliability of the English Reading Comprehension Test .................................... 79. VIII.

(12) List of Figures Figure 1. A schematic diagram of the major processes and structures in reading comprehension.. ............................................................................................................... 17 Figure 2. Kintsch’s (1988) Construction-integration Model.. ................................................. 22 Figure 3. Relationship between PISA 2012 Reading Framework and the Aspect Subscales. . 48 Figure 4. Research Procedure .................................................................................................. 59 Figure 5. The scatter plot of item difficulty of different cognitive targets in two text types... 75 Figure 6. The scatter plot of item difficulty of different cognitive targets .............................. 76. IX.

(13) Chapter One Introduction This chapter provides the background and depicts the purpose of the present study. Three sections are included: background and significance of the study, purpose of the study and research questions, and definitions of terms used in the study. Background and Significance of the Study Reading literacy is one of the most important abilities for learning. It is not only the foundation for future learning across all subjects as well as for personal growth and recreation, but also a skill that helps students cultivate their problem-solving abilities. David Blunkett, the former Britain Secretary of State for Education and Employment, once mentioned that reading is the basis for future learning, so he launched the National Year of Reading from September, 1998 to August, 1999, with the stated aim of engaging the whole community in reading in order to build a nation of readers. In America, several reading campaigns were proposed as the government noted the importance of reading ability. “The America Reads Challenge” is a campaign led by Bill Clinton, the former President of the United State, and the other President Bush proposed “No Child Left Behind (NCLB)” and “Reading First” as the core of his policy to be implemented. Having seen that many countries put their emphasis on reading literacy, Taiwan’s government also proposed several projects of reading, such as elementary school children’s. 1.

(14) reading programs which involve three hundred elementary schools located at the areas short of cultural resources; the reading promotion programs of elementary, junior, and senior high schools of remote districts; fun reading 101-the reading promotion programs of basic education; and root reading and room improvement-the innovative service development plan of libraries (Wu, 2010), and so forth. On the other hand, according to PIRLS’s reports, Taiwan’s 4th grade students rank 22 (Mean=535) in 2006 PIRLS assessment, and 9 (Mean=553) in 2011 PIRLS assessment. As for PISA’s reports, Taiwan’s 9th grade teenagers rank 23 (Mean=495) in 2009 PISA assessment, and 8 (Mean=523) in 2012 PISA assessment. It is noted that after the implementation of reading projects as mentioned, reading literacy of Taiwan’s students is getting better and better. Although the results of the assessments showed some progress in reading performances, there is still room left for us to improve. Reading is a process that involves different aspects of reasoning strategies. Readers usually tend to search for information they need while browsing through the article. After forming an overview of the article, readers will conduct different levels of interaction with the article. According to readers’ prior knowledge and previous experiences, they begin to form some hypotheses about the ideas they get from the article and change their schemata as the reading process continues. Then, comprehension occurs when readers integrate their new understandings about the article into this ongoing process.. 2.

(15) A number of reading theories pointed out that in the course of the reading process, readers will come by different levels of understandings. Kintsch (1988) noted that reading comprehension is a complex cognitive process. Readers will have three levels of comprehension during their reading process: microstructure, macrostructure, and situation model. Microstructure refers to the phenomenon that readers understand the meanings from the sentences in the article, construct the propositional representation of a text, and then have an initial understanding of the article. Microstructure can be seen as the first level of comprehension building. Macrostructure refers to the phenomenon that readers integrate the microstructure of the article and form a whole understanding of the content and the topic of the article after they have finished reading the article. This can be seen as a further level of comprehension. The highest level of comprehension building is the situation model, and this refers to the phenomenon that readers connect their prior knowledge to the content, and come up with higher comprehension of the article. Swaby (1989) suggested that reading comprehension is a skill, and reading skills to certain extent can develop four reading levels. The first one is literal comprehension, and this means readers can understand the main ideas from the sentences in the article. Next one is inferential comprehension, and this refers to the fact that readers can infer the meanings behind based on their own experiences and intuition from the description of the article. The third level is the evaluative comprehension, and this means readers will come up with their own perspectives, according to the messages from the. 3.

(16) article. Last one is the critical comprehension, and this refers to the fact that readers are able to analyze the article with its writing style or content (as cited in Lin, 2010). According to Kintsch (1988) and Swaby’s (1989) theory, reading comprehension contains understandings of the literal meanings of words and sentences, the integrative meanings of the article and the inference of the meaning behind, and of the critical thinking about the content and the structure of the article. The assessment of reading comprehension should include these aspects, so the ability of reading comprehension will be effectively measured. Nowadays, many reading frameworks were set up, based on the integration of reading theories and related empirical studies. For example, the Programme for International Student Assessment (PISA) contains three broad aspect categories in their reading assessment, namely, access and retrieve, integrate and interpret, and reflect and evaluate; Progress in International Reading Literacy Study (PIRLS) assesses four broad-based processes of comprehension in their reading framework, including focusing on and retrieving explicitly stated information, making straightforward inferences, interpreting and integrating ideas and information, and evaluating and making critiques on content and textual elements. The first two levels are the processes of direct comprehension and the other two are the processes of explaining comprehension; National Assessment of Educational Progress (NAEP) distinguished two text types and defined three cognitive targets in their reading framework,. 4.

(17) the former refers to informational text and literary text, while the latter includes locating and recalling what readers have read, integrating the information and interpreting what readers have read, and critiquing and evaluating text. As mentioned above, these international reading assessments constructed their reading frameworks based on the aspect of cognitive processes, and many studies related to reading assessment tend to accept that reading comprehension contains literal comprehension and integrative comprehension, connecting one’s prior knowledge to the text, and Kintsch’s (1988) model, therefore, was widely used in many reading studies. Despite the fact that Taiwan’s government has put emphasis on developing reading ability, few studies of reading except large-scale reading assessments (e.g.,PISA, PIRLS) are discussed with cognitive levels involved. In PIRLS 2011 report of Taiwan, Ko (2013) and her colleagues reported that Taiwanese students’ reading performance on direct comprehension process performed better than interpretative comprehension process. Although the reading performance on these two processes made an improvement, compared with PIRLS 2006 report, the researchers suggested that reading comprehension items designed in cognitive processes may improve students’ overall reading comprehension (Ko, et al., 2013). In Taiwan PISA 2009 national report, Hung (2011) and her colleague also reported that Taiwanese students’ interpretative and evaluative comprehension processes are weaker, and the educational assessments in Taiwan could use PISA framework as a reference in item design. 5.

(18) (Hung, et al., 2011). Owing to the reasons mentioned above, reading items designed in cognitive levels play an important role in English reading comprehension tests. Although it is a common practice for test developers to construct reading comprehension test according to the cognitive levels entailed by reading theories as described above, very few studies have incorporated these levels into writing item distractors. In other words, few tests have built item distractors reflecting different or hierarchical reading cognitive levels. Among the few studies, Pearson Inc. (2004) has developed a system for using distractors to collect and report data that reveals a student’s level of understanding of a subject area. These distractors include a rationale that explains why the distractor may lead to the wrong answer. When organized into a taxonomy that reflects levels of understanding, distractor rationales enable the development of multiple-choice items that yield a broader view of students’ incorrect choices. However, traditional reading comprehension tests were usually developed into multiple-choice items, and these distractors do not provide the information necessary to determine how close for a student to meet the content standard. Instead of simply identifying whether a student understands a concept or not, ordered multiple-choice items provide a diagnostic function to identify possible misconceptions or misunderstandings with students’ cognitive process being considered. This kind of information could help teachers adapt their instructional and intervention strategies to the needs of some particular groups of students.. 6.

(19) The Purpose of the Study and Research Questions The purpose of present study is to validate an English reading comprehension test, modified from the sample items of the North Carolina Testing Program, with ordered multiple-choice items based on reading framework for 2013 NAEP. According to the purpose of the study and the research design, several research questions are drawn up as follows: 1. Does the English reading comprehension test have satisfactory reliability? 1-1 Does the English reading comprehension test have satisfactory internal consistency (Cronbach alpha)? 2. Does the ordered multiple-choice English reading comprehension test have satisfactory validity? 2-1 Is the content (item/distractors) of the English reading comprehension test relevant to the representative sample and its cognitive levels? (content-based validity) 2-2 Is the difficulty of items/distractors of various cognitive levels of reading comprehension test increased as the cognitive level of reading comprehension increases? (construct-based validity) 2-3 Is the item difficulty of literary text and informational text different? (construct-based validity) 2-4 Do boys and girls perform differently on various cognitive levels of the English reading comprehension test? (construct-based validity). 7.

(20) Definitions of Terms 1. Reading Comprehension test “Reading is an active and complex process that involves understanding written texts, developing and interpreting meaning, and using meaning as appropriate to type of text, purpose, and situation” (NAGB, 2012, p.2). The reading comprehension process includes three cognitive targets: locate/recall, integrate/interpret, and critique/evaluate. The first target indicates that readers “locate or recall information from what they read, identify clearly stated main ideas or supporting details, or find essential elements of a text” (NAGB, 2012, p.38), such as time, setting, or characters; the second target refers to “what readers do as they integrate new information into their initial sense of what a passage says, often interpreting what they read in the process” (NAGB, 2012, p.38); The last target requires readers to “stand back from what they read and view the text objectively” (NAGB, 2012, p.39). The focus remains on the text itself but the readers have to consider the text critically by assessing it from numerous perspectives and synthesizing what is read with other texts or experiences. The present study used the assessment adapted from the reading sample items of the North Carolina Testing Program and by modifying the items based on these three cognitive targets to measure participants’ reading comprehension performances. It was shown that higher scores participants received, the better they comprehend the reading passages.. 8.

(21) 2. Ordered Multiple-Choice item (OMC) A traditional reading comprehension question designed by using the Distractor Rationale Taxonomy (DRT) is called “Ordered Multiple-Choice item.” These multiple-level distractor rationale descriptions entail and classify errors associated with each option through the Understanding Hierarchy system (Lin, Chu & Meng, 2010). Each item has four levels of understanding: minimal (level 1), moderate (level 2), significant (level 3), and complete (level 4). Level one includes the most fundamental errors; levels two and three correspond to responses that, while incorrect, indicate increasing sophistication in the student’s response; and level four represents the correct response. Each item, according to the intended cognitive level to be measured might consist of different number of levels of distractors (e.g., for a level-two item with four options, three of the four options will be level-one distractors). Besides the three cognitive targets from the NAEP, the present study used Ordered Multiple-Choice item not only to enhance the function of distractors, but also to diagnose students’ possible misconceptions or misunderstandings. 3. English as a Foreign Language (EFL) students EFL is an abbreviation for "English as a Foreign Language". The term is mainly used for non-native English learners who learn English in non-English speaking countries (e.g., Taiwanese students who learn English in their country are EFL students) (Peng, 2019).. 9.

(22) Chapter Two Literature Review According to the purpose of the study and the research questions, the researcher reviewed related theories and empirical studies of reading comprehension as the basis of the study. There are three sections in this chapter: the theories of reading comprehension, factors that influence L2 reading comprehension, and the assessments of reading comprehension.. Reading Comprehension The purpose of the study is to validate an English reading comprehension test for senior high school students in Taiwan, and therefore, the definitions and theories of reading comprehension will be reviewed first to form an overview of the meaning of reading comprehension. Five sections: the definitions of reading comprehension, the models of reading comprehension, the theories of reading comprehension processes, the levels of reading comprehension, and the stages of reading development will be illustrated below. Definitions of Reading Comprehension Reading is a complex cognitive process. No matter what spelling and writing system is, reading can be divided into two parts: word recognition and comprehension. Word recognition is the basic element of comprehension, and the ultimate goal of reading is comprehension. As for the definition of reading comprehension, many researchers hold different views of it.. 10.

(23) In the traditional view, reading is the ability to understand meaning from the printed page and interpret it appropriately (Grabe & Stoller, 2011). Swaby (1989) asserts that reading comprehension is a demonstration of reading skills. When the reading skills are acquired, readers are seen as fluent readers who are able to comprehend what they read. During this process, reading comprehension can be seen as rapid and accurate recognition of information in the text. Readers in this process are passive receivers because meaning is in the text and readers’ task is to reproduce it (Dole, Duffy, Roehler & Pearson, 1991). In the cognitive view, Anderson and Pearson (1984) suggested that reading comprehension is readers construct meanings through the interaction of the information in the text with their prior knowledge. Alderson (2005) states that during the process of reading, besides the literal meaning of words, implied meaning and readers’ evaluation of the text are important elements for comprehension. Gagné, Yekovich, and Yekovich (1993/1998) claimed that successful reading comprehension depends on three component skills: conceptual understanding, automated basic skills, and strategies. Conceptual understanding includes topics that readers are reading, text schema, and vocabulary and so on; automated basic skills are decoding skills of words and propositional ability of strings of words; strategies contains personal reading methods and it depends on reader’s reading purpose and their comprehension monitoring. Conceptual understanding are stored in the declarative memory while the skills and strategies are stored in the procedural memory, and the process of reading. 11.

(24) are presented as sets of productions. The definitions of reading comprehension mentioned above are rooted from different perspectives. Meanwhile, many reading comprehension assessments are developed based on these reading theories. However, the definitions of reading and reading comprehension have changed over time in parallel with changes in culture, society, and economy. The National Assessment of Educational Progress (NAEP), which is a representative measure of trends in academic achievement of U.S. elementary and secondary students, redefined their reading framework as NAEP 2013. The definition of reading for NAEP 2013 is derived from several sources and grounded in scientific research on reading. Among the sources, the definitions of reading literacy come from the RAND Report, the Progress in International Reading Literacy Study (PIRLS), and the Programme for International Student Assessment (PISA) influenced the development of reading framework for NAEP 2013. The RAND Reading Study Group (2002) defined reading comprehension as “the process of simultaneously extracting and constructing meaning through interaction and involvement with written language” (p.11). In their definition, comprehension consists of three elements: “the reader who is doing the comprehending, the text that is to be comprehended, and the activity in which comprehension is a part” (p.11); PISA modified their definition by adding engagement with the texts, and the definition is “understanding, using, and reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in. 12.

(25) society” (OECD, 2013, p.9); and PIRLS defined reading literacy as “the ability to understand and use those written language forms required by society and/or valued by the individual. Readers can construct meaning from texts in a variety of forms. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment” (Mullis & Martin, 2013, p.14). The definitions of the RAND Report, PISA, and PIRLS have contributed greatly to NAEP 2013 reading framework, and all of them pointed out that “reading is an active, complex, and multidimensional process undertaken for many different purposes” (NAGB, 2012, p.6). Therefore, the NAEP 2013 Reading Assessment, guided by a definition of reading that reflects scientific research, draws on multiple sources, and conceptualizes reading as a dynamic cognitive process, proposed the definition that “reading is an active and complex process that involves understanding written text, developing and interpreting meaning and using meaning as appropriate to type of text, purpose, and situation” (NAGB, 2013, p.2). According to the three points of view mentioned above, traditional view of reading comprehension focused on reading skills. Readers are passive recipient of the text; cognitive view of reading comprehension stressed that readers construct meanings by interacting with their prior knowledge; assessment view of reading comprehension further extended the definition in an operational way. The purpose of the study is to validate an English reading comprehension, and therefore, the clear definition of reading has to be made. The reading. 13.

(26) comprehension test of the present study will be modified based on NAEP 2013’s definition of reading, because it is derived from many sources and grounded in empirical studies in reading. Models of Reading Comprehension There are three metaphorical models of reading that reflect main processing assumptions about how comprehension occurs: bottom-up model, top-down model, and interactive model. These three models are illustrated below. Bottom-up model, also called data-driven model, is a linear model for comprehension, and it suggests that readers recognize letters, words, and sentence structures to build up meanings (Nuttall, 2005; Lin, 2010). During this process, readers translate what they have read piece-by-piece “with little interference from readers’ background knowledge” (Grabe, 2009, p.89). The information in the text is more important than readers’ background knowledge. However, word meanings depend on its context. Readers usually infer meanings from the context to connect the information. Therefore, Ko (1993) suggested that using decoding to explain the interaction between readers and texts is not enough (as cited in Lin, 2010). Top-down model, also called conceptually driven model, emphasizes on readers’ background knowledge as a guide for comprehension. This model assumes that “reading is primarily directed by readers’ goals and expectations” (Grabe & Stoller, 2011, p.25). Readers. 14.

(27) will have expectations about the text based on their background knowledge during the ongoing process and confirm these expectations according to what they have read in the text. During this process, when readers encountered unknown words, their prior knowledge is compensation for this issue. One problem of this model is “what a reader can learn from a text if the reader must first have expectations about all the information in the text” (Grabe, 2009, p.89). In fact, when readers are reading some articles that they are not familiar with (i.e., topics they are not familiar with), they can still decode word meanings to understand the article. Perfetti and McCutchen (1986) pointed out that good readers will come up with meanings immediately, without activating their schemata, when they read words in the text (as cited in Lin, 2010). Therefore, simply use top-down model to explain comprehension process is not enough. Interactive model is the combination of top-down model and bottom-up model. Word recognition and syntactic parsing are major part of comprehension; context and prior knowledge are supports for former lower-level processes; and prediction and inferencing help improve word recognition. In other word, fluent reading comprehension requires fast and automatic lower-level (i.e., word recognition, syntactic parsing) and higher-level (i.e., text model, situation model, inferencing) processes. That is to say, when readers have some background knowledge about the text, even if they encounter unfamiliar words, they can still use top-down model to understand the text; when readers do not have background knowledge. 15.

(28) about the text, they can use bottom-up processing to comprehend the text. Reading Comprehension Processes According to the different focuses of reading processes, there are four theories: recycling model, schema theory, information processing theory, and construction-integration model discussed in this section. These four theories are illustrated below. Just and Carpenter (1980) used eye trackers to examine college students’ eye movements while reading and they found that readers will anticipate the interpretations of the words they read. Once the interpretation matches the meaning in the text, proposition will occur, and then readers will integrate all the propositions form an overall interpretation of the text. If a word has more than one interpretation and the meaning the reader comes up with does not match the other propositions, the reader will form another interpretation of the word. Therefore, reading comprehension is a recycling model that follows decoding the words, forming propositions, and integration of the propositions. The theory that Just and Carpenter (1980) proposed contains two models of reading comprehension: contextual effects (top-down processing) and lexical access (bottom-up processing), and they proposed that top-down processing cannot replace the interpretations from the sentences. Top-down processing helps readers decode the sentences, not to replace bottom-up processing. In other words, readers will mainly use bottom-up processing and topdown processing will be a support while reading. By going through decoding the word. 16.

(29) meanings, forming propositions and integration of the propositions, the recycling model will be formed by readers.. Figure 1. A schematic diagram of the major processes and structures in reading comprehension. (Solid lines donate data-flow paths, and dashed lines indicate canonical flow of control) Adapted from “A Theory of Reading: From Eye Fixations to Comprehension,” by M. A. Just, and P. A. Carpenter, 1980, Psychological Review, 87(4), p. 329-354. Copyright 1980 by the American Psychological Association. Schema theory plays an important role in reading comprehension. The role of schema is like a container that provides us a place to store experiences that we have been through. As long as we need it, we can retrieve the ideas from our experiences to deal with what we. 17.

(30) encountered. In Bartlett’s (1932) study, he found that when readers encountered ideas that are new to their schemata, they would try to relate the previous ones to the new, and adjust these new ideas to accommodate to the original schemata (as cited in Ashcraft, 2004). Anderson (1984) also made a clear elaboration that schemata help assimilate text information, determine what is important in a text, and improve inferential elaboration. In other words, with schemata’s supports, readers are not just receiving new information. Instead, they will use their schemata to help build up their comprehension. If readers are not able to integrate the new ideas with original schemata, readers may have wrong interpretations on the text. As discussed above, schema theory put emphasis on readers being active roles during the reading processes. No matter content schema, which means content related to the text, or text schema, which refers to text structures, these factors are involved in reading comprehension. In Gagné, Yekovich, and Yekovich’s (1993/1998) theory of reading comprehension process, they divided the process into four parts: decoding, literal comprehension, inferential comprehension, and comprehension monitoring. During this ongoing process, every part will probably be processing in a parallel way. The first part of the processing is decoding, which means to crack written symbols and make them become meaningful. There are two processes involved in this processing: matching and recoding. Matching refers to readers will match the interpretation of written. 18.

(31) words with their sight vocabulary (i.e., a set of words that readers can automatically recognize); recoding refers to sounding out the written words to retrieve the interpretation from their long-term memory. Written languages and sounds are part of input in decoding process, and they will stimulate literal comprehension processing. The purpose of literal comprehension is to infer meanings from written words, and it contains two processes: lexical access and parsing. Lexical access is a set of productions that uses the products of decoding process to figure out appropriate meanings in the text. Gagné, Yekovich, and Yekovich (1993/1998) pointed out that people have different mental dictionary, so this will cause different product of lexical access, and therefore, inferential comprehension will be affected; Parsing refers to readers recognize syntactic patterns and linguistic rules of the language and combine grammatical information such as phrasal groupings, word ordering information and so on in a logical connection to understand the meanings in the text. Although readers need to understand the meanings presented in the text in the process of literal comprehension, lexical access and parsing are fundamental requirements for higher-level processes. In our daily lives, sometimes people will encounter some reading tasks such as reading the menu or reading the time table of trains. In these reading tasks, readers will complete the decoding processing in the literal comprehension process. However, if readers want to know interpretations beyond the text, the process of inferential comprehension will provide a. 19.

(32) further and deeper level of comprehension of the text. There are three processes involved in this process: integration, summarization, and elaboration. The function of integration is to make a coherent connection between ideas presented in the text, discover the relationship between words and sentences, and combine two or more propositions in a logical way; summarization refers to building a macrostructure to include major ideas of the text and form a mental outline with propositions hierarchically listed after reading it; elaboration means that readers connect their prior knowledge with the text and construct meanings beyond the text. The last part of the processing is comprehension monitoring. This process ensures that readers can effectively and efficiently achieve reading goals. Comprehension monitoring includes goal setting, strategy selection, goal-checking, and remediation. To a skilled reader, comprehension monitoring starts at the beginning of reading processes, and keep working during whole reading processes. First, the readers will set up a goal and choose appropriate strategies to achieve the goal. Then, the readers will use some methods to check whether they achieve the goal or not, such as asking oneself a question related to the text and see if oneself can correctly answer the question. During the reading processes, goal-checking may interrupt the ongoing process, and therefore, remediation will be activated to deal with the problem that caused interruption. Kintsch (1988) proposed the construction-integration model based on the interaction between textbase and readers’ previous knowledge and experiences. The researcher thought. 20.

(33) that readers will “form propositional units according to the words of the text and their syntactic relations (p.72)” to build the textbase. Then, in order to achieve a real understanding of a text, readers connect the textbase to their previous knowledge and experiences to build up a situation model. If the situation models that readers built can be effectively used in other situations they encounter later and help them solve the problem, then it is called meaningful text learning. Therefore, Kintsch (1988) suggested that readers will have three processes of comprehension processes in reading: microstructure, macrostructure, and situation model. In order to form microstructure of the text, readers need to decode the sentences first, form propositional representations, and then have an initial understanding of the text. It can be seen as the first level of building comprehension. After readers finish reading the text, they integrate the microstructure they formed to construct macrostructure of the text. In this process, readers are able to form a global understanding of the topic and the content. The last process is situation model, which means that readers connect their previous knowledge and experiences to the text to build up a higher and deeper comprehension of the text. It requires the integration of propositions with readers’ relevant experiences or knowledge and readers’ goals of reading. During the reading processes that Kintsch (1988) proposed, construction and integration will keep recycling for comprehension. In the process of construction, readers keep. 21.

(34) integrating old propositions and coming up with new ones to build up a textbase, while in the process of integration, readers match the text with their own knowledge and experiences, and infer the meanings behind the text. By going through these processes, readers proceed beyond the text and reach a complete level of comprehension.. Figure 2. Kintsch’s (1988) Construction-integration Model. Adapted from “Analysis of Reading Comprehension Levels of 4th to 6th Grade Students in Pingtung County Using the Rasch Model,” by C. C. Li, 2014, p. 11. According to the theories discussed above, it is noted that although each theory defines reading comprehension processes from different point of view, they all show that reading comprehension is a complex process. Readers need to not only decode the surface structure of the text, integrate propositional representations in different aspects, but also activate their own schemata and prior knowledge to achieve a higher level of comprehension. Levels of Reading Comprehension According to the definitions and theories of reading comprehension discussed so far, these levels contain from surface comprehension (i.e., literal comprehension) to deeper. 22.

(35) comprehension (i.e., inferential comprehension, critique and evaluation of the text). No matter what point of view is highlighted, the levels of reading comprehension suggested by every researcher are quite similar. Only the categorizations that every researcher used are slightly different. In Pearson and Johnson’s (1978) taxonomy of reading comprehension, there are three aspects included: literal comprehension, inferential comprehension, and comprehension involved in personal experiences. Meanwhile, there are three types of reading comprehension questions developed based on the interaction between questions, answers, and readers: textually explicit question, textually implicit question, and scriptally implicit question (as cited in Lin, 2010; Joselito, 1984). At the level of literal comprehension, textually explicit question is used to test reader’s performance. The description of the question is quite similar to what is stated in the text, and reader should be able to point out the answer directly from the text. As for inferential comprehension, textually implicit question is used at this level. The answer of this type of question involved the factual information stated in the text and reader’s background knowledge. Readers need to infer meanings between the sentences to find out the answer. At the last level of comprehension, scriptally implicit question is used to test whether reader achieve this goal or not. The stem of the question and answer at this level is developed beyond the text. Readers have to retrieve related schemata and connect personal experiences. 23.

(36) to the text in order to make correct responses to the question. Swaby (1989) suggested that reading comprehension is a skill, and reading skills to different extent can be developed into four comprehension levels: literal comprehension, inferential comprehension, evaluative comprehension, and critical comprehension (as cited in Lin, 2010). Readers can understand the main idea from sentences is literal comprehension. Related skills contain retrieving the facts, details or orders presented in the text, and knowing the meanings of words. While inferential comprehension refers to descriptions of the texts plus readers’ own knowledge and experiences for understanding the meanings behind. Related skills include inferring the central idea, appropriate interpretations, prediction, and comparisons. Next one is evaluative comprehension and this means that readers will come up with their own perspectives according to the messages presented in the texts. Related skills contain value judgment, supporting or rejecting opinions in the texts. At the last level of comprehension, readers are able to analyze text formats and content. Related skills include figuring out the logic and consistency of the ideas, and discriminating the syntactic patterns of the texts. The levels that Swaby (1989) proposed not only contain surface and deeper comprehension, but critique and evaluation of the text also included. Compare to Pearson and Johnson’s (1978) taxonomy of reading comprehension, Swaby (1989) developed a more. 24.

(37) extensive level on reading comprehension. The NAEP reading assessment illustrated that reading is an active and complex process by three cognitive targets, and used them as a guide to develop reading comprehension questions. These questions assess learners’ comprehension of literary and informational text (NAGB, 2012). In the following section, each cognitive target used in the NAEP assessment and their connections with other theories will be illustrated. The cognitive targets of the NAEP reading assessment refer to “the mental processes or kinds of thinking that underlie reading comprehension (p.37)”, and these targets “serve to guide the test development process in that item writers target these processes or kinds of thinking as they write items (p.40).” These cognitive targets are: locate and recall, integrate and interpret, and critique and evaluate. On the first target, readers retrieve factual information, clearly identify the major concepts and supporting ideas or any other essential elements of the text, such as times, characters, or settings. In the process of answering items on this cognitive target involves “matching information given in the item to either literal or synonymous information in the text (p.38).” Items that assess the first target “focus on information contained in relatively small amounts of text: a sentence, a paragraph, or two or more adjacent paragraphs (p.38).” “These items provide information about the most basic comprehension skills, those that ultimately form the foundation for a more elaborate understanding of the text (p.38).”. 25.

(38) On the second target, readers “integrate new information into their initial sense of what a passage says, and often interpret that they read in the process (p.38).” When readers start to integrate and interpret the ideas of the text, “they make comparisons and contrasts of information, examine relations across aspects of text, or consider alternatives to what is presented in the text (p.38)” This process is “critical to comprehension and can be considered the stage at which readers really move beyond the discrete information, ideas, details, themes, and so forth presented in text and extend their initial impressions by processing information logically and completely (p.38).” Items that assess the second target might contain “asking students to form generalizations about a piece of informational text, making statements about how the settings of a story contributes to the creation of theme, or interpretations of author’s reasons for attempting to persuade readers about an issue or of an implied message or moral from a story (p.39).” The last target needs “readers to stand back from what they read and view the text objectively (p.39).” The focus still “remains on the text but the reader’s purpose is to consider the text critically by assessing it from numerous perspectives and synthesizing what is read with other texts and other experiences (p.39).” Items that assess this target may include “evaluating the quality of a text, determining what is most significant in a passage, or judging the effectiveness of specific textual features to accomplish the purpose of the text (p.39).” Certain items might “ask for the likelihood that an event could actually have taken place, the. 26.

(39) plausibility of an argument, the adequacy of an explanation for an event, the level of language choices, or the broader level of the entire text (p.39).” As discussed above, it is noted that the NAEP reading assessment developed items based on three cognitive targets (i.e., locate and recall, integrate and interpret, critique and evaluate) to measure reader’s understanding of a text. The researcher thinks that the reading processes that NAEP proposed have similar patterns with Gagné’s et al. (1993/1998) information processing theory, Kintsch’s (1988) construction-integration model, Pearson and Johnson’s (1978) taxonomy of reading comprehension, and Swaby’s (1989) levels of reading comprehension. As shown in Table 1, the framework of NAEP reading assessment was formed based on several reading theories. Among these theories, Kintsch’s (1988) situation model and Pearson and Johnson’s (1978) comprehension involved in personal experiences need readers to connect information to their prior knowledge and experiences. In NAEP reading assessment, “integrate and interpret” and “critique and evaluate” these two cognitive targets also require readers connecting what they read to their previous experiences, and integrating the information for critique and evaluation. Therefore, situation model and comprehension involved in personal experiences correspond to NAEP’s two cognitive targets: “integrate and interpret” and “critique and evaluate.” . 27.

(40) Table 1 The comparison of processes of reading comprehension from different sources Sources NAEP (2013) Gagné Yekovich, and Yekovich (1998) Kintsch (1988). Processes/Levels of reading comprehension Locate and Recall Literal comprehension Microstruture. Pearson and Johnson (1978). Literal comprehension. Swaby (1989). Literal comprehension. Integrate and Interpret Inferential comprehension Macrostructure Situation model Inferential comprehension Comprehension involved personal experiences Inferential comprehension. Critique and Evaluate Comprehension monitoring Situation model Comprehension involved personal experiences Evaluative comprehension Critical comprehension. Developmental Stages of Reading Comprehension Reading is an active and complex process that involves multiple behaviors. Students change their reading behaviors as they grow up. Chall (1983) proposed six developmental stages to explain child’s reading development. These six stages are separated into “learning to read” and “reading to learn”. Stage 0-2 can be categorized as the time of “learning to read”, and stages 3-5 as the time of “reading to learn”. The former means that students learn how to read while later means that through reading, students can acquire new knowledge. One thing should be noticed that these stages are not discrete; they are continuous and overlapping (Chall, 1983). Table 2 shows the six developmental stages of reading processes.. 28.

(41) Table 2 Chall’s stages of reading development Stage. Reading to learn. Learning to read. Stage 0: Pre-reading “pseudo reading”. Approximate age/grade. 6 months – 6 years old Preschool. Stage 1: Initial reading and decoding. 6–7 years old 1st grade – 2nd grade. Stage 2: Confirmation and fluency. 7–8 years old 2nd grade – 3rd grade. Stage 3: Reading for learning the new. 9–13 years old 4 grade – 8th grade. Stage 4: Multiple viewpoints. Stage 5: Construction and reconstruction. th. 14–17 years old 10 grade – 12th grade th. 18+ years old College and beyond. Characteristics Child “pretends” to read, retells story when looking at pages of book previously read to him/her, names letters of alphabet; recognizes some signs; prints own name; plays with books, pencils and paper. Child learns relation between letters and sounds and between printed and spoken words; child is able to read simple text containing high frequency words and phonically regular words; uses skill and insight to “sound out” new one syllable words. Child reads simple, familiar stories and selections with increasing fluency. This is done by consolidating the basic decoding elements, sight vocabulary, and meaning context in the reading of familiar stories and selections. Reading is used to learn new ideas, to gain new knowledge, to experience new feelings, to learn new attitudes, generally from one viewpoint. Reading widely from a broad range of complex materials, both expository and narrative, with a variety of viewpoints. Reading is used for one’s own needs and purposes (professional and personal); reading serves to integrate one’s knowledge with that of others, to synthesize it and to create new knowledge. It is rapid and efficient.. Note. Adapted from “Stages of Reading Development,” by J. S. Chall, 1983. Retrieved from http://online.warner.edu/pluginfile.php/335772/mod_resource/content/1/challs_stages_of_reading_development.pdf. 29.

(42) The reading theories mentioned above illustrated the processes of reading comprehension and developmental stages of reading comprehension. However, in EFL context, some variables will affect students’ reading comprehension performances. These variables will be discussed in the third section.. The Assessments of Reading Comprehension In many researches and assessments, researchers adapted or developed different kinds of instruments according to their research topics. However, the major purpose of developing an instrument is to objectively measure participants’ reading comprehension ability. In this section, the methods of measuring reading comprehension, ordered multiple choice items and several international reading comprehension assessments will be introduced below. Methods of Measuring Reading Comprehension Multiple-Choice Items Multiple-Choice items are widely used in many standardized reading comprehension tests, such as TOEFL, TOEIC, and GEPT. Typically, a multiple-choice item consists of one or more question stems and three or more possible answers. The correct or the most appropriate answer is called the key, and the incorrect choices are referred to as the distractors (King, Gardner, Zucker & Jorgensen, 2004). The administration of multiple-choice reading comprehension test is to let students read an article first, and then answer the multiple-choice questions related to the content they read. The advantage of multiple-choice items is that test. 30.

(43) scores can be easily calculated, either by the researchers or by the computers. However, Kintsch and Kintsch (2005) pointed out that some higher order comprehension cannot be measured by using multiple-choice items. Short Answers Short answer questions include two formats: short constructed-response items and extended constructed-response items. Short constructed-response questions require answers in one or two phrases or in one or two sentences. This kind of items may contain questions of literal comprehension whose answers can be found in the text; questions involving reorganization or reinterpretation which the student need to reinterpret the context or to retrieve information from different part of the text and reorganize in a new way, perhaps using a little inferencing skill; and questions of inference which oblige the student to think about the meanings behind the text. Questions of inference also need the student to reorganize pieces of information from various parts of the text, but this type of questions requires more sophisticated inferencing skills. Extended constructed-response items require more elaborated answers in one or two paragraphs. This kind of items usually asks the student to evaluate or judge the whole text, such as the writer’s bias, the quality of evidence and so on (Nuttall, 2005). Free-recall In free-recall test, students are asked to read a text first, put the text aside once they. 31.

(44) finished reading, and then write down as much information from the text as possible (Nuttall, 2005). The score will be calculated based on the idea units from the text that the student wrote down. The more information that students recalled, the better comprehension they got. Summary The summary test requires students’ summarizing skills and organizing skills. In order to accomplish the task, students have to locate the main ideas of the text and distinguish relevant from irrelevant ideas. The score will be calculated, as in free-recall, according to the main ideas reproduced by students. One thing should be noticed when administrating the summary test is that students may have difficulty expressing their answers adequately in the target language because reading skills and writing skills are both involved in the summary test. Therefore, instructors have to be aware of the validity of the summary test. Cloze The cloze test is another way to measure students’ understanding of the text. Typically, the teacher may erase keywords or cohesive devices to test students’ overall understanding of the text. The test developer may leave at least five or six words between the gaps to make sure these deleted words are restorable. This method of constructing the cloze test is termed gap-filling procedure. There are many methods to test reading comprehension in different aspects, but these methods share one goal: to objectively and effectively measure participants’ reading. 32.

(45) comprehension ability. The purpose of present study is to validate a suitable English reading comprehension test for EFL senior high students in Taiwan. In order to build an acceptable validity and reliability, NAEP 2013 reading framework was used to develop reading comprehension items. These items include multiple-choice items, short constructed-response items and extended constructed-response items. However, in order to achieve another research purpose, only multiple-choice items were used and the rationale of ordered multiplechoice items was applied for modifiying reading comprehension items.. 33.

(46) Ordered Multiple-Choice Items As the researcher mentioned above, a multiple-choice item consists of question stems and distractors. Popham (2000) and Nitko (2004) pointed out that these distractors are designed to “draw uninformed or unskilled students away from the correct answer (p.3, as cited in King et al.).” However, these two researchers noticed that the incorrect responses that students choose may provide diagnostic insight into student’s misunderstanding. Therefore, a new purpose of distractors has been suggested: “identifying the nature of a student’s misunderstanding (King et al., 2004).” An ordered multiple-choice (OMC) item is a traditional multiple-choice (TMC) item written with distractors designed to measure different or hierarchical levels of understanding. In order to help teachers and practitioners to develop diagnostic reading comprehension assessments, Pearson Inc. (2004) developed distractor rationale taxonomy (DRT) as guidelines in writing reading comprehension items. Items written in this rationale “reveal a student’s breakdown in understanding through their incorrect answers (p.2).” Specifically, each of the distractors in an OMC item includes a rationale that reflects four levels of understanding. Table 3 shows the levels of understanding and associated student errors for reading comprehension. Level one includes the most basic errors, that is, reader retrieved the information that is not necessarily related to the text or item; level two and three involves the increasing sophistication in student’s response although the answer is not good enough; and. 34.

(47) level four represents the correct answer. Table 3 Levels of understanding and associated student errors for reading comprehension Level of Understanding. Level 1 (Minimal). Level 2 (Moderate). Level 3 (Significant). Level 4 (Complete). Student Error Makes errors that reflect focus on decoding and retrieving facts or details that are not necessarily related to the text or item. Student invokes prior knowledge related to the general topic of the passage, but response is not text-based. These errors indicate that the student is grabbing bits and pieces of the text as he or she understands them, but the pieces are unrelated to the information required by the question being asked. Makes errors that reflect initial understanding of facts or details in the text, but inability to relate them to each other or apply them to come to even a weak conclusion or inference. The student may be focusing on literal aspects of a text or on superficial connections to arrive at a response. Makes errors that reflect analysis and interpretation, but conclusions or inferences arrived at are secondary or weaker than ones required for correct response. A distractor may be related to the correct response in meaning, but be too narrow or broad given the circumstances. Correct response.. Note. Adapted from “The Distractor Rationale Taxonomy: Enhancing Multiple-Choice Items in Reading and Mathematics,” by K. V. King, D. A. Gardner, S. Zucker, and M. A. Jorgensen, 2004, p. 6. Copyright 2004 by Pearson Education Inc.. The difference between TMC and OMC is that OMC items “provide information about the developmental understanding of students that is not typically available with TMC items (Briggs, Alonzo, Schwab & Wilson, 2006, p.34). Two example items are demonstrated below to show the comparison of an OMC item and a TMC item. The distractors in a TMC do not provide enough information to know a student’s level of understanding. Although the. 35.

(48) instructor may know whether the student has made a correct response, the instructor still does not know why the student selected an incorrect answer. Instead of knowing whether a student understands a concept, the OMC items “delve into the cognitive process of the student to identify a possible misconception or misunderstanding (King et al.,2004, p.8).” Table 4 Comparison of an Ordered Multiple-Choice item and a Traditional Multiple-Choice item (Example one: main idea question) Q: What is the main idea of the passage “Frogs and Toads”? (A) Frogs and toads are cute. Level 1 (Minimal): prior knowledge, not text-based (B) Toads have shorter legs than frogs have. Level 2 (Moderate): text-based detail unrelated to main idea (C) Frogs are different than toads. Level 3 (Significant): only part of main idea (D) Frogs and toads share many differences and similarities. Level 4 (Complete): correct response Q: What is the main idea of the passage “Frogs and Toads”? (A) Frogs live closer to water than toads.. All distractors are essentially Level 3: Each is related to the main idea but is not the best answer.. (B) Frogs and toads are like cousins. (C) Frogs are different than toads.. (D) Frogs and toads share many differences and similarities. Note. Adapted from “The Distractor Rationale Taxonomy: Enhancing Multiple-Choice Items in Reading and Mathematics,” by K. V. King, D. A. Gardner, S. Zucker, and M. A. Jorgensen, 2004, p. 7. Copyright 2004 by Pearson Education Inc.. . 36.

(49) Table 5 Comparison of an Ordered Multiple-Choice item and a Traditional Multiple-Choice item (Example two: vocabulary-in-context question) Read this sentence from the passage “Frogs and Toads.” Both frogs and toads have a tail at first that disappears when they get older. Q: Which word has the same meaning as disappears as it is used in this sentence? (A) disagrees Level 1 (Minimal): look-alike word (B) .vanishes Level 4 (Complete): correct response (C) can be seen Level 2 (Moderate): antonym (D) becomes small Level 3 (Significant): related to the meaning, but not precise Q: What is the main idea of the passage “Frogs and Toads”? (A) turns green. All distractors are related to frogs, not to the meaning of the word. (B) vanishes (C) jumps. “disappears.”. (D) breathes. Note. Adapted from “The Distractor Rationale Taxonomy: Enhancing Multiple-Choice Items in Reading and Mathematics,” by K. V. King, D. A. Gardner, S. Zucker, and M. A. Jorgensen, 2004, p. 8-9. Copyright 2004 by Pearson Education Inc.. 37.

(50) International Reading Comprehension Assessments National Assessment of Educational Progress (NAEP) The NAEP has been an ongoing national indicator of what American students know and can do in major academic subjects since 1969. The NAEP reading assessments have been administered on a regular schedule to students in grades 4, 8, and 12. The latest reading framework for the 2009-2013 NAEP was the second reading framework approved by National Assessment Governing Board (NAGB) to replace the reading framework for the 1992-2007 NAEP. The major difference between new framework and previous framework is that the new one was designed based on current scientific research in reading, including some of the nation’s leading figures in reading research, assessment, and instruction. In the following section, the reading framework for the 2013 NAEP, text types, item types, and results report will be introduced. Reading Framework for the 2013 NAEP NAEP used three cognitive targets: locate and recall, integrate and interpret, and critique and evaluate to illustrate that reading is an active and complex process that involves multiple different behaviors, and pointed out that different text type may cause different reading behaviors. As shown in Table 6, it is noted that text types will cause different reading skills during the reading processes.. 38.

(51) Table 6 Cognitive targets for the NAEP reading assessment. Specific to Literary Text. Both Literary and Informational Text. Text type. Cognitive targets Locate/Recall. Integrate/Interpret. Critique/Evaluate. Identify textually explicit information and make simple inferences within and across texts, such as:  Definitions  Facts  Supporting details. Make complex inferences within and across texts to:  Describe problem and solution or cause and effect  Compare or connect ideas, problems, or situations  Determine unstated assumptions in an argument  Describe how an author uses literary devices and text features. Consider text(s) critically to:  Judge author’s craft and technique  Evaluate the author’s perspective or point of view within or across texts  Take different perspectives in relation to a text. Identify textually explicit information within and across texts, such as:  Character traits  Sequence of events or actions  Setting  Identify figurative language. Make complex inferences within and across texts to:  Infer mood or tone  Integrate ideas to determine theme  Identify or interpret a character’s motivations and decisions  Examine relations between theme and setting or characters  Explain how rhythm, rhyme, or form in poetry contribute to meaning. Consider text(s) critically to:  Evaluate the role of literary devices in conveying meaning  Determine the degree to which literary devices enhance a literary work  Evaluate a character’s motivations and decisions  Analyze the point of view used by the author. 39.

(52) Specific to Informational Text. Identify textually explicit information within and across texts, such as:  Topic sentence or main idea  Author’s purpose  Causal relations  Locate specific information in text or graphics. Make complex inferences within and across texts to:  Summarize major ideas  Draw conclusions and provide supporting information  Find evidence in support of an argument  Distinguish facts from opinions  Determine the importance of information within and across texts. Consider text(s) critically to:  Analyze the presentation of information  Evaluate the way the author selects language to influence readers  Evaluate the strength and quality of evidence used by the author to support his or her position  Determine the quality of counterarguments within and across texts  Judge the coherence, logic, or credibility of an argument. Note. Adapted from “Reading Framework for the 2013 National Assessment of Educational Progress,” by NAGB, 2012, p. 41. Copyright 2012 by the American Institutes for Research.. Text types Passages selected for inclusion on the NAEP reading Assessment are authentic: students would encounter in their in-school and out-of school reading; unbiased: potential bias and sensitive issues will be avoided; and interesting to read: materials collected from a variety of sources familiar to students nationwide, and at least two research-based readability formulas will be used to gather additional information about passage difficulty. The NAEP reading framework includes two major types of text: literary and informational. Literary texts include fiction, literary nonfiction, such as essays, speeches, and autobiographies or biographies, and poetry. Informational texts include exposition, 40.

(53) argumentation and persuasive text, and procedural text and documents. Exposition texts, such as textbooks, news stories, and informational trade books, present information, provide explanations and definitions, and compares and contrasts. Argumentation and persuasive texts, such as political speeches, editorials, and advertisements, tend to influence through appeals that direct readers to specific goals or try to win them to specific beliefs. Procedural texts and documents convey information in the form of directions for accomplishing a task such as manuals, directions for activities and so on. Table 7 shows the passage lengths and distribution of literary and informational passages on the NAEP reading assessment. The change across the grade levels reflects changes in the kinds of reading that students do as they progress through school. Table 7 Passage lengths and percentage distribution of literary and informational passages Grade. Literary. Informational. Passage lengths (number of words). 4 8 12. 50% 45% 30%. 50% 55% 70%. 200-800 400-1,000 500-1,500. Note. Adapted from “Reading Framework for the 2013 National Assessment of Educational Progress,” by NAGB, 2012, p. 11 & 29. Copyright 2012 by the American Institutes for Research.. Item types The NAEP Reading Assessment contains multiple-choice and constructed-response items. Multiple-choice items include a key and three distractors. It is assumed that one multiple-choice item will take students approximately one minute to complete. Short constructed-response items can be answered by one or two phrases or sentences. This kind of. 41.

(54) items should take students nearly two to three minutes to answer. Extended constructedresponse items should elicit longer, more elaborated answers of a paragraph or two. They may take students about five minutes to finish answering. Scoring rubrics for constructedresponse items will focus on the content included in answers, not on spelling or grammatical problems. However, students need to answer constructed-response questions by using information from the text to receive credit. The distribution of multiple-choice and constructed-response items will vary across the grades. Students at grades eight and twelve will spend more time on constructed-response items while students at grade four will need 50 percent of test time answering multiple-choice items and the other half for constructed-response questions. Table 8 shows the percentage distribution of time to be spent on specific item types. Table 8 Percentage distribution of time to be spent on specific item types Constructed-response item Short Extended. Grade. Multiple-choice item. 4. 50%. 40%. 10%. 8 12. 40% 40%. 45% 45%. 15% 15%. Note. Adapted from “Reading Framework for the 2013 National Assessment of Educational Progress,” by NAGB, 2012, p. 43. Copyright 2012 by the American Institutes for Research.. The distribution of cognitive targets will also vary across the grades. As shown in Table 9, items at locate and recall level will decrease (30%→20%) while items at critique and. 42.