英語閱讀能力測驗之內涵與閱讀能力之評析：以大學學科能力測驗與指定科目考試為例

全文

(1)國立台灣師範大學英語學系碩士. 論文. Master’s Thesis Graduate Institute of English National Taiwan Normal University. 英語閱讀測驗之內涵與閱讀能力之評析: 以大學學科能力測驗與指定科目考試為例 An Analysis of the Reading Skills Measured in Reading Comprehension Tests on the Scholastic Achievement English Test (SAET) and the Department Required English Test (DRET). 指導教授：陳秋蘭博士 Advisor: Dr. Chiou-lan Chern 研究生：陳秀娟撰 Hsiu-chuan Chen. 中華民國九十八年七月 July 2009.

(2) 摘要本研究使用語言學家 Nuttall 的閱讀技巧分類之修正版來探討近六年（2002-2007 年）大學學科能力測驗及指定科目考試英文閱讀測驗試題中欲測試的閱讀技巧為何，以及歷屆考生在各類題型上的表現。本研究採質化與量化分析法。質的分析採內容分析法，將一百六十七個考題依照 Nuttall 的閱讀技巧分類之修正版分類。量化分析則使用電腦統計軟體 SPSS 13.0 來檢測各種閱讀技巧類型在考試中出現的頻率及分布，雙因子變異數分析用來個別檢測二種考試中各類題型之答對率間有無顯著差距及其差距是否每年相同，同時並了解高分組與低分組學生於各類題型上表現的差距。本研究之主要研究發現如下：（一）研究結果顯示，在二種考試中，試題欲測量的閱讀技巧可以分為八類（「由上下文推測字彙意義」、「辨認連接詞」、「辨認細節」、「辨認功能價值」、「辨認文章組織結構」、「辨認作者的預設立場」、「推論」以及「辨認主旨」。（二）「辨認細節」這類考「由下往上」(bottom-up)的技巧的題目類型在二種考試中最常被考，因此可推論二種考試都比較偏好考「由下往上」(bottom-up) 類型的技巧，而出現次數最少的題型則為「辨認文章組織結構」。此外，大學學科能力測驗及指定科目考試的英文閱讀測驗的最大差異在於閱讀技巧類型出現的頻率、出現處以及分佈。在大學學科能力測驗中每年都會出現的技巧是「由上下文推測字彙意義」以及「辨認細節」此二種題型，而在指定科目考試的英文閱讀測驗每年都出現的則只有「辨認細節」此類題型。（三）二因子變異的分析研究顯示在大學學科能力測驗以及指定科目考試中，不同題型對於答對率並沒有顯著影響，亦即不同題型答對率的高低在六年當中並不一致。（四）在二種考試當中，低層次的閱讀技巧(local skill)最能區隔高低分組的學生。在大學學科能力測驗當中，所有的題型在高低分組的答對率差距方面都有達到最低標準，然而在指定科目考試方面有二類題型的鑑別度低於最低標準，像. i.

(3) 是「推論」以及「辨認作者的預設立場」。此結果暗示這二類題型可能對所有的考生來說太難，以致於無法適當區隔高低分組的表現。根據上述分析結果，本研究最後提出一些教學建議以供參考。. ii.

(4) Abstract The present study aimed to adopt a revised version of Nuttall’s taxonomy to investigate the reading skills measured in the SAET (Scholastic Achievement English Test) and the DRET (Department Required English Test) administered from 2002 to 2007, and to explore how test takers (all examinees, high achievers, and low achievers) performed on different types of items. Both qualitative and quantitative analyses were adopted. The qualitative analysis was conducted by categorizing each of the 167 reading comprehension items into reading skill type in the revised Nuttall’s Taxonomy. For the quantitative analysis, SPSS 13.0 statistical package was used to examine the frequency distribution of the item types. The two-way ANOVA test was applied to the SAET and the DRET to see whether there were significant differences among the passing rates of different question types and to investigate whether these differences were consistent throughout the years. Also, the discrimination indexes were analyzed via the two-way ANOVA to see how the high achievers and low achievers differed while answering different types of reading questions each year. The results of this study are summarized as follows: First, the findings showed that in both tests, eight types of reading skills were measured: “Word Inference from Context,” “Recognizing Cohesive Devices,” “Recognizing. and. Interpreting. Details,”. “Recognizing. Functional. Value,”. “Recognizing Text Organization,” “Recognizing Presuppositions Underlying the Text,” “Recognizing Implications and Making Inferences,” and “Recognizing and Understanding the Main Idea.” Second, the most frequent items tested in the SAET and DRET are items on “Recognizing and Interpreting Details,” which indicated that this type of reading skill is favored in both tests. However, “Recognizing Text Organization” is the least tested skill. In addition, the similarities and differences between the SAET and DRET lay in the frequency, occurrences, and distribution of reading skill item types. Two types of items occurred every year in the SAET, including local items on “Word Inference from Context” and “Recognizing and Interpreting Details.” However, only “Recognizing and Interpreting Details” occurred every year in the DRET. Third, the ANOVA analysis showed that there was no significant effect of items. iii.

(5) on the examinees’ average passing rates. In other words, the ranking of passing rates of different item types in the SAET and DRET were not consistent throughout the years. Finally, in both SAET and DRET, items on local skills best discriminated high and low achievers. In the SAET, the discrimination indexes of all item types reached the ideal discrimination index whereas in the DRET two types of items had unsatisfactory discriminatory power: items on “Recognizing Implications and Making Inferences” and “Recognizing Presuppositions Underlying the Text.” This indicated that these two types of items were probably too difficult for most examinees and did not appropriately distinguish the high and low achievers. Based on the aforementioned analysis of results, some pedagogical implications for reading instruction and testing in senior high schools were provided.. iv.

(6) Acknowledgements I owe my deepest gratitude to my advisor Dr. Chou-lan Chern, chair of Department of English of National Taiwan Normal University, whose inspiring guidance, encouragement, patience and support helped me in all the time of research and writing of this thesis. I would also like to extend my heart-felt thanks to Dr. Hsi-chin Chu and Dr. Chin-kuei Cheng for serving on my thesis committee and for their valuable comments, suggestions, and advice. I would like to thank all the faculty members in the English Department for making my Master’s program a good experience. Also, I am very grateful to my family back home for always standing by my side. I am thankful to all my friends and my boyfriend Tsung and his family who always have supported and encouraged me. Finally, I would like to extend my warmest thanks to two wonderful teachers, Elaine and Julie, for being my raters and for their support and encouragement.. v.

(7) TABLE OF CONTENTS Page CHINESE ABSTRACT..............................................................................................i ENGLISH ABSTRACT.............................................................................................iii ACKNOWLEDGEMENTS.......................................................................................v TABLE OF CONTENTS...........................................................................................vi LIST OF TABLES.....................................................................................................vii. CHAPTER ONE INTRODUCTION.........................................................................1 Background and Motivation.......................................................................................1 Research Questions ....................................................................................................5 Significance of the Present Study...............................................................................6 Organization of the Thesis..........................................................................................6. CHAPTER TWO LITERATURE REVIEW ..............................................................9 The Rationale of Reading ...........................................................................................9 Reading Process Models .....................................................................................9 Reading Comprehension and Reading Skills.............................................................14 Theoretical Reading Skill Frameworks..............................................................15 The Original Nuttall’s Taxonomy.......................................................................17 Reading Skill Instruction............................................................................................24 Testing ESL/EFL Reading Comprehension................................................................26 Why Testing (The Relationship of Teaching and Testing)..................................27 Testing Comprehension with Multiple-choice Questions...................................28 Studies of Reading Tests Analysis (Item Analysis).............................................30. vi.

(8) Summary .....................................................................................................................38. CHAPTER THREE METHOD...................................................................................41 Materials......................................................................................................................41 Participants..................................................................................................................42 Instrument....................................................................................................................43 Data Analysis...............................................................................................................46 Formal Analysis...........................................................................................................46 Analysis of the Data Coded and the Passing Rates....................................................48 Summary.....................................................................................................................49. CHAPTER FOUR DATA ANALYSIS AND RESULTS.............................................51 Qualitative and Quantitative Data Analysis and Results.............................................51 The Reading Skills Measured in SAET & DRET...............................................51 Similarities and Differences between the SAET & the DRET...........................53 Similarities and Differences between Reading Skills Measured in the SAET and the DRET...........................................................................................55 Examinees’ Performance on Each Question Type .....................................................59 SAET...................................................................................................................59 DRET...................................................................................................................63 Comparisons of High and Low Achievers’ Performances on Different Item Types...........................................................................................66 Results of the SAET............................................................................................67 Results of the SAET............................................................................................74 Summary .....................................................................................................................80. vii.

(9) CHAPTER FIVE DISCUSSION AND CONCLUSION.............................................83 Discussion of Major Findings......................................................................................83 Reading Skills Measured in the SAET & DRET.................................................84 Similarities and Differences between the Reading Skills Tested in the SAET and the DRET...........................................................................................84 Examinees’ Performances on Various Item Types...............................................86 Comparisons of High and Low Achievers’ Performances on Different Item Types............................................................................................................88 Conclusions..................................................................................................................90 Summary of the Major findings...........................................................................90 Pedagogical Implications.....................................................................................91 Limitation Suggestions for Future Research .......................................................93. REFERENCES............................................................................................................95 APPENDIX 1: Principles for Classification of the Reading Skills Based on the Revised Nuttall’s Taxonomy.........................................................102 APPENDIX 2: Results of the Formal Item Analysis..................................................107. viii.

(10) LIST OF TABLES Table 1 Structure of Nuttall’ Original Taxonomy. 22. Table 2 The Number of Reading Passages and Comprehension Test Items. 42. Table 3 The Number of Examinees Taking the SAET from 2002 to 2007. 42. Table 4 The Number of Examinees Taking the DRET from 2002 to 2007. 43. Table 5 Structure of the Revised Nuttall’s Taxonomy. 45. Table 6 Reading Level Tested in the SAET & DRET. 51. Table 7 The Frequencies of Reading Skills Measured in the SAET & the DRET 52 Table 8 Reading Level Tested in the SAET & DRET from 2002 to 2007. 53. Table 9 Reading Skills Measured in both SAET & DR ET. 54. Table 10 Reading Skills Measured in 2002-2007 SAET & DRET. 58. Table 11 Average Passing Rates of Items Measuring Different Reading Skills in 2002-2007 SAET and DRET. 60. Table 12 The ANOVA Analysis of the Passing Rates in the SAET. 63. Table 13 The ANOVA Analysis of the Passing Rates in the DRET. 66. Table 14 Passing Rates of High & Low Achievers and the Discrimination Indexes on Different Reading Levels in the SAET & DRET. 68. Table 15 Passing Rates of the High & Low Achievers and the Discrimination Indexes on Different Item Types in 2002-2007 SAET Table 16 The ANOVA Analysis of Discrimination Indexes in the SAET. 69 74. Table 17 Passing Rates of the High & Low Achievers and the Discrimination Indexes on Different Item Types in 2002-2007 DRET Table 18 The ANOVA Analysis of Discrimination Indexes in the DRET. ix. 76 79.

(11) CHAPTER ONE INTRODUCTION Background and Motivation English reading is an essential skill for those who use English as a second or foreign language. It is probably by far the most important of the four skills (i.e., listening, speaking, reading and writing) for English as a second language (ESL) & English as a foreign language (EFL) learners (Carrell, 1988a). For senior high school students in the EFL context of Taiwan, English reading ability is of paramount importance since English is taught as an academic subject in the school setting. In addition, a good command of English reading is important in passing nationwide English tests such as the Scholastic Achievement Test (SAT)1 and the Department Required Test (DRT)2. The College Entrance Examination Center (hence CEEC) implements a two-stage nationwide examination every year to determine who goes to which university. Therefore, both tests have a huge impact on teaching and learning for senior high school teachers and students in Taiwan. The SAT is an achievement test aiming to assess senior high school students’ basic scholastic knowledge and abilities. It is designed to choose potential candidates for universities at an early stage. The DRT, on the other hand, serves as a proficiency test to identify competent candidates who possess certain abilities to do well in certain subject areas required by particular universities. Usually, students need the results of tests on three to six different subjects to apply for schools of their choice. The goals of 1. The SAT was firstly administered in 1994. The DRT was originally called the Joint College Entrance Examination (JCEE) before 2002. It was first implemented in 2002 when the new college admission policy was introduced. 2. 1.

(12) the SAT are: (1) to see whether the candidates have the abilities required of senior high school students in fundamental subjects, (2) to see whether candidates have acquired abilities for college education, (3) to see whether candidates have abilities to combine knowledge they learn at school with real-life experience or to integrate it with knowledge in other fields, and (4) to see whether candidates have the abilities to comprehend and apply knowledge. On the other hand the goals of the DRT are (1) to see whether candidates understand the important knowledge in the discipline, (2) to see whether candidates have the abilities to read, to judge, to infer, and to analyze materials, (3) to see whether candidates have abilities to express their ideas, and (4) to see whether candidates have abilities to apply the knowledge in the disciplines. Comparing the purposes and goals of the SAT and DRT, the major difference between these two tests is that the SAT focuses more on measuring students’ basic abilities whereas the DRT emphasizes more on measuring the students’ higher-order cognitive abilities—to judge, to infer and to analyze materials. Thus, the DRET (i.e., Department Required English Test), with its aim to discriminate more proficient students in the English subject, is relatively more difficult than the SAET (i.e., Scholastic Achievement English Test). Both Scholastic Achievement English Test and Department Required English Test play vital roles in university admission and both tests exert considerable influence on English teaching and learning. As a result, what is measured in the reading comprehension section in both tests has drawn interest among teachers and students (Lu, 2002). On both tests, the reading comprehension section, which usually occupies the largest proportion of scores in the whole English test, is a major component used to assess reading ability and the reading comprehension test items are assumed to tap different reading skills. Since both tests exert a great influence on. 2.

(13) teaching and testing, both tests are analyzed annually to help teachers and students better understand the tests. Researchers have been interested in what should be and is tested in reading comprehension tests and have noted the importance of assessing reading skills. According to Cohen (1994), in addition to the types of meaning assessed by reading comprehension questions, test writers and users should be aware of the skills being tested. Alderson (in Nuttall, 1996, p. 219) points out that it is important to define the reading skills to be tested and to write questions which involve such skills. As Yu (2006) pointed out, senior high school teachers in Taiwan tend to see vocabulary teaching and structure analysis as the focus of reading instruction. In order to turn this around, the best way is probably to raise teachers’ awareness of the concept and application of reading skills tested in the SAET or the DRET. The CEEC makes an annual report on the item analysis of both SAET and DRET. The statistical analyses by the CEEC showed that most of the items had good statistical values, but test items with good statistical values might not necessary be good items (Huang, 1994). Also, the analysis has been focused mainly on the statistical values of items and yet in-depth qualitative analysis of the items which measure reading comprehension skills in SAET and DRET is rare. Most of the previous studies on the SAET and DRET aimed to provide an overview of the test construction or statistical results of item analysis rather than a thorough qualitative analysis on the reading skills tested on each comprehension test item in particular (e.g. CEEC Web site; Huang, 1994; Jiang & Lin, 1999; Jeng, 1992; Xu & Lu, 1998). In recent years, three researchers in Taiwan have more thorough research on the reading comprehension questions—one analyzed reading comprehension test items in the SAET (Lu, 2002), the other focused on the reading comprehension test items in the DRET (Hsu, 2004), and still another analyzed reading comprehension items in both SAET and DRET (Lan, 2007). However, the first two studies have been limited to the 3.

(14) use of the same taxonomy which is Mo’s taxonomy (Mo, 1987). Mo’s scheme (Mo, 1987), derived by summarizing six types of skills from both Carver’s (1878) and Raygor & Raygor’s (1985) taxonomies, appeared to be very general and failed to provide clear and detailed descriptions and examples of each skill. What’s more, these two studies exhibited some problems with the categorization of test items. It was found that some of the categorizations of test items in both studies were in appropriate because Mo’s taxonomy is too general, making the statistical analyses and the studies questionable. The third study by Lan (2007), adopted a revised Bloom’s taxonomy to analyze the cognitive levels and knowledge types measured in both SAET and DRET. The taxonomy has proved to be useful in examining the cognitive levels and knowledge types of reading comprehension items; however, it is not specifically designed for ESL/EFL context. Both SAET and DRET are extremely important, yet previous studies concerning reading skills measured in SAET and DRET were far from conclusive. Only two taxonomies were used to examine the reading skills measured in these two tests and problems have been found with the two studies which analyzed the SAET and DRET by adopting Mo’s taxonomy. Nuttall’s taxonomy is similar to Mo’s taxonomy, but it does not have the problem exhibited in Mo’s taxonomy. In Nuttall’s taxonomy, the ability to recognize cohesive devices and the ability to recognize text organization are treated as separate skills. Therefore, Nuttall’s taxonomy would be a better taxonomy in identifying the skill to recognize cohesive devices and the skill to recognize text organization. Moreover, Nuttall divided reading skills into bottom-up (lower-order) and top-down higher order) skills. By adopting Nuttall’s taxonomy, the researcher is able to find out whether global or local reading skills are more tested in the SAET and the DRET. Also, Nuttall’s taxonomy contains skills that were measured either on the SAET or the DRET but items which measure these skills could not be identified by Mo’s 4.

(15) taxonomy. For example, one skill on Nuttall’s taxonomy—Recognizing Functional Value—had been measured on the DRET (See Appendix 1 for the principle and example question of this skill); however, by using Mo’s taxonomy, items which test this skill were classified as items on inference. Thus, we can assume that Nuttall’s taxonomy could reflect the skills tested but were not identified according to Mo’s taxonomy. Nevertheless, a few limitations were found with Nuttall’s taxonomy and therefore a revised version of Nuttall’s taxonomy was formed for the present study. The original Nuttall’s taxonomy is presented in Chapter Two and the revised taxonomy is presented in Chapter Three. Though it is still undecided whether the reading ability can be measured by reading comprehension tests, so far no definite answer can be given to prove otherwise. Thus, the items in the reading tests are still worthy of analysis. The present study, viewing reading as a cognitive activity which can be broken down into component skills, attempts to apply a revised Nuttall’s taxonomy to analyze the reading comprehension test items in both SAET and DRET from 2002 to 2007. The taxonomy adopted for the present study is specified in the chapter which describes the method.. Research Questions The present study attempts to answer the following questions: 1. What are the reading skills measured in the SAET and DRET in the reading comprehension section from 2002~2007? 2. What are the similarities and differences between the SAET and DRET with regard to the reading skills measured? 3. How did high school students perform on test items that measure different skills? 4. How did high achievers and low achievers differ on test items that measure 5.

(16) different skills?. Significance of the Present Study Over the past decades, several studies have been conducted with regard to the reading comprehension item analysis in SAET and DRET. Most of them focused on the statistical values of distractors, topics of passages, and so on. In view of the importance of the role reading skills play in reading instruction and assessment, the purpose of the present study is to examine the reading skills measured in the SAET and DRET so as to provide teachers and students with a better understanding of these two tests and to provide implications for reading instruction in senior high schools. This study, different from previous studies, attempts to investigate the reading skills measured in the reading comprehension questions in the SAET and DRET from 2002 to 2007 by adopting a taxonomy based on Nuttall’s rubrics. It seeks to examine the reading skills measured in both tests, to see what reading skills were tested and to see if there’s any difference between the two tests regarding the type of reading skills measured. The study is significant in that it provides both teachers and students in Taiwan with a better understanding of reading skills measured in SAET and DRET. It is hoped that the findings in the present study will help teachers gain some insights of the reading skills measured in both tests and help students develop reading skills appropriate for the exams.. Organization of the Thesis The remainder of this study is structured as follows. Chapter Two deals with the literature review regarding the rationale of reading (e.g. reading process models), existing frameworks of reading skills, the original Nuttall’s taxonomy, studies on reading skill instruction, and studies of reading comprehension tests. Chapter Three. 6.

(17) describes the method and procedures for the collection of data on reading skill analyses. The fourth chapter presented the results, with a thorough description of the data analyses. Finally, results are discussed, conclusions are drawn with pedagogical implications, followed by limitations of the current study and suggestions for future study.. 7.

(18) CHAPTER TWO LITERATURE REVIEW This study aimed to investigate the reading skills measured in the reading comprehension section in SAET and DRET. Thus, the theoretical foundations for the investigations of reading skills in tests are reviewed. First of all, section one explores the rationale of reading—the reading process models. Then, section two accounts for the relationship between reading comprehension and reading skills and existing reading taxonomies. Next, the third section presents Nuttall’s original taxonomy. Furthermore, the fourth section reviews studies on reading skill instruction. Finally, the last section discusses issues of assessing reading in an ESL/EFL context, including the advantages and disadvantages of the multiple-choice question technique and a review of recent studies on SAET and DRET.. The Rationale of Reading Considerable research has been done to gain insights into the nature of reading and various theories exist to explain what is involved when we read. Generally speaking, reading is a cognitive activity since “it largely takes place in the mind” (Urquhart & Weir, 1998). To comprehend and interpret a text appropriately, a reader must be actively engaged in a mental process with the text. In order to look for ways to describe the interaction between the reader and the text, researchers have created models that describe what happens when people read. It is commonplace to classify reading process models into three types: the bottom-up model, the top-down model, and the interactive model. A general review of literature of these three models is provided as follows. Reading Process Models The Bottom-up Model. The bottom-up model, essentially data-driven, was 9.

(19) initially proposed by Gough (1972). This model involves the reader paying close attention to every detail of the text. In bottom-up processing, the meaning is arrived at, from bottom to top, where the reader begins with the printed text by decoding letters, words, phrases, and sentences and then builds up comprehension in a somewhat linear way from the incoming language data (Omaggio, 2001). The bottom-up model typically consists of lower-level processes in which the reader focuses on the processing of smaller linguistic units (e.g., letter, words ) and then works towards larger units such as sentences, arriving at meaning from parts to whole. Therefore, in this view, readers are seen as passive identifiers of sequential graphic, phonemic, syntactic, and semantic systems of language (Alderson, 2000). According to Eskey (1973) (cited in Carrell, 1988a, p.3), this model was inadequate because it failed to recognize the readers’ contribution during the reading process. It failed to recognize that the readers brought prior knowledge and expectations to bear on the task of understanding the written language. The Top-down Model. In place of the bottom-up approach, Goodman (1967) offered a different kind of model which has been characterized as a top-down model of the reading process. Goodman (1967) claimed that reading is a “psycholinguistic guessing game” in which readers sample from the text3 and then confirm and correct their hypotheses about the predicted meaning of the text based on the message conveyed in the text. In other words, readers use guessing as a basis for comprehension and reconstruct meaning by predicting and confirming. By top-down processing, readers create meaning by relating to what the text says to what they already know. Smith (1971) and Goodman (1967) are major proponents of this model.. 3. Sampling implies that the reader does not read every feature of every word, but rather focuses on. certain features, certain words or certain phrases to get a sense of the meaning from the text.. 10.

(20) In this view, readers become active participants in the reading process in which the readers make predictions and process information by utilizing their prior experience or. background. knowledge. (Carrell,. 1988b).. Elaborating. on. Goodman’s. psycholinguistic model, Coady (1979) argued that reading comprehension required three components: background knowledge, higher-level conceptual abilities (e.g., intellectual capacity), and process strategies (e.g., grapheme-morpho-phoneme correspondences, syllable-morpheme information, syntactic information, lexical meaning, and contextual meaning. The top-down (reader-driven or conceptually-driven) model, which involves the reader sampling text information to verify hypothesis and predictions, emphasizes the importance of the readers’ schemata (Alderson, 2000). Carrell & Eisterhold (1983) claimed that there are two types of schemata—formal schemata, i.e., knowledge of language and linguistic conventions (e.g., knowledge of rhetorical structures, how a text is organized), and content schemata, which refers to knowledge of the world or knowledge about the content area of a text. The schemata the reader brings to the reading process is said to guide the reader’s processing of text. According to schema theory, comprehending a text is an interactive process between the reader’s background knowledge and the text, requiring the reader to activate relevant schemata to process and interpret new information quickly and efficiently (Richards & Schmidt, 2002). The Interactive Model. By bottom-up processing only, readers rely too much on the lower-level skills, and by top-down processing only, readers rely too much on their prior knowledge. Thus, neither of these two models is an adequate characterization of the reading process (Alderson, 2001). More recent research on teaching reading has shown that the interactive model, a combination of top-down and bottom-up processing, is nearly always a prerequisite for successful reading 11.

(21) instruction (Brown, 2001). “Good reading — that is, fluent and accurate reading — can result only from a constant interaction between these processes (bottom-up & top-down)” (Eskey, 1988, p.95). The interactive model is a process model involving the combination and integration of bottom-up processing and top-down processing skills alternatively or at the same time (Carrell, 1988; Grabe, 1991; Nassaji, 2002). According to Rumelhart (1977, 2004), the interactive process model stresses that reading comprehension is reached when various sources of knowledge interact with sensory input, including orthographic knowledge, lexical knowledge, syntactic knowledge, semantic knowledge and pragmatic knowledge. Stanovich (1980), on the other hand, had proposed an interactive compensatory model. Stanovich’s model is interactive in that readers employ both bottom-up and top-down processing skills whereas it is compensatory in the sense that readers process texts by utilizing information from various resources of knowledge and a deficit in one knowledge source will result in a heavier reliance on other knowledge sources. In other words, readers with poor linguistic knowledge may use their top-down skills (e.g., background knowledge) to facilitate reading. Similarly, in Eskey’s model (1988, p. 96), “interactive refers to the interaction between information obtained by means of bottom-up decoding and information provided by means of top-down analysis both of which depend on certain kinds of prior knowledge and certain kinds of information processing skills.” The interactive model assumes that the bottom-up and the top-down processes are equally important. Readers with different reading styles may prefer one mode of processing over the other. Kirby (1988) classified reading (cognitive) styles into three types—analytic, global, and synthetic. Analytic readers adopt the bottom-up process approach to reading in which the lower-level decoding skills are utilized (e.g., decoding the words). On the other hand, global readers adopt the top-down process approach to 12.

(22) reading in which higher-level cognitive skill is of importance (e.g., employment of background knowledge). However, Kirby (1988) noted that synthetic readers are more successful readers than analytic and global readers. Synthetic readers, according to Kirby, adopt a synthesized version of the bottom-up and top-down approach, that is, an interactive approach. Readers with preference to one approach over the other may not be effective readers. Effective reading requires an adoption of both top-down and bottom-up processing, i.e., the interactive processing. Carrell and Eisterhold (1983) maintained that bottom-up processing ensured that the readers would be sensitive to information that was new or that did not fit their anticipation or prediction of the content or structure of the text, whereas top-down processing helps the readers to resolve ambiguities or to choose alternative possible interpretations of the text. Viewing reading comprehension as involving an interaction of text-based (bottom-up) processing and knowledge-based (top-down) processing, Carrell (1988b) also noted that two kinds of skill deficiencies, linguistic deficiencies and reading skill deficiencies, may lead to inefficient interaction between these two types of processing in ESL reading. Linguistic competence, which involves skills in decoding the vocabulary and syntactic structures of a text, is required for text-based processing (Carrell, 1988b) and successful reading (Eskey, 1988). In reviewing Spiro’s (1978) study, Carrell (1988b) accounted for how reading skill deficiencies affected interaction of the two types of processing in comprehension. Lacking text-based skills (e.g., decoding) or knowledge-based skills (e.g., pragmatic inferencing) may result in reading difficulties and the same skill deficiency may lead to overreliance on either text-based or knowledge-based processing. For example, when confronted with reading skill problems, readers may focus on one particular type of processing or they may escape from the problem by shifting to another type of processing to compensate for the processing. In other words, when readers have difficulties understanding texts 13.

(23) by using top-down processing skills, they may try to use bottom-up processing skills to facilitate understanding of texts. For instance, when readers fail to understand the message of a paragraph from context, they may try to break the paragraph into sentences and analyze each sentence in order to interpret the meaning of the paragraph. Hence, as noted by Carrell (1988b), efficient readers shift from one process to the other whereas readers with lower proficiency level tend to focus more on one particular model of processing. Since efficient and effective reading requires both bottom-up reading (lower-level processing) and top-down reading (higher-level processing) operating interactively, it is vital for teachers to teach students to adopt a synthesized style of reading (global & analytic) and to design reading tasks for them to practice and to become familiar with the interactive reading process.. Reading Comprehension and Reading Skills Two theories are common for describing reading comprehension: one views reading comprehension as a unitary skill which cannot be broken down into separate sub-skills (e.g., Alderson, 2000; Wallace, 1992) whereas the other views reading comprehension as an aggregate of sub-skills such as finding the main idea or interpreting inferences in a text (e.g., Carver, 1978; Grabe, 1986; Grellet, 1981; Mo, 1987; Nuttall, 1996; Raygor & Raygor, 1985). Though there is inconsistency of the evidence for distinct reading skills, it has not prevented material writers to include some of them in EFL reading materials (Williams, E. & Moran, 1989) and so far there has been no evidence to prove that reading subskills do no exist. Reading comprehension is such a complex cognitive process that researchers attempt to explain the process by analyzing it into a set of component skills. It has been assumed by researchers, teachers, and test writers that reading can be broken down into “underlying skills components” for the purposes of teaching and testing (Grabe, 1991). Researchers suggested that reading comprehension test items be 14.

(24) written to measure reading skills so that they call for different types of reading (e.g., Cohen, 1994; Mo, 1987). Many different lists, taxonomies and even hierarchies of skills have been drawn up by researchers, yet little consensus concerning the content and terminologies can be found (Williams & Moran, 1989). What’s more, there has been a fair amount of confusion regarding the use of the terms “skill” and “strategy” in literature. The distinction between these two terms is not clear and they often appear to be used interchangeably. “In EFL reading material, the tendency is for the term “skill” to be used to the exclusion of ‘strategy’ and even of ‘style’ (Williams, E. & Moran, 1989, p. 223).” Urquhart & Weir (1998) made a distinction between “skill” and “strategy.” They argued that strategies are “reader-oriented” whereas skills are “text-oriented.” Richards & Schmidt (2002) identified reading skills as “abilities required for skillful reading” and reading strategies as “ways of accessing the meaning of texts, which are employed flexibly and selectively in the course of reading.” In principle, skills are what the reader automatically and unconsciously contributes to the text (text-oriented) while strategies represent conscious decisions taken by the reader to respond to problems during the reading process (e.g., failure to understand a word or sentence) (Williams, & Moran, 1989; Urquhart & Weir, 1998). The present study sees reading as consisting of an aggregate of subskills rather than strategies which readers use to check, monitor, and evaluate their thinking and performance. Detailed discussion of what distinguishes a ‘skill’ from a ‘strategy’ is outside the scope of this study. Theoretical Reading Skill Frameworks Second/foreign language reading ability was viewed as the mastery of specific reading subskills and this view has continued to inform approaches to the teaching of reading today (Richards, 2002). The investigations of ESL/EFL reading skills have produced a wide variety of skill inventories and classification schemes. Although 15.

(25) there are differences among various classification schemes, most schemes contain one or more of the reading skills listed below, even when the term “strategy” is used rather than “skill” (e.g., Abbott, 2007; Alderson, 1991; Brown, 2003; Grabe, 1986; Grellet, 1981; Mo, 1987; Munby, 1978; Nuttall, 1996; Phakiti, 2003; Purpura, 1997; Raygor & Raygor, 1985; Richards, 2002). 1. 2.. Deducing the meaning and use of unfamiliar words Noticing specific details. 3. 4. 5. 6. 7.. Recognizing cohesive devices Recognizing discourse markers Identifying the main idea Identifying the writer’s point of view Recognizing the organization of text. 8. Drawing inferences and conclusion 9. Determining functional values 10. Making predictions As aforementioned, two researchers in Taiwan (Lu, 2002; Hsu, 2005) adopted Mo’s taxonomy to categorize the reading comprehension items on the SAET and DRET. Mo summarized six types of reading skills by reviewing the categories proposed by Carver (1878) and Raygor & Raygor (1985). Mo (1987) excluded skills unrelated to text structure such as skills related to reading speed, habit and pleasure, and identified six reading skills: (1) identifying the main idea, (2) finding specific details mentioned in the passage, (3) finding implications and drawing inferences and conclusions from the text, (4) recognizing style and tone, (5) determining the special techniques used by the author to achieve his effect, and (6) determining the meaning of strange words or phrases as used in the text. However, as briefly mentioned in Chapter One, Mo’s categorization is so general that it could possibly make it difficult for raters to categorize items appropriately. According to Mo, the fifth skill refers to the skill to understand the organization of a text, the skill to interpret the cohesive. 16.

(26) devices used by the writer and any other skills relevant to interpreting the main idea of an article. Thus, the fifth skill in fact involves various different skills. According to Mo (1978), items that measure the reader’s ability to recognize cohesive devices, items that measure the reader’s ability to recognize the organization of a passage, or items that measure any other techniques that the writer uses to develop the main idea of an article should all be categorized as one skill. This could possibly result in difficulty of item categorization and make the results of categorization unclear. Usually, in most taxonomies, recognizing cohesive devices, which deals with more local information in texts, and recognizing the organization of texts, which deals with more global information in texts, are treated as separate skills. Nevertheless, Mo combined these two skills and other skills into one skill, making the skill ill-defined and complicated. This could possibly be the reason why the researchers in two previous studies, Lu (2002) and Hsu (2005), categorized items that measured cohesive devices and text organization into other types of skills such as items on details or items on inference. Another weakness with Mo’s taxonomy was that items which measure test takers’ ability to recognize the functional values of sentences or paragraphs were categorized as items on inference since Mo’s taxonomy was not able to reflect this skill. Thus, being too general and ill-defined, Mo’s taxonomy seemed ineffective in classifying several types of reading comprehension test items on the SAET and DRET. The Original Nuttall’s Taxonomy Nuttall (1996), in a book designed to teach ESL/EFL reading, developed a reading skill taxonomy which included two major skill categories: word-attack skills and text-attack skills. In terms of work-attack skills, Nuttall (1996) identified three skills required for coping with difficult words while reading: (1) interpreting structural clues, (2) inference from text, and (3) using a dictionary. Knowing what a 17.

(27) word means is recognized as a necessary skill for reading comprehension by researchers. Text attack skills are divided into skills necessary to read for plain sense (bottom-up skills) and skills necessary to understand discourse (top-down skills). Skills to reading for plain sense involve: (1) understanding syntax, (2) recognizing and interpreting cohesive devices, and (3) interpreting discourse markers. Skills necessary to understand discourse are: (1) recognizing functional value, (2) recognizing text organization, (3) recognizing the presuppositions underlying the text, (4) recognizing implications and making inferences, and (5) prediction. A detailed description of Nuttall’s taxonomy is presented as follows and modifications made to the taxonomy will be presented in Chapter Three. Word Attack Skills. Word attack skills are applied when readers need to figure out the meaning of unknown words or phrases. Three word attack skills were identified by Nuttall: interpreting structural clues, inference from context, and using a dictionary. Word Attack Skill 1: Interpreting Structural Clues This skill is applied to facilitate understanding when readers are required to determine words by interpreting either of the two following aspects of a word: (1) the grammatical function of the word, and (2) the morphology of the word. By looking at the position of a word, namely its place in the sentence, we can establish its grammatical category and this tells us the kind of meaning to look for. Nuttall also maintained that by adopting an analytic approach to morphology, that is, analyzing the bases, prefixes and affixes of words, readers are able to work out the meanings of new words. Word Attack Skill 2: Inference from Context This skill is used when readers try to look for contextual clues in the text in order to infer the meaning of unfamiliar lexical items. According to Nuttall, readers get a 18.

(28) rough idea of the meaning of a word from the context in which it occurred; and with all of the subsequent occurrences of the word, the meaning became more precise to the readers. Nuttall pointed out that this skill is invaluable for less fluent readers. Word Attack Skill 3: Using a Dictionary This skill requires readers to look up unfamiliar words in a dictionary. By this skill, readers have to be trained what words should be looked up and what should be done to make use of the information in the dictionary. For example, some readers may find it difficult to find the words they want to quickly enough. Therefore, as Nuttall stated, exercises on alphabetical order and using the guide words at the end of each page will help. Text Attack Skills. There are a total of eight types of text attack skills: Understanding Syntax, Recognizing and Interpreting Cohesive Devices, Interpreting Discourse Markers, Recognizing Functional Value, Recognizing Text Organization, Recognizing the Presuppositions Underlying the Text, Recognizing Implications and Making Inferences, and Prediction. These skills will be discussed in details in this section. Text Attack Skill 1: Understanding Syntax Texts with long sentences or difficult syntax could be difficult to understand even when vocabulary is familiar to the readers. This skill is employed when readers are required to understand long sentences or sentences of difficult syntax. Therefore, students need to break down long sentences into smaller units. Text Attack Skill 2: Recognizing and Interpreting Cohesive Devices This skill is applied when readers are required to recognize and interpret cohesive devices in order to facilitate understanding of texts. Items that measure the readers’ ability to interpret pro-forms, elliptical expressions or lexical cohesion are considered as measuring this skill. To identify pro-forms, readers are required to 19.

(29) identify the meanings of words like it, our, this, those, then, one (as in the wrong one), so/no (as in I think so, it appears not), and comparatives (smaller, same, additional, such, other, etc). Words like these are used by writers to avoid needless repetition. With this skill, readers should be able to identify the person, object, or anything referred to and retrieve the reference from the text. This skill is applied when readers are required to identify either anaphoric reference (looking backwards in the text) or cataphoric reference (look forwards). As for interpreting elliptical expressions, readers are required to retrieve the meaning of something left unsaid but has already been mentioned in the text previously. To interpret lexical cohesion, readers are required to interpret the relationship between a lexical item and other parts of the discourse, e.g., synonyms, hyponyms, metaphors, etc.). In sum, items that measure the readers’ ability to interpret pro-forms, elliptical expressions or lexical cohesion are considered as measuring this skill. Text Attack Skill 3: Interpreting Discourse Markers This skill is concerned with signaling relationships between different parts of the discourse or between the writer and his or her message. Discourse markers often indicate the functional value of the sentence in which they occur and show the relationship the writer intends between two parts of the text. This skill involves interpreting either one of three types of discourse markers: (1) markers that signal the sequence of events (e.g., then, first, at once, next, the following day), (2) markers that signal discourse organization (in conclusion, that is to say, for example, to resume, in short), and (3) markers that signal the writer’s point of view (e.g., moreover, incidentally, similarly, however, as a matter of fact, in any case, therefore, in order to, if, certainly, more importantly). Text Attack Skill 4: Recognizing Functional Value Nuttall (1996, p. 100) stated that understanding discourse depends on 20.

(30) understanding the functional value of the text sentences. Nuttall divided functional value into three categories: (1) independent functions (associated with propositional meaning; e.g., defining, classifying, generalizing, naming, describing, reporting, speculating, predicting, etc.), (2) text-dependent functions (associated with contextual meaning;. e.g., asserting,. exemplifying,. explaining, reinforcing,. explicating,. hypothesizing, commenting, concluding, etc.), and (3) interaction-dependent functions (associated with pragmatic meaning; e.g., conceding, evaluating, inviting, instructing, apologizing, suggesting, complaining, complimenting, warning, etc.). Text Attack Skill 5: Recognizing Text Organization Skill of this type is used when the reader is required to trace the rhetorical development of a text and to analyze the rhetorical organization of a text. In general, paragraphs can be organized in many patterns (e.g., definition, description, cause-effect, classification, comparison-contrast, process, hypothesis, argumentation, exemplification, etc.). It is easier for readers to interpret difficult sentences if they can identify the principle by which the text is organized and see how the ideas hang together (Nuttall, 1996, p.106). Text Attack Skill 6: Recognizing the Presuppositions Underlying the Text This skill is applied when the reader is required to figure out presuppositions such as the knowledge and experience that the writer expects the reader to have or the opinions, attitudes, emotions that the writer expects the reader to share, or at least to understand. Text Attack Skill 7: Recognizing Implications and Making Inferences This skill is necessary when the reader is expected to understand information implicitly stated in the text or draw certain unstated conclusions, points in an argument from the text. The answer of a question of this type cannot be directly located but has to be inferred. 21.

(31) Text Attack Skill 8: Prediction This skill involves readers using their understanding of the passage and their background knowledge to determine what might happen next or after a text ends. Prediction involves using schemata about the way stories work, the way texts are constructed, and the way people tend to think. By prediction, readers are able to predict what the writer is likely to put forward next. In sum, Nuttall’s taxonomy consists of three word attack skills and eight text attack skills. The structure of Nuttall’s taxonomy is presented in Table 1 below. Table 1.. Structure of Nuttall’s Original Taxonomy. Word attack skills 1. Structural Clues 2. Inference from Context 3. Using a Dictionary Text attack skills Skills necessary to read for plain sense (Bottom-up skills) 1. Understanding Syntax 2. Recognizing and Interpreting Cohesive Devices 3. Interpreting Discourse Markers Skills necessary for reading beyond plain sense and into discourse (Top-down skills) 4. Recognizing Functional Value 5. Recognizing Text Organization 6. Recognizing Presuppositions Underlying the Text 7. Recognizing Implications and Make Inferences 8. Prediction Note. Adopted from Nuttall’s original taxonomy: Teaching Reading Skills in a Foreign Language, by Nuttall, 1996. However, a few weaknesses and practical limitations could be found in the original taxonomy. The skills proposed by Nuttall are important for efficient and effective reading; however, skills which are not supposed to be measured on the reading comprehension section and skills which are not able to be measured in testing. 22.

(32) situations should be excluded for the purpose of the present study. In terms of word attack skills, the first word attack skill “Interpreting Structural Clues” is usually measured in the section which tests vocabulary knowledge, not in the reading comprehension section. Therefore, this skill is excluded from the analysis of the current study. In addition, the second word attack skill “Using a Dictionary” is obviously not applicable in testing situations as the examinees are not allowed to use a dictionary during testing. Therefore, this skill is also excluded. Thus, the only skill left is the second word attack skill “Inference from Context.” As for the skills under text attack skills, some skills are not measured or are usually measured in other sections on the SAET or DRET rather than in the reading comprehension section. First, “Understanding Syntax” requires the reader to break down long sentences in order to arrive at meaning. According to Nuttall, when a reader encounters structural difficulty, he or she must take an analytic approach to analyze long or difficult sentences. For example, the reader is suggested to identify the cohesive elements, rewrite the sentence, find the nouns, and identify the verbs, etc. These are more related to analyze the grammar rules of a sentence, but reading comprehension tests do not require an examinee to analyze the grammar of sentences. Therefore, this skill is excluded as well. Secondly, “Interpreting Discourse Markers” which requires the reader to identify and interpret discourse markers is usually measured in the cloze test instead of the reading comprehension test. Thirdly, “Prediction” involves readers using their understanding of the passage and their background knowledge to determine what might happen next or after a text ends. In testing situations, this skill cannot be used and thus is excluded as well. Thus, the original taxonomy was revised to fit the purpose of the present study and the revised taxonomy is presented in Chapter Three.. 23.

(33) Reading Skill Instruction With strengthened English reading skills, ESL/EFL readers will make greater progress and attain greater development in all academic areas of English (Anderson, 1999). As many researchers have assumed that reading skills can be analyzed as involving subskills, the way to help learners improve their reading is to practice those subskills (Dubin, Eskey & Grabe, 1986; Grellet, 1981; Mo, 1987; Nuttall, 1996). Researchers have stressed the importance of teaching reading skills to facilitate reading comprehension and to achieve successful reading by teaching one or many reading skills in various existing taxonomies (e.g., Duke & Pearson, 2002; Dunmore, 1989; Mo, 1987; Nuttall, 1996; Pressley, 2000; Rubin, 1982; Su, 2001; Williams, 1983). Mo (1987) pointed out that English reading instruction in Taiwan placed little emphasis on the comprehension of textual structure; therefore, students often understand every sentence of a text yet fail to understand textual organization and the main idea of a paragraph. Therefore, he proposed that reading skills should be taught to help students understand textual organization and to improve their reading abilities. He also proposed that reading tests be constructed by incorporating reading skills into test items. Empirical studies have showed that explicit reading skill instruction enhances EFL reading (e.g., Chen, 2005; Su, 2001). Chen (2005) investigated the effects of explicit instruction of reading skills on EFL senior high school students’ reading comprehension and also examined the students’ comprehension of different types of questions. What’s more, the students’ perceptions of reading skills, reading attitudes, and their responses to the skill instruction were investigated. A total of 89 year-three students from two classes of a senior high school in Kaohsiung were recruited as participants in Chen’s study. A three-month explicit skill instruction was provided with a focus on five skills: (1) skimming for main ideas, (2) identifying topics and 24.

(34) main ideas, (3) making predictions, (4) making inferences, and (5) guessing the meanings of unfamiliar words from context. These skills were taught in an explicit approach which emphasized modeling, guided practice, independent practice, and application. Furthermore, reading comprehension tests and questionnaires were conducted before and after the treatment to collect the related data for analysis and discussion. The results showed that the skill instruction proved effective in building up the participants’ knowledge and use of the skills. According to Chen, most of the participants did not have a clear understanding of those skills before the treatment; however, they had learned to use them and liked to use them after the treatment. The results also revealed that the skill instruction helped enhance the participants’ learning interests toward English reading and their confidence in their English reading ability. Also, it was found that the effect of skill instruction was more significant on the intermediate and the lower proficiency participants in promoting reading comprehension. The lower proficiency subjects benefited most from the treatment and most participants had a positive response toward the skill instruction. In terms of the effectiveness on each reading skill, the skill instruction helped improve the participants’ comprehension for main idea, detail, and inference questions, but not for word-guessing questions. Among the skills instructed, “guessing the meanings of unfamiliar words from context” and “skimming for the main idea” were considered the most practical skills by the participants. Su (2001) conducted an empirical study to evaluate effectiveness of teaching English reading skills and the cognition of the students on the reading skills they learned. Su introduced fourteen reading skills to her students in the first-year college English reading course whenever her students need to apply those skills in their reading. The reading skills Su introduced to the participants were (1) questioning, (2) previewing and predicting, (3) scanning, (4) recognizing topics, (5) resolving 25.

(35) ambiguities in the text, (6) looking for key words, (7) guessing vocabulary from context, (8) locating topics sentences, (9) understanding main idea & details, (10) skimming, (11) reading aloud, (12) small group discussion, (13) paraphrasing, and (14) role-playing. According to Su, these skills were selected for students who need “elementary” literacy skills to help them read and think in English. After the reading skills were introduced to the students, Su evaluated the students’ feelings about the reading skills. A questionnaire was designed for the participants to fill out at the first and the final classes of the second semester of instruction to rate the usefulness of the reading skills they learned. The results showed that all the students in Su’s study agreed that the reading skills they learned were helpful for the improvement of reading ability except for one skill—the skimming skill. In addition, most of the students had difficulties with reading skills such as skimming, paraphrasing and guessing vocabulary from context. Although researchers have different opinions regarding how many reading skills there are, in general, it is agreed that the teaching of reading skills is helpful in promoting reading comprehension. Also, it is very important for teachers to be aware that if reading skills are teachable, then they are probably testable (Nuttall, 1996). Nuttall (1996, p. 220) augued that “If we do no test what we are trying to teach, there is a danger that students will not pay much attention to what is taught, or that teachers will cease to teach it.”. Testing ESL/EFL Reading Comprehension In this section, the relationship between teaching and testing is discussed. Also, literature related to using multiple choice questions to measure reading skills is presented.. 26.

(36) Why Testing (The Relationship between Teaching and Testing) Testing is an important part of every teaching and learning experience (Madsen, 1983).The proper relationship between teaching and testing is surely that of partnership (Hughes, 2003). Testing should not only follow teaching but should support good teaching and exerts a corrective influence on bad teaching. Therefore, understanding of tests could provide information about whether our teaching needs improvement or certain part of our teaching needs more attention. The importance of testing has been emphasized by a number of researchers. As McNamara (2000) pointed out, language tests can serve as a valuable tool for teachers by providing information that are relevant to language learning: such as (1) evidence of the results of learning and instruction as well as feedback on the effectiveness of the teaching program, (2) information relevant to decision-making about individuals, and (3) information for teachers to clarify instructional objectives. Tests can provide information about teaching and learning and thus tests can be helpful to both teachers and students. According to Madsen (1983), well-made tests can help students develop positive attitudes toward instruction by giving them a sense of accomplishment and a feeling that the teacher’s evaluation of them corresponds to what has been taught. In addition, tests also help foster learning since they help the teachers confirm which part of the learning each student has mastered and which part needs further attention and improvement. What’s more, good English tests can assist the students in learning the language by requiring them to study hard, emphasizing course objectives, and showing them where they need to improve. The qualities desirable for a good test are validity, reliability, and practicality (Bachman, 1990; Brown, 2001; Harries, 1969; Hughes, 2003). These qualities are generally regarded as the basic requirements of a good test. Validity is considered as 27.

(37) the most complex criterion of a good test (Brown, 2003). A test is said to have validity when it actually measures what it intends to measure. For example, if the purpose of a reading test is to examine the reading ability of a particular group of students, the test results should reflect their true reading ability. Validity is an important factor in designing good reading comprehension tests (Sequera, 1995). Three types of important validation of a test are content validity, face validity, and construct validity. The core element of test validity is the construct—the theoretical representation of the skill or knowledge that the test purports to measure (Slomp, 2005, p. 149). As such, the validity of a test rises or falls in accordance with the degree to which its scores are a reflection of students’ ability in relation to the construct. Hughes (2003) defined construct in language testing as “any underlying ability (or trait) that is hypothesized in a theory of language ability.” (p. 31). Hence, in the case of reading comprehension, tests should reflect the theoretical assumptions under which reading teachers operate (Sequera, 1995). Therefore, it is assumed that the tests reflect the objectives of reading abilities stipulated in the curriculum guidelines and it is one of the issues that the present study aims to address. Testing Comprehension with Multiple-choice Questions Many textbooks on language testing (e.g., Heaton, 1988; Hughes, 1989; Weir, 1990, 1993) have given examples of testing techniques that might be used to assess language abilities. Among various test techniques, multiple-choice (hence MC) format is a common device for testing students’ reading comprehension. Multiple-choice (MC) is considered to be an objective technique which “requires intellectual discrimination skills, a versatile test capable of probing a variety of areas and different types of cognitive activities such as acquisition of knowledge, understanding, application, analysis and evaluation” (Green, 1975; Marshall, 1971, cited in Nevo, 1989). The most obvious advantage of MC questions is probably in that 28.

(38) the scoring can be reliable, rapid, and economical. In addition, MC format allows for more items to be included in a given period of time. A further advantage is that MC format allows the testing of receptive skills without requiring the test-takers to produce output. Nevertheless, given the virtues of multiple-choice questions, this format has been criticized by researchers on a number of grounds. A serious disadvantage with MC questions is that guessing may have a considerable but unknowable effect on test scores (Hughes, 2003). Test-takers may get an item correct for the wrong reason without actually understanding the text. Another objection to the use of multiple-choice questions is that they are often passage-independnet which means that the items could be answered without reference to the reading passages accompanying them (Bernhart, 1991; Teale & Rowley, 1984; Weir, 1997). Evidence has been found that test-takers do not always recourse to the reading passages in response to MC questions (Cohen, 1998; Nevo, 1989); it is suspected that explicit teaching of certain techniques will help students become test-wise and thus improve their scores (Richards, 1997). Nevo (1989, p. 212) argued that “it would appear useful to devote attention, time, and effort to guiding and training students in coping effectively with a test format like [the multiple-choice test]”. A further concern with the use of multiple-choice (MC) format is that there has been much doubt with respect to the validity of multiple-choice (MC) questions. Weir (1990) and Urquhart & Weir (1998) argued that answering multiple-choice questions is an unreal task since in real life communications one is rarely required to choose one answer from four choices to show understanding. In a multiple-choice test, distractors can be used to trick the test-takers by presenting choices that the test-takers may not have otherwise have thought of (Alderson, 2000; Richards, 2000; Weir, 1990). Test-takers could be deliberately tricked into confusing dilemmas. Therefore, it is 29.

(39) difficult to know whether failure of questions is due to lack of comprehension of the text or lack of comprehension of the question. Despite of the criticisms against the use of the MC technique, multiple-choice questions has been widely used to assess reading ability and therefore issues have been raised about what MC questions actually measure in reading tests and whether they are valid measurements of reading ability (Cummings, 1982; Farr, Prichard & Smitten, 1990; Urquhart & Weir, 1998; Weir, 1997). Weir (1997) proposed a four-level version of reading comprehension for testing purposes: reading expeditiously. for. global. comprehension,. reading. expeditiously. for. local. comprehension, reading carefully for global comprehension, and reading carefully for local comprehension. In Weir’s version of operations in reading, skills such as understanding the syntactic structure of a sentence and clause, understanding lexical and/or grammatical cohesion, understanding word meaning and locating specific details are bottom-up skills operated when reading at local levels, while identifying the main idea and making inferences are top-down skills operated at global levels. Similarly, Harrison (1983, in Navo, 1989) argued that through the MC format, it is considered possible to check all reading levels (the semantic and syntactic aspects of the test), the discourse level (cohesion and coherence connections among various parts of the text), and the pragmatic level (the writer’s point of view). In other words, bottom-up reading skills and top-down reading skills could be tested via MC questions. Studies of Reading Tests Analysis (Item Analysis) As noted previously, researchers agree that tests can provide information about teaching and learning and thus tests can be helpful to both teachers and students. An analysis of the reading comprehension test items also reveals the knowledge and skills valued by the reading comprehension tests. Research into reading tests has received 30.

(40) much interest in the field of reading research in Taiwan. Both SAET and DRET exert considerable influence on English teaching and learning in Taiwan because the scores a student receives on each exam is a determining criterion for admission to desired universities. Thus, with the importance of both tests, they have been topics for discussion and research reports on the overall test content analysis and statistical analysis of both tests have been conducted annually by the College Entrance Examination Center (CEEC Web site; Huang, 1994; Jeng 1992; Jiang & Lin, 1999; Xu & Lu, 1998). In particular, most of the studies on the SAET and DRET aimed at providing an overview of the test construction or statistical results of difficulties of items, distractors, and passing rates, etc. rather than a thorough qualitative analysis on the reading skills tested on each comprehension item in particular (e.g., Huang, 1994; Jeng, 1992; Jiang & Lin, 1999; Xu & Lu, 1998). Most statistical analyses have often focused on the numbers of items and distribution, length of text, vocabulary, topics, discriminatory power, and examinees’ test performances (e.g., passing rates). Huang (1994) conducted a qualitative analysis of the Joint College Entrance Examination (hence JCEE, renamed as DRT in 2002) English test items from 1985 to 1994. The results of the reading comprehension item analysis showed that roughly over 90% of the items were well-designed, yet a few were not well-constructed or not even reading comprehension items at all. Items designed to test examinees’ vocabulary in context are reading comprehension questions on account that the readers have to look for contextual clues in the texts. However, Huang found that some items designed to test examinees’ vocabulary were not well-written because they could be answered without referring back to the text. What’s more, he found that some items were designed to test examinee’s grammar knowledge instead of their reading abilities. He indicated items designed to test examinees’ knowledge of vocabulary and grammar without referring back to the text are ill-written and thus should be excluded. He also said that 31.

(41) cautions should be taken when constructing items that involve arithmetic. Jeng (1992) conducted a statistical analysis of the English test of JCEE in 1991. This analysis focused on the difficulty of items, discriminatory power, and distractors in the test. There were three formats of questions, including sections on conversations, cloze, and reading comprehension. It was found that 76% of all of the items in the test, a total of 34 items, were well-written. Among the well-written items, 13 items were reading comprehension items, which amounted to 86% of the well-written items in total. The results also showed that the overall reliability of the test items reached .90. Xu and Lu (1998) studied various elements of the JCEE English test content in 1998, including the topics, text length, syntactic complexity, vocabulary, distractors, and question types. The researchers found that these reading comprehension test items could usually be classified into four types: vocabulary, main idea, detail, and inference. However, they did not further identify those elements item by item or examine the frequency and distribution of different items. The aforementioned studies aimed at exploring the overall test construction or statistical results rather than a thorough qualitative analysis on the reading skills tested on each comprehension item in particular. Recently, two studies (Hsu, 2005 ; Lu, 2002) used Mo’s taxonomy as the coding scheme to analyze the reading comprehension test items on SAET and DRET. Lu (2002) conducted both qualitative and quantitative analyses of the reading skills measured in reading comprehension section on the SAET from 1995 to 2002. Qualitatively, Lu categorized the reading comprehension test items by using Mo’s classification as the coding scheme. Lu also examined the texts and the variables that affected the passing rates of high achievers and low achievers. With respect to quantitative analysis, she computed the frequency distribution of question types and the correlation between question types and passing rates. Results showed that the 32.