國民中學基本學力測驗英語科回沖效應研究 - 政大學術集成

全文

(1)國立政治大學英國語文學系碩士班碩士論文. 指導教授：尤雪瑛博士 Advisor: Dr. Yu, Hsueh-ying. 國民中學基本學力測驗英語科回沖效應研究. 政治大. A Study on the Washback Effect of the Basic Competence English Test. 立. on Junior High School Students in Northern Taiwan. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i Un. 研究生：林銀姿撰 Name: Lin, Yin-tzu 中華民國九十九年七月 July 2010. v.

(2) A STUDY ON THE WASHBACK EFFECT OF THE BASIC COMPETENCE ENGLISH TEST ON JUNIOR HIGH SCHOOL STUDENTS IN NORTHERN TAIWAN. A Master Thesis Presented to. Department of English,. 學. National Chengchi University. Nat. n. al. er. io. sit. y. ‧. ‧ 國. 立. 政治大. Ch. engchi. i Un. v. In Partial Fulfillment of the Requirements for the Degree of Master of Arts. by Lin, Yin-tzu July 2010.

(3) Acknowledgments. I would like to thank my advisor, Dr. Hsueh-ying Yu, for her constant guidance and patience.. With her expertise, wisdom, and kindness, Dr. Yu led me to. the final completion of this thesis. was far beyond the call of duty.. I believe that what she has done along the way. I am also grateful for valuable insights and. constructive comments given by the other committee members at proposal and thesis oral defense stages, Dr. Chieh-yue Yeh, Dr. Yuen-mei Yin, and Dr. Chen-kuan Chen.. 政治大. My greatest appreciation goes to all the students who willingly helped. 立. complete my survey. Also, my thesis would not have been made possible without. ‧ 國. 學. the generous help from the 24 English teachers in northern Taiwan, especially Ms. Chiao-yun Hsu, who readily offered me her classes for the pilot study.. ‧. I am indebted to Dr. Huei-ling Lai, Dr. Chin-chi Chao, and my colleague Betty. y. Nat. io. sit. Chi-Ju Teng, for their continuous encouragement and tolerance at work.. My. n. al. er. gratitude also goes to Dr. Hsin-fen Wu, whose academically rigorous attitudes. Ch. inspired me a lot when I took her course.. engchi. i Un. v. And finally my heartfelt gratitude goes to my family.. I would like to thank. my parents for their understanding, and particularly, my boyfriend for going through the whole process with me and assuring me of the completion of this academic journey. I also want to say a special thankyou to my loving pets, Mimi and CPA, who buoyed me up through these difficult times.. iii.

(4) Table of Contents. Acknowledgments ....................................................................................................... iii Table of Contents .......................................................................................................... iv List of Tables and Figures.......................................................................................... viii Chinese abstract ............................................................................................................. x. 政治大. English abstract............................................................................................................xii. 立. CHAPTER 1 INTRODUCTION ................................................................................... 1. ‧ 國. 學. Context of the Study .......................................................................................... 7. ‧. English Language Education in Taiwan ................................................ 7. y. Nat. er. io. sit. The Basic Competence Test (BCT) for Junior High School Students... 9 The Basic Competence English Test (BCET) for Junior High. al. n. iv n C School Studentsh ................................................................................... 10 engchi U Purpose of the Study ........................................................................................ 11 Significance of the Study ................................................................................. 12 CHAPTER 2 LITERATURE REVIEW ...................................................................... 13 The Nature of Washback: Positive vs. Negative.............................................. 13 Negative Washback.............................................................................. 13 Positive Washback ............................................................................... 16. iv.

(5) The Nature of Washback and Context ................................................. 19 The Washback Mechanism .............................................................................. 20 Alderson and Wall’s Washback Hypothesis ........................................ 20 The Hughes Trichotomy Framework ................................................... 22 Bailey’s Model of Washback ............................................................... 23 Studies on BCET and BCT Washback............................................................. 25. 政治大. CHAPTER 3 METHODOLOGY ................................................................................ 29. 立. Selection of the Participants ............................................................................ 29. ‧ 國. 學. Instruments ...................................................................................................... 31. ‧. Content of the Questionnaire ............................................................... 32. y. Nat. er. io. sit. Reliability and Validity ........................................................................ 34 Data Collection Procedures ............................................................................. 36. al. n. iv n C hengchi U Data Analysis ................................................................................................... 38 CHAPTER 4 RESULTS AND DISCUSSION ............................................................ 40 Students’ Characteristics and Their Learning Contexts................................... 41 Students’ Perceptions of the BCET ................................................................. 44 Perceptions of the BCET Test Difficulty and Test Format .................. 45 Perceptions of the BCET Test Purpose ................................................ 46 Perceptions of the BCET Test Difficulty in Relation to Understanding. v.

(6) the BCET Test Format and Test Purpose ............................................. 47 Summary on Students’ Perceptions of the BCET ................................ 49 Students’ Motivation to Learn English ............................................................ 51 Perceptions of the Importance of Learning English ............................ 53 Motivation in Relation to Grade 1-9 Curriculum ................................ 55 Motivations that Students Find Less Strong ........................................ 57. 政治大. Summary on Student Motivation ......................................................... 61. 立. Students’ Learning Activities ........................................................................... 62. ‧ 國. 學. Learning Activities that are Less Directly Related to the BCET ......... 63. ‧. Learning Activities that are More Directly Related to the BCET ....... 66. Nat. io. sit. y. Summary on Students’ Learning Activities ......................................... 71. er. Students’ Perceptions of Class Activities ........................................................ 72. al. n. iv n C th h e Activities Common Classroom n g c h ini 9U-Grade English. Language Courses................................................................................ 74 Classroom Activities Used Occasionally in 9th-Grade English Language Courses................................................................................ 76 Classroom Activities Rarely Used in 9th-Grade English Language Courses................................................................................ 79 CHAPTER 5 CONCLUSIONS ................................................................................... 83. vi.

(7) Summary of the Findings ................................................................................ 83 Implications of the Study ................................................................................. 87 Limitations of the Study .................................................................................. 89 Suggestions for Future Research ..................................................................... 90 REFERENCES ............................................................................................................ 91 APPENDICES ............................................................................................................. 98. 政治大. A. List of Selected Schools and Number of Classes ....................................... 98. 立. B. Questionnaire ............................................................................................ 101. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. vii. i Un. v.

(8) List of Tables and Figures. Figure 2.1 A Basic Model of Washback ...................................................................... 24 Table 3.1 Structure and Categories of the Questionnaire ............................................ 32 Table 4.1 Questionnaire Return Rates by Administrative Level ................................. 41 Table 4.2 Students’ Characteristics and Their Learning Contexts ............................... 42. 政治大. Table 4.3 Students’ Perceptions of the BCET ............................................................. 44. 立. Table 4.4 Students’ Perception of the BCET Difficulty and Their Knowledge of. ‧ 國. 學. the BCET Format ............................................................................................. 48. ‧. Table 4.5 Students’ Perception of the BCET Difficulty and Their Knowledge of. y. Nat. io. sit. the BCET Purpose ........................................................................................... 49. er. Table 4.6 Students’ Motivation to Learn English ........................................................ 52. al. n. iv n C Table 4.7 Students’ Perceptionshofethe nImportance g c h i Uof Learning English and Their. English Achievement ....................................................................................... 54 Table 4.8 Students’ Perceptions of Learning English Mainly for the BCET and Their English Achievement ............................................................................. 58 Table 4.9 Students’ Perceptions of Learning English out of Personal Interests and Their English Achievement ............................................................................. 60. viii.

(9) Table 4.10 Students’ English Learning Activities Less Directly Related to the BCET ......................................................................................................................... 64 Table 4.11 Students’ English Learning Activities More Directly Related to the BCET ......................................................................................................................... 67 Table 4.12 Students’ Perceptions of Learning Words and Phrases Beyond the Word List of 1,000 Basic English Vocabularies and Their English Achievement .... 70. 政治大. Table 4.13 Students’ Perceptions of Classroom Activities .......................................... 73. 立. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. ix. i Un. v.

(10) 國立政治大學英國語文學系碩士班碩士論文提要. 論文名稱：國民中學基本學力測驗英語科回沖效應研究指導教授：尤雪瑛博士研究生：林銀姿論文提要內容：. 立. 政治大. 國民中學基本學力測驗（簡稱「基測」）英語科屬高代價之語言測驗，測驗. ‧ 國. 學. 結果攸關每年三十萬左右國中畢業生的升學依據，影響層面甚廣，測驗對教學的. ‧. 影響即語言測驗文獻中所探討之回沖效應。. 本研究旨在探討基測英語科對台灣北部地區九年級學生英語學習的回沖效. y. Nat. er. io. sit. 應，以問卷調查台灣北部地區（基隆市、台北市、台北縣、桃園縣、新竹市、新竹縣）九年級學生，從以下四個面向探討基測英語科所造成的回沖效應：（1）. n. al. Ch. i Un. v. 學生對基測英語科試題的了解及看法；（2）學生的學習目的；（3）學生的學習. engchi. 方式；（4）課堂活動。本研究以四個行政層級（直轄市、市、鎮、鄉）作為抽樣依據，共計發出 1,490 份問卷。研究主要結果如下：一、大多數學生了解基測英語科之題型及測驗目的。僅有半數學生認為試題簡單，進一步分析顯示，比較了解題型及目的的學生認為試題簡單。二、學生學習目的與《國民中小學九年一貫課程綱要》英語課程目標一致。不以基測作為主要英語學習目標的學生英語成績較好。三、學生最常從事的學習活動是聽英語歌曲、看英語影片或節目，也會學習千字. x.

(11) 表之外的字詞。大部分的學生沒有藉由參考書、參加校內課後輔導或校外補習來準備基測英語科。四、最常見的課堂活動是老師帶唸課文或單字、文法練習，及寫測驗卷或聽老師檢討內容。相較之下，口說聽力的教學活動較少。結果顯示基測英語科對學生的學習影響不大，但對老師的課堂教學產生負面回沖效應，進而影響學生學習。綜合研究結果，本研究建議教師應增加課堂教學活動的多樣性，減少測驗相關教材及練習，以培養學生學習英語的興趣與方法。. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. xi. i Un. v.

(12) Abstract. The effect of testing on teaching and learning is generally known as washback in the field of language testing and assessment.. Despite a growing attention to the. concept of washback, most washback studies focus on teachers, not on learners as test takers, who are the most important stakeholders in testing. The purpose of this study is to investigate the washback effect of the Basic Competence English Test (BCET), a high-stakes language test, on 9th-graders in. 政治大. northern Taiwan. The method of questionnaire survey was used to collect students’. 立. perspectives on the following four areas: (a) perceptions of the BCET, (b) learning. ‧ 國. 學. motivation, (c) learning activities, and (d) class activities. A convenient sample of a proportionate size was selected based on four administrative levels (i.e., national,. ‧. municipal, town, and village levels) from Keelung City, Taipei City, Taipei County,. y. Nat. Altogether 1,490 ninth-graders. io. sit. Taoyuan County, Hsinchu City, and Hsinchu County.. n. al. er. from 42 classes at 22 schools were selected for the survey across northern Taiwan.. Ch. i Un. The major findings of the study are as follows: 1.. engchi. v. Most 9th-graders in northern Taiwan were aware of the BCET test format and the BCET test purpose.. Nearly half of them found the BCET not to be easy.. The. more the students understood the BCET format, the more likely they thought that the BCET was easy.. Similarly, the more the students understood the BCET. purpose, the more likely they regarded the BCET as an easy test. 2.. Three top motivations for the students to learn English coincide with the overall objectives of English language education in Grade 1-9 Curriculum.. Students’. English achievement gets better if they do not regard the BCET as the primary goal of language learning. xii.

(13) 3.. Students’ most common learning activities include listening to English songs, watching movies/programs in English, and learning words and phrases beyond the word list of 1,000 basic English vocabularies. The BCET does not drive students to undertake the following practices: using practice books for examinations, attending cram schools, and attending self-study classes for the test at school.. 4.. It was common for the 9th-graders to do practice tests or mock examinations in class, and listen to their teachers explaining the test items.. Instruction relating. to formal aspects of English also occurred frequently in English language. 治政 courses, while few aural/oral practice activities 大were undertaken in classroom 立 teaching. ‧ 國. 學. Based on the findings of the BCET washback effects on students and teachers,. ‧. this study suggests that English teachers should attempt to employ various types of. sit. y. Nat. classroom activities, as proposed in Grade 1-9 Curriculum, rather than relying on. io. al. er. examination-oriented materials and practices, so as to promote learning interests and. n. develop effective methods of learning.. Ch. engchi. xiii. i Un. v.

(14) CHAPTER 1 INTRODUCTION. A language curriculum generally involves a number of components: identifying goals and objectives, selecting materials and procedures, teaching, testing, and evaluating the overall curriculum (Brown, 2005). These components are viewed A change in one part of the system has 政治大. as forming a system of interrelated parts.. 立. effects on the other parts of the system.. Among these elements, tests play a crucial. ‧ 國. 學. role in curriculum development. They are frequently used to gather information for. ‧. the reference of stakeholders, such as students, teachers and administrators, and. y. Nat. io. sit. Stakeholders are likely to know how well students have. er. parents of students.. mastered the course content, and how effectively the overall language program has. al. n. iv n C h e ngained Information test results can be used to maintain i U g c hfrom. reached its objectives.. and adjust the other parts of the curriculum. Given the central role that testing plays in the curriculum, it is necessary to evaluate its use and consequences for stakeholders, particularly students as test takers. Tests are used to make inferences about test takers’ abilities, and more importantly, to make decisions about placement, admissions, or graduation.. These decisions imply. values and goals, and certainly have effects on individuals, primarily students and 1.

(15) 2. teachers, and wider communities, from a particular education system to whole societies.. In the field of language teaching, it is widely believed that tests,. particularly high-stakes tests that have important consequences for individuals and institutions, can exert influence on the curriculum in terms of content and method, and these effects are known as washback (also known as backwash, Alderson & Wall, 1993; Bachman & Palmer, 1996; Bailey, 1996; Cheng, 2008; Cheng & Curtis, 2004; Hughes, 2003; Wall, 1997).. 政治大. Washback generally refers to “the influence of testing. 立. on teaching and learning” (Bailey, 1996, p. 259).. Following Alderson and Wall’s. ‧ 國. 學. (1993) focus of influence on classroom behaviors of teachers and learners, Messick. ‧. (1996) paraphrases the definition of washback as “the extent to which the. y. Nat. io. sit. introduction and use of a test influences language teachers and learners to do things. al. er. they would not otherwise do that promote or inhibit language learning” (p. 241).. n. iv n C U h e ncompetitive Previously in Taiwan, highly examination system was g c h i entrance. criticized to have negative washback on the curriculum. tremendous stress on learners.. The public examination put. They fretted about the entrance examination because. test scores would have a direct impact on their future academic success.. Many. students devoted much time to preparing for the examination. In addition to regular education, they often attended short-term private supplementary schools, or so-called cram schools, to assist them in test preparations. The anxiety caused by entrance.

(16) 3. examination system had reached the extent to which it had debilitating impact on students’ learning attitudes and motivation. The Ministry of Education (MOE) in Taiwan introduced a reformed entrance examination in school year 2001 (September 2001 to June 2002), called the Basic Competence Test (BCT) for Junior High School Students, to meet the requirements of the revised Curriculum Standards for Junior High Schools.. The BCT is intended. 政治大. that students would get rid of the cramming and emotional tension that accompanied. 立. the previous public entrance examination. According to Lin (2001), ex-director of. ‧ 國. 學. the BCT commissioned center, the BCT has great potential to produce positive. ‧. washback on the curriculum. One great benefit is that students do not need to attend. y. Nat. er. io. sit. cram schools to learn complicated grammar rules through extensive drills, which was common when the previous entrance examination was in operation.. al. n. iv n C Changes to the entrancehexamination i U were typically motivated by the e n g c hsystem. desire to engineer beneficial educational effects on student learning.. However, it. seems that the BCT falls short of its intended objectives. Three years after the implementation of BCT, Wu and Chang’s (2005) national survey of nearly seven thousand 7th- and 8th-graders reported that approximately 53 percent of them learned English outside of school, primarily by attending cram schools.. Similarly, a more. recent survey by China Times (“Eighty Percent of Junior High Students,” 2007).

(17) 4. shows that 77 percent of the parents sent their junior high children to cram schools in the hope that they may achieve higher scores on the BCT. English and mathematics are the most common courses of studies in cram schools.. It seems that despite the. reforms in curriculum and entrance examination system over the past decades, heavy pressure for entrance examination still exists, and it gets even worse. From the language testing and assessment literature, Shohamy’s (1992). 政治大. washback definition can be applied to Taiwan’s context: “the utilization of external. 立. language tests to affect and drive foreign language learning” and “this phenomenon. ‧ 國. 學. [washback] is the result of the strong authority of external testing and the major. ‧. impact it has on the lives of test takers ”(p. 513).. Nat. y. For the present study, the English. er. io. sit. language component of the BCT, i.e., the Basic Competence English Test (BCET), is a language test that junior high graduates need to take for advancing into the next. al. n. iv n C U h esongreat Its stakes are exerts effects on how h iit inevitably g cthat. level of education.. students learn English, whether inside or outside of formal schooling system. Although the relationship between testing and teaching/learning is commonly held to exist, it seems not clear whether this connection really occurs, and if it does exist, what its effect is actually like. The question of whether washback effects exist was first raised by Alderson and Wall (1993), who argued that many researchers had made assertions about the effects of tests on teaching and learning, and those.

(18) 5. influences were perceived to be mainly detrimental.. However, according to. Alderson and Wall, little empirical research findings have been presented about the existence and nature of washback. They consequently proposed 15 hypotheses regarding the potential impact of language testing on aspects of teaching and learning. Since then, an increasing number of empirical studies, either directly or indirectly influenced by Alderson and Wall’s work, have been undertaken to investigate. 政治大. different tests within different educational contexts, and have indicated that washback. 立. does exist (e.g., Alderson & Hamp-Lyons, 1996; Cheng, 2005; Ferman, 2004; Qi,. ‧ 國. 學. 2005, 2007; Shohamy, Donitsa-Schmidt, & Ferman, 1996; Wall & Alderson, 1993;. Nat. y. ‧. Watanabe, 1996, 2004).. er. io. sit. Despite a growing body of research into washback, research attention has been, however, somewhat focused on the effects of testing on teaching, such as what. al. n. iv n C U (Bailey, 1996; Cheng, 2007; hconduct teachers teach, and how teachers e n g ctheir h ilessons. Wall, 2000). Cheng (2007) reviewed the empirical studies over the past 15 years in the field of language testing, and found that washback studies on student learning are limited, compared with those on teachers and their classroom practices. Likewise, Hamp-Lyons (2000), in light of the increasing realization of the importance of stakeholder perspectives, contended that of all stakeholders for tests, the voices of students as test takers are those least heard in the language testing.

(19) 6. literature, though they receive the most direct impact of tests. She persuasively commented that. of the many millions of people who will sit down to take (English) language test . . ., virtually none will have participated in the test’s design, in writing test items, in critiquing the test methods, in setting cut scores or in writing or commenting on the performance descriptions that tie to their all-important. 政治大. score. Of all stakeholders in testing events, test takers surely have the highest. 立. ‧. ‧ 國. 學. stake of all. (pp. 580-581). To sum up, considerable attention in the washback literature has been paid to the. y. Nat. er. io. sit. washback of tests on teachers; however, relatively little research has explored the washback effects on learners, who are also important stakeholders in testing.. al. n. iv n C h eTaiwan’s With the research gap and h i U context in mind, this study n g ceducational. was designed to examine the washback effects imparted by the BCET on students as English learners, and more importantly, to examine how successful the BCET, after several years’ use, has been in achieving one of its main goals—to produce beneficial washback on English language education..

(20) 7. Context of the Study. Before proceeding to the main research topic, a description of Taiwan education system, English language education, and the BCET is offered in the following sections, to help readers understand the context of the study.. English Language Education in Taiwan. 政治大. English is learned as a foreign language (EFL) in Taiwan. With the wave of. 立. socio-economic mobility.. 學. ‧ 國. globalization, having a good command of English is generally associated with upward English, regarded as the first foreign language, is taught. ‧. Nat. The traditional. er. io. sit. elementary education and three-year junior high education.. y. as a subject in the Taiwanese compulsory education system, comprising six-year. curriculum focuses on reading, with an emphasis on lexical items and grammar rules,. al. n. iv n C U fostering students’ ability to h e criticism which has come under considerable n g c hfori not. communicate. The revised English Curriculum Standards for Junior High Schools and accompanying new textbooks were issued in the mid-1990s. The revised curriculum was more communication-oriented, but it lasted for only few years, owing to the immediate introduction of Grade 1-9 Curriculum in 2001, the purpose of which is to fill the gap between elementary and junior high school curriculums. Within the framework of Grade 1-9 Curriculum, English has become.

(21) 8. compulsory from the fifth grade on since school year 2001, two years earlier than before.. Later, the initial grade of English instruction has been further lowered to the. third grade starting from school year 2005.. Besides, schools are allowed to choose. their own textbooks that best meet the demands of students in their particular contexts. This has caused some people to question the fairness of the BCT, for fear that this high-stakes test may favor a particular version of the textbooks. The BCT. 政治大. commissioned center states that the test content is developed on the basis of the. 立. competence indicators in Grade 1-9 Curriculum, which apply to all certified versions. ‧ 國. 學. of textbooks.. ‧. According to Grade 1-9 Curriculum Guidelines, implemented by the Ministry. y. Nat. er. io. sit. of Education in Taiwan (MOE, 2003a), three overall objectives are identified for English curriculum in elementary school and junior high school: (a) to develop basic. al. n. iv n C communicative competence forhauthentic i Ulanguage use situations; (b) to e n g cor hactual. learn English autonomously and effectively by developing interests and methods; and (c) to understand native and foreign cultures so as to compare and respect cultural differences. These three curricular objectives guide the English language teaching and learning in formal education. The design of the BCET is based on the competence indicators derived from these objectives as well..

(22) 9. The Basic Competence Test (BCT) for Junior High School Students. In Taiwan, junior high school graduates have to take public entrance examinations to enter secondary schools (i.e., senior high school, senior vocational school, or five-year junior college), with very few exceptions.. Previously, each. county administered its own district joint entrance examinations.. In 2001, the BCT,. a reformed nationwide entrance examination, was implemented. The purpose of the. 政治大. BCT is to assess junior high graduates’ basic competences, which are core knowledge. 立. and capabilities that should be acquired through junior high education. This is a. ‧ 國. 學. distinguishing feature between the BCT and the previous public entrance. ‧. examinations for senior high school.. The previous examination, intended to. y. Nat. al. er. io. sit. discriminate among test takers for selection purposes and thus to measure the. v. n. capabilities of those at the higher end, typically contains items that are more difficult. Ch. i Un. e n g c h i By comparison, the BCT is an. or challenging for most junior high graduates.. achievement test, concerning whether test takers have mastered basic competences identified in the curriculum.. Thus, the BCT aims for a set of items with a spread of. facility values within the range of 50%-75% (Lin, 2001).. In other words, a higher. pass rate is expected, and the majority of test items are not too difficult for junior high school graduates..

(23) 10. The Basic Competence English Test (BCET) for Junior High School Students. The English language component of the BCT, i.e., the Basic Competence English Test (BCET), is developed based on the reading competence indicators of junior high English curriculum, as identified in Grade 1-9 Curriculum Guidelines (MOE, 2003a).. In line with the trend of communicative approaches in English. language teaching, the BCET underlines language use in contexts by adopting topics. 政治大. familiar to the students, such as school life, sports, peer interaction, etc. This focus. 立. is also reflected in the various text types that the BCET employs: short stories, notices,. ‧ 國. 學. signs, e-mails, letters, timetables, posters, maps, advertisements, forms, diaries,. ‧. weather reports, etc.. y. Nat. al. er. io. sit. As for the coverage of lexical and grammatical items, further standards are set.. v. n. Vocabulary selection is based on the word list of one thousand basic English. Ch. vocabularies (MOE, 2003b).. engchi. i Un. The word list contains high frequency vocabulary that. junior high graduates should at least recognize.. Words and phrases not included in. the word list are given with their Chinese translation on the test.. All test items are. checked against every version of the textbooks in terms of phrases, sentence patterns, and grammar points, in order to ascertain that what is on the BCET can be found in all versions of textbooks. Regarding the format of the BCET, two parts are involved. The first part,.

(24) 11. assessing linguistic knowledge (i.e., vocabulary, idioms, and grammar), consists of 15-20 items in the blank filling format. The second part measures reading competence, comprising 25-30 items in cloze and reading comprehension format. The number of items for each part may vary slightly, and the total number overall is 40-45. All test items are in a four-option multiple-choice format.. Time for the test. is 70 minutes.. 立. 治政 Purpose of the Study 大. ‧ 國. 學. The researcher would like to know how the test affects students’ motivations. ‧. and learning activities, and how it influences teachers’ classroom practices.. Nat. y. The. majority of 9th-graders to enter secondary schools.. al. er. io. sit. BCET as a high-stakes language test is worth close scrutiny because it is taken by the It is certain that due to the. n. iv n C U views and behaviors are high-stakes decisions associatedhwith e nthis h istudents’ g ctest, influenced.. Also, the study could be used to examine to what extent the intended. beneficial washback effects of the BCET on student learning has achieved. Specifically, the research questions the study addressed were:. 1.. What are students’ perceptions of the BCET in terms of its difficulty, test format, and test purpose?.

(25) 12. 2.. What are the motivations for students to learn English?. 3.. What are students’ learning activities, including test preparation practices?. 4.. What classroom activities do students have in their English lessons?. Significance of the Study. This study intends to provide research evidence of the washback effect of a. 政治大. high-stakes public examination on junior high students in northern Taiwan.. 立. aims to identify students’ perspectives on their learning.. It also. It is hoped that the research. ‧ 國. 學. findings will contribute to the understanding of the impact of the BCET on students’. ‧. learning. Furthermore, the study may offer educators some insights into English. y. Nat. er. io. sit. language teaching and learning in junior high education. Although whether a test is successful depends on a variety of factors, test developers may draw on the findings. al. n. iv n C U objectives have been h edegree from this study to examine to what h i intended n g ctheir achieved..

(26) CHAPTER 2 LITERATURE REVIEW. This section presents an overview of the theoretical background of washback research, including the discussion on the nature of washback, and how washback works. Finally, the studies related to the BCET washback effects are reviewed.. 政治大. The Nature of Washback: Positive vs. Negative. 立. ‧ 國. 學. The term washback itself is neutral, which has the potential to become either. ‧. positive or negative (Alderson & Wall, 1993; Bailey, 1996; Hamp-Lyons, 1997;. y. Nat. er. io. sit. Hughes, 2003). For example, if the use of oral interview in a proficiency test encourages learners to practice communication skills, then the effect can be seen as. al. n. iv n C However,h if e thenskill ofhspeaking g c i U is tested only by reading aloud,. positive washback.. and learners overlook other important oral production skill such as conversational exchanges, then its washback effect on learning is limited.. Negative Washback. The undesirable effects of testing on teaching and learning are referred to as negative washback. Negative impact of testing has long been identified as a. 13.

(27) 14. potential problem in language education. From the literature, a number of harmful washback effects have been observed.. They include: narrowing the curriculum and. the educational process (Alderson & Hamp-Lyons, 1996; Cheng, 2005; Madaus, 1988; Shohamy, 2001; Qi, 2005; Spolsky, 1995; Wall & Alderson, 1993), increasing anxiety among teachers and students (Ferman, 2004; Shohamy et al., 1996), additional test-preparation classes or tutorials (in addition to or in place of other language. 政治大. classes) (Alderson & Hamp-Lyons, 1996; Bailey, 1996; Shohamy et al., 1996),. 立. reduced emphasis on skills that require complex thinking or problem-solving. ‧ 國. 學. (Alderson & Hamp-Lyons, 1996; Qi, 2007), and demeaning teachers’ professional. Nat. y. ‧. judgments (Madaus, 1988).. er. io. sit. Another example of negative washback is that testing may lead students to engage in inappropriate learning processes and fail to bring about expected learning. al. n. iv n C h eeducational It is referred to in the literature as test score i U n g c h measurement. outcomes.. pollution (Alderson & Hamp-Lyons, 1996; Haladyna, Nolen, & Hass, 1991).. Test. takers’ scores increase, but their ability being measured in fact does not develop to that extent. Test score gains may be a result of rote memorization of material, instead of acquiring the target construct. This may result from the influences that are irrelevant to the construct of interest, and thus contaminate the interpretations based on test scores..

(28) 15. Considering construct validity, Messick (1996) argued for avoiding two major threats to a test’s validity: construct under-representation and construct-irrelevant variance, both of which may result in the negative washback effects mentioned above. The former occurs when the test content does not adequately reflect the construct as defined for the test. For example, if a test claims to assess communicative competence but does not measure speaking or writing abilities, then it would. 政治大. under-represent the construct of communicative competence. On the other hand,. 立. other than the construct of interest.. 學. ‧ 國. construct-irrelevant variance refers to the variance of test scores that is due to factors For instance, if a communicative language test. ‧. requires test takers to perform decontextualized grammar analyses, teachers might put. y. Nat. al. er. io. competence.. sit. emphasis on irrelevant tasks instead of helping them acquire communicative. n. iv n C U and test formats fail to reflect hthat Many writers have noted e nifgthectest h icontent. the curriculum goals or the course objectives to which they are supposed to be related, then negative washback effects on students’ learning processes and behaviors are likely to occur. Moreover, a test may be seen as lack of validity due to its negative washback effects on teaching and learning..

(29) 16. Positive Washback. Beneficial impact of a test on teaching and learning is referred to as positive washback.. Positive washback has been viewed as an important criterion in the. development and evaluation of language tests in recent years, and a number of suggestions have been made for ways to promote positive washback.. Among these. suggestions, communicative methods of language testing are referred to most frequently.. 政治大. As Bailey (1996) pointed out, one distinctive feature of communicative. 立. language tests is the emphasis on promoting positive washback. Communicative. ‧ 國. 學. language tests are aimed to establish a closer link with the principles of. ‧. communicative language teaching.. In other words, communicative language tests. y. Nat. al. n. precise specifications of the learners’ proficiency. informed about test results.. Ch. engchi. er. io. sit. reflect the language learning goals of the test takers and they are designed based on. v. Moreover, test-takers are better. i Un. Descriptive score reporting, rather than a single score, is. offered so that test-takers would better interpret their performance and problem areas, which can potentially promote the occurrence of washback on test-takers’ subsequent language learning. Swain (1985) proposed “work for washback” as one of the four general principles relevant to the design of communicative language tests (p. 36).. Bailey. (1996), in a review of relevant literature on washback, suggested four areas that foster.

(30) 17. beneficial washback. They are “the incorporation of (1) language learning goals; (2) authenticity; (3) learner autonomy and self-assessment; and (4) detailed score reporting” (p. 268). Similar advice was given by Shohamy (1992) in her diagnostic feedback model for assessing foreign language learning.. In addition to the needs for using. direct methods and authentic language tasks, she argued for involving teachers and. 政治大. administrators in the planning and decision making of the testing process because. 立. they are the people who will have to make changes. Shohamy particularly stressed. ‧ 國. 學. the importance for providing “detailed, innovative, relevant and diagnostic”. ‧. assessment information that addresses “a variety of dimensions rather than being. Nat. er. io. sit. y. collapsed into one general score” (p. 515).. Messick (1996) linked the characteristics of communicative language. al. n. iv n C h e n gthe testing—authenticity and directness—to i U that most likely induce c hproperties positive washback. He considered them as safeguards against two possible threats to a test’s validity: construct under-representation and construct-irrelevant variance. The former leaves out important features of the measured construct; the latter includes features irrelevant to the interpreted construct.. Messick maintained that. tests with positive washback are likely to include tasks which are “authentic and direct samples of the communicative behaviors of listening, speaking, reading and.

(31) 18. writing of the language being learnt” (p. 241). He argued that for a test with beneficial washback, exercises for learning and testing should match to a great extent. Also, learning practices for mastering the language and learning practices for preparing for the test should be similar. Hughes (2003) encouraged test developers, either language teachers in classroom situations or test designers in testing agencies, to work toward achieving. 政治大. positive washback by using the following principles (pp. 53-56):. 立. Test the abilities whose development you want to encourage.. 2.. Sample widely and unpredictably.. 3.. Use direct testing.. 4.. Make testing criterion-referenced.. 5.. Base achievement tests on objectives.. ‧. ‧ 國. 學. 1.. n. er. io. 6.. sit. y. Nat. al. Ch. engchi. i Un. v. Ensure the test is known and understood by students and teachers.. 7. Where necessary, provide assistance to teachers.. As one might have noticed, some of the recommendations listed above, such as direct testing of abilities and teacher training, are costly in terms of time and money, and this might make a test impractical.. Hughes, however, taking practicality into. account, still highly values tests that facilitate beneficial washback.. He contends.

(32) 19. that it is certainly sensible and worthwhile to develop a test having positive washback, given the waste of time and effort in teaching and learning practices that may be inappropriate to the goals of language education. The discussion about achieving positive washback has centered on the inclusion of features of communicative testing in test design and test content, such as designing the test to measure what the language program intends to teach, ensuring. 政治大. that important stakeholders understand the purpose of the test, using authentic tasks,. 立. employing direct testing, and providing detailed reporting of test results.. ‧ 國. 學. The Nature of Washback and Context. ‧. Like most studies conducted in educational contexts, a complex interplay of. sit. y. Nat. io. n. al. er. various contextual factors seems to be involved in the process of generating washback.. i Un. v. Alderson and Wall (1993) maintained that the quality of washback, whether classified. Ch. engchi. as positive or negative, might be independent of the quality of a test. Washback is mediated by complex interrelationships between testing, teaching and learning. Such complexity was highlighted by Messick (1996) as well: “a poor test may be associated with positive effects and a good test with negative effects because of other things that are done or not done in the educational system” (p. 242).. Whether the. effect of testing on teaching and learning is regarded as positive or negative should.

(33) 20. depend largely on where and how it exists and manifests itself within a particular educational context, as most empirical washback studies have demonstrated.. This. understanding implies the importance of unraveling these contextual factors when the researcher attempts to describe and explain washback effects.. The Washback Mechanism. 政治大. In order to investigate washback phenomenon, it is important to turn our. 立. attention to the mechanisms through which washback operates. Three models of. ‧ 國. 學. examining how washback works will be illustrated: (a) Alderson and Wall’s (1993). ‧. Washback Hypothesis; (b) Hughes’ trichotomy (1993, as cited in Bailey, 1996); and (c). Nat. al. n. Alderson and Wall’s Washback Hypothesis. Ch. engchi. er. io. sit. y. Bailey’s (1996) model of washback.. i Un. v. Alderson and Wall (1993), based on the findings of their own washback study in Sri Lanka, focused on micro aspects of teaching and learning influenced by tests. Given the scarcity of empirical research into washback, they put forward a series of washback hypotheses, ranging from the most general to the more specific (pp. 120-121):. 1.. A test will influence teaching..

(34) 21. 2.. A test will influence learning.. 3.. A test will influence what teachers teach; and. 4.. A test will influence how teachers teach; and therefore by extension from 2 above,. 5.. A test will influence what learners learn; and. 6.. A test will influence how learners learn.. 7.. A test will influence the rate and sequence of teaching; and. 8.. A test will influence the rate and sequence of learning.. 9.. A test will influence the degree and depth of teaching; and. 立. 政治大. ‧ 國. 學. 10. A test will influence the degree and depth of learning.. ‧. 11. A test will influence attitudes to the content, method, etc. of teaching and learning.. y. Nat. er. io. sit. 12. Tests that have important consequences will have washback; and conversely, 13. Tests that do not have important consequences will have no washback.. al. n. iv n C h eallnlearners Tests will have washback on i Uteachers. g c hand. 14.. 15. Tests will have washback effects for some learners and some teachers, but not for others.. In summary, a test can influence learning as well as teaching in terms of what to learn and teach, how to learn and teach, the rate and sequence of teaching and learning, the quantity and quality of teaching and learning, and attitudes to the content.

(35) 22. and methodology of teaching and learning.. The Hughes Trichotomy Framework. Hughes (1993, as cited in Bailey, 1996), in an unpublished paper, suggested a model of washback on the basis of a distinction between three elements in an educational system: participants, processes, and products. Participants include students, teachers, administrators, materials writers and publishers, and process is. 政治大. defined as any actions taken by participants which may contribute to the process of. 立. Lastly, product refers to the outcomes and the quality of the learning.. ‧ 國. 學. learning.. Hughes explained his model as follows:. ‧ y. Nat. er. io. sit. The nature of a test may first affect the perceptions and attitudes of the participants toward their teaching and learning tasks. These perceptions and. al. n. iv n C h ewhat attitudes in turn may affect participants i U do in carrying out their work n gthec h (process), including practicing the kind of items that are to be found in the test, which will affect the learning outcomes, the product of that work. (p. 2, as cited in Bailey, 1996, p. 262). For the present study on the BCET washback, it may be assumed that students’ perceptions of learning English may first be influenced by the BCET itself, which in.

(36) 23. turn affects how they are involved in learning activities, and then in turn affects ultimate learning outcomes.. Bailey’s Model of Washback. By connecting Hughes’s (1993) ideas of tripartite distinction and Alderson and Wall’s (1993) list of hypotheses mentioned above, Bailey (1996) proposed a basic model of washback of three major categories: participants, processes, and products.. 政治大. Her model illustrates that a test may affect products, i.e., learning, teaching, materials. 立. ‧ 國. engage in.. 學. and research findings, through the participants and the processes the participants Furthermore, these outcomes may in turn provide feedback to the test. ‧. and bring about changes to the test (see Figure 2.1).. n. er. io. sit. y. Nat. al. Ch. engchi. i Un. v.

(37) 24. PARTICIPANTS. TEST. PROCESSES. PRODUCTS. Students. Learning. Teachers. Teaching. Materials writers and curriculum designers. New materials and new curricula. 政治大. Researchers. 立. Research results. ‧ 國. 學. ‧. Figure 2.1 A Basic Model of Washback (reprinted from Bailey, 1996, p. 264). y. Nat. al. n. products, as dotted lines indicate.. Ch. er. io. sit. The model also allows for possible influences on a test from participants and certain. v. In the context of current study, the BCET may. engchi. i Un. directly affect the students, who, in turn, engage in the processes that will eventually lead to the learning outcomes, including their actual language development in English, and their performance on the BCET. Bailey also categorized the impact of tests into two dimensions: washback to the learners, and washback to the program.. The former refers to the direct impact of. the test on learners as test takers, and the latter refers to the impact on teachers, administrators, curriculum developers, and counselors.. Bailey notes that this.

(38) 25. distinction is consistent with Alderson and Wall’s washback hypotheses mentioned above.. Washback to the learners refers to hypotheses 2, 5, 6, 8, and 10; while. washback to the program addresses those of 1, 3, 4, 7, 9, and 11.. Furthermore, she. particularly points out that washback to the learners is a relatively under-explored area, according to her review of related literature.. Studies on BCET and BCT Washback. 立. 政治大. Two washback studies of the BCET in Taiwan EFL context are found to date.. ‧ 國. 學. One is the doctoral dissertation by Chen (2002), and the other is the master’s thesis by. Nat. y. ‧. Huang (2004). They will be discussed below.. er. io. sit. Chen (2002), using questionnaires and focus group interviews with junior high school English teachers, investigated how the teachers perceived the washback effects. al. n. iv n C U curricular planning and h ethenBCET, of the revised public examination, g c honi their instruction.. It was found that the BCET has considerable effect on teachers, but this. impact is quite superficial; namely, the BCET may affect teachers what to teach, but not how to teach.. It seemed highly probable that changes to the teaching content. were caused by the 1997 revised English mandated textbook based on the Junior High School Curriculum Standards in English education, issued by the MOE in 1994. The study indicated that most junior high school English teachers lack knowledge of.

(39) 26. how to adjust their teaching methods in alignment with the revised curriculum which emphasized communicative approaches to teaching and learning.. This study offers. insights into the BCET washback on teachers and their attitudes at the very beginning of introducing this reformed entrance examination. The other BCET washback study by Huang (2004) explored the BCET washback effects in terms of teaching materials, teaching methods, classroom A set of 82 teacher questionnaire and 351 student 政治大. assessment, and student learning.. 立. questionnaire was distributed across Taiwan. Although the study focused mainly on. ‧ 國. 學. teachers’ perceptions of the BCET in relation to their beliefs and attitudes, students’. ‧. perspectives were also investigated.. Nat. y. These data provided rich information for the. er. io. sit. present study; however, the data gathered might have been more useful if the author had used four-point Likert scale, rather than five-point scale, for the student. al. n. iv n C h e n gindicated The students’ responses c h i Uthat most of them hold neutral. questionnaire.. opinions, probably due to the inclusion of “undecided” options. Also, the wording of some items is biased.. For instance, one of the questions is stated: “The BCET. does not assess my writing ability; therefore, I do less writing exercise” (p. 103). These areas provide valuable insights into the design of the questionnaire for the present study. Two relevant BCT studies which may contribute to the understanding of the.

(40) 27. effects of BCT on teaching and learning are worth mentioning here. Yu, Lai, and Liu’s (2004) national survey of students found that the administration of BCT does not reduce the pressure of entrance exam, nor does it help mitigating the problem of private coaching. The issue of fairness was raised as well, and most respondents suggested that the BCT tends to favor certain groups of test takers. Yu, Lai, and Liu (2005) also conducted a nationwide survey study on the BCT. 政治大. from the points of view of high school administrators and senior teachers.. 立. It was. competencies effectively.. 學. They argue that the results of the BCT do not accurately. indicate students’ academic competence.. Comparing the students’ and the teachers’. Nat. y. ‧. ‧ 國. found that the administrators and teachers doubt that the BCT measures students’ true. High. er. io. sit. views, the researchers found different perceptions of the BCT test format.. school senior teachers and administrators suggest that the BCT should include new. al. n. iv n C h e nwhereas item types, like writing and listening, i U students prefer to maintain g c h most current multiple-choice format.. To conclude, Yu, Lai, and Liu’s (2004, 2005) studies explored the BCT’s effectiveness in terms of its test content and test format, the use of scale scores, and fairness, etc. from both learners’ and teachers’ perspectives.. Although the voices of. students as test takers were heard in these studies, the effect of testing on learning was addressed to a little extent.. As for the BCET washback studies reviewed previously,.

(41) 28. Chen’s (2002) study emphasized teachers’ views and teaching practices, which have been primary research attention in washback studies (Bailey, 1996; Cheng, 2007; Hamp-Lyons, 2000; Wall, 2000).. Huang’s (2004) study tended to focus on teachers’. perceptions, rather than students’ voices, since the number of the student sample was slightly small.. It would seem, therefore, that further investigations are needed in. order to know more about washback effects on students.. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i Un. v.

(42) CHAPTER 3 METHODOLOGY. The current survey study aimed to investigate the washback effect of the BCET on junior high school students in northern Taiwan.. For this purpose, a. questionnaire survey was developed as data collection instrument, since it allowed the. 政治大. researcher to collect a large amount of information from a variety of students and to. 立. In the following sections, the rationale for. selecting participants will be explained first.. 學. ‧ 國. process the data in a more efficient way.. Then the design of questionnaire and. ‧. procedures for collecting and analyzing survey data will be illustrated.. er. io. sit. y. Nat. Selection of the Participants. al. n. iv n C U The target population ish the junior high school students egraduating n g c h ifinal-grade (i.e., 9th-graders), for they are faced with the immediate BCET.. The present study. focused on the 9th-graders in northern Taiwan, which covers Keelung City, Taipei City, Taipei County, Taoyuan County, Hsinchu City, and Hsinchu County. They account for approximately 40% of the 9th-graders all over the country (MOE, 2008, 2009). All the junior high schools and integrated junior-senior high schools in northern Taiwan were further classified according to their locations. Four 29.

(43) 30. administrative levels were employed to select sampled classes:. (a) National level (直轄市) : i.e., Taipei City, which is under the direct governance of the central government of Taiwan; (b) Municipal level (市) : e.g., Keelung City, Hsinchu City, Tucheng City in Taipei County, and Taoyuan City in Taoyuan County; (c) Town level (鎮) : e.g., Jhudong Town in Hsinchu County, and Yangmei Town in Taoyuan County; and. 立. 政治大. ‧ 國. 學. (d) Village level (鄉) : e.g., Shenkeng Township in Taipei County, and Luchu Township in Taoyuan County. ‧ y. Nat. io. sit. Research design of the present study was initially based on school year 2007. al. er. educational statistics (MOE, 2008). According to the MOE statistics (2008), 9th. n. iv n C h e n gfor grade classes at national level accounted i U those at municipal level c h25.74%; accounted for 53.53%; town-level, 8.99%; and village-level, 11.74%. This distribution was an important criterion for sample selection.. The access to classrooms was another important consideration. To obtain a high return rate, the present study recruited the 9th grade English teachers who were willing to help with questionnaire administration in their own classes.. For the. present study, a convenient sample of a proportionate size was selected from each of.

(44) 31. the four administrative levels. Meanwhile, sample size was decided by a general rule of selecting 1% of the population for survey sample (Dörnyei, 2003).. There were 3,850 ninth-grade. classes, and 132,543 ninth-graders in northern Taiwan in school year 2007 (MOE, 2008), so the estimated number of selected classes was 39, and the estimated number of sampled students were 1,325. The final sample size slightly increased to 42. 政治大. classes so as to avoid the case of not having sufficient respondents at town and village. 立. In total, 11 classes were sampled at national level, 22 classes at municipal. 學. ‧ 國. levels.. level, 4 classes at town level, and 5 classes at village level (see Appendix A for the list. ‧. of participating schools).. Among these classes, 29 were from public junior high. y. Nat. io. sit. schools, 10 were from public integrated junior-senior high schools, and 3 were from. er. private integrated junior-senior high schools. The total number of the 9th-graders in. n. al. i n C U hengchi the 42 selected classes was 1,490.. v. Instruments. A questionnaire was developed to gather 9th-graders’ self-reported perceptions of the BCET and their learning. the questionnaire:. The following sources were employed to construct.

(45) 32. (a) Official BCET-related documents (e.g., The Committee of the Basic Competence Test for Junior High School Students, 2007a, 2007b) (b) Relevant empirical studies (e.g., Alderson & Hamp-Lyons, 1996; Cheng, 2005; Ferman, 2004; Huang, 2004; Shohamy et al., 1996; Yu, Lai, & Liu, 2004; Wu & Chang, 2005). Content of the Questionnaire. 政治大 The resulting questionnaire (see Appendix B) had 39 items altogether. 立. The. ‧ 國. 學. content consisted of three parts, and the first part was further divided into three. ‧. categories. The questions were designed and expressed in respondents’ native. sit. y. Nat. language, Mandarin Chinese, to ensure that the meanings of items were fully. n. al. er. io. understood. The organization of the questionnaire is described in Table 3.1 below.. Ch. engchi. Table 3.1 Structure and Categories of the Questionnaire. i Un. v. Structure. Category. Item Number. Part One. Perception of the BCET. 1-4. Motivation to learn English. 5-11. Learning activities and test preparation. 12-20. Part Two. Class activities. 1-15. Part Three. Background information. 1-4. Part One consisted of three categories and 20 items altogether, which dealt.

(46) 33. with (a) students’ perceptions of the BCET; (b) motivation for learning English; and (c) learning activities and test preparation. All of them were closed-response questions designed on a four point Likert scale of agreement.. The respondents’. attitudes and opinions were recorded by having them check on a scale of four categories: Strongly Agree, Agree, Disagree, and Strongly Agree. The response format did not include the neutral option of “neither agree nor disagree”; thus all. 政治大. participants had to take the position that was either for or against.. 立. Four items on students’ perceptions of the BCET in terms of test difficulty,. ‧ 國. 學. test format, and test purpose were designed first on the assumption that learners are. ‧. io. sit. Then seven items were used to explore the factors that may. motivate 9th-graders to learn English.. al. er. Nat. nature of washback.. y. directly affected by the test, and their understanding of the test may influence the. Lastly, nine items were designed to examine. n. iv n C h ebyn asking the participants’ learning processes i Uthey do for learning English g c hwhat outside of regular English language lessons.. Part Two comprised 15 items, which were designed for students to assess how much time their teachers devote to different classroom activities.. The first 14. questions were designed on a four point Likert scale of frequency: Often, Sometimes, Seldom, and Never. The last question was designed as open-ended, allowing participants to answer in their own words, in case there are classroom practices not.

(47) 34. mentioned in the previous items.. It is assumed that classroom contexts are. important areas where student learning is fostered. English teachers’ selection of class practices can be influenced by the BCET, and these teaching activities may, in turn, have impact on student learning. The final part, Part Three, elicited demographic information of the student respondents.. They were also asked to provide English achievement scores of the. 政治大. spring semester, school year 2007; and their English grades of last mock examination.. 立. were placed at the end of the questionnaire.. 學. ‧ 國. A short thank-you note for cooperation, and request for checking blank responses In addition, the respondents were. ‧. invited to volunteer for a follow-up telephone interview, which could be useful for. y. Nat. al. er. io. sit. further data triangulation.. n. Reliability and Validity. Ch. engchi. i Un. v. Regarding reliability and validity of the questionnaire, four important principles were followed so as to maximize its effectiveness: “(1) simple, uncluttered formats; (2) unambiguous, answerable questions; (3) review by several researchers; (4) piloting among a representative sample of the research population” (Mackey & Gass, 2005, p. 96). The survey questions were first examined by three proposal supervisory committee members, who are TESOL experts from the Department of.

(48) 35. English at National Chengchi University, to ensure the appropriateness and clarity of the items. The final version of the questionnaire was in the format of double-sided A4 sheet, with an orderly layout. The questionnaire was then piloted among a small sample to obtain information regarding the relevance and clarity of the items, the format, and the amount of time required to answer the questions. The pilot study was conducted in. 政治大. two classes at a junior high school at Taoyuan County on December 10, 2008.. 立. 學. ‧ 國. Altogether 63 students completed questionnaires during class time. They were given a brief introduction of the study and assurance of confidentiality before filling. ‧. out questionnaires.. It took the respondents approximately 10 minutes to complete. y. Nat. er. io. sit. their questionnaires on their own. Then the researcher interviewed the students about the problems and ambiguity that they encountered in filling up the forms.. al. n. iv n C Group discussion with the two h classes e n lasted g c hfori 30Uminutes respectively.. The. respondents were allowed to react to various aspects of the questionnaire. Based on the feedback from the pilot study, two adjustments were made. One is the addition of telling jokes in Part Two, for several respondents in one class said that their English teacher told jokes in English, and asked them to tell jokes in English, too. Since telling jokes was similar to telling stories in terms of listening/speaking training, telling jokes was added to the item of story-telling. The.

(49) 36. other modified area is an item in Part Three, where students were requested to put down their English achievement score of last semester. At first, the researcher thought that achievement scores were represented by average numbers.. But the. respondents said that their achievement scores were labeled by five rankings: You (優), Jia (甲), Yi (乙), Bing (丙), and Ding (丁).. After confirming this scoring. policy with other English teachers, the researcher rewrote this item to reflect reality.. 政治大. The reliability of questionnaire instrument was examined in the pilot study.. 立. The Cronbach’s  coefficient for Part One items and Part Two items were .75 and .78. ‧ 國. 學. respectively. Altogether, the pilot study questionnaire reached an internal As for the main study, the Cronbach’s  coefficient for Part. ‧. consistency level of .84.. y. Nat. er. io. sit. One items and Part Two items were .87 and .78 respectively. The reliability value for the 34 items overall was .88. According to Dörnyei (2003), acceptable internal. al. n. iv n C h e nisgabove consistency reliability for L2 research c h i.70.U Therefore, the questionnaires used for both the pilot and main studies reached acceptable reliability.. Data Collection Procedures. As mentioned in the decision of selecting participants, questionnaires were distributed by the 9th grade English teachers who volunteered to spare class time for questionnaire administration.. The researcher made phone calls to a number of.

(50) 37. English teachers to invite their participation in November 2008, and asked whether it was appropriate to mail out questionnaires one month later according to school schedule. After obtaining their consent to assist, a survey packet was mailed to each teacher in late December 2008. The contents of packet were: (a) a cover letter to the teacher to explain how to conduct the questionnaire survey, (b) questionnaires with code numbers specifying school and class, (c) a stamped self-addressed envelope, and In the cover letter, the teachers were 政治大. (d) a gift in appreciation of the teacher’s help.. 立. asked to return the completed questionnaire within two weeks, and to follow several. ‧ 國. 學. guidelines, which include informing the students of the purpose of the study prior to. ‧. the questionnaire distribution, and promising them all the responses would be kept. er. io. sit. y. Nat. confidential.. The data for the study were collected before the end of the fall semester,. al. n. iv n C U returned in December 2008 and Most of thehquestionnaires e n g c h iwere. school year 2008.. January 2009, but the data of three classes returned late. One class at municipal level completed questionnaires in winter vacation in February 2009, due to the English teacher’s health problem. Since the school did not require all students to attend supplementary classes in winter vacation, the return rate of that class is 70 percent.. The other two classes at a town-level school did not complete. questionnaires until March 2009 because of tight schedule, according to their English.

(51) 38. teacher. A problem arose from the data collection process regarding confidentiality. One English teacher at a municipal level school asked her students to write down their student numbers on the questionnaires and gave the researcher a copy of the grades of the recent mock examination. According to the teacher, some of her students said that they had forgotten their mock examination scores, so she decided to. 政治大. save the students and the researcher trouble of getting correct scores.. 立. Although this. well-intentioned move might render the data biased, the researcher decided not to. ‧ 國. 學. discard these questionnaires, mainly because most of the respondents had completed. ‧. Part One and Part Two before knowing that they had to write down their student. y. Nat. n. al. er. io. sit. numbers for identification.. Ch. i Un. Data Analysis. engchi. v. All the data were recorded by the researcher personally into computer files. The data were double-checked on hard copy to ensure the accuracy of what was recorded before the actual analyses were undertaken. SPSS (13.0 version in 2004) was used for analyzing the students’ responses.. Descriptive statistics were first. calculated for all the questionnaire items, such as frequencies and percentages, central tendencies, and dispersion, to present general patterns emerging from the data.. If.

(52) 39. necessary, inferential statistics, like multiple means comparisons, was used for the analysis of any other information. Missing values were excluded for calculations, and all percentages were reported as valid percentages with missing data excluded. The significance level for all statistical tests was set at .01. Regarding the students’ semester achievement scores, it should be noted that they were reported in the form of the ranking of five scales, not average numbers.. 政治大. Thus, it would be appropriate to use the median and the mode to represent averages,. 立The former is the result of ranking all the values from high. not the arithmetic mean.. ‧ 國. 學. to low, then choosing the middle value, or the 50th percentile; the latter is the value. ‧. that occurs most often in a set of numbers.. Both statistics were used for analyzing. y. Nat. n. al. er. io. sit. the participants’ semester achievement scores.. Ch. engchi. i Un. v.

(53) CHAPTER 4 RESULTS AND DISCUSSION. Findings of the survey study on how the BCET, a high-stakes public examination, has impacted on student learning in Taiwan junior high schools will be reported in this chapter.. As explained in the previous chapter, for the questionnaires,. 政治大. altogether 42 classes from 22 schools across northern Taiwan were selected by four. 立. administrative levels: national, municipal, town, and village levels. A total of 1,490. ‧ 國. 學. questionnaires were issued, and 1,432 questionnaires were completed, among which. ‧. 1,388 questionnaires were valid for data entry and analysis.. y. Nat. Several reasons may explain this high return rate.. al. er. io. sit. The overall return rate was 96% (1,432 out of the 1,490 questionnaires issued). First, the questionnaire itself is. n. iv n C U of the questionnaire is closely fairly short and easy to fill out. h In e addition, n g c hthei topic related to the students’ immediate concerns. Particularly important is the attitude conveyed by the teachers within their questionnaire administration contexts. Since the teachers give clear instructions to complete the questionnaires and convey potential significance of the results, their students might sense a positive attitude toward the questionnaires, thus promoting their willingness to respond.. The. response rates by administrative level are summarized in Table 4.1 below. 40.

(54) 41. Table 4.1 Questionnaire Return Rates by Administrative Level Administrative Level. Number sent out Number returned Return rate (%) Percent of all questionnaires (%). National. Municipal. Town. Village. Total. 423 410 96.93. 747 710 95.05. 150 144 96.00. 170 168 98.82. 1,490 1,432 96.11. 28.63. 49.58. 10.06. 11.73. 100%. The last row in the table indicates that most of the questionnaires came from. 政治大 municipal level (49.58%) and 28.63% from national level, while 10.06% came from 立. ‧ 國. 學. town level and 11.73% from village level.. ‧. Students’ Characteristics and Their Learning Contexts. sit. y. Nat. n. al. er. io. Prior to presenting the study results, characteristics of the students who. Ch. i Un. v. responded to the questionnaires are described in terms of gender, school types, and administrative level. well.. engchi. Whether they were going to take the BCET will be reported as. Table 4.2 provides demographic information for the study participants and. various elements of their learning contexts..

(55) 42. Table 4.2 Students’ Characteristics and Their Learning Contexts Characteristic. Number of Students Percent (%). Total. Gender Male. 712. 51.82. Female. 662. 48.18. Public school. 1,259. 90.71. Private school. 129. 9.29. 100% (N = 1,388). 68.66. 100%. 31.34. (N = 1,388). 100% (N = 1,374). School Type (1). School Type (2) Junior high school. 立. 政治953 大 435. 學. ‧ 國. Integrated junior-senior high school. Administrative Level of School Location 391. 28.17. Municipal level. 698. Town level. 141. ‧. Village level. 158. n. al. No. y. 10.16 11.38. sit. io. Planning to take the BCT Yes. 50.29 100% (N = 1,388). er. Nat. National level. Ch. 1,257. i e n g c h35. Undecided. 80. i Un. v. 91.62 2.55 5.83. 100% (N = 1,372). Note. N = number of respondents/students.. The distribution of the male and female students was quite balanced.. The majority. of the students (90.71%) were from public schools, only 9.29 % studied in private schools. Two thirds of the participants (68.66%) studied at junior high schools, while nearly one third was from integrated junior-senior high schools.. Regarding.

(56) 43. administrative level, municipal level had the highest portion of the students sampled, whereas town level had the lowest ratio. This distribution is close to that of the 9th-grader population in northern Taiwan. As for the participants’ attendance of the coming BCT in 2009, the majority of the students responded that they were going to take the BCT, and only 8.38% of the students gave “no” or “undecided” responses. Some stated that they would not seek. 政治大. secondary education (i.e., senior high school, senior vocational school, or five-year. 立. junior college), and may enter the job market upon graduation. Still some said that. ‧ 國. 學. they have joined the Practical Technical Program, which offers technical training. ‧. Nat. It should be pointed out that nearly half of. io. sit. schools without submitting a BCT score.. y. courses beginning in the ninth grade, and they are allowed to enroll in vocational. al. er. the “no” and “undecided” responses on this item were given by the cases from two. n. iv n C U that they considered getting h eCity. classes at a private school in Taipei h i stated n g cSome admitted to the original private school without taking the BCT. Given that the study focused on the washback effects imparted by the BCET on learners, the main analyses of questionnaire results excluded those who did not intend to take the BCET.. Consequently, 115 cases were ruled out.. Thus, 1,257. cases of the questionnaire are examined for the following four areas: (a) perceptions of the BCET, (b) learning motivation, (c) learning activities, and (d) class activities..