運用多面向羅許測量模式分析指考和學測翻譯試題

全文

(1)國立臺灣師範大學英語學系碩. 士. 論. 文. Master’s Thesis Department of English National Taiwan Normal University. 運用多面向羅許測量模式分析指考和學測翻譯試題. Applying Many-Facet Rasch Measurement Model to Analyzing Translation Items in the GSAT and AST Tests. 指導教授：曾. 文. 鐽博士. Advisor: Dr. Wen-Ta Tseng 研究生：蘇. 芷. 瑩. Graduate Student: Jhih-Ying Su. 中華民國一百零四年七月 July 2015.

(2) 摘要在台灣的英語教學現況中，「翻譯」不僅列入普通高級中學英文課程綱要中的核心能力，也是大學入學考試中的考試題型之一。因此，教與學當中，翻譯為不容忽視的語言能力。在翻譯試題中，評分者扮演極其重要的關鍵角色，因為評分者可以用分數去評斷考生能力，然而評分包含眾多複雜的因素，像是評分者本身內在學術知識、過往評閱試卷經驗，或是評分者個人特質，皆有可能影響評分的嚴厲或寬鬆程度。因此，本研究應用多面向羅許模式 (Many-Facet Rasch Measurement Model)去檢驗評分者特質(評分者經驗)對於翻譯試題評分的影響程度以及多面向羅許模式如何看出四個因素間(考生能力、評分者嚴厲程度、評分者經驗、考題難易度)的交互作用。參與此研究的受試者為二百二十五名來自北台灣的四所高三學生。研究結果顯示評分者經驗在某些程度的確會影響評分，包含過往批改經驗促進了評分效率、對考生答案更為敏銳與有較大的彈性空間…… 等。然而，評分經驗並不是評分品質優劣的指標，即使是新手評分者，若能仔細詳閱批改說明與標準答案，並且謹慎批改試題，也能提升自身的評分品質。本研究對於在中學擔任英語科老師能有極大的啟發，英語科老師不僅只有教學，也要懂得如何批改翻譯試題。在翻譯考試中，只依循原始分數去評斷學生翻譯能力與翻譯考題難易度並非客觀，若能在施測中，同時使用多面向羅許模式去檢驗四大因素(考生能力、評分者、評分者經驗、考題難易度)，相信不僅能提升教師教學品質、學生學習狀況，更能隨時調整自身的評分狀況。. 關鍵字: 多面向羅許測量模式、翻譯、評分者嚴厲度. i.

(3) ABSTRACT In Taiwan, an EFL context, English is a core component of the national senior high school curriculum, of which translation skill is a key objective. As for senior high school, translation is also a testing method in advanced subject tests and general scholastic ability tests. In translation items, raters play a critical role, because they have to judge rater’s ability by giving them scores. The essence of rating is still a complicated process, including rater’s inner knowledge, prior rating experience, or rater characteristics, and all of these reasons might cause variance in performance ratings, that is, harshness or leniency. Therefore, the current study applies Many-Facet Rasch Measurement Model to examine to what extent do the characteristics of raters, in terms of their experience, affect scores on the translation items and to find out the interactions of the four facets of rater severity, rater experience, test taker proficiency level, and item difficulty. Participants in this study were 225 third-year senior high school students from northern Taiwan. From the result, it may only be surmised that rater experience indeed causes differences in rater severity. But, it is hard to make a strong conclusion as to which group is more severe. Even within groups, there are rating differences. Though two groups of raters have different prior knowledge, given careful adherence to the scoring criteria, experts and novices can reach agreement on item scores. From this study, it is hoped that English teachers can gain some insight. In translation tests, using raw scores is not objective to judge learners’ ability or item difficulty. If teachers can make use of MFRM to examine the relationships between and among the facets of the estimates of trait ability, it can help teachers understand more about students, items, and even himself/herself.. key words: Many-Facet Rasch Measurement Model, translation, rater severity. ii.

(4) Acknowledgements The road of research is like an enduring journey which leads me to enter a profound academic domain and also gives me an opportunity to know more about TESOL. From this process, I find out one thing important is that I should always use a humble heart to learn everything. Now, I am here to write the acknowledgements of my thesis, and I really feel grateful to all the people who had helped me through this research road. I am indebted to the following people who made the birth of this thesis possible. First, I would like to express my deepest gratitude to my advisor, Dr. Wenta Tseng, who guided me through the whole research process patiently. His valuable suggestion and insightful comments enlightened me a lot and his encouragement also helped me overcome the difficulties through the process. His passion and enthusiasm for academic research really makes me feel admirable. Second, I am grateful to the committee members, Dr. Shiping Wang and Dr. Boshi Chen, for giving me their insightful suggestions and critical comments to revise the thesis further. Despite their tight schedule, they read my thesis thoroughly, offered me valuable feedback, and helped me modify the thesis. It was really my pleasure to have these two professors as my committee members. Third, my sincere thanks go to six raters. They rated test items efficiently and correctly in such a short time constraint, without their help, finishing this work of thesis is totally impossible. Also, special thanks also go to senior high school teachers who helped me collect the data. From the short period of time, they efficiently distributed and collected all the tests. Fourth, the constant support and encouragement from my family gives me power to stick to this road. It would not have been possible to finish the thesis without the help and support of them. They are always a strong backup and don’t let me worry about anything; therefore, I can fully concentrate on my studies and job at the same time. iii.

(5) TABLE OF CONTENTS ABSTRACT (CHINESE) .............................................................................................i ABSTRACT (ENGLISH) .......................................................................................... ii ACKNOWLEDGEMENTS ........................................................................................iii TABLE OF CONTENTS ............................................................................................ iv LIST OF TABLES .........................................................................................................v LIST OF FIGURES .......................................................................................................v CHAPTER ONE: INTRODUCTION .........................................................................1 Background of the Study .................................................................................... 1 Significance of the Study and Research Aims ..................................................... 4 Research Questions .............................................................................................. 5 Organization of the Thesis ................................................................................... 6 CHAPTER TWO: LITERATURE REVIEW.............................................................. 7 Translation..............................................................................................................7 Interdependence Hypothesis.................................................................................12 Contrastive Analysis, Conscious Awareness, and Noticing…………………….16 Code-switching, Cross-lingual Transfer...............................................................18 Raters and Writing Assessment............................................................................23 Expert and Novice Raters....................................................................................26 Many-Facet Rasch Measurement Model (MFRM)…………………………..31 CHAPTER THREE: METHOD ................................................................................39 Participants ......................................................................................................... 39 Data Collection Procedures (I)..............................................................................39 The Pilot Test I.....................................................................................................40 The Outcome of Pilot Test I.................................................................................42 The Pilot Test II....................................................................................................42 The Outcome of Pilot Test II................................................................................43 Rewrite Translation Items ...................................................................................43 Data Collection Procedures (II)............................................................................44 Formal Test Administration..................................................................................45 Scoring and Coding..............................................................................................45 Data Analysis........................................................................................................48 CHAPTER FOUR: RESULTS & DISCUSSIONS.................................................... 50 Overview of the Study .........................................................................................50 Major Findings ....................................................................................................50 CHAPTER FIVE: CONCLUSIONS ................................................................68 Summary of Major Findings ...............................................................................68 iv.

(6) Implications ........................................................................................................ 69 Limitations of the Study.......................................................................................71 Directions for Future Research.............................................................................71 REFERENCES .......................................................................................................... 73 Appendix A: The Pilot Test I (A-1) ....................................................................... 81 Appendix B: The Pilot Test I (A-2) ......................................................................83 Appendix C: The Pilot Test I (B-1) ......................................................................85 Appendix D: The Pilot Test I (B-2) ......................................................................87 Appendix E: The Formal Test (A-1) …...................................................................89 Appendix F: The Formal Test (A-2) ......................................................................90 Appendix G: The Judging Plan..................................................................................91. LIST OF TABLES Table 1. The Characteristics of Translation Items from 2009 to 2014........................ 41 Table 2. The Characteristics of Revised Version of Translation Items.........................44 Table 3. The Judging Plan of Six Raters’ Grading.......................................................47 Table 4. Students Measurement Report .......................................................................52 Table 5. The Implication for Measurement of Mean-square Value................................53 Table 6. Raters Measurement Report…........................................................................54 Table 7. Raters’ Experience Measurement Report........................................................55 Table 8. Task Measurement Report .............................................................................55 Table 9. The Interaction Between Items and Experience...............................................58 Table 10. The Grammatical and Syntactic Features of Item 8......................................57 Table 11. The Grammatical and Syntactic Features of Item 2.......................................60 Table 12. The Interaction Between Judges and the Task................................................61 Table 13. The Grammatical and Syntactic Features of Item 1.....................................63 Table 14. The Grammatical and Syntactic Features of Item 10.................................. 64. LIST OF FIGURES Figure 1. Facets Variable Map…………………...........................................................33 Figure 2. Measurement Model for the Writing Assessment..........................................36 Figure 3. The Flow Chart of the Research Procedure of the Study................................45 Figure 4. Facets Variable Map of the Study...................................................................51. v.

(7) CHAPTER ONE. INTRODUCTION. Background of the Study Language learning and teaching involve many aspects related to the four basic skills of listening, speaking, reading, and writing. Among these aspects, translation is an integrative process which needs the combination of language knowledge and language use. According to Liao (2006), translation is an integrative thinking process and can help learners comprehend, remember, and produce a foreign language. In practice, learners’ learning can be facilitated and translation is also commonly employed in Taiwan and other EFL contexts. Translation skills are becoming increasingly important, because it could be used to interact and communicate with others, to mediate and exchange information, and to understand and comprehend others’ thoughts. These language functions are under the framework of communicative competence, and the competence is essential in language learning. According to Hymes (1967), he referred that the definition of communicative competence is the ability that speakers can convey and interpret information and negotiate meanings with others within specific contexts. Therefore, translation skills are basic communicative skills which are necessary in nowadays society. There is another advantage using the translation is that by practicing translation, learners can know more about the language and discover similar and different aspects of linguistic forms (Lado, 1957) and can develop improved translation skills and a deeper knowledge of the target language. By acquiring and contrasting languages, learners can simultaneously strengthen both their native language and second language in a reciprocal procedure. On the one hand, translation helps learners improve their own language system and develop the foundation of communicative competence; on the other hand, based on the first language system, translation can formulate a new 1.

(8) language system of the second language (Coelho, 2008). In this way, translation is beneficial to two language developments reciprocally. During learners’ learning process, translation in language teaching can also facilitate effective teaching and can enable effective learning in the long run. According to Bransford (2000), making use of translation can engage learners’ prior knowledge, integrate practical knowledge with conceptual frameworks, and take effective control over the learning process by using metacognitive strategies. As a result, translation in language learning and teaching is also a necessary process. In practice, translation is not only a learning strategy and skill but also a testing method in advanced subject tests and general scholastic ability tests. Translation test items are examples of performance testing, whose validity depends on criteria used to measure the learner behavior as represented by the produced translation. However, the validity of performance tests are often simultaneously and interactively influenced by multiple facets, such as rater severity, item difficulty, and domain difficulty. From the discussion mentioned above, it is clear that the relationships between testing, teaching, and learning regarding translation are intertwined, especially in testoriented countries, like China, Japan, Korea, and Taiwan. From the literature review, translation has been studied in the contexts of both teaching and learning. In EFL school contexts, given that translation is a common component in English teaching and learning, if translation test items can have a positive washback on English learning, the relationship between learning and teaching can be mutually beneficial. Washback is used to refer to the impact of tests on teaching (McNamara, 1996). Also, a valid procedure for assessing writing assessment must be established in order to have a positive washback for the teaching and learning of writing (Huot, 1996, pp. 549-566). In second language writing assessment, raters need to assess writing skills, and hence there is no avoiding of a certain amount of subjective evaluation. Even with 2.

(9) scoring criteria, raters have to make a judgment on the writers’ performance and score them as fairly and consistently as possible, but there are still many factors which might affect a rater’s grading, such as background, rating severity, training experience, etc. These factors need to be carefully examined to show the extent that these factors affect the rating. Since translation is one form of writing assessment, they also require raters to follow deliberate rating methods and criteria for grading. In addition, in advanced subject tests and general scholastic ability tests, raters also use analytic marking to give test takers scores based on the performance for each writing trait, such as coherence, spelling, and content. Therefore, raters plan a major role in grading translation items. In this study, raters’ experience (novice vs expert) will be examined to see its influence on the rating. In this twenty-first century, the increased interest and support for the use of one’s own language has been proved in many foreign language studies. Also, this thesis will argue for the role of translation and its educational value from the relevant literature and will also try to synthesize the relevant concepts to prove that translation is not only ubiquitous but also inevitable in international affairs, educational domains, or even school teaching. The educational value and the role of translation will be examined and to this end, the Many-Facet Rasch Measurement Model (MFRM) will be used to examine the quality of the test items and to find out to what extent the experience of raters (novice vs expert) affects their ratings (Linacre, 1996). According to Cook (2010), the term “native” language is not precise enough to be used, because language of infancy, expertise, and identity do not necessarily go together (Rampton, 1990). In addition, “mother” tongue is also inaccurate because many people’s mother tongue is not their mother’s mother tongue. Hence, in this thesis, the term “own language” and “first language” would be used to mention leaners’ native language. As for leaner’s second language, the term “target” language and “second” 3.

(10) language would be used.. Significance of the Study and Research Aims This study will first survey the literature on relevant concepts of translation, writing assessment, raters in academic writing, and the Many-Facet Rasch Measurement Model, and considering issues such as rater bias, rater background, and rating quality. The reason for the focus on translation items is that few studies focus on translation items and use of MFRM. As a result, this study will examine translation items using MFRM in order to determine the relationships between test taker ability, the difficulty of items, and rater experience. By using MFRM as a fit statistics analysis, factors which might affect test taker scores can be minimized so that a more accurate evaluation can be more efficiently obtained. If MFRM, in this study, could successfully examine translation items, to some extent, MFRM could be widely applied to examine various factors that might negatively affect an examinee’s scores in many different academic domains. In this thesis, four facets will be simultaneously examined, rater severity, examinees, items, rater expertise, and the grammatical features of each translation item. From the calibration of a Rasch map, the relationship of these facets can be demonstrated and displayed, which will allow us to have a clear picture of the interaction among different factors. From 1979 to 2008, translation items in college entrance exam had been examined by some researchers. Based on Zhou (1996), he gave a theoretical review on translation items for Joint College Entrance Examinations from 1979 to 1994, including difficulty, length, scores, and development of items. Pan (2001) also put her focus on the development of test types from 1979 to 1999, especially on topics of meaningconsecutive sentences and verb forms. Liou (2009) studied contents of translation items from 1998 to 2008, including percentage, tense, voice, and sentence structures, and 4.

(11) gave a clear explanation on translation items in recent years. From these studies mentioned above, items from 1979 to 2008 had been studied. Therefore, In this study, test items in advanced subject tests and general scholastic ability tests from 2009 to 2014 would be used to fill the research gap. Therefore, this current study can fill the research gap and the findings shown here could be applied in the domains of SLA and language testing. The anticipated aim of this study would be divided into two aspects, one is for the teaching domain, and the other is for the testing. In the teaching domain, the importance of translation training in the formal education would be highlighted in this study. Because from the literature review, it is clear that the translation is a helpful method and a learning strategy both on teaching and learning. In the testing domain, the study shows that MFRM can be widely applied to different academic domains; therefore, it is a useful tool for testing. MFRM can convert raw scores into a more linear and measureable measurement. Also, MFRM can put more testing aspects into considerations to decide test takers’ true ability more accurately. If learners’ ability could be effectively shown by this statistical method, we could have more information on how to teach students much more properly.. Research Questions (1) How does MFRM show the interactions of the four facets of rater severity, rater grading experience, test taker’s proficiency level, and item difficulty (in terms of grammatical features)? (2) To what extent do the characteristics of raters, in terms of their experience, affect scores on the translation items?. 5.

(12) Organization of the Thesis The thesis contains six chapters and is organized as follows. Chapter one provides the introduction and backgrounds of the study. Chapter two reviews important concepts in translation, second language assessment, theories of measurement, and expertise and novice raters. It discusses the general view of translation from educational and pedagogical perspectives. Chapter three describes the methodology, and includes the method, participants, data collection, and data analysis procedures. In chapter four, the results and findings are presented, and chapter five discusses the major findings. Chapter six concludes the study with its major findings, implications, limitations, and suggestions for future research.. 6.

(13) CHAPTER TWO. LITERATURE REVIEW. Translation Translation is still widely used in many contexts and common in classroom situation (Benson, 2000). Learning and teaching are like a coin of two sides, with each one complementing the other. Translation can be used as an instructor’s teaching technique or learning activity in the course and as a leaner’s learning strategy during his or her learning process. Cook (2010, p. 16) pointed out that translation “is a living, moving activity” which integrates the language teaching and learning process as a whole. The definition of translation is that from words, phrases, to sentences, speakers or readers convert the target language expression into the native language or to alter the native language into the target language (Oxford, 1990). Likewise, Chamot (1987, p. 77) described the translation as “using the first language as a base for understanding and/or producing the second language.” These definitions of translation suggest that second language learners use their first language as a basis to facilitate the target language learning from the word level to the discourse level, even though there will be obvious similarities or differences between the two languages. Through translation, learners can make use of their prior knowledge, the first language, to convert the two languages. Based on the theorizing of communicative competence, by using leaners’ first language to translate, translation enables students to powerfully and authentically communicate with various people in different contexts and in both languages (Cummins, 2007). Dickson (1992) also observed that to attain the goal of converting input to intake, learners need to interact with the language and to learn how to use the language in real situations. As a result, second language input without proper interaction and comprehensive understanding cannot be successfully converted into intake. Therefore, 7.

(14) to draw upon students’ first language to help them communicate in the target language is reasonable, but to ignore language learners’ own language to learn a brand new language is unreasonable. Learners’ first language plays a crucial role in the process of language learning. If the teacher only uses the second language in the classroom, this does not guarantee successful learning in the long run. Translation can be utilized to help learners communicate and interact with people in different contexts. Based on Fromkin (2013), the essence of knowing a language is to have the ability to create and understand numerous sentences which have never been spoken or heard before. This kind of creative aspect of language use can be applied to translation. Translation also can help speakers understand numerous sentences which have never been spoken or heard before. Logically, in the authentic interaction among people of two different first languages, translation could be applied in various contexts. A two-language-transfer process is the essence of the real communication and it is this dynamism that makes translation so interesting and so stimulating (Cook, 2010). In the classroom context of translation processes ranging from word-to-word translations to paraphrases or interpretations, learners bring their own language to the classroom to acquire the target language. By engaging in this dynamic process, learners actually come to realize what the essence of knowing a language is. When leaners become aware of the meaning of the communication and what language is, learning a target language should be more effective (Schmidt, 1990). In this sense, translation can be seen as the embodiment of real communication in an authentic context, and this strategy can lead to a more successful learning. In addition, to achieve the goal of effective learning, translation may also aid memory retention during the language learning process (Källkvist, 2004). As Laufer and Girsai (2008) also pointed out translation is a good learning strategy especially for vocabulary learning. In language teaching and learning process, the use of translation can also make 8.

(15) teaching more effective, which can in turn lead to more effective learning in the long run. According to Bransford (2000), the definition of effective learning is for learners to know how to simultaneously make use of prior knowledge and integrate practical experience with theoretical frameworks in order to control their own learning process by metacognition. Therefore, the teacher uses students’ first language via translation so as to activate not only learners’ prior knowledge of language world, but also cognitive thinking processes. Based on Cook (2010), learning a new language is a two-way process, in which teachers make use of the learners’ own language to help them understand and learn the target language. Neglecting the use of students’ first language in the classroom makes language learning a one-way process, and this neither facilitates efficient learning nor reduces linguistic power differences. The language-only perspective is normally described as the realization of language power or nationalism (Phillipson, 1992; Cook, 2010). From these arguments, effective learning would be facilitated by translation. If an effective learning could be attained, the goal of second language acquisition could also be achieved more easily. In Sridhar and Sridhar’s (1986, p. 5) words, ideally, the goal of second language acquisition is bilingualism. From this perspective, it is untenable to ignore the importance of translation. Cook (2007) also mentioned that bilingualism is the ideal goal of second language acquisition. He observed that bilinguals are more successful in choosing between competing stimuli, and this is because they know how to plan and focus, and how to avoid distractions. Though the response times of bilinguals may sometimes be slower in these psycholinguistic tests (Cook, 2007), for language learners, appropriate translation practice can provide the opportunity to slow down and carefully consider what they have been exposed to and then try to formulate this in words. This process can be beneficial for learners to produce better output in the target language. Besides, as Pavlenko and Lantolf (2000) noted, the 9.

(16) relationship between a first and second language is more accurately one of “participation” rather than only acquisition. The successful language learner needs to build up a bilingual identity, and translation can be an effective way to foster the participatory language relationship necessary for implementing this. Scholars have theorized that translation practices own a number of manifest educational justifications. First, translation in language teaching is a means rather than an end. In multilingual and multicultural societies, there are more chances for people to be exposed to different languages. As a result, translation is widely used in everyday life because of daily needs, and not for a specialized activity at all (Cook, 2007). In a world of cross-cultural communication, translation is a constant mediation between languages. According to Catford (1965, p. 20), “translation replaces textual material in one language by equivalent textual material in another language.” A translator is therefore like a bilingual agent who can mediate communication between two different language communities (House, 1977). From everyday communication to practical needs, learning a language is a useful tool for not just individuals but also societies to achieve linguistic aims and to fulfill practical work-related needs. In this light, Cook pointed out that “Societies need individuals with certain skills in order to function and conversely individuals need these same skills in order to gain and succeed in employment” (Cook, 2007, p. 111). Given that employment is an important means of personal fulfillment and development in life, if formal educational curriculum neglects the significance and training in translation, both the individual learning translation of students and the progress of society are likely to be. A second reason to give translation a strong justification is that translation is one of the foundations of peacemaking and understanding between nations. Although Goethe (1982, p. 353) said these words over 20 years ago, they are as true today as back 10.

(17) then, “Say what one will of the inadequacy of translation, it remains one of the most important and worthiest concerns in the totality of world affairs.” Through translation, languages are understood among different countries or communities, and through the awareness of differences and similarities between languages, mutual understanding increases and tensions and misunderstandings decrease between countries (Cameron, 2000). Also, translation in language teaching can preserve the presence of the students’ own language identity in learning, though political realities may cause inequalities. However, translation practices in language teaching can be deployed to help preserve the identities of speakers of minority languages (Cook, 2007). Translation can be used reduce language inequality and promote more balanced language identities. As a result, translation including in teaching can help learners not only build a new language identity but also to preserve their own identity. Third, when the translation is appropriately used in academic education, it enables the analysis of any languages and a deeper understanding of the nature of language and language use (Cook, 2010). From this process, students are not passive receivers, but rather, active converters of input into output that demands the simultaneous understanding of two languages. Translation thus develops learners’ critical and independent thinking by the analysis and the meta-cognition of two languages, which in turn clarifies language knowledge (Cook, 2010). Based on the reasons mentioned above, translation practices can be seen as having strong educational justifications and should be included in the teaching in the formal school context. Besides, translation in testing is also needed. The issue of washback is important in criterion-referenced language tests, which includes translation items (Brown & Hudson, 2002). As a result, careful design of tests, items and rating procedures are essential for the positive washback on learning, teaching, and also curriculum design. 11.

(18) In addition, the conventional way of viewing the role of translation in second language classroom is to connect the Grammar Translation Method (GTM) and translation together. Since the GTM has long been downgraded and criticized, translation is also criticized by many linguists. However, contemporary applied linguists argued that there is a role for translation in the ESL classroom, e.g., Cook (2010) and Catford (1965). Actually, translation and GTM are different in essence. Translation, on the one hand, is inherent in both teachers’ and students’ innate ability and is manifest in a variety of contexts. GTM, on the other hand, is mostly used in the classroom environment which traditionally and exclusively tends to focus on discrete grammar rules and grammatical accuracy and writing ability development. However, translation refers to a broader linguistic process that involves whole language use instead of just discrete language forms only. Therefore, it is inaccurate to simply conflate translation with the GTM. Therefore, to use disadvantages of GTM to evaluate translation isn’t fair as well. Cook (2010) also highlighted the importance of the concept of componential learning as applied to language learning. To achieve this learning or atomization, learners have to first acquire discrete skills separately and then integrated them. Cook (2010) claimed that translation is like a process of learning components, and that learners gradually attain the ability of holistic language use. To some extent, explicit grammatical explanation is still supported in componential learning and is also applicable to the teaching of translation.. Interdependence hypothesis Cummins developed The Linguistic Interdependence Hypothesis in 1979, and he mentioned that when leaners are in the second language acquisition process, they can receive certain positive transfer from their first language knowledge. The first language 12.

(19) knowledge is like a schema, which can be conducive to the development of second language. Therefore, learners’ own language and target language are interdependent and complement with each other. According to the schema-theoretic approaches proposed by Nassaji (2002), schema is the pre-existing knowledge stored in mind that can help people understand external phenomena or information. In second language acquisition, students use their own language to function as a prior knowledge, a schema or a scaffolding, to acquire the second language. It is natural that language learners use their first language to think about or translate the second language. In this way, students can have a better understanding between two languages and promote acquisition in the long run. Coelho (2006) also argues that to ensure producing a better second language ability, having a strong basis in the first language is the first step. There is thus a necessary interdependence between first and second languages. Likely, Cummins (1979) also proposed that bilingualism can be more likely achieved cognitively and academically on the basis of well-developed first language skills. In the developmental interdependence hypothesis, he remarked that the development of the first language plays a facilitative role in second language learning. The second language acquisition can be built upon the basis of well-developed first language. Donovan and Bransford (2005) also claimed that a foundation of understandings and experiences can lead to new comprehension. Hence, the first language is an important existing understanding and experience to help develop the target language system. Learners make an effort to understand the second language by their first language. Under the communicative language teaching framework, in order to improve the efficiency of first language and second language interaction, the students’ own language should be used. To interact with others, negotiate the meaning, and exchange the information in a classroom, the native language plays a crucial role to improve these 13.

(20) skills. Translation practice can improve learners’ social interaction and facilitate successful communication. As a result, making proper use of a learners’ first language is a good way to learn a target language, and translation can be a means to this end. From the process of translation, it can improve learners’ second language use to communicate through the first language. In Anton and DiCamilla’s (1999) words, the first language is a tool to provide learners a social and cognitive guidance to benefit their learning throughout the task and the benefit can be extended into the real life situation gradually. When the target language is extended into the real life situation, translation is considered to be a good means to provide opportunities for the interaction and mediation between first and second languages in different contexts within a communicative language framework. That is to say, from the learning tasks accomplishing in a classroom, learners should be able to extend the learning experience to a real context. With the adequate exposure to the other language, learners can be more possible to acquire a new language. When there is adequate exposure to two languages, instruction in one language can promote proficiency in that language itself, and through the same process, no matter whether in a school or real life context, this instruction can arouse learning effect for the second language. At this given moment, two languages are intertwined together, because two of them are complemented by each other. Also, bilinguals attain their high second language competence through intensive exposure, and this ability is similar with learners have in their native language (Dickson, 1992). In other words, to learn a language well, having enough exposure is imperative, not only in the school context but also in the linguistic environment outside of the school, which can contribute to higher levels of second language competence without sacrificing the native language competence. With enough exposure to the target language, it is clear that the first language and the second language are equally 14.

(21) important in the process of second language learning process. Another echoing concept of interdependence hypothesis is the instruction of Focus on Form. Focus on Form generally refers to the simultaneous attention which is brought to both meaning and how the meaning is encoded (VanPatten & Benati, 2010). To some extent translation can be likened to the instruction of Focus on Form, which Lambert (1974) referred to as a means to focus learners’ attention on not only linguistic elements, but also on how these resources interact with each other. In addition, translation promotes a focus on form as a beginning of a communicative need (Cook, 2010). Therefore, translation can be functioned as a integration of the second language’s linguistic elements and to promote learners’ understanding of these elements. Moreover, the meaning of translation in the classroom includes not only translating spoken language into spoken language but also translating written text into spoken language, such as the interaction about texts during conversations between teachers and students. Lemke (1989) pointed out that when learners engage in understanding a written text, they have the opportunity to paraphrase, restate, and translate its meanings into several patterns of spoken language. Written and spoken modalities interact with each other by means of translation tasks and this kind of interaction is common in classrooms where learners can verify their own understanding by communicating with teachers and peers. Therefore, trying to understand how to use a language and what the written translation task activates the knowledge of both spoken texts and written texts at the same time. As a result, students’ own language and target language are connected together even though both of them have different text types. To sum up, the relationship between learners’ own language and target language is a kind of interdependence. From the beginning of language learning, the first language already provides the learners’ schema that serves as the basis to facilitate second language learning. Especially in the real life situation outside the classroom, 15.

(22) translation is also essential under the framework of communicative society. Also, to convert written texts into spoken texts by the other language is also a concept of interdependence. From the understanding of two different text types, the language knowledge can be truly attained.. Contrastive Analysis, Conscious Awareness, and Noticing The Contrastive Analysis Hypothesis predicted acquisition difficulties. In essence, the hypothesis claimed that when there were similarities between first language and target language, acquisition would be easier than those with the most differences (Lado, 1957). There are always surface aspects that are shared between languages, though there are also underlying cognitive and academic differences (Cummins, 2007). Through the comparison between similarities and differences, which involves human’s conscious awareness, students are expected to generate and make connections of the first and the target language. Contrastive analysis and conscious awareness are helpful for learners to contrast two languages and then construct two language systems. Conscious attention directed to material to be learned is crucial for learning, and through translation, learners are forced to focus on the material which they are not fully familiar with. Also, attention is related to conscious experience, insofar as when people focus on something, they are aware of it (Baars, 1988). As a result, through consciousness-raising instruction, learners start to focus on what they’re not familiar with and to improve their language ability and increase their learning autonomy. In terms of conscious awareness, the allocation of attention helps convert the object or stimulus of attention from the shortterm to long-term memory (Logan, 1988), thereby lengthening the retention of learning. Therefore, through translation, which is a consciousness-raising instruction, it can arouse learners’ conscious attention and improve their learning efficacy. 16.

(23) Also, noticing is the other innate ability which is relevant to the conscious awareness. Noticing is both necessary and “highly student-centered in nature” (Amirian & Abbasi, 2014) to help learners transfer the input into intake, so that they can one day produce natural and fluent output. According to the Noticing Hypothesis proposed by Schmidt (1990), conscious noticing of linguistic forms and meanings is necessary to convert input into intake. A further extension of the noticing hypothesis is that “what must be attended to and noticed is not just the input in a global sense but whatever features of the input are relevant for the target system” (Schmidt, 1990). As a result, the feature of the input which is relevant to language learning should be included in the consciousness-raising instruction, just as translation. Translation can help students notice the differences and similarities of two languages and can increase their awareness and to facilitate the transfer of language. When language learners are aware of language differences, their innate ability to convert two languages will be activated, i.e., they will translate the second language into their native language. Lambert (1972) mentioned that translation is “learnercentered in a radical, psycholinguistic sense: it respects learners’ internal syllabus.” As Lörscher (2012) proposes, translation competence is the rudimentary and innate capacity in the human intellect to mediate languages, and includes the sub-abilities to categorize, compare, and differentiate similarities and dissimilarities. These subabilities enables translation to “express sense and/or connections of signs in different ways” (Lörscher, 2012). As learners translate unfamiliar second language into their own language, their innate ability will be activated to categorize and compare two languages to help them acquire the language gradually. Translation can thus be seen as a kind of innate linguistic ability which can help learners produce translation equivalences. According to the tabulation of translation equivalence proposed by Catford (1965, p. 39), translation is fundamentally a choice about what to include and what to exclude. 17.

(24) Therefore, translators have to be thoughtful and deliberate about pragmatic, functional, discoursal, and cultural equivalences between languages (Cook, 2010). Some studies indicate that across unusually different languages and cultures, equivalent translations can still be achieved (Brislin, 1970; Malakoff & Hakuta, 1991). By translating in a realistic context or within specific situations can raise students’ conscious awareness by noticing similar and different properties of languages, and this can have a significant effect on language learning. From tasks of translation, learners can distinguish rules and formulate their own to improve their familiarity toward a new language and internalize the target language into their own brain ( Amirian & Abbasi, 2014). Language learning is a process for learners to actively interact with the language from the input they have been exposed to in real life or in the classroom. Students’ use of translation can help them use the language more consciously and become more aware the differences of first and second languages.. Code-switching, Cross-lingual Transfer In an EFL or ESL context, code-switching is prevalent and common in both teachers’ teaching and students’ learning. Code-switching can be defined as the use of the first language in the second language utterance, and this phenomenon frequently occurs in bilingual and immigrant contexts. Just as Belz’s (2002, p. 61) words, codeswitching refers to “the use of more than one language or code in a single stretch of discourse.” Hagen (1992) even justified code-switching as being a fundamental language skill that needs to be developed because code-switching is a normal part of interacting in multilingual situations. As Stern (1992, p. 282) pointed out, “The L1-L2 connection is an indispensable fact of life”, and learning a new language through students’ existing language system can help the learning process to resemble the acquisition process of the mother language 18.

(25) (Celik, 2003). Cook also argued that excluding the first language in language teaching goes against the natural process in students’ minds, and for this reason, to make sure that learners can learn a language cognitively, emotionally, or socially, the second language should not be isolated (Cook, 2001). Translation makes use of learners’ schema, i.e., first language, to acquire a second language, and this process is what the code-switching refers to. It has also been recognized that in the teaching environment, teachers often use written text in the target language to teach spoken language to learners. Therefore, to translate written texts into speakers’ language is important for language learners. As Lemke (1989, p. 136) pointed out, “When we approach written text, we need to be able to do more than just decode letters to sounds.” We should have the capacity to infer the meaning from the written text and then find the equivalence of the target language, which is similar to the translation. Teachers and students translate written texts of the second language into spoken texts of the first language, which is a process that should be encouraged. Also, teachers usually have reasons for using code-switching. Studies show that code-switching has different aims in the language teaching, and these can be divided into cognitive, affective, and classroom managing domains (Samar & Moradkhani, 2014). According to Samar and Moradkhani (2014), the first aim could be discussed from the cognitive domain. Improving and checking students’ comprehension and avoidance of breakdowns in communication are two frequently cited reasons for using the first language. By explaining, elaborating, paraphrasing, and reiterating, teachers can evaluate students’ understanding by using students’ native language. To further improve and monitor students’ comprehension, teachers can refine the meanings of their language by providing contextualization cues and to check students’ 19.

(26) understanding. Furthermore, when teaching grammar or sentence structure, technical and abstract concepts can be explained in students’ first language in order to achieve the simultaneous goals of explanation and understanding (Samar & Moradkhani, 2014). Since it is more complicated and difficult to explain terminology in the target language, at this given time, code-switching can be observed in the classroom. It is likely that some tasks and technical terms need more elaboration from teachers (Kim & Elder, 2005), and for this reason teachers might avoid using the target language so that students’ comprehension will not be impeded (Samar & Moradkhani, 2014). Another common functional use of the code-switching is for comparing and contrasting the first language and the target language, also known as the concept of Contrastive Analysis Hypothesis which maintains that students can learn better when they are conscious of the similarities and differences between the first language and the target language (James, 1981). Stern (1992) pointed out because the first language can serve as a reference to help learners recognize differences and similarities between first and second languages, the status of the first language should not be denigrated. By comparing the two languages, translation can also be used to predict areas of difficulty that a language learner might encounter (Eckman, 1977) which Lado (1957) had in mind with his Contrastive Analysis Hypothesis that can be used to predict learners’ errors and to select items to teach. According to this logic, the new language will be easier to learn when the linguistic elements are similar to the learners’ native language and more difficult to learn when the elements are different (Lado, 1957). When teachers highlight the differences between two languages, therefore, they help students overcome difficulties and increase their retention of the target language material (Samar & Moradkhani, 2014). To avoid breakdowns in communication, when moving on to the next section or going back to the course, translation can help students pay attention to the course again. 20.

(27) And as a form of code-switching, translation can help learners compensate for communication breakdowns (Macaro, 2001) and overcome comprehension problems. Regarding the second of the four domains, the affective domain, code-switching has positive effects on establishing the student’s identity and stabilizing the learner’s emotions. Many scholars maintain that code-switching can promote students’ learning motivation and success as well as being able to facilitate a sense of class unity and shared identity (Addendorff, 1996; Camilleri, 1996; Dornyei & Murphey, 2003) by creating the second language identity. “Translation is an integral part of creating dual language identity texts. Translation also plays an important role in enabling bilingual and newcomer students to participate actively in instruction” (Cook, 2010). Therefore, translation, part of code-switching process is an important teaching and learning resource in the classroom. By providing learners a sense of comfort zone, proper use of translation can help them ease their learning anxiety. Translation can also encourage students to consider themselves as true language users in both languages (Cook, 2001). Levine found that when students’ used more of their own language, they felt less anxious about the new language (Levine, 2003). Code-switching can thus ease students’ in-class anxieties, which is especially essential for low-proficiency level students. As Krashen realized long ago when students learn a new language with high level of motivation and self-confidence, a good self-ego, and a lower anxiety, they will be more likely successful in second language acquisition. Giving students’ assistance in their own language can help them “feel solidarity and overcome the stressful situations” (Samar & Moradkhani, 2014) which usually happen in the second language classroom. Mitchell (1988) similarly observed that using the new language all the time is too overwhelming for lower level students. When the students’ proficiency level are lower, it is common for the teacher to predominantly use the students’ own language (Duff & Polio, 1990). And this can establish a genuine and friendly relationship with students 21.

(28) (Lin, 1996). The third, function of code-switching is classroom management. Teachers usually employ the first language to discipline students or show disapproval of their behavior (Samar & Moradkhani, 2014) to maintain discipline (Cook, 2001) and to keep control of the group (Levine, 2003).When teachers undertake classroom management, the native language is the most efficient way to show the teachers’ teaching pause and to change their focus on students’ behaviors. Not only can the first language be used to establish classroom rules, but it can also be used to praise students and to establish a cozy and friendly atmosphere (Lin, 1996). The appropriate deployment of translation strategies can give students encouragement, praise, or reproof to individual pupils (Gumperz, 1982). Lastly, but very importantly, when learners are outside of the classroom, do they employ translation during self-study with bilingual dictionaries or when watching second language films with the first language subtitles? If they do, translation is useful in learning at any time, and it is common for them to exploit the first language or codeswitching to process the meanings for the second language (Cook, 2001). In all, translation is a moving activity which integrates the language teaching and learning process as a whole. Especially for second language learners, leaners can draw upon students’ first language, which is their innate ability, to help them communicate in the target language. Through translation, learners can convert two languages and improve learning in the long run. Translation can not only improve a successful learning but also facilitate an effective teaching. Teachers make use of translation to help learners notice similarities and differences of two languages. Based on Interdependence Hypothesis, first language knowledge can be positively transferred during the process of second language acquisition. Teachers’ teaching can arouses leaners’ conscious awareness to 22.

(29) differentiate similarities and differences of two languages. By the process of noticing, leaners activate the mechanism of code-switching to help them learning. In Taiwan, translation is a testing method in advanced subject tests and general scholastic ability tests, which are two high-skate tests. When the translation becomes a testing method, the rater issue becomes very important.. Raters and Writing Assessment Translation is not only a learning strategy or a teaching technique but also mode of testing. Translation test items are a kind of writing assessment, which can show the test takers’ writing performance and their knowledge of the target language. In translation items, candidates have to read the sentences written by the native language first, and then translate them into the target language. Although the ideal translation refers to “the same events in the real world as the original”, it may be impossible to produce something in one language which means exactly the same as something in another language (Cook, 2011). Since there might be some gaps between two languages’ translation conversion, a good translator should know how to identify the essential meaning of one language and express it into another language. In Taiwan, translation is not only a learning strategy but also a test item on advanced subject tests as well as general scholastic ability tests. Over 20 years ago, Buck showed that translation can be one kind of test item that was satisfactorily reliable and valid (Buck, 1992). As a result, translation items are also included in high-stake exams in Taiwan, and in a large scale assessment, students’ ability must be determined through technical and statistical rigor. Hence, the development of scoring guidelines or rubrics, the training of raters, and the test takers’ ability are crucial factors in the performance assessment. Brown and Hudson (1998) proposed that there are three test main item types on 23.

(30) language tests: selected response, constructed response, and personal response. In terms of definitions, constructed-response items are those in which a student has to actually produce language by four skills of a language, including writing or speaking, rather than answering one-on-one answer only. Here, the three sub-types of constructedresponse items are fill-in, short answer, and performance and they can show students’ receptive and productive knowledge simultaneously while minimizing most of the guessing risk. Writing assessment, such as essay writing, translation items, or composition, therefore belongs to the type of the constructed response and especially in the sub-type of the performance assessment. In the performance assessment, the test items design may include real life contexts and candidates have to integrate their language knowledge to examine their true language ability (Brown & Hudson, 1998). And by the definition of performance assessment, examinees have to perform some tasks that should be as authentic as possible, and candidates’ performances should be rated by qualified raters (Brown, Hudson, Norris, & Bonk, 2001). Therefore, in the writing assessment, candidates generate appropriate content by making use of the knowledge they have in their knowledge base. Again, performance assessment includes complex human performance and involves acts of interpretation from test takers’ responses. For this reason, to reduce the disagreement of raters these interpretation to acceptable levels is the main goal in the performance assessment (McNamara, 1996). Writing assessment, belonging to the performance assessment, is a type of assessment which needs to take many dimensions into consideration. Researchers recognize that rater judgments include rater’s subjective opinion, so even the same writer would vary his judgements when correcting different items. Many studies have showed that the assessment of linguistic performance may be influenced by various factors with the scoring procedure and the traits of the writing samples to be scored, such as content, organization, sentence 24.

(31) structure, or genre, might influence the judgment made by raters (Schoonen, 2005; Weigle, 2007; Cooper, 1984). Because raters play a crucially important role in the writing assessment and because raters need to temper the subjectivity of their evaluations, rater performance requires an empirical examination. Based on the arguments mentioned above, in the testing process, there are three main factors of raters, candidates, and test items from a continuum. From one end of this continuum, test takers’ writing samples can indicate items’ complexity and candidates’ ability. From the other end, a standardized and reliable measurement is used as a rating basis. Raters lie in the pivotal position of this continuum to turn performances into outcomes. However, a number of elements might simultaneously cause variability in writing scores. Rater characteristics seem to affect raters grading as well, such as rating experience, the assessment process knowledge, familiarity with the rating criteria, or the amount of rater training (Kuiken & Vedder 2014). Similarly, task characteristics and types of assignment can also influence scores. Shaw and Weir (2007) noted that in order to make sure the tasks are obviously valid in all other aspects, scoring validity is an important guide. Since the rating of exams should be examined for a valid scoring, the rater is the main focus in the thesis. Raters have to be able to give scores appropriately and consistently, and therefore, rating scales are used to improve grading’ consistency. However, rating is a complex issue. Even if there are rating criteria and rater training, the essence of rating is still a complicated cognitive process which might lead to variance in performance ratings, so there is a need to study rater effects (Myford & Wolfe, 2004). Raters can bring the biases of raters to exams, and McNamara listed three factors which might cause differences in raters’ grading, rater severity, rater characteristics, and rater consistency. First, the interaction between raters, items, and candidates may be either severe or lenient. Linacre (1989) used the term severity to refer both to the overall 25.

(32) severity of the rater and the difference between raters in the way they evaluate constructed responses. There may also be some interactions among raters or some other aspects of the rating situation, like rater-item interaction and rater-candidate interaction. In addition, raters may interpret rating scales in different ways, which can lead to different grading rationales. McNamara and Adams pointed out that differences of interpreting rating scales might result in two systematic ways of grading writing assessment, one is a centralizing tendency and the other is avoiding scores in the middle of the scale (McNamara & Adams, 1991). Another potential source of differing interpretations involve different rater characteristics, such as gender, background, or even the rating time of a day. A third source of difference involves the rater’s own consistency. The extent of random error associated with their ratings might change according time, sometimes being harsher and sometimes being more lenient. In short, it is a challenge to maintain the consistency of rater scoring for writing assessment. Many studies have made an effort to eliminate interrater problems and to try to improve scoring validity. Rater training is one way to solve these problems and to show scoring validity. Training typically includes “familiarization activities, practice rating, and feedback and discussion” (Lane & Stone, 2006). Weigle (1998), however, referred that differences between experienced and inexperienced raters could not be totally eradicated by training. However, inexperienced raters can make some progress after training, relatively speaking. In other words, rater training cannot achieve complete consistency among raters, especially among expert and novice raters. These rater characteristics thus need to be taken into account. Expert and Novice Raters In different kinds of professional domains, such as medicine, education, science, the process of maturing from novice to expert is ongoing. Ideally, the novice can attain 26.

(33) the ultimate goal of becoming an expert by passing through stages. Dreyfus and Dreyfus (1986) proposed a five-stage model of skill acquisition from novice to expert (novice, advanced beginners, competent, proficient, and expert) where expertise is characterized by “knowing how” rather than “knowing that” and expert knowledge is embedded in action. The expert have a clear understanding on not only what to do but also how to do to achieve the goal of the action and is more capable of focus on goals. According to Scribner (1985), the expert has learned to distinguish different situations from the experience beforehand and has the ability to interact with those contexts. With amount of experience in various contexts, experts see things from the same point of view but make different skillful decisions. Tactically, they can divide the group of situations into many subgroups and to deal with them specifically. Experts deal with problems based on their prior experience, and they do not view these problems as brand new challenges; instead they know how to revise problems to fit in the solutions they already have. From the accumulation of existing experience, experts can immediately and automatically call upon specific solutions from the long-term memory, which is a conscious processing (Bohr, 2001). Cognitive psychologists also hypothesize that though experts and novices deal with the same problem, they can see it with different perspectives (Alexander & Judy, 1988; Chi, Glaser & Rees, 1982). Unlike novices, experts can infer more often from the coming information and cluster sets of information into meaningful patterns and abstractions (Chi, Feltovich, & Glaser, 1981; Feltovich, Prietula, & Ericsson, 2006). According to Benner (1982), experts can also subconsciously show what they have already learned or known in their actions, and this knowledge has become an innate knowledge, which is existed in their brain. They also can adjust themselves quickly to conform to the new situation without too much consideration. From this body of relevant literature, experts can be described as schema-driven 27.

(34) rather than data-driven; they use the prior knowledge to adjust themselves to fit in different situations. Moreover, the schema of experts seem to include more procedural knowledge and more knowledge about how to apply the main principles underlying a given problem (Chi, Glaser, & Rees, 1982). Experts also can be able to categorize the presented problem and know how to give an appropriate solution in proper ways, whereas novices easily focus on a problem’s surface features. Because experts have a higher-order processing mode to cluster each event (Chi, Glaser, & Rees, 1982), they can monitor the problems and know how to reflect on them. Experts pay attention to contextual details and cues from the task, while simultaneously being able to monitor and gather information from the problem they face. In contrast, novices cannot achieve this kind of atomization and more easily only focus on the surface aspects of a problem (Hobus, Schmidt, Boshuizen, & Patel, 1987). Berliner (1994) also proposed that the difference between novices and experts is that the latter learn new things in a highly motivated way and can reflect on the experience. Therefore, from these findings, main characteristics for expertise can be said to be intuition, experience, automatic performance, and a deliberative processing mode. To some extent, while the process from the novice to expert is continually ongoing, it is can still be seen as a continuum with two poles. On the one hand, beginners have fewer relevant experiences on the specific domain than experts, and they are taught in terms of objective attributes which can be recognized without situational experience. Novices consciously thus need to follow the rules to take action and are guided by principles to connect their judgment to an appropriate action (Benner, 1982), and they may be overly eager to come up with a solution even though they may not be able to see what the whole picture of the task is. On the other hand, experts tend to gather, analyze, and evaluate information to formulate out a better way to understand 28.

(35) the problem (Ross, Shafer, & Klein, 2006; Voss, Tyler, & Yengo, 1983). At the same time, the experts know how to monitor and self-monitor themselves and have a greater control on their performance, and they have a greater sense to detect errors and adjust their behaviors according to the context. This ability comes from not only their cognitive capacity but also their mental models, and this capacity makes them more able to make interpretative and evaluative comments. From the task experience, novices can gradually develop expert-like behaviors if they can access and process an extensive and well-structured knowledge base. In this way, novices under the right conditions can have the ability to proficiently deal with complicated tasks (Chi, 2006). Many researches have showed that differences of cognitive structures and information processing methods between novices and experts lead to the different qualities of task outcomes (Govaerts, Schuwirth, Van der Vleuten, & Muijtjens, 2010). From the above discussion, it is clear that compared to a novice’s knowledge, that of an expert is more complex, and not only more extensive but also more integrated. With accumulated experiences from real situations, trainees are able to recognize more complexities and various cues at one time, and they can have more opportunities to interpret cues in the context (Swanson, 1981). Therefore, the ongoing developmental process from novice to expert depends on learning from experience. In terms of teacher experiences, the main difference between novice teachers and expert teachers is that the former have little or no teaching experience (Tsui, 2003), and typically they are considered either student-teachers or teachers in their first year of teaching. However, the definition of experts can take other factors into consideration, such as years of teaching experience, reputation, classroom observation, and recommendation by school administrators. Also, Farrell (2013) also identified five characteristics of experts, that is, “knowledge of learners and learning, engage in critical reflection, access past experiences, informed lesson planning, and active student 29.

(36) involvement.” The relationship between experience and expertise is not necessarily a direct correlation, because these two qualities are intrinsically different. Building on Bereiter and Scardamalia’s theory of expertise shows that experience is often mistaken as expertise (Bereiter & Scardamalia, 1993). Tsui pointed out that only when practitioners learn from experience with a constant reflection on their performance can the accumulation of experience will translate into the expertise (Tsui, 2003). Experience and expertise are related, but it is not necessary that these prior experience will lead to expertise (Weigle, 1998). In the assessment of writing, the literature has shown that raters play a crucial role in grading. In this study, only the rater characteristics of expertise versus novice is focused on. According to Cummins (1990), expertise refers to “raters whose rating performance is consistently good.” In Wolfe, Kao, and Ranney’s (1998) study, they also classified raters into three levels that based on how highly rater’s ratings corresponded with others, that is, competent, intermediate, or proficient. Based on their clarification, proficient raters are similar in their expertise, and they can judge the writing based on the general comments on the whole text. They also have the ability to consider all criteria equally and use more rubric-related languages to justify their ratings (Wolfe, Kao, & Ranney, 1998). In a standardized and reliable measurement, candidates and test items are like on two sides of a continuum, and raters lie in a critical position who can turn performances into outcomes. However, rater judgments include raters’ subjective point of views, thus rater judgments of the same writers often vary. Many factors might cause differences in raters’ grading, rater severity, rater characteristics, and rater consistency. The essence of rating is still a complicated cognitive process which might lead to variance in performance ratings. In this study, one of rater’s characteristics is chosen to be the 30.

(37) research goal, expert and novice. Based on Cummins (1990), expertise also has the ability to consider all criteria equally and use more rubric-related languages to justify their ratings. To what extent do the characteristics of raters, in terms of their experience, affect scores on the translation items? This research question could be answered by using MFRM. Many-Facet Rasch Measurement Model (MFRM) In the area of performance assessment, such for writing and speaking tests, there are different factors or facets that simultaneously and obviously interact with each other, such as, test takers, items, and raters. Candidates with different backgrounds, proficiency levels, or genders may have different performances on the performance assessment based on their backgrounds alone. It is also likely that raters, tasks, and other relevant aspects of the exam setting also cause a variability of grades and affect the interpretation for the true ability of the candidate. Because of the multiple sources of variability, a stable and fair estimate of how well learners can manage relevant tasks is necessary. For these reasons, the Many-Facet Rasch measurement model is an important tool to compensate for deficits of raw scores and can be used to map the relative ability of candidates, relative difficulty of items, and toughness of raters in a logit scale. Also, the model can form a model to predict a candidate’s odds from a rater with given leniency on a given item. The is especially the case in performance tests, where the goal of valid and reliable scoring is very important because scoring is often influenced by multiple facets simultaneously and interactively. (Davies, 1999; Bachman, 2004; McNamara, 1996). The Many-Facet Rasch measurement model (Linacre, 1996) is an extension of the basic Rash measurement model (Wu, 2010) and can be used to examine dichotomous data, whose two values are more than nominal data. The value of 1 is meaningfully 31.