Methodology
In this chapter, two separate parts of questionnaires, translation test and translation strategies survey, are carried out to examine the relations and strategies between
multiple-choice (MC) and constructed-response (CR) tests. A total of 411 students from twelve schools participate in the study. The schools are located in the northern, middle, and southern parts of Taiwan. This chapter is divided into five sections, including the pilot study, subjects, instruments, procedures, and data analysis.
Pilot Study
The pilot study was conducted through two versions of translation, from English to Chinese. The subject were 24 English majored college students in Nanya Institute of Technology with an average age of 22.5. Two versions of translation tests, test A and test B, selected randomly from the previous two-year technological and vocational education (TVE) entrance examination from 2001 to 2006, were adopted to test the students’
translation ability and were distributed randomly to the participants. Each version consisted of two sections of tests, MC translation section and CR translation section.
There were totally six MC questions and six CR questions in the two versions of
questionnaires. Total points available for the test were 120: 60 points for the MC tests and 60 points for the CR tests. Each MC question was worth 10 points with no penalty
for incorrect answers, whereas each CR question was worth from 0 to 10 points, depending on the quality of the translation. Two raters were invited to grade the CR tests. The tests were carefully assessed based on the answers provided by the administration of the two-year TVE joint college entrance examination and Test for English Majors 8 (TEM8), a criteria list that was set up as a reference for the raters to evaluate the translations (Table 1).
Each CR test score was the average score from the two raters.
Table 1
TEM8 Criteria for Translation Assessment Scale
Rank Grade Description
5 10-9
Excellent: It contains only one or two errors with respect to the norms of the target language and it is a functionally acceptable translation of the source text.
4 8-7
Good: This is basically a good translation. Most of the translations are loyal to the source text. It does have some minor errors, but they do not hinder the
comprehension of the translation.
3 6-5
Qualified: This translation is roughly qualified. It has several errors and would require a modification on it. Most translations are readable.
2 4-3 Poor: This is a poor translation. It would require major improvements.
1 2-0 Unacceptable: This is a totally unacceptable translation
The main difference between test A and test B was that the six MC tests in version A were equivalent to the six CR tests in version B, and the six CR tests in version A were
equivalent to the six MC questions in version B. This difference was spelled out in the following two summaries:
1. The first six MC tests in version A, referred to hereafter as MCA1 to MCA6, differed from the first six MC tests in version B, referred to hereafter as MCB1 to MCB6.
2. The first six CR questions in version A, referred to hereafter as CRA1 to CRA6, were equivalent to MCB1 to MCB6, except that no choosing items were offered.
To illustrate this, the third MC test, MCA3, on version A and its corresponding CR test on version B, CRB3, were shown below:
MCA3. The mother felt great anxiety when told of her son’s accident.
(A) 當談到兒子的意外事故時，母親都會感到焦慮。
(Mothers always felt great anxiety when talking about their sons’
accident.)
(B) 當這位母親談到兒子的意外事故時，她深感焦慮。
(The mother felt great anxiety when talking about her son’s accident.)
(C) 當母親們被告知兒子的意外事故時，都會感到非常焦慮。
(Mothers felt great anxiety when told of their sons’ accident.)
(D) 當這位母親被告知兒子的意外事故時，她感到非常焦慮。
(The mother felt great anxiety when told of her son’s accident.)
CRB3. The mother felt great anxiety when told of her son’s accident. (Translate this sentence into Chinese.)
Similarly, the sixth MC test, MCB6, on version B and its corresponding CR test on version A, CRA6, were shown below:
MCB6. The importance of national safety cannot be overemphasized.
(A) 國家安全的重要被過渡的強調。
(The importance of national safety is overemphasized.)
(B) 人們太強調國家安全的重要。
(People overemphasize the importance of national safety.)
(C) 國家安全的重要無論多麼強調，亦不為過。
(The importance of national safety cannot be overemphasized.)
(D) 國家安全的重要不該被強調。
(The importance of national safety shouldn’t be overemphasized.)
CRA6. The importance of national safety cannot be overemphasized. (Translate this sentence into Chinese.)
The overall results revealed that there was a significant difference between the MC and CR tests, that is to say, a learner’s translation abilities could not be equally assessed by the two test forms.
Subjects
Translation Test Section
Seven classes from six schools, Tajen University, Fortune Institute of Technology, Fooyin University, Tzuhui Institute of Technology, the Overseas Chinese Institute of Technology, and the Tahwa Institute of Technology were included. The total number of subjects was 203. There were 22 incomplete questionnaires, leaving a total valid sample size of 181 (Table 2). All the participants were fifth grade junior college students
majoring in English in the Applied Foreign Language Department. There were 22 males and 159 females, all of whom had had at least ten years of formal English education in Taiwan (Table 3). Their average age was 19.5.
Table2
Total #umber of Participants in Translation Test Section
School Total
Table 3
#umber of Participants for Versions A and B
School Version A Version B
Male Female Male Female
Tajen 1 2 11 1 8
Tajen 2 0 14 0 15
Fortune 2 14 6 12
Fooyin 1 8 0 6
Tzuhui 4 12 0 16
Overseas Chinese Institute
0 11 0 14
Tahwa 4 10 2 8
13 80 9 79
Strategies Survey Section
Eight schools participated in this part, from north to south Taiwan, including Ching Yun University, Nanya Institute of Technology, National Taichung Institute of Technology, Chia Nan University of Pharmacy and Science, Fooyin University, National Kaohsinging First University of Science and Technology (NKFUST), Kao Yuan University, and Tajen University. All the participants were second grad of two-year college students, who were also majoring in Applied Foreign Language of English and had experienced the two-year TVE joint college entrance examination. A total of 208 students were recruited in this part, including 39 males and 169 females with the average age of 21.3. They all had had at least ten years formal English education in Taiwan (Table 4).
Table 4
Total #umber of Participant of Strategy Survey Section
NKFUST Taichung Kaoyuan Nanya Chianan Fooyin Chingyun Tajen
Male 3 3 0 3 6 13 5 6
Female 14 25 22 22 34 13 19 20
Total 17 28 22 25 40 26 24 26
Instrument
Translation Test Section
The translation test used in this current study basically remained the same frameworks as the tests used in the pilot study, except that each version was consisted of four sections of tests— testing vocabulary, grammar, MC translation, and CR translation. There were eight vocabulary and eight grammar items in the beginning of the two versions (Appendix A and B). The main purpose of testing the students’ vocabulary and grammar abilities was to verify that the students’ language ability was more or less on the same level, even though they were from different schools. There were a total of sixteen translation questions in the translation parts, for a total 160 points, half for MC questions and half for CR questions. Two raters were invited to assess the CR tests using the criteria list of TEM8 (Table 1).
Survey section
A well designed and organized questionnaire, about the test-taking strategies that the students would use while taking the translation test by MC and CR items, was presented in
order to understand the relation and differences between the strategies used for the two types of tests. There were three main sections in the questionnaire. This first dealt with the students’ personal background information, including their school number, gender, age, and number of years spent learning English. The second section was comprised of three subsections, including 20 strategies used to facilitate comprehension and expression of translation tests in MC and CR tests, and 10 common strategies used in MC tests (Appendix C). The Likert scale (strongly agree, agree, disagree and strongly disagree, marked with 4, 3, 2 and 1, respectively) was used in the questionnaire to rate the student’s response toward the statements. The participants needed to carefully read the statements and circle the numbers based on their attitudes toward each statement. Only one choice could be circled for each statement. The last section of the questionnaire contained four open-ended (OE) questions, intended to see the students’ personal opinion about the two types of translation tests, and they were free to answer each of them. The questionnaire was confirmed and verified by two professional experts, and was modified several times.
Procedure
The data collection took place in two stages, the translation test survey stage and the translation strategy survey stage. Data were collected in May, 2006. In order to obtain the permission of distributing the surveys, all the class teachers were informed in advance that the class was going to participate in a survey test. After obtaining the authorizations,
the researcher personally delivered the questionnaires to each school and administered the test. Students who were unwilling to participate in the research had the right not to answer the questionnaire; they were free to join the study or not. The purpose of the study was mentioned at the top of the questionnaire, reminding the students that this survey would have no affect on their scores in school, and therefore they need feel no pressure as to what their response was. This was also explained orally by the researcher on the day of the test. In the translation test survey, the participants randomly received either version A or version B. All subjects in a school were tested at the same time. They were told in advance that they would totally have thirty minutes to answer the test. Any assistant tools were forbidden to use during the test. In the translation strategy survey, twenty minutes were allowed for the responding of survey. Before the timed test started, the subjects were given two to three minutes to ask questions about any ambiguities. The students were reminded 3 minutes before the end of the testing time to ensure that all the testing items were completed. Finally, all the surveys were collected on the spot and checked for any missing items (Table 5).
Table 5
Data Collection Procedures 1. Carried out the pilot study.
2. Designed the two questionnaires.
3. Recruited English majored students from junior and two-year college students.
4. Contacted the class teachers for the permission of distributing the test or survey to the classes.
5. Stated the purpose of the study to the students.
6. Distributed two versions of the tests randomly.
7. Passed out the translation strategy survey.
8. Notified the students of the total time for completing the test or survey.
9. Gave time to raise any questions.
10. Administered the process.
11. Reminded the students to review uncompleted items.
12. Collected the tests or surveys on the nail and checked for missing items.
13. Analyzed the data.
Data Analysis
All the test data were analyzed with the standard item analysis procedures with SPSS 12.0 statistical software package. The Independent-Samples t-test was adopted as the main tool to identify if there was a significant difference between the scores of the two test forms. If the result showed that there was a significant difference between these two test types, then the null hypothesis would be therefore rejected; on the contrary, if there was no significant difference between the tests, the null hypothesis would be sustained.
It was also used to examine the consistency of difficulty between the tests. All the CR tests were graded by two experienced experts. The Spearman Rank Correlation was run to examine the inter-rater reliability between the two raters. The Pearson Correlation was applied to check for correlation between the MC and CR tests. The comparability
among different students’ language ability in different groups was certified by One-way ANOVA. Finally, the Frequency analysis was used in analyzing the frequency of the strategies employed in the MC and CR translation tests, respectively.