Results
The results of the study will be presented and discussed in this chapter. The data collected are examined according to the results obtained from the translation test and translation strategy survey. The first part revealed the results of the two forms of translation tests. The second part displayed the findings of translation strategies used in multiple-choice (MC) and constructed-response (CR) tests. Participants’ opinions
towards the two types of tests were shown in the last part. All the data are processed with the computer program of Statistical Package for the Social Sciences (SPSS) version 12.0.
Results of Translation Test Section
The researcher developed an English ability test which included vocabulary and grammar sections in order to verify that the English abilities of the different groups of students were more or less on the same level. The null hypothesis was phrased “there is no significant difference among the seven groups of students” (H0: μ1 - μ2 = 0), and the alternative hypothesis was phrased “when null hypothesis is false” (H1: μ1 - μ2 ≠ 0).
In all cases, the null hypothesis for both MC and CR tests was rejected at the 5%
significance level. By running the One-Way ANOVE with students’ vocabulary and grammar scores, the data showed F(48.235, 1746.118) = .801, p = .570 > .05. In other words, the null hypothesis was accepted, indicating that there was no significant difference
in the language abilities of the different groups of student (Table 6). Second, the
researcher determined the acceptable range of skewness and kurtosis to be -1 < skewness <
1, -3 < kurtosis < 3. The frequency distribution showed the skewness (-.076) and kurtosis (-.318) values to be within the acceptable range, indicating that the students were in the normal distribution. Accordingly, from the results of the data above, the participants could be concluded as independent, normal distributed, and homogeneous.
Table 6
A#OVA on Testing of the Language Ability of the Groups of Participants
SS df MS F Sig.
(p<.05) Between
Groups 48.235 6 8.039 .801 .570
Within
Groups 1746.118 174 10.035 Language ability among
groups of students
Total 1794.353 180
The inter-rater reliability between the two CR raters was evaluated by using the Spearman Rank Correlation method to examine the inner-consistency between them. The value of correlation coefficient was .971 at p = .000 < .05. This result revealed there was a high consistency between the grading criteria of the two raters (Table 7).
Table 7
Spearman Rank Correlations between the Two Raters
Correlation Coefficient Rater 1 Rater 2
Rater 1 1 .971**
.000
Rater 2 .971**
.000 1
#ote. Correlation is significant at the 0.01 level (2-tailed).
In terms of the difficulty of the two test versions A and B, the null hypothesis was formulated as H0: μ1 - μ2 = 0 and the alternative hypothesis as H1: μ1 -μ2 ≠ 0. By running the Independent Samples t-test with the total scores of MC and CR in version A and B respectively, the result showed t(179) = .012, p = .99 > .05, which meant that there was no significant difference between the two versions of tests (Table 8). Based on the data, it could be concluded that the difficulty of the two translation versions received no difference.
Table 8
Independent Samples T-Test Results for Version A and B
F Sig. t df Sig. (p < .05)
Two tests results of versions A and B
.120 .729 .012 179 .990
Analysis of research question 1: Is the result of multiple-choice test the same as that of
constructed-response test in translation assessment?
To answer this question, the null hypothesis was formulated as H0: μ1-μ2 = 0, and the alternative hypothesis as H1: μ1 - μ2 ≠ 0. The average scores of MC and CR tests in version A were 59.46 and 40.31, whereas the average scores of MC and CR in version B were 53.52 and 46.2, respectively (Table 9). Independent Samples t-test was used to perform the scores of MC in version A and CR in version B and the scores of CR in
version A and MC in version B. For the former, t(179) = 5.26, p = .00 < .05, while for the latter t(179) = 6.12, p = .00 < .05, both rejected the null hypothesis (Table 10). Based on the result, it could be ascertained that there was a significant difference between the two forms of tests. Therefore, the conclusion of the research question one was that the two forms of tests could not equally assess learners’ translation abilities.
Table 9
The Average Scores of MC and CR Tests in Version A and B
Average score Number of participants
MC CR
Version A 93 59.46 40.31
Version B 88 53.52 46.2
Table 10
Independent Samples T-Test Results for the MC and CR Tests
F Sig. t df Sig. (p < .05) MC in Version A and CR in version B 4.42 .037 5.26 179 .00 MC in version B and CR in version A .704 .402 6.12 179 .00
This result echoed previous studies. Many studies have reported that the average score of MC tests are higher than the average score of CR tests (Bridgeman, 1992; Wolf, 1993; O’Neill, 2001; Nixon & Kennedy, 2002; Cheng, 2004). Participants generally performed better on the MC tests than on the CR tests. One important conclusion from the literature review was that format effects could vary across different types of tests. In some cases, MC questions appeared easier, but in others, MC and CR versions of the same question appeared to be nearly difficult. The result of the study was consistent with this general conclusion. However, there were two interesting findings for the pairs of MCA4 and CRB4, and MCB1 and CRA1. In MCA4 and CRB4, the average scores of CRB4 were higher than MCA4 (Table 11). The item MCA4 and its equivalent counterpart CRB4 were shown below:
( )MCA4. Don’t turn to him for help. He is a penny pincher.
(A) 他一毛不拔。
(He is stingy and ungenerous.)
(B) 他自私自利。
(He is selfish and full of himself.) (C) 他鐵面無私。
(He is inflexibly just and fair.)
(D) 他剛正不阿。
(He is sturdy and honest.) CRB1. Don’t turn to him for help. He is a penny pincher.
Table 11
Average Scores for MCA, CRB, MCB and CRA
Q 1 Q 2 Q 3 Q 4 Q 5 Q 6 Q 7 Q8
MCA 850 740 710 460 700 600 650 820
CRB 691 452 493 616 328 315 481 651
MCB 160 500 640 800 650 440 700 800
CRA 312 342 461 773 614 274 332 569
#ote. Q = question
Among the 93 examinees in version A, 38 of them selected answer B, 5 selected C, and 4 selected D. Analysis of the raw data showed that in MCA4, if the test-taker knew the meaning of the keyword ‘penny’, they could pick out the correct answer, but if they did not, a guessing or other type of strategy might be used. It was the well-designed distractors that caused the high percentage of selecting answer B. For instance, from the whole context of the question, it could be inferred that the description of the person tended to be negative, and hence item C and D could be easily ruled out because of their positiveness,
meaning in Chinese. From the semantics aspect, it also had a perfect meaning of “Don’t turn to him for help. He is selfish and full of himself.” Examinees who only knew the meaning of first sentence might choose answer B based on the intuition of semantics. In terms of CRB4, if the test-takers understood some aspect of penny, they could arrive ar a rough answer as “他很小氣(He is stingy)”, “他是個吝嗇鬼(He is a miser)”, “他是個鐵公 雞(He is a niggard)” and so on. Therefore, totally 55 examinees received a perfect score
of the question. Furthermore, the item MCB1 and its equivalent counterpart were shown below:
( )MCB1. It pays to work hard.
(A) 努力是要付出代價的。
(It costs to work hard.)
(B) 要花時間用功。
(Take time to study hard.) (C) 用功可以得到錢財。
(Working hard earns you money.)
(D) 努力工作有好處。
(It pays to work hard.) CRA1. It pays to work hard.
Among the 88 examinees in version B, 62 of them chose answer A, 10 chose B, and none of them chose answer C. This result revealed that the examinees who selected answer A hardly knew the deeper meaning of the sentence and were likely to translate the sentence by word-for- word in mind. In CRA1, many participants translated the sentence as “工作 辛苦是有代價的(It is valuable to work hard)”, “努力是有回報的(It is repayable to work
hard)”, and “努力是值得的(It is worthy to work hard)”. Unlike the MC form, there was
not any confusion in the CR form. In other words, the distractors associated with MC tests could make the MC tests more difficult by causing students to worry about the erroneous factors.
Analysis of research question 2: Is there a high correlation between the result of
multiple-choice and constructed-response translation tests?
The Pearson Correlation was run through the scores of MC in version A and CR in version B and the scores of CR in version A and MC in version B to answer the second research question. The data showed that the correlation coefficient was -.077 at p = .477
> .05 and .082 at p = .466 > .05, respectively (Table 12). It could be concluded from the data that there were no significant correlations between the two comparisons. This result echoed the result of research question one.
Table 12
Correlation of MC and CR Tests
Correlation Coefficient MC A CR A
MC B .082
.466
CR B -.077
.447
#ote. Correlation is significant at the 0.01 level (2-tailed).
Analysis of research question 3: Is there a high correlation between the result of learners’
translation ability evaluated by MC test and the result of vocabulary and grammar tests?
The Person Correlation was also performed to answer the third research question.
The data of the correlation coefficient was .065 at p = .384 > .05, showing that there was no significant correlation in learners’ translation ability tested by MC and vocabulary and grammar tests. This result differed from Pan’s (2001) findings. According to Pan, translation tested in the MC format evaluated nothing but grammar and idioms, which meant there was a high positive correlation between the two variables. However, the finding indicated that there was no significant correlation between them.
Results of the Translation Strategy Survey Section
Analysis of research question 4: Do the strategies used in multiple-choice test differ from
those in constructed-response test in terms of understanding the translation questions and
expressing the answers?
A strategies survey, including translation strategies, common MC tests strategies and participants’ opinion towards the two forms of tests, was designed by the researcher in order to compare the differences between the strategies used in MC and CR test. The specific translation strategies were developed based on Chang’s (2005) book: Practical Translation: An Introductory Coursebook, and the common MC test-taking strategies were
adapted and modified according to McDonough’s (1995) book: Strategy and Skill in
Learning a Foreign Language. Two professional experts confirmed and approved the
questionnaire before its distribution. There were four parts in the section. The first and second parts dealt with the strategies in helping learners understand and produce the translations in both MC and CR tests. In the third part, ten common MC test-taking strategies were developed to investigate whether, when doing MC translation tests, examinees tended to adopt these strategies. Four open-ended (OE) questions made the last part up. The reliability of the strategies survey was estimated by inter-item
consistency. An alpha reliability index was calculated for the questionnaire. Table 13 presented the reliability of the MC and CR strategies. The index of inter-item consistency for the MC translation strategies section was alpha .755, and for the CR translation section was alpha .708. Hence, the internal consistency of the survey was tenable.
Table 13
Reliability Analysis of MC and CR Translation Strategies
Sections MC CR
N of participants 208 208
N of items 30 20
Alpha .755 .708
The first ten questions in the translation strategies survey were the questions that aiming at investigating participants’ translation strategies in understanding translation questions. The mean and frequency of the questions were shown in Table 14 and
Meanwhile, by running Independent Sample t-test with the mean scores, the result was t(18)
= -.105, p = .918 > .05, also indicating that when doing a translation task, the participants tended to agree to adopt the same strategies to understand the questions in both MC and CR translation tests. The result disclosed that the strategy of using the contexts to infer the unknown words was the most adopted strategy in MC (1.61) and CR (1.58) translation tests. Moreover, the strategy of using their sense of English to deduce the unclear words was less adopted strategy in MC (2.22) and CR (2.42) translation tests.
Table 14
Translation Strategies for Understanding the Questions in MC and CR Tests (#=208) Mean
MC CR
1. I make use of the parts of speech to facilitate my understanding of the
meaning of the words. 1.93 1.70
2. I make use of affixes to the words to facilitate my understanding of the
meaning of the words. 1.80 2.01
3. I make use of the context of the sentence to facilitate my understanding
of unclear words. 1.61 1.58
4. I make use of the relations between words to facilitate my
understanding of the question. 1.88 1.96
5. I make use of the extension of key words to facilitate my
understanding of the questions. 1.75 1.75
6. I make use of the concepts of grammar to facilitate my understanding
of the questions. 2.09 1.75
7. I make use of the relations between sentences to facilitate my
understanding of the questions. 1.87 1.72
8. I write down the meanings of the words when reading the question. 2.29 2.27 9. I make connects with prior knowledge to facilitate my understanding
of the questions. 1.91 2.31
10. I make use of my sense of English to facilitate my understanding the
2.22 2.42
The second set of ten questions was focused on exploring the strategies participants used to answer translation questions. The mean and frequency of the questions were shown in Table 15 and Appendix E. According to the data, there was an evident contrast between the agreements of adopting the strategies to answer the questions in MC and CR translation tests. The mean data of translation strategies in MC test, on one hand, were higher than the 2.5 average level, showing that the participants tended not to used these specific translation strategies to help them answer the translation questions. The mean data of translation strategies in CR test, on the other hand, were lower than 2.5, indicating that the participants did use the specific translation strategies to respond to the translation questions. Moreover, via performing the Independent Sample t-test, the value t(18) = 9.232, p = .00 < .05 showed the significant difference between the two sets of strategies.
It could be therefore inferred that it was the free writing test form, namely, CR, which allowed examinees to use and synthesiz various strategies into the tests. In MC test, the testees passively accepted the four tricky items which had already been translated by the designers.
Table 15
Translation Strategies for Answering MC and CR Questions (#=208)
Mean
MC CR
11. I make use of adding word. 2.65 1.62
12. I make use of transforming the parts of speech. 2.71 1.84
13. I make use of changing tense. 2.66 2.01
14. I make use of switching positive words to negative words and vice
versa. 2.64 2.11
15. I make use of repeating words. 2.78 1.89
16. I make use of reversing the order of words. 2.74 1.82 17. I make use of adopting semantic translation, not word-for-word
translation. 2.83 2.35
18. I make use of shifting the active voice to passive voice and vice versa. 2.64 2.29
19. I make use of using Chinese idioms. 2.69 1.81
20. I make use of splitting or combining words. 3.07 1.75
The following set of ten questions was the questions that stressed on investigating participants’ translation strategies for answering translation questions, particularly in MC form. The mean and frequency of the questions were shown in Table 16 and Appendix F.
The data showed the strong agreement of using these general strategies, such as guessing, eliminating, selecting positive statement and so forth. From all these data, learners clearly expressed a preference for using common strategies rather than specific translation
strategies when taking MC translation tests. If a translation test did not allow test-takers to use the specific translation strategies, it would be surprised that it allowed them to use other strategies.
Table 16
Translation Strategies for Answering the MC Questions (#=208)
Mean 21. I make use of random guessing to choose answers. 1.63 22. I make use of eliminating incorrect items as the basis for choosing an
answer. 1.71
23. I make use of choosing the length that is closest to the stem as the
basis for choosing an answer. 2.21
24. I make use of selecting positive statement items for choosing the
answer. 2.21
25. I make use of calculating the average distribution of correct answers as
the basis for choosing the answer. 2.15
26. I make use of picking the statement that is nearest to common sense as
the basis for choosing the answer. 1.94
27. I make use of identifying the two statements that are closest to or
contradict the question as the basis for choosing the answer. 2.12 28. I make use of selecting key words which also appear in the stem as the
basis for choosing the answer. 2.11
29. I make use of the grammar pattern which is most in accordance with
the question as the basis for choosing the answer. 2.21 30. I make use of selecting the tense which is most in accordance with the
question as the basis for choosing the answer. 2.17
Analysis of OE questions
The four OE questions at the end of the survey were developed to probe the
doing a translation task, what kind of translation test, MC or CR, would you prefer to take?
Why?” about 90% of the participants expressed the preference of MC test. The most common reason was that they could guess if they failed to understand the translation questions. Once they guessed, they had at least one quarter of luck to reach the right answer, and there was no penalty if they guessed wrongly. The followings were reasons given by three participants to the first OE question.
I prefer MC test, because when I don’t understand the question, I can guess. In CR test, I have to come out the answer by myself. (Participant 1)
Of course I will choose MC test, in that I don’t have to make something out of nothing.
(Participant 2)
My preference is MC test, not only because it saves time to answer, but also because I won’t be afraid of spelling wrongly. (Participant 3)
When being asked the second question “Based on your own opinion, what kind of translation test, MC or CR, do you think that can really evaluate a person’s translation ability? Why?” over 95% of the participants reported CR test. Because there were no pre-translated items as a reference for them to select, they had to produce the answers on their own. There were no lucky guesses; everybody was equal in CR test. The
followings were three participants’ opinions to the second OE question.
It is the CR test that can authentically assess learners’ translation ability. A translation question could have many different ways to translate it. It concerns learners’ first and second languages abilities. (Participant 1)
I think CR test can really test out a learner’s translation ability, because it requires learners to translate every single word by themselves. (Participant 2)
I consider that CR test can really evaluate learners’ translation abilities, because people cannot guess in CR. (Participant 3)
The last two OE questions were designed to ask participants whether they had participated a translation test by MC form before joined the two-year TVE joint college entrance examination and what they thought about it. About a half of participants
answered yes. They considered that the translation test evaluated by MC format was easy to prepare for, easy to respond to, and easy to obtain a higher score. They felt more comfortable and less pressure when writing an MC than a CR translation test. The followings were opinions of three participants who had joined the MC translation test.
An MC translation test is easy to prepare and easy to obtain a higher score. I don’t have to memorize lots of vocabulary and grammar rules…… (Participant 1)
What I need to do in MC translation test is simply to recognize the specific words from Chinese into English among the four choices, and vice versa. (Participant 2)
What I need to do in MC translation test is simply to recognize the specific words from Chinese into English among the four choices, and vice versa. (Participant 2)