國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
40
CHAPTER 4
RESULTS AND DISCUSSION
Findings of the survey study on how the BCET, a high-stakes public
examination, has impacted on student learning in Taiwan junior high schools will be reported in this chapter. As explained in the previous chapter, for the questionnaires, altogether 42 classes from 22 schools across northern Taiwan were selected by four administrative levels: national, municipal, town, and village levels. A total of 1,490 questionnaires were issued, and 1,432 questionnaires were completed, among which 1,388 questionnaires were valid for data entry and analysis.
The overall return rate was 96% (1,432 out of the 1,490 questionnaires issued).
Several reasons may explain this high return rate. First, the questionnaire itself is fairly short and easy to fill out. In addition, the topic of the questionnaire is closely related to the students’ immediate concerns. Particularly important is the attitude conveyed by the teachers within their questionnaire administration contexts. Since the teachers give clear instructions to complete the questionnaires and convey potential significance of the results, their students might sense a positive attitude toward the questionnaires, thus promoting their willingness to respond. The response rates by administrative level are summarized in Table 4.1 below.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 4.1
Questionnaire Return Rates by Administrative Level
Administrative Level
National Municipal Town Village Total
Number sent out 423 747 150 170 1,490
Number returned 410 710 144 168 1,432
Return rate (%) 96.93 95.05 96.00 98.82 96.11
Percent of all
questionnaires (%) 28.63 49.58 10.06 11.73 100%
The last row in the table indicates that most of the questionnaires came from
municipal level (49.58%) and 28.63% from national level, while 10.06% came from town level and 11.73% from village level.
Students’ Characteristics and Their Learning Contexts
Prior to presenting the study results, characteristics of the students who responded to the questionnaires are described in terms of gender, school types, and administrative level. Whether they were going to take the BCET will be reported as well. Table 4.2 provides demographic information for the study participants and various elements of their learning contexts.
‧
Students’ Characteristics and Their Learning Contexts
Characteristic Number of Students Percent (%) Total Gender Integrated junior-senior high school 435 31.34
Administrative Level of School Location
National level 391 28.17
100%
(N = 1,388)
Municipal level 698 50.29
Town level 141 10.16
Village level 158 11.38
Planning to take the BCT
Yes 1,257 91.62
100%
(N = 1,372)
No 35 2.55
Undecided 80 5.83
Note. N = number of respondents/students.
The distribution of the male and female students was quite balanced. The majority of the students (90.71%) were from public schools, only 9.29 % studied in private schools. Two thirds of the participants (68.66%) studied at junior high schools, while nearly one third was from integrated junior-senior high schools. Regarding
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
administrative level, municipal level had the highest portion of the students sampled, whereas town level had the lowest ratio. This distribution is close to that of the 9th-grader population in northern Taiwan.
As for the participants’ attendance of the coming BCT in 2009, the majority of the students responded that they were going to take the BCT, and only 8.38% of the students gave “no” or “undecided” responses. Some stated that they would not seek secondary education (i.e., senior high school, senior vocational school, or five-year junior college), and may enter the job market upon graduation. Still some said that they have joined the Practical Technical Program, which offers technical training courses beginning in the ninth grade, and they are allowed to enroll in vocational
schools without submitting a BCT score. It should be pointed out that nearly half of the “no” and “undecided” responses on this item were given by the cases from two
classes at a private school in Taipei City. Some stated that they considered getting admitted to the original private school without taking the BCT.
Given that the study focused on the washback effects imparted by the BCET on learners, the main analyses of questionnaire results excluded those who did not intend to take the BCET. Consequently, 115 cases were ruled out. Thus, 1,257 cases of the questionnaire are examined for the following four areas: (a) perceptions of the BCET, (b) learning motivation, (c) learning activities, and (d) class activities.
‧
Students’ Perceptions of the BCET
This section addresses the first research question: What are the students’
perceptions of the BCET in terms of its difficulty, test format, and test purpose.
There were four items, all of which were scaled 4 = Strongly Agree, 3 = Agree, 2 = Disagree, and 1 = Strongly Disagree. Students’ responses are summarized in Table
4.3 below.
Table 4.3
Students’ Perceptions of the BCET
Item No. and Statement
Response Percentage (%)
M SD
Strongly
Agree Agree Disagree
Strongly Disagree 1. I think the BCET is easy. 2.48 .79 7.80 43.41 37.78 11.01 2. I know the format of the BCET. 2.57 .82 10.56 46.09 32.72 10.64 3. I think the purpose of the
BCET is to assess whether I have achieved basic
competence in English. 3.20 .72 34.38 55.04 6.94 3.63 4. I think the purpose of the
BCET is to measure my actual
English proficiency. 2.75 .81 17.43 46.41 29.94 6.21
Note. M = mean; SD = standard deviation.
Most students demonstrated that they were aware of the BCET test format and test purpose, with both Item 2 and Item 3 achieving the mean scores above the neutral mean 2.5. As for the BCET test difficulty, the overall mean for Item 1 was 2.48,
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
slightly lower than the neutral mean. The students’ perceptions of the BCET difficulty will be further discussed in relation to their understanding of the BCET format and purpose.
Perceptions of the BCET Test Difficulty and Test Format
The BCET is intended to identify whether test takers have achieved basic competence in English at the end of junior high education. This purpose is reflected in its test design: facility values of BCET items are set in the 50%-75% range.
Trialing of test items is undertaken to ensure that this is the case, according to the BCT commissioned center. In other words, the BCET is designed in a way that the majority of items are not difficult for most test takers. According to the students’
responses to Item 1, 51.21% (n = 637) either strongly agreed or agreed that the BCET was easy, whereas 48.79% (n = 607) either strongly disagreed or disagreed.
Regarding the understanding of the BCET test format (Item 2), it is assumed that students should have a certain degree of familiarity with the BCET format, including the time available to complete the test, the number and type of sections, and the item and task types used. Information about the test format is presented in the BCT test handbook published by the BCT commissioned center. Besides, English teachers often introduce the overall design of the BCET in class as part of test-taking
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
instruction. According to the results, approximately ten percent of the students (10.56 %, n = 131) strongly agreed that they knew about what item types to appear on the BCET, and 46.09% (n = 572) agreed. In comparison, 10.64% (n = 132) of the students strongly disagreed that that they knew the BCET test format, and 32.72% (n
= 406) disagreed. Given that the BCET has been in operation for several years, this disagreement percentage is somewhat high.
Perceptions of the BCET Test Purpose
Items 3 and 4 are designed to know about the respondents’ understanding of the BCET test purpose. The intended purpose of the BCET is to assess test takers’
basic competence in English that should be attained via junior high education based on the curriculum regulated by the MOE, rather than to measure test takers’ general English proficiencies. In other words, Item 3 is intended to be a true statement of the BCET test purpose, while Item 4 is not.
The results of Item 3 indicates that a large number of the students (89.42%, n
= 1,108, including Strongly Agree and Agree respondents) agreed that the purpose of the BCET was to assess whether they have achieved basic competence in English.
Mean score of 3.20 on this item demonstrated the students’ knowledge of the BCET as a test that assesses basic competence in English. However, it is interesting to note
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
that most of the students (63.84%, n = 791, for both Strongly Agree and Agree
respondents) also held Item 4 to be a correct statement (M = 2.75). It seems that the students thought both two statements as true to varying degrees. Though most of the students agreed that the BCET assessed their basic competence in English, some still believed the BCET could measure their true English proficiency. The results indicate that the respondents might feel confused about these two statements and could not tell the difference. Even so, further analyses showed that the group mean of Item 4 was statistically significantly lower than that of Item 3 (t = 18.56, p < .001).
To a certain extent, it could be concluded that the students perceived test purpose of the BCET as intended by test designers: to measure basic competence in English.
Perceptions of the BCET Test Difficulty in Relation to Understanding the BCET
Test Format and Test Purpose
Cross-examination of the answers to Item 1 and Item 2 indicates that the students’ perception of the BCET test difficulty was related to their understanding of the BCET format. Table 4.4 reports the students’ perception of the BCET test difficulty according to their responses to whether they understood the BCET test format.
‧
Students’ Perception of the BCET Difficulty and Their Knowledge of the BCET Format
Response to Knowledge of the BCET Format (Item 2)
Perception of the BCET Difficulty (Item 1)
Note. N = number of respondents/students; M = mean; SD = standard deviation.
As shown by Table 4.4, mean scores for the BCET test difficulty are 1.60 for those who strongly disagreed with knowing the BCET format and 2.34 for those who disagreed, compared with 2.65 for those who agreed that they knew the BCET format and 3.07 for those who strongly agreed. The students who agreed that they knew the BCET format also agreed that the BCET was easy; and the better they knew the test format, the more likely they viewed the BCET as an easy test. On the other hand, the students who admitted lack of knowledge of the BCET format thought that the BCET was difficult. The less they knew the test format, the more likely they regarded the BCET as a difficult test.
Furthermore, results of Item 1 and Item 3 indicate that the students’ perception of the BCET test difficulty was related to their awareness of the BCET purpose.
Table 4.5 reports the students’ perception of the BCET test difficulty according to
‧
their responses to the understanding of the BCET test purpose.
Table 4.5
Students’ Perception of the BCET Difficulty and Their Knowledge of the BCET Purpose
Response to Knowledge of the BCET Purpose (Item 3)
Perception of the BCET Difficulty (Item 1)
Note. N = number of respondents/students; M = mean; SD = standard deviation.
Mean scores for the BCET test difficulty are 1.67 for those who strongly disagreed that the BCET was intended to assess whether they have achieved basic competence in English, 2.17 for those who disagreed, and 2.37 for those who agreed, compared with 2.81 for those who strongly agreed. Only those who strongly agreed that they knew the BCET purpose considered the BCET to be an easy test. A tendency can be observed that the less the students knew the BCET purpose, the more likely they found the BCET difficult.
Summary on Students’ Perceptions of the BCET
This category explores students’ perceptions of the BCET in terms of test
difficulty, test format, and test purpose. In summary, three major findings can be
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
observed from the questionnaire results. Firstly, a mismatch occurs between students’ perception and intended washback effect of the BCET. The BCET is intended to have positive washback effects on learning by avoiding arousing students’
feelings of high anxiety and fear of potential failure in the test. However, nearly half of the 9th-graders in northern Taiwan found the BCET not to be easy. This may affect their learning motivation. Those who found the BCET beyond their abilities may be less motivated to devote time and efforts to test preparation.
It is possible that the students held false assumptions about the BCET difficulty; after all, they had not taken the BCET at that time, and they might exaggerate its difficulty. Closer analyses showed that the more the students
understood the BCET test format, the more likely they regarded the BCET as an easy test. In a similar vein, an obvious pattern was found between the students’
perception of the BCET difficulty and their awareness of the BCET test purpose.
The more the students understood the purpose of the BCET, the more likely they thought that the BCET was easy. These findings suggest that as students become familiar with the BCET test format and test purpose, they would experience less anxiety for the test.
Second, as for the students’ understanding of the BCET test format, up to 43%
of the students reported that they did not know its format. This percentage was quite
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
high. Since the BCET is not a completely new test, students should have certain familiarity with the test format, which is an essential part in test preparation. Efforts should be made on the part of both English teachers and the BCT commissioned center to enhance 9th-graders’ knowledge of the BCET test format.
Finally, regarding test purpose, the BCET is designed to assess whether junior high students have achieved basic competence in English that they should acquire via English curriculum. According to the students’ reactions, more students thought that the BCET assessed their basic competence in English than those who thought that the BCET assessed their actual English proficiency. It may be concluded that most students in northern Taiwan understood the BCET purpose, as intended by the BCET test designers.
Students’ Motivation to Learn English
With regard to the second research question of students’ motivations to learn English, seven items were proposed, all of which were designed on a four-point Likert scale, where 4 = Strongly Agree and 1 = Strongly Disagree. Table 4.6 provides results of the seven items about students’ learning motivation. They are listed below according to the mean scores to show the degree of agreement.
‧
Students’ Motivation to Learn English
Rank Item No. and Statement
Response Percentage (%)
M SD
Strongly
Agree Agree Disagree
Strongly
native and foreign cultures
and customs. 2.83 .88 23.35 44.04 24.64 7.97 4 7. I learn English to listen to
English songs, read English books/newspapers/magazin es, and watch English
movies. 2.81 .90 23.07 43.97 23.71 9.24
5 9. I learn English because I
like English. 2.40 .92 11.92 34.30 35.59 18.20 6 8. I learn English primarily
because it is tested in the
BCT. 2.37 .93 13.39 28.31 40.40 17.90
7 6. I learn English to visit
English websites. 2.02 .77 4.60 16.88 54.36 24.15
Note. M = mean; SD = standard deviation.
Table 4.6 shows that most students thought learning English to be very important, with Item 5 achieving an overall mean score (i.e., 3.41) highly above the neutral mean 2.5. Also, three top motives, i.e., Items 10, 11, and 7, coincide with the overall objectives of English language education in Grade 1-9 Curriculum. On the other
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
hand, most students learn English not because they liked English, nor because English was tested in the BCT. Visiting English websites was found to be least motivating for learning English. The students’ responses will be further discussed in relation to their English achievement in the following sections.
Perceptions of the Importance of Learning English
Among all the seven items, Item 5 is the general question about the importance of learning English. It can be seen clearly that a large number of the students (90.49%, n = 1,123, comprising both Strongly Agree and Agree respondents) attached importance to mastering English, while only 9.51% (n = 118) either strongly disagreed or disagreed that it is important to learn English. This indicates that students have positive feelings about English, and this may lead to their increased motivation and to successful achievement.
Further analyses of Item 5 and the students’ academic achievement in English demonstrated that the students’ attitudes toward the importance of learning English are positively related to their English academic achievement, as indicated by their scores of last semester and last mock examination (see Table 4.7).
‧
Students’ Perceptions of the Importance of Learning English and Their English Achievement
Response to Importance of Learning English (Item 5)
English Achievement
Semester Scorea Mock Exam Scoreb
Median Mode Mean SD
Note. SD = standard deviation.
aEnglish score of the spring semester, 2008, with five rankings: You, Jia, Yi, Bing, and Ding (from the highest to the lowest). bMaximum mock exam score = 80.
Judging from the students’ English achievement scores for the previous semester, both Strongly Agree and Agree respondents outperformed Disagree respondents, who in turn outperformed Strongly Disagree respondents. In addition, Strongly Agree respondents scored highest with a mock examination mean of 64.56 out of 80, and Strongly Disagree respondents scored lowest with a mean of 35.39. One-way
Analysis of Variance (ANOVA) was conducted to examine whether the differences in mock examination score means were significant. The results indicate that the
differences in score means were statistically significant, F (3, 1078) = 56.74, p < .001.
Dunnett’s C post hoc test showed that Strongly Agree respondents outperformed Agree respondents, who in turn outperformed both Strongly Disagree and Disagree
respondents (p < .01). These results show that students’ attitudes toward the
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
importance of learning English are associated with their academic achievement in English. Those who place more value on English acquisition tend to have better English achievement scores than those who do not.
Motivation in Relation to Grade 1-9 Curriculum
Items 6-11 present six factors that may motivate 9th-graders to learn English.
Among these items, three items were found to motivate the students most, as they achieved the mean scores above the neutral mean 2.5: Items 7, 10, and 11. As shown in Table 4.6, the students agreed that they learned English in order to foster communicative competence (Item 10, M = 3.16), to understand foreign and native cultures (Item 11, M = 2.83), and to listen to English songs, read
books/newspapers/magazines in English, and watch English movies (Item 7, M = 2.81). In percentage terms, a large number of the students (83.53%, n = 1,039, inclusive of both Strongly Agree and Agree respondents) thought that it is important for English learners to develop communicative competence (Item 10).
Approximately 67 % of the students either strongly agreed or agreed that learning English was to understand native and foreign cultures (Item 11). They also agreed that they learned English to be able to listen to English songs, read English texts, and see movies in English (Item 7).
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The most common motivations for learning English stated in Items 7, 10, and 11 correspond to the overall objectives of the English language curriculum.
According to General guidelines of grade 1-9 curriculum of elementary and junior high school education (MOE, 2003a), overall curriculum objectives for English
language education are:
(a) to develop basic communicative competence for authentic or actual language use situations,
(b) to learn English autonomously and effectively by developing interests and methods, and
(c) to understand native and foreign cultures so as to compare and respect cultural differences.
The results of Items 10 and 11 confirmed that most students recognize the importance of using English for communication, and the value of understanding the notion of culture through comparisons of other cultures and their own. The results of Item 7 supported that students also recognize the significance of effective methods and positive attitudes, like exposing themselves to English movies, songs, radio programs, books, etc. These findings suggest that the curricular objectives of English teaching
The results of Items 10 and 11 confirmed that most students recognize the importance of using English for communication, and the value of understanding the notion of culture through comparisons of other cultures and their own. The results of Item 7 supported that students also recognize the significance of effective methods and positive attitudes, like exposing themselves to English movies, songs, radio programs, books, etc. These findings suggest that the curricular objectives of English teaching