國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Chapter 5
Results and Discussion
The results of the data analysis procedure are presented in this chapter, and they reflect the multi-method approach utilized in this research. First, the results of the questionnaire completed by employees of the CBC, MOEA, and the MOF are presented, focusing first on the descriptive statistics summarizing the responses of the participants, and then discussing the significance of these results. The results of statistical analysis of the questionnaire data are offered next. Then the results of the interviews with representatives of seven central government agencies are summarized. The key issues discussed in the different interviews and gathered from official documents are presented with reference to the G-S policy analysis framework, first focusing on the government goals, and then the successive stages of the policy cycle, namely objectives, methods, activities, performance, and evaluation.
Questionnaire
The questionnaire results are presented and discussed in the following sections. See Appendix F for the complete results.
Demographics. The results from this section are found in Table 3. The participants in the survey are employees ofthe CBC, the MOF, and the MOEA. According to CPA data from 2009 Q2, the total number of CBC employees at the elementary, junior, or senior rank totaled 1,060. This number is far less than the number of employees of similar status at the other two agencies. If the employees of state-run enterprises are included in the totals, the workforce of the MOEA numbers 24,078, while the MOF comprises 24,854. The sampled population of the CBC workforce accounts for 10% of the total number of employees, while the MOEA and MOF samples account for just 0.3% and 0.4%, respectively. If the total number of employees of the
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
MOEA is calculated as including only those working in staff units or administrative agencies, the total workforce totals 4,506. Based on this number, the sampled population from MOEA makes up 1.7% of the total number of civil servants employed by that agency.
Comparing the data for the study’s sample population to CPA data reveals significant differences. Women make up a greater percentage in the sample than in the CPA data, 56.7%
compared to 43.5%, respectively. Civil servants at the elementary rank comprise a smaller
percentage in the sample population than in the civil service as whole, 11.3% compared to 40.5%, respectively. In terms of age, younger people made up a greater portion of the total in the study sample than in the CPA data. Civil servants between the ages of 25 and 39 made up 51.2% of the sample population compared to just 27.6% of the civil service as a whole. The education level of the study sample was considerably higher than the population of civil servants as a whole. Participants with master’s or doctorate degrees made up 43.7% of total, while in the civil service as a whole, only 19.4% of employees possess such degrees. It was anticipated that the demographics of the study sample would differ significantly from that of the civil service as a whole. It is assumed that the three agencies, all concerned with finance and economics, require personnel with a relatively high level of both education and English ability due to the highly professional nature of their work.
‧
MOF Local Tax Agencies 9.9%
MOF National Level Agencies 12.1%
MOF Customs 9.1%
MOEA Taiwan Power Co. 10.7%
MOEA Staff Units 7.9%
MOEA Administrative Agencies 8.3%
Civil Service Gradeb (N=282)
Elementary A Elementary B Junior A Junior B Senior
1.8% 10.5% 36.3% 47.4% 5.8%
aThis variable was transformed by combining members of different departments to produce groups of comparable size in which the members have similar areas of responsibility.
bThe civil service grades reported by the participants were transformed to reduce the number of grades (from 14 to 5) and to allow for comparison among agencies with different grade systems.
cThe participants reported this variable as a specific number of years. The groups were formed after collection to facilitate presentation.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
English language. This section of the questionnaire includes information about the participants’ self-assessed English ability, the importance of English for their work, the experience taking English language proficiency tests, and their study of English. The general aim of this section is to: (a) present data that demonstrates general trends and average figure in order to accurately characterize the sample population, (b) suggest explanations for the observed data, and (c) provide evidence to support conclusions about the effectiveness of testing policy measures.
The self-assessed English data demonstrate the participants’ abilities for each of the language skills and indicate their relative strengths and weaknesses. This data represents the supply of English language resources that the government may draw on. The data about the importance of English indicates the relative importance of each of the English language skills for the participants’ work and represents the government’s demand for English language resources.
The participants’ comments about the work activities for which they most commonly use English are also reported.
The self-assessed English and importance of English results are compared to each other in order to demonstrate whether there is a surplus or shortage of English language resources for each of the language skills. Similarly, overall averages for both self-assessed English and importance of English are compared to each other to indicate whether there is a relative shortage or surplus of English language resources at lower, middle, and higher proficiency levels.
The English proficiency testing experience data includes the participants’ reasons for taking or not taking an English test, the number of tests they have taken, and the participants’
CEFR levels based on their self-reported test scores. Finally, the English study data helps to characterize how the participants’ study of English is related to English proficiency testing. Of
‧
particular interest is whether taking an English test leads participants to attend test preparation courses or continue studying English after taking a test.
Self-assessed English ability. The participants assessed their English ability on a 5-point Likert scale for each of the four language skills (listening, speaking, reading, and writing). The choices were labeled from worst to best: very poor, poor, not bad, good, very good. To facilitate comparison, these responses have been combined to form three groups: below average includes responses of very poor and poor; average includes only not bad responses; above average is comprised of good and very good. Refer to Figure 5 for the results.
16
Self-assessed Importance Self-assessed Importance Self-assessed Importance Self-assessed Importance
Listening Speaking Reading Writing
% of Responses
Below average Average Above average
Figure 5. Comparing self-assessed English with importance of English by language skill. For all self-assessed English, N = 281. For importance of English: listening, speaking, and writing, N = 282; reading, N = 281.
Approximately one-half of the participants rated their English ability as average in all four language skills, ranging from a high of 52% for listening and reading to a low of 45% for speaking. Speaking and writing had the highest percentage of participants who rated their ability to be above average, at 43% and 37%, respectively. Reading had the highest percentage of
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
participants reporting that their English ability was below average, at 28%. These results suggest that participants are most confident in their speaking and writing and least confident in their English reading ability.
Importance of English. The participants indicated the importance of the four English language skills for their work by selecting from one of five choices on a Likert scale. The choices were labeled from least important to most important: no importance, not very important, no opinion, important, and very important. These responses have been combined to form three groups: below average includes responses of no importance and not very important; average includes only no opinion responses; above average is comprised of important and very important.
The most frequent response for each of the language skills was below average, ranging from a high of 58% for reading to a low of 45% for writing. These results suggest that approximately one-half of the participants estimate English ability to be of little importance to their work, while approximately one-quarter find English to be of average importance and one-quarter believe it to be of above-average importance.
Comparing importance of English to self-assessed English by language skill, a clear trend emerges. For each of the language skills, the participants indicate that their supply of English language resources is generally more than adequate for the demand for those skills in their jobs.
Only for reading is the figure for above-average self-assessed English, 20%, less than the accompanying figure, 21%, for importance of English. However, 52% of the participants report their English reading ability is average, while only 22% claim that English is of average
importance for their work. For the other three language skills, the above-average self-assessed figure is greater than the above-average importance figure. The greatest difference is noted for
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
speaking, with 43% of participants rating their speaking ability to be above average versus only 26% who judge English speaking to be of above-average importance for their work.
In addition to rating the importance of English for their jobs, the participants briefly commented about how they used English for their work. In all, 125 participants, or 44.3% of the total, provided 166 responses to this item. Their responses were translated into English and grouped into 7 categories. Of those who responded to this item, 67% reported just one activity requiring English language proficiency, while 29% reported two activities, and just 4% reported three. Reading was the activity reported most frequently, making up approximately 37% of all the responses. Attending international conferences and talking with foreign visitors each comprised 21% of the total. Translating made up 8% of the total, while writing reports
contributed 7%. Writing letters or email and using the telephone each made up approximately 3% of the total.
The responses to this item give a significantly different view of the importance of English language skills for the work done by the participants than do the responses to the quantitative items reported above. One reason for the difference could be that the response rate to the quantitative items is much higher than for the qualitative item, but the divergence may also be explained by the fact that the qualitative item required the participants to think more deeply about their actual duties than they did for the quantitative item. It would be valuable to carry out a more thorough English language needs analysis for civil servants. Such a study could help to identify appropriate English proficiency levels for different jobs and civil service grades, to aid in the design or selection of English proficiency tests for civil servants, and to guide the
development of curricula for English language training programs.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Overall averages: Self-assessed English and importance of English. The choices for each language skill for self-assessed English and importance of English were assigned a numeric value, with very poor/no importance equal to 1, and very good/very important equal to 5. The sum of each participant’s responses for the four skills for both self-assessed English and importance of English were divided by four to produce an average figure. The frequencies of these averages are presented in Figure 6.
0 5 10 15 20 25 30
1.00 1.25 1.50 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00
Averages
% of Participants
Self-assessed English average Importance of English average
Figure 6. Overall Averages: Self-assessed English and Importance of English.
Comparing the average figures for self-assessed English and importance of English reveals three distinct trends. The percentage of participants in jobs for which English is of below-average importance, from 1.25 to 2.0, is approximately equivalent to the percentage of participants with English proficiency that is below average. The percentage of participants who
Approximately equivalent
Self-assessed greater than importance
Importance greater than self-assessed
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
report that English is of average importance, from 2.25 to 3.25, is lower than the percentage who report that their English proficiency is average. Finally, the percentage of those who report that English is of above average importance for their work, from 3.5 to 5.0, is greater than that of those who claim their English proficiency is above average.
These results suggest that in these three agencies, there is a surplus of civil servants with an average level of English proficiency but a shortage of those with English proficiency that is above average. In terms of promotion scoring, the scoring values associated with basic English proficiency levels may be adequate to motivate employees to improve their English. This conclusion is supported by the fact that there is surplus of participants with English proficiency of an average level. On the other hand, the apparent shortage of participants with above-average English proficiency suggest that the promotion scoring values associated with higher levels of proficiency could be insufficiently high to motivate further improvement. There is a potential danger, however, of assigning promotion scoring values that are too high, since such an action could result in civil servants ignoring their duties and devoting an inordinate amount of time to improving their English. Therefore, it may be prudent to increase opportunities to improve English proficiency for civil servants in positions in which English is of average or above-average importance without appreciably altering the weighting of promotion scoring values.
Testing experience. A total of 270 participants provided information about whether they had taken an English proficiency test recognized by the MOE. Anticipating potential difficulties in calculating the number of participants who had taken an English test, several methods were employed to arrive at this figure. First of all, in answer to Item 9a, 182 participants, or 64.5% of the total study sample, reported the name of at least one English test they had taken. Another item (10a), found that 31.9% of the participants reported they had never taken an English
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
proficiency test. The sum of these two numbers is 96.4%, and while it is not equal to 100%, the fact that it is relatively close may suggest that the 64.5% figure may have some basis in reality.
It should be emphasized that this number represents the percentage of the participants who have reported having taken, not having passed, an English proficiency test. Of the 282 participants in the study, 124 (or 43.9%) provided a score that could be equated with a CEFR proficiency level.
It is not possible to say what percentage of those who did not report a score on a recognized English proficiency test achieved an A2 on the CEFR scale.
The figure of 43.9% is significantly higher than the 25% figure reported by the CPA.
This discrepancy could have several explanations. Firstly, the number of participants who claim to have reached the A2 level may be either higher or lower than the actual figure by an
undetermined degree. Secondly, the sample may not reflect the population of civil servants as a whole based on the fact that the study focused on three agencies whose need for
English-speaking employees was estimated to be relatively higher than average. Overall, the margin of error could be substantial, and it is difficult to conclude that the figures arrived at by this study have a high degree of confidence.
Of those participants who reported not having taken an English proficiency test, 43.1%
claimed that taking such a test was unnecessary. There was no information provided about why taking an SELPT was not needed, but it may be assumed that English proficiency is not
important for these participants’ jobs. If that were the case, then English examination promotion scoring points might have little impact on the likelihood of receiving a promotion. Alternatively, those participants may believe that English proficiency would have little impact on their
opportunity to receive a promotion. In that case, the promotion scoring associated with
achieving the CEFR A2 level may have little ability to motivate them to improve their English.
‧
A further 29.3% reported that they were not yet ready to take an English test. It seems likely that these participants may plan to take an English examination in the future. A further 12.1% of the participants reported that they had no time to prepare to take an English proficiency test, while 10.3% were not interested. Of these two groups, the former is more likely than the latter to decide at some point in the future to take an English test.
A total of 52.2% of those participants who provided information about having taken an English test reported taking just one, while 11.1% reported having taken two. Fewer than 4%
took more than two English tests, with one participant reporting having taken five tests, the highest number of tests taken by any participant.
In all, the participants reported taking 182 separate tests. Of these, 48.9% were TOEIC, with GEPT-E the next most commonly cited, making up 12.1%. Overall, 67% of those
participants who reported having taken an English proficiency test took a test developed by ETS, while 27% took an LTTC-developed test, and just 6% took one of Cambridge ESOL’s tests.
Refer to Figure 7 for a detailed breakdown of the tests taken.
48.9
TOEIC GEPT-E TOEFL PBT GEPT-I TOEFL CBT FLPT GEPT-HI IELTS BULATS TOEFL iBT
% of Tests Taken
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The participants were not asked to give a reason for choosing to take a particular English proficiency test. It would be useful for future studies on this topic to enquire into the factors that influenced this decision. Given this limitation, the observed preference for taking TOEIC by the study’s participants is likely attributable to several factors. First, TOEIC maintains a high visibility in Taiwan due to advertising and the marking efforts of ETS’s local agent.
Furthermore, TOEIC’s brand image benefits from its association with TOEFL, which has a long history in Taiwan. Thirdly, it is claimed that TOEIC is particularly suited to testing the English skills that are used in the workplace. It is likely that a narrower focus for the intended uses of an English proficiency test would support a claim that a test is better suited to a particular use. A testing system such as the GEPT, with a more broadly defined general intended use,
encompassing both academic and workplace contexts, could be perceived as less suited to a specific use than a test with a narrower context of use. Finally, the fact that TOEIC did not include a speaking test until 2006 may have given test-takers the impression that it would be easier to earn a higher score on TOEIC than tests that assessed all four language skills. The importance of this factor is undoubtedly decreasing now that TOEIC assesses all four skills.
Approximately one-half as many participants reported taking one of the GEPT levels as those taking TOEIC. Over one-half of those who took the GEPT reported having taken the
Approximately one-half as many participants reported taking one of the GEPT levels as those taking TOEIC. Over one-half of those who took the GEPT reported having taken the