THE UNIVERSITY OF HONG KONG LIBRARIES
This book was a gift from
The Hong Kong Institute of Education
PROJECT TEAM MEMBERS
Dr Peter Storey, Centre for Language in Education, Hong Kong Institute of Education Dr Jasmine Luk, Department of English, Hong Kong Institute of Education
Dr Jeremy Gray, Department of English, Hong Kong Institute of Education
Mrs Elena Wang-Kho, Centre for Language in Education, Hong Kong Institute of Education Dr Angel Lin, Department of English, City University of Hong Kong
Research Support Staff
Dr Lester Chan, Research Officer, Centre for Language in Education, Hong Kong Institute of Education
Dr Rita Shuk Yin Berry, Research Officer, Centre for Language in Education, Hong Kong Institute of Education
Ms Anna Tsang Ka Mei, Research Assistant, Centre for Language in Education, Hong Kong Institute of Education
Miss Cynthia Mok Ka Lai, Research Assistant, Centre for Language in Education, Hong Kong Institute of Education
Mr fed Kwok Ming Chi, Research Assistant, Centre for Language in Education, Hong Kong Institute of Education
Miss Iddy Lee Ho Shan, Research Assistant, Centre for Language in Education, Hong Kong Institute of Education
Mr Stephen Wan Ho Ming, Research Assistant, Centre for Language in Education, Hong Kong Institute of Education
PROJECT TEAM MEMBERS*
INTRODUCTION
1 HISTORICAL BACKGROUND OF THE NET SCHEME ~~~~ ~~.~~ « 6
1.1 INVOLVEMENT OF NATIVE ENGLISH SPEAKERS IN HONG KONG 6 1.2 RECOMMENDATIONS OF THE VISITING PANEL 6 13 THE EXPATOIATE ENGLISH LANGUAGE TEACHERS PILOT SCHEME 7 1.4 THE EXPATRIATE ENGLISH LANGUAGE TEACHERS MODIFIED SCHEME 7 1.5 THE NATIVE SPEAKING ENGLISH TEACHER SCHEME 8
2 SELECTING AN APPROPRIATE EVALUATION MODEL 10
2.1 THE CONTEXT ADAPTIVE MODEL . - 10
2.2 THE PROJECT FRAMEWORK APPROACH... 11
3 IMPLEMENTING THE EVALUATION MODEL 12 3.1 DETERMINING THE PURPOSE OF THE EVALUATION .'. 12 3.2 WIDER AND IMMEDIATE OBJECTIVES AND OUTPUTS OF THE NET SCHEME 12 3.2.7 Documentary Analysis 12 3.2.1.1 Selection of Documents for Analysis,—•. •> .. 13
33 WIDER OBJECTIVES THE SCHEME WILL HELP TO RESOLVE ... .— 15
3.4 INTENDED BENEFITS AND CHANGES THE SCHEME WILL BRING ABOUT 16 3.5 OUTPUTS, WAYS OF ASSESSING THEIR ACHIEVEMENT AND SOURCES OF INFORMATION 16 3.6 PARAMETERS OFTHE PRESENT STUDY 17 4 OBJECTIVES OF THE NET SCHEME ., ^^»«w«w«ww~w~«w.~.~. 18 4.1 PURPOSES OF THE EVALUATION . .." 18 4.2 RESEARCH QUESTIONS ... «...., 18
5 RESEARCH METHODOLOGY ~~~ ~~ ~ 20 5.1 TRACING THE PROFESSIONAL AND SOCIAL EFFECTS. . . 20
5.2 MEASURING LANGUAGE GAIN . ... . 20
5.2.7 Hypothesised Language Gains ... 21
5.2.1.1 Reading and Writing. ... . . 21
5.2.1.2 Speaking „ .. . ... 22
5.2.13 Listening M. . ... . 22
5.2.1.4 Attitude™ . . 22
5.2.2 Existing Instruments .. 23
5.2.2.1 language Assessments , . . 23
5.2.2.2 AttitudinalQiiestioiinaiTO,,..,...,.^^ ...23
5.3 SAMPLING PROCEDURES ... .... . ... . 24
5.3.7 Selection of Secondary Schools...* ..24
53.1.1 Sampling Criteria. . 24
53.1.2 Drawing up die Sample ... 26
53.13 Second Stage Sampling . . . 28
5.5.2 Selection of Primary Schools... 28
5.3.3 Sampling for Case Study Schools 29 € LANGUAGE PROFICIENCY ASSESSMENTS __ ^ 30 6.1 RESEARCH INSTRUMENTS ... . . ... .... ..,.. .. 30
6.1.1 Hong Kong Attainment Tests ... ... ... . 30
6.1.2 Oral Assessment Instruments... . ... . ... . ....32
6.1JL1 Secondary Oral Assessment ... .... . ... .%... , „.,., 32
6.1.2.2 Primary Oral Assessment. ... . ... . .. .. „. 35
6.1.23 Standardisation of Oral Assessment Procedures... ... . ... „ 38
6J.3 Listening & Writing Tests . . . .. ..„., ..„ „ 38
6,13.1 Secondary Listening and Writing Tests... . . ... ... „ „. 38 2
6.1.3.2 Primary Listening and Writing Tests...., 40
6.2 ANALYSIS OF LANGUAGE ASSESSMENT DATA . 43 6.2.1 Hong Kong Attainment Tests 43 6.2,1.1 Secondary Schools 44 6.2.2 Cross-sectional Analysis of Language Assessment Scores 48 6.2.2.1 Objective of Analysis... ... .„„ 43
6.2.2.2 Methodology. . , ..,48
6.2.2.3 Analysis of Coordinated Oral and Listening Scores 49 6.2.2.4 Analysis of Writing Scores ..61
6.2.2.5 Alternative Analysis of Writing Assessments .. 62
6.2.3 Longitudinal Analysis of Language Assessment Scores Using Repeated Measures Analysis 63 6.2.3.1 Objective 63 6.2.3.2 Methodology .. . 64
6.2.3.3 Assumptions ... ...~... ..„...„ 64
6.2.3.4 Limitation.. . . ... . 65
6.2.3.5 Findings... ... 65
6.2.4 Longitudinal Analysis of Language Assessment Scores Using ANCOVA 68 6.2.4.1 Objectives 68 . 6.2.4.2 Methodology. 68 6.2.4.3 Findings ., . 69 6.2.5 Alternative Analysis of Oral and Listening Scores 72 6.2.5.1 Paired Samples T-tests 72 6.2.5.2 * Multiple Regressions * 81 - 6.2.6 Follow-up research and analysis. ,. ....; 85
6.2.7 SecondStage Analysis of'Primary and Secondary Oral Assessment... 85
6.2.7.1 Review of First Stage Analysis..; ,.... 85
6.2.7.2 Variables Studied in the 2nd Stage Analysis 86 6.2.7.3 San^ling for Second Stage Analysis 87 6.2.7.4 Preparation of Scripts for Second Stage Analysis ... 88
6.2.7.5 Correlation Analysis between Ist and 2nd Stage variables. 89 6.2.7.6 Objective and Methodology . . . .. ... 90
6.2.7.7 Findings . ; .. . . . 91
6.3 LIMITATIONS OF INSTRUMENTS AND PROJECT PROCEDURE... 99
6.4 RECOMMENDATIONS , .... . 99
7 CASE STUDY __ ... - 101
7.1 PROFILES OF CASE STUDY SCHOOLS AND PARTICIPANTS _________ .... _________________ ... ________ . --- ... — 101
7.2 AcnvmEs UNDERTAKEN DURING CASE STUDY VISITS... ______ . __________ ... _______ . — . — ,.,...,... 101
73 DATA ANALYSIS OF CASE STUDY ., ___________________________ ... --- ... — ... — ... --- . --- 102
7.3 J Sources of Data ... ... ... . ... .... ... , ... . ... -.-.... ____ 102 7.3.2 Findings ______________________________ . ... . ... . ... ... ... ... ... 103
7.3.2.1 NET Effects through the Eyes of the School Personnel __________ ... --- .. --- 103
7.3.2.2 NET Effects through the Eyes of the NETs _____________________________ ... _____________________ .... --- .. — 105
7.3.2.3 NET Effects through the Eyes of die Students ... ... .. ... ... --- ... .. 107
7.3.2.4 NET Effects in the Classroom _______ .... ... . --- ... ... ... ... 109
7.3.2.5 NET effects through Extra-curricular Activities. --- ... --- . --- .... --- . --- ... 1 13 7.3.2.6 Unique Role and Positive Impact of NETs on the Teaching and Learning of English --- ...114
7.3.2.7 Optimal Conditions for a Successful Implementation of the NET Scheme --- . --- 1 17 8 STUDENT QUESTIONNAIRES^ - ~ - -- - - ~~ - 8. 1 INSTRUMENT DESIGN . ______________________________ . --- . --- ... ... 1 19 8.2 RELIABILITY ___________ ________________ . __________ . ________________________________ ... ... ______ . ... ... ... ... 122
8.3 ANALYSIS OF STUDENT QUESTIONNAIRE --- ... --- . --- . ... -..«.». 122
8.3.1 Atoto^/^gv... ________ ... ________ ... ___________ .. — . ... . ... . ... . ... -. ... 122
8.3.2 Findings. ______ . ... ... ... _________ . --- . — ... ... ... ... ... ... . ... 124
83.2.1 Cross-sectional Analysis of Student Questionnaire. — . --- . --- . --- --- . --- 124
8.3.2.2 Longitudinal Analysis of Student QuestionnaireUsingRepeatedMeasures Analysis... 132
8.3.23 Longitudinal Analysis of Student Questionnaire Using ANCOVA ... — .... --- ... ... 134
9 QUESTIONNAIRES FOR SCHOOL PERSONNEL^ - - ...— --- - 137
9.1 INSTRUMENT DESIGN __________ ... — .... --- . --- ... — ... . — 137
9.2 OBJECTIVE OF THE SURVEY ... 138
93 SAMPLING FOR THE SCHOOL PERSONNEL QUESTIONNAIRES ... 138
9.4 ANALYSIS OF QUESTIONNAIRE RESULTS ... . ... 138
9.4.1 Desirable qualities and skills ofNETs ... . ... . ... 139
9.4.2 Allocation of duties in school ... ... ... 139
9.4.3 Impact of the NET scheme on schools ... . ... 141
9.4.4 Collaboration between NETs and local teachers ... 142
9.4.5 Beliefs and practices regarding the teaching and learning of English ... ... . ... 142
9.4.6 Perceptions of students' English proficiency gain as a result of instruction by NET ... 143
9.4.7 Performance evaluations of NETs... — . ... 143
10 QUESTIONNAIRES FOR PARENTS --- . --- 145
10.1 INSTRUMENT DESIGN ... . ... ... 145
10.2 SAMPLING FOR THE PARENTS* QUESTIONNAIRES --- ... 145
103 OBJECTIVE OF THE SURVEYS ____ , ____________________________________ . ... 146
10.4 ANALYSIS OF THE QUESTIONNAIRE RESULTS ... . ... 146
10.4.1 Teacher qualities /skills ... . ... . ... 146
10.4.2 Allocation of duties ... . ... . ... .. ... 146
10.4.3 Impact on school.. ... . ... . ... . ... ... 146
10.4.4 Collaboration between NETs and Local Teachers ... . ... ... 147
10.4.5 Issues about teaching and learning English ... ... .*. ... 147
10.4.6 General. ... . ... ... 147
11 DISCUSSION^ _ ... .-. ... :... ... . _____ . __ , _______________ 149 11.1 OBJECTIVE 1 ________________________ ... ... ... ______________ . ... ... . ... ... ... 149
//././ Evidence from the Questionnaire Findings.... ... . ... . ... . ... 149
11.1.2 Evidence from the Case Study Findings ... . ... 151
1 1.1.2.1 Views of School Personnel ________________ . _________ . ___________________ . ____________ . ... . ... *... 151
1 1.1.24 View of Students . _____ ... _____________________________________ ... __________ . ... . __________ . ... ... 152
11.1.23 NET Effects tirn>ughExta<^ __________________ . ... . _______ ... ... 155
11.2 OBJECTIVE2.. _________________________________________________ ... _______ , ______________ ________ ...~. _______________ . ... . ... . ______ 156 11. 2 J Evidence from the Questionnaire for School Personnel and Parents ... ... ... ... ... 156
11.2.2 Evidence from the Case Studies.... ________ _____________________ ... ... ... ... 158
113 OBJECTIVES _______________________ . _____________________________________ __________ . _____ . _________________ . _______________________ . _________________ 161 11.3.1 Evidence from the Questionnaire Surveys for School Personnel and Parents ... 161
113.2 Evidence from the Case Study __________ . ___________ . ... ... ....162
11.3.3 Evidence from the Language Assessments. _________ . ________ . _________ ... . ... 163
1 1.33.1 Evidence from Cross-sectional Analyses ... 164
11334 Evidence from Longitudinal Analyses _____ ... _________ . ______ ... _________ .... ____________ . ___________ ... _________ 166 1L3.4 Evidence from the Attitudinal Questionnaires for Students... ... 168
12 RECOMMENDATIONS _ _ r i l l..1 ( l (_J. _ _ 172
176
APPENDICES Appendix!
Appendix E AppendixlII AppendixIV AppendixV Appendix W Appendix VH
INTRODUCTION
The Native English Teacher (NET) scheme provides funding for the recruitment of native-speaking teachers of English to serve in Hong Kong secondary schools. The present scheme commenced in the 1998/1999 school year.
Evaluation of the scheme was commissioned by the Standing Committee on Language Education and Research(SCOLAR) in June 1998. TheHong Kong Institute of Education (HKIEd), after a tender exercise, was given funding support from the Language Fund to conduct the evaluation which is known as The Monitoring and Evaluation of the Native-speaking English Teacher Scheme (MENETS). The evaluation commenced in November 1998 and covered a two-year period to November 2000.
The results of the evaluation are reported in two documents: the Evaluation Report which provides a non-technical summary of the major findings of the evaluation, and this document, the Technical Report, which provides more complete details of the evaluation.
After providing background information, including the identification of an appropriate evaluation model, the Technical Report goes on to describe the complete research process including the sampling procedures, instrument design and analysis of the data generated by the instruments1. Findings are reported and discussed in relation to the three major objectives of the scheme, i.e.
to enable native-speaking English teachers to enhance the teaching of English by:
1. acting as English language resource persons in the schools 2. assisting in school-based teacher development
3. Helping to foster an enabling environment for students to speak English and practise their oral skills.
Recommendations are made for the more effective operation of the scheme and for further investigation. Finally, the limitations of the present study are analysed and areas for possible further investigation are identified.
1 Readers interested in viewing the statistical data generated from the evaluation are requested to a|>ply in writing to MENETS Research Team, Centre for Language in Education, Hong Kong Institute of Education, 10 Lo Ping
1 Historical Background of the NET Scheme
1.1 Involvement of Native English Speakers in Hong Kong
A relatively recent rationale for the systematic recruitment of expatriate teachers for Hong Kong schools can be traced back to the recommendations of the Visiting Panel in 1982.
However, there is a 150-year tradition of involvement of native speaking English teachers in Hong Kong coinciding with the colonial phase of Hong Kong history (Sweeting, 1990).
Initially the influx of English speaking educators to Hong Kong was associated more with a desire to spread the gospel rather than the English language. This was related to missionary involvement in Canton, Malacca and Macau, and was not confined to educators of British origin (ibid. p. 20). Later, missionary zeal was transformed into a mission to civilise the local residents by freeing them of what were perceived as the shortcomings of traditional Chinese education. Frederick Stewart, founding head in 1862 of the Central School, recruited English native speakers from Britain (Bickley, 1997). Though Stewart's concern in this was with the provision of a British style education through the English language, it is interesting to note that he required staff recruited to the Central School to acquire at least a working knowledge of
Chinese (Bickley, 1998),
1.2 Recommendations of the Visiting Panel
In the period between these early beginnings and the 1980s when the large scale recruitment of expatriates was once again considered, the teaching of English in Hong Kong schools had undergone a process of what the Visiting Panel in 1982 called "localisation" which the Panel considered had led to deterioration.
We consider the 'localisation of staffing* policy ougiht to be amended so that children in their first years of schooling might be exposed to native English speakers, engaged as ancillary staff either on a contract basis or accepted as helpers (e.g. the non-working spouses of British expatriates or other suitable English speakers).
(Visiting Panel, 1982. m.L9)
In reality, in the period up to 1982, many local schools had employed native speaking teachers of English to enhance the teaching and learning of English, but this was done on an individual and ad hoc basis. The recommendations of the Visiting Panel sparked an interest, taken up in Education Commission Report No. 1 (ECR1), in the more widespread employment of expatriate teachers (Education Commission, 1984). ECR1 noted that provision already existed in the Code of Aid for up to three expatriates to be recruited for each school. This was not widely utilised, probably because of difficulties in providing adequate housing. The writers of the report recommended that schools should be encouraged to employ "locally available native English speakers5*, but stopped short of recommending large-scale external recruitment Continued concerns at falling standards of English and a desire to enhance the quality of English teaching in schools prompted the Government to consider extending the employment of expatriates to a broader range of schools and a proposal was put forward for a scheme which
"would in due course make available to each secondary school up to 2 or 3 qualified expatriate teachers of English, subject to fiinds being available" (Education Department, 1986 quoted in British Council, 1989), This prototype NET scheme began in 1987, was continued to 1991 and was revived in 1997 in the form of the current scheme.
1.3 The Expatriate English Language Teachers Pilot Scheme
The Expatriate English Language Teachers Pilot Scheme (EELTPS) began in August 1987 with the recruitment of 30 Expatriate English Language Teachers (EELTs). The Pilot Scheme was designed as a means of evaluating the impact of EELTs in a range of teaching situations in a representative cross-section of schools (British Council, 1989).
EELTPS was systematically evaluated through the exploration of multiple aspects of the impact of the EELTs on language learning and on the life of the school taking into consideration the views of various stakeholders regarding the contribution of the EELTS to school language education.
Quantitative measures employed in the evaluation focussed on Hong Kong Attainment Test results supplemented with results from specially designed tests of listening and speaking as well as questionnaires for students, teachers, principals and parents. Qualitative measures focussed on case studies of a sub-sample of schools.
Despite considerable initial problems (Boyle, 1997), reported outcomes indicated a generally positive impact for the scheme on English language learning. While refraining from drawing any conclusions as to the cost-effectiveness of the scheme, the evaluators concluded that EELTPS had had a positive effect on the teaching and learning of English in the Hong Kong secondary school and that educationally the continued employment of expatriate English teachers was viable (British Council, 1989 p. 182).
The EELTPS evaluation was rigorous and the conclusions drawn are objective. Alan Davies, who was responsible for the quantitative evaluation, is extremely careful in reporting results and points out from the outset that the identification of language gain is '^hugely fugitive" (ibid, p. 20). Despite having been able to capture this fugitive effect of the EELTs, Davies does not propose that the employment of more EELTs would necessarily solve the problems of falling English standards. He reminds the readers that there may be advantages to investing in the further training of local teachers as an alternative. He recommends further research, part of which would be an investigation of "the sense of local attachment and of continuity in the Hong Kong culture which by definition are not available to expatriates" (ibid p. 181).
This recommendation appears to be a veiled reference to problems associated with the acculturation (or lack of it) of the expatriate teachers, and the advantages which local Hong Kong Chinese teachers have over expatriates in this regard According to Boyle (1997), these problems were detailed in the Interim Report on the scheme, however this report is not in the public domain and so any interpretation of Davies* recommendation is speculative. It is interesting to surmise however, that two aspects of the present scheme may have drawn directly on some of the less positive experiences of the past scheme. These are the inclusion of issues of acculturation in the implicit objectives of the present scheme (see 1.5 below); and the strategic attempts made to effect a smooth transition into the local workplace and to encourage harmonious collaboration between local English teachers and NETs.
1.4 The Expatriate English Language Teachers Modified Scheme
EELTPS was a two-year pilot scheme which was completed at the end of the 1989 school year, The systematic involvement of native speaking teachers of English in Hong Kong secondary schools was not discontinued at that time, but continued in the form of a "modified" scheme
The Expatriate English Language Teachers Modified Scheme (EELTMS) involved 19 schools in the first year of operation and 14 schools in the second year. Thirty-three EELTs were employed in the 1989-1990 school year (on one-year contracts) and twenty-three in the
1990-1991 school year.
Evaluation of the scheme was less comprehensive than that for the EELTPS and was conducted internally by the Education Department. It comprised a quantitative analysis of HKAT results for students taught by BELTS. This analysis, however, did not make comparisons with results for students taught by local English teachers. The main part of the evaluation focussed on a series of questionnaire surveys of stakeholders addressing issues related to the implementation and effectiveness of the scheme with a focus on the effectiveness of the scheme management.
The conclusions drawn from the evaluation are positive. Principals and Panel Chairpersons were generally happy with the management of the scheme, and with the performance of the EELTs in their teaching and non-teaching duties.
1.5 The Native Speaking English Teacher Scheme
The systematic recruitment of native speaking teachers of English was re-introduced in 1998 as one of a series of measures designed to address the problems of perceived declining language proficiency identified in Education Commission Report Number Six (Education Commission,
1996, p. 18),
Though the findings of the EELTPS evaluation had provided a rather limited basis for recommending continuation, they were cited in the fourth Education Commission report as the justification for a decision to introduce a "permanent scheme'' in 1991 (Education Commission,
1990). This is however, the sole hint of permanence in any new scheme and the sixth Education Commission report suggests that this latest NET scheme was an interim measure designed to make up for a shortfall in the numbers of trained English teachers in the schools.
",. .before an adequate number of local teachers are trained, schools should be encouraged to employ on local terms more native English teachers who are qualified English language teachers, to teach the subject"
Education Commission, 1996. p. 4
The introduction of the scheme was announced by the Chief Executive of the Hong Kong Special Administ^ Region (HKSAR) Mr Tung, Chee Hwa in October 1997 on the occasion of the first policy address of the new HKSAR Government (Hong Kong Special Administrative Region, 1997). The scheme was implemented in September 1998 with the recraitaent of a total of 388 teachers in the 1998/1999 school year. The total number of teachers recruited undo: the scheme rose to 440 in the 1999/2000 school year and stands at 441 at the time of reporting.
Qmcuirentiy with the NET scheme proper, which was confined to secondary schools and special schools in the secondary sector, two separate school organisations received funding from the Language Fund and the Quality Education Fund to enable them to introduce expatriate teachers into their primary schools during the period 1998 to 2000, Under these two primary school schemes a total of 16 teachers were recruited In the current report, the term the NET scheme is used to refer to both the primary and secondary applications of the scheme in the period from September 1998 to July 2000.
The two primary schemes had a two-year life span in their original form (NETs continue to be employed in the primary sector in different ways, though not commonly simply as teachers).
The NET Scheme proper, however, was granted a five-year term when the ftmding for it was voted, with the implicit understanding that renewal of the funding would depend on achievement of expected outcomes.
2 Selecting an Appropriate Evaluation Model
The evaluation literature is very substantial indeed. The Hong Kong Institute of Education library, the largest in Hong Kong with a collection dedicated to education, lists 2677 titles which include the word "evaluation". This literature is replete with different models for the design of evaluation studies including Decision-oriented, Goal-free, Naturalistic, Illuminative, Transactional, Intrinsic, Responsive, Adversarial, and Judicial (Walberg and Haertel, 1990).
Rea-Dickins and Germaine (1998) in an overview of the current state of the art in educational evaluation, identify a movement away from empirical evaluation models to a more holistic approach drawing on Cronbach's view of the evaluator as "an educator whose success is to be judged from what others learn" (Cronbach et al, 1980). This entails moving away from "a narrow judgemental definition** which "presents evaluation as an objectives-driven undertaking associated with the conventions of experimental design" towards a view of evaluation as a "curriculum-focussed inquiry grounded in professional practice of an illuminative, responsive and developmental rather than recommendatory nature" (Rea-Dickins and Germaine, 1998 pps, 11-12).
Although modem educational evaluation studies would tend to adopt the holistic orientation outlined by Rea-Dickins and Germaine (op. cit), in policy-driven evaluation there are other priorities that need to be taken into account This is the case in the present study, where the client, while valuing the formative information which holistic evaluation can provide, requires a more traditional objectives-driven approach in addition. The financial commitment involved in the NET scheme is substantial. Moreover, the scheme is perceived to cany an implicit threat to the job security of existing English teachers and hence is being kept under close scrutiny.
These factors militate against an evaluation which is solely "illuminative, responsive and developmental" and will certainly require account to be taken of cost effectiveness when the policy makers come to evaluate the evaluation.
Funding for the scheme was approved on the basis of a rationale setting out the expected benefits it could bring (see section 3*2.1.1 and Appendix I below). Evaluation therefore needs to include judgement of the extent to which the expected benefits have been realised.
In seeking an appropriate model to structure the evaluation, therefore, reference was made to two relatively recent approaches which together satisfy both the modern call for a more formative evaluation, as well as the policy-makers need for a more objectives-driven study.
These are the models proposed by Dynch (1990,1996), a^
2.1 The Context Adaptive Model
Lynch (1990) proposes an evaluation model focusing in particular on the evaluation of language programmes. This Context-Adaptive Model (CAM) provides a simple heuristic for the present study unencumbered by many of the constraints which some of the more rigid models proposed in the literature entail (Lynch, 1996). It provides seven simple steps which can structure the evaluation process as follows.
L Determine the purpose of the evaluation 2. Determine what is being evaluated
3. List the themes wMch will structure the evaluation 4* Design the evaluation
10
5. Collect the necessary data 6. Analyse and interpret findings
7. Communicate the evaluation findings 2.2 The Project Framework Approach
Weir and Roberts (1994) also focus specifically on the evaluation of language programmes and initiatives. They draw reference in particular from the extensive experience of the British Council and the United Kingdom Overseas Development Agency in evaluating the effectiveness of sponsored language enhancement initiatives and projects. The Project Framework approach derived from this experience (ibid. p. 245) provides a matrix for analysing evaluation problems which can usefully complement the CAM procedure.
In the Project Framework approach, (i) immediate and (ii) wider objectives are identified as well as (Hi) outputs from and (iv) inputs to the project Indicators of achievement and means of verifying achievement are identified and underlying assumptions are clarified. The table below is a worked example of a project framework tailored to the present evaluation, focusing on the wider and immediate objectives, the project outputs, achievement indicators and the means of verification. The questions posed in each cell, serve as a checklist for assessing the evaluation design. .
Questions addressed in the Project Framework approach for MENETS
Project narrative Achievement indicators Means of verification Wider objectives
What are the wider objectives or problems which the NET scheme will help to resolve?
What are the quantitative or qualitative ways of assessing whether the wider objectives have been achieved?
What sources of information exist or can be provided cost-effectively?
Immediate objectives (project purpose) What are the intended b enefits and changes which the NET scheme will bring about?
What quantitative or qualitative evidence will be used to judge effects and benefits?
What sources of information exist or can be provided cost effectively?
Outputs
What outputs (kind, quantity and by when) are to be produced by the Scheme in order to achieve the objectives?
What are the sources of information?
Project Framework adapted from Weir and Roberts, 1994, p. 245
The seven-stage CAM procedure, together with the prompts provided in the project framework approach template above provide a useful structure for reporting the evaluation of the NET scheme and will be adopted in the following-
3 Implementing the Evaluation Model 3.1 Determining the purpose of the evaluation
The evaluation of the NET scheme was put out to tender. The tender document is not a detailed specification and leaves several assumptions unstated. In bidding for the tender, the evaluation team set out an interpretation of these assumptions in the form of research questions and areas for investigation in the project After some negotiation with representatives of the steering body, the Standing Committee on Language Education and Research (SCOLAR), the bid was accepted. This acceptance was an implicit recognition that the purpose of the evaluation had been addressed by the project design.
This acceptance of the implicitly agreed purpose of the evaluation does not provide a clear and unambiguous foundation upon which to develop the project. A more explicit statement o f purpose was needed to guide the implementation of the proposal. Before finally determining the purpose of the evaluation of the NET scheme, it was necessary to identify the purpose of the scheme itself.
The prompts of the Project Framework approach serve as useful guidelines in the process of identifying the puipose of the evaluation. The wider objectives or problems which the project will help to resolve and the intended benefits and changes which the project will bring about as well as the outputs to be produced by the scheme to achieve the objectives (Weir and Roberts, 1994 p. 245) can be identified by documentary analysis of the various pronouncements, reports and other documents which led to the introduction of the scheme.
3.2 Wider and Immediate Objectives and Outputs of the NET scheme
Official statements of the objectives of the NET scheme are characteristically terse and tend to take the benefits of employing native speaking language teachers as understood. This is evident in statement,going back to the report of the Visiting Panel in 1982 and culminating in ECR6 in 1996 (see section 3*2.L1 and Appendix I). On the other hand, there is a growing body of literature which has questioned the implicit assumptions underlying the employment of language teachers on the basis of first language or racial origin and which has characterised the role of the English native speaker as a fonn of linguistic imperialism (Boyle, 1997; Cook, 1999;
Kachru, 1985; Medgyes, 1992; Phillipson, 1992; Pennycook, 1994; Hampton, 1990;
Widdowson, 1994,1999).
The rationale for the introduction of the scheme gave the appearance of being anchored in an earlier tradition, one that saw native speakers as necessarily the best teachers of second language. On the other hand, it is difficult to see the scheme as a form of linguistic imperialism since it was introduced by a post-colonial regime which had just instituted Chinese as an official language of Government for the fast time. Nevertheless, the scheme has been interpreted as a form of linguicism (Law, 1997; Lai, 2000) and hence it is necessary to unpack the implicit rationale which underlies its introduction. Documentary analysis is a useful means ofdoingso.
3JL1 Documentary Analysis
In order to be able to make the expected outcomes of the NET scheme explicit and set parameters for evaluating the achievement of these outcomes, it is necessary to trace from the
12
available evidence, what the assumed benefits of the scheme are. The following section therefore attempts to trace the assumptions underlying the NET scheme in order to identify the implicit objectives of the scheme. This is done by identifying key phrases from the relevant documents and pronouncements.
A similar process was undertaken by Johnson and Tang (1993) two of the evaluators of the EELTPS. They conducted an analysis of documents, pronouncements and views from various sources including the British Council (who were responsible for the organisation and management of EELTPS), the Education Department, the media and key stakeholders in the scheme. Based on this analysis, Johnson and Tang identify two levels of aims for the scheme — official stated aims, and 'controversial* unofficial aims (ibid, p.206).
The official aims they identified wore two-fold, namely:
1) to raise the quality of English language teaching in the schools;
2) to improve the standard of English of the students taught by the expatriate teachers.
What Johnson and Tang consider more important however, were the unofficial aims, namely To change the sociolinguistic climate of the workplace: Le. to change the school from an ethnocentric, monolingual, Hong Kong, Cantonese-speaking environment, to a bilingual, bicultural, internationally-orientated workplace:
a. by providing a need for Chinese staff to use English for professional, administrative and social purposes; and
b. by providing opportunities for students to interact with non-Chinese, non-Cantonese-speaking members of staff outside as well as inside the classroom.
(ibidp.20S-6)
3.2.1.1 Selection of Documents for Analysis
In the present study, the documents analysed are as follows:
Report of the Visiting Panel,1982
Education Commission Report No. 1,1984
Report of the Working Group on Language Improvement Measures, (Education Department 1989)
Education Commission Report No. 6,1996 Chief Executive's Policy Address, October 1997
Finance Committee Agenda Item papa: for Finance Committee Meeting on 21 November 1997 Net-working: Examples of Good Professional Practice within the NET Scheme, May 2000
Since the intention is to unpack the implicit rationale for policy makers' decision to introduce the scheme, the selection of documents is narrower than that of Johnson and Tang (op> cit) and has concentrated on official papers.
Relevant extracts from these documents are reproduced in Appendix I with phrases giving clues as to 1he objectives of the NET scheme underlined. Inferring from the key phrases in
these documents allows for the identification of the purpose of the NET scheme which can be stated as follows,
NETs are seen as a solution to the problem of low standards of English among teachers in Hong Kong schools and the quality of language teachers must first be improved before language standards can be raised. There is definite educational benefit to be gained from teaching English in schools by native English speakers and NETs can help to raise fee quality of the teaching and learning of English in schools.
The long-term solution may be to have adequate training for local teachers, however, before an adequate number of local teachers are trained it is expedient to recruit large numbers of NETs. It is expected that local teachers and expatriate teachers will work together in the teaching of English, To make an immediate impact on improving the English language standard of our students^ NETs will therefore be introduced to all schools. They will help to enhance the teaching of English in schools adopting Chinese as the MOL They will act as English language resource teachers, assist in school-based teacher development and help foster an enabling environment for students to speak English and practise their oral skills. The scheme can play an important role in enhancing English language proficiency in furthering Hong Kong's status as a world-class international centre. The key roles of the NET are: to enhance the English language proficiency of individual students; to demonstrate contemporary approaches to the teaching and learning of English and to share professional ideas with their fellow English teachers.
Further clues as to the thinking underlying the re-introduction of the scheme announced in the October 1997 Policy Address can be found in a circular issued to schools soon after the announcement, listing possible duties for NETs. An extract from this circular is reproduced below.
The exact duties of NETs should be determined by the principals according to the specific needs of the school It is proposed that the duties of NETs should cover the following:
1) To be responsible for classroom teaching and assessment;
2) To provide support to the English Panel Chairperson, including assisting in the tailoring of the curriculum and preparation of teaching materials;
3) To assist in conducting extra-curricular activities related to the English language, e.g.
speech, drama, debates, choral spealdng and extensive leading;
4) To assist in nuining more oral activities for students after school;
5) To assist in setting up an English comer in the school where students can come together to practise oral English and read English books under their guidance; and
6) T\> act as an BogBsh language r^^
GuMelines on the duties o^
Education Department, November 1997
In the conceptualisation of the role of the NETs which underlies this specification, we can discern a possible clarification of the "educational benefit" which was mentioned in ECR1.
According to the Education Department document, the primary benefit to students from being exposed to a NET would appear to bem the area of oral proficiCTicy. While the NET has a clear role in cuniculum tailoring and materials development, the emphasis of the suggested activities
14
the NET would undertake is on oral skill development with a secondary emphasis on extensive reading.
A final source which was referred to in identifying the implicit purpose of the NET scheme and hence of the evaluation of the scheme was a letter from SCOLAR to Heads of Institutions dated 23 June 1998, in which bids were invited for the evaluation of the project A relevant extract from this document is reproduced below.
Researchers must consider all aspects of the scheme including induction programmes for the Native-speaking English teachers, the contents and syllabus of their teaching, the teaching and assessment methods used, student benefits, and professional exchanges between local and native-speaking teachers, etc-... The research methodology employed should allow inferences to be made on the effectiveness of the NET scheme towards students* development of English language proficiency and in terms of integration of NETs into the school adTT"Tristration« with colleagues etc.
Letter from SCOLAR to Heads of Institutions, 23 June 1998 In this tender specification the expected student benefits are more general, not being stated in terms of oral proficiency alone. Moreover there is a clear expectation that the effectiveness of the scheme should be determined partly in terms of how well NETs integrate into the life of the school. - .
From the above exercise we can draw conclusions which provide answers to the questions posed in the Project Framework grid on page 17 above.
33 Wider objectives the scheme will help to resolve
The following question is posed in the Project Framework approach:
What are the wider objectives or problems which the NET scheme will help to resolve?
The scheme is seen as one of a number of possible solutions to the problem of the falling standard of English in Hong Kong. The fall in standards is attributed in part to the poor qiiality of language teaching in the schools which is in turn a result of the imufficient numbers of qualified language teachers. The NET scheme is a short- to mid-term measure to raise the quality of language teaching by increasing the number of subject-trained, graduate English teachers. The expected educational outcome of this is improvement in the language standards of secondary school students with a particular focus on their oral ability. The scheme can also play a role in compensating for the impact of loss of an English-speaking environment in CMI schools. The NETs can save as resource persons in the schools contributing to the development of teaching materials and to the professional development of other staff both implicitly, by serving as a model and sharing professional ideas, and explicitly by providing training.
There is not a great deal of evidence in the official documentation to support the sociolinguistic aims' detected by Johnson and Tang (1994) Le. to change the school from an ethnocentric, monolingual, Cantonese-speaking environment, to an inteniationally-oriaitated, bilingual, bicultural workplace. However, the emphasis on integration of NETs into the life of the school noted in the official documentation indicates that the acculturation problems experienced by expatriate teachers in the EELTPS and EELTMS had been recognised. There was recognition
that effort needed to be made to avoid a repetition of the same socio-cultural problems seen in the earlier schemes.
3.4 Intended benefits and changes the scheme will bring about The following question is posed in the Project Framework approach:
What are the intended benefits and changes which the NET scheme will bring about?
Based on the identified objectives, we can infer that the scheme is expected to result in improvement in the quality of language teaching through a system where NETs are integrated into the life of the school, serve as models of good practice, produce teaching resources and effect gains in student language proficiency.
Not all*>f these benefits would be measurable within the term of a two-year evaluation project such as that reported here. The recruitment of the planned 700 NETs would have an immediate effect in increasing the proportion of qualified English teachers in the school. A measurable increase in the quality 6 f English language teaching is unlikely to occur in the short term however solely by the addition of one or two qualified native speakers to the school staff.
Indeed the impact of theHET on the enhancement of language teaching in general is'likely to be difficult to quantify due to the range of intervening factors at play, including other Government initiatives to raise the quality of teaching in general and language teaching in particular.
3*5 Outputs, ways of assessing their achievement and sources of information The following questions are posed in the Project Framework approach:
What outputs are to be produced by the scheme to achieve the objectives?
What are the quantitative or qualitative ways to assessing their achievements?
What sources of information exist or can be provided cost effectively?
Using the terms of the Project Framework approach we can characterise the objectives, outputs, achievement indicators and means of verification as follows.
Project Framework—achievement indicators and means of verification
Objectives Achievement Indicators Means of Verification
Enhanced quality language teaching
Better public examination results; greater employer satisfaction; NET inputs to the school curriculum, resources and language learning provision;
involvement of NETs in teacher development activities
HKEA Statistics; Employer feedback;
Quality of NET contribution to the curriculum, resources and language learning provision Integration
of NETs into the life of the school
Good professional and social relations between NET and local teachers; NET job satisfaction;
positive reports from school personnel
Case study interviews and observations;
Questionnaire responses Enhanced
student proficiency
Enhanced performance on HKAT; gain on language tests administered before and after exposure to NET teaching; improved attitudes to the language and its speakers leading to greater motivation to learn
HKAT results;
Results on specially designed language tests;
Improved student attitudes as measured by questionnaire 16
3.6 Parameters of the Present Study
The table above serves as a specification grid analysis of which can serve as a means of setting parameters for the present study. The next section presents the outcome of this analysis in the form of the scope of the study, what it can achieve, the research questions formulated to drive the investigation and the methodology adopted to carry it out
In order to view the present study from the correct perspective, it is worthwhile reminding ourselves of the difficulties which it is has to attempt to overcome. In particular, the measurement of language gain unambiguously resulting from a particular treatment has been recognised in the academic literature as being a notoriously difficult exercise. Recent evaluation models have emphasised a more holistic approach in which the impact of an innovation on the quality of educational outcomes is charted by means of narrative description rather than by quantitative means. Berretta (1992) recognises these difficulties but points out that studies of the impact of an innovation on the quality of education are not what policy makers need. "Comparative outcomes based on tests are precisely what the policy making community needs to know about" Berretta advises that, in this circumstance, "in spite of known difficulties, it is appropriate to make the best use of whatever tools are available."
Alderson (1992) raises a serious concern which quantitative evaluators need to bear in mind when communicating results. This is related to the fact that the audience for evaluation results may not be fully aware of what can reliably be shown by means of quantitative evaluation.
According to Alderson, policy makers "often., .require test data, understandably enough if they are interested in outcomes, yet lack the professional expertise to recognise the appropriacy (sic) of instruments, the misapplication of tests, or to interpret test results with suitable caution.
Tests yield figures, figures look like hard data, and tend to be believed, naively." (ibid. p. 284).
In a situation where large sums of public money have been invested in a not uncontroversial innovation, there will be considerable public interest in results which can show unequivocal benefits. This poses a particularly serious problem when the innovation in question is not likely to yield concrete outcomes in the short to medium term.
4 Objectives of the NET Scheme
Three major objectives were set for the Native-speaking English Teacher Scheme in the 1997 Finance Committee paper (Education Department, 1997a). According to that document, the objectives of the native-speaking English teacher scheme are:
To enable native-speaking English teachers to enhance the teaching of English by:
a) acting as English language resource persons in the schools b) assisting in school-based teacher development
c) helping to foster an enabling environment for students to speak English and practise their oral skills.
4.1 Purposes of the Evaluation
The major purpose of the evaluation is to assess the extent to which the objectives of the scheme had been met within the first two years of its operation. Drawing on the contextual analysis reported in the previous section, the purposes of the Monitoring and Evaluation of the Native English Teacher Scheme can be stated as follows:
1) to conduct an in-depth and comprehensive investigation into how the NET scheme operates in the ecology of local schools;
2) to identify the effects of the scheme on the key players and how they react to it; and 3) to carry out a quantitative evaluation of the effectiveness of the scheme in generating
language proficiency and attitude gains among the pupils involved.
These can be stated more specifically as follows:
• to examine, over a two year period, the teaching and learning approach adopted by NETs including their approach to the school syllabus, their influence on it and contributions to it, in the form of teaching, learning and assessment materials, methods and approaches;
• to investigate the effects of the introduction of NETs on the social and professional life of the school*
• to investigate the effects of the scheme on English language proficiency and attitude towards English among pupils in the schools involved;
4.2 Research questions
The purposes of the evaluation were operationalised in the form of three major research questions which served to guide the investigation:
1. What is the nature of the teaching and learning approach adopted by NETs and how does it differ from that of local teachers in tennsof:
a, approach to and influence on the school syllabus?
b. teaching Jeaniing and assessment materials, methods and approaches adopted?
2. What are the effects of the introduction of NETs on the social and professional life of the school?
How do NETs and local teachers collaborate and exchange expertise on professional issues such as developing the syllabus, production of instructional materials, organising extra-curricular activities, and management of students?
18
3. What is the effect of the NET Scheme on student language learning in terms of the English language proficiency and attitudes towards English among pupils in the schools involved?
In addition, in order to monitor the induction programmes, a fourth research question was formulated as follows:
4. How effective are the NET induction programmes in:
a. helping NETs understand the situation in the schools?
b. ensuring that NETs adapt to the school situation and the local environment?
c. enhancing the work performance of NETs?
5 Research methodology
The research questions were investigated by means of language proficiency assessments, questionnaire surveys, and case studies. The applications of the various methods are shown in the table below. Detailed descriptions of the various instruments and of the case study methodology is provided in section 6 below.
Research methods employed to investigate the research questions Research Questions
1. Effectiveness of induction
2. Nature of NET teaching and learning approach
3. Effects of NETs on the social and professional life of the schools 4. Effect of NET Scheme on language . learning
Investigated by:
Examination of materials & evaluation reports Observation of sessions
Follow up interviews Questionnaire surveys Questionnaire surveys Case studies
Questionnaire surveys Case studies
Measurement of language gain Questionnaire surveys
The overall intention was to identify the nature of the influence of Native speaking teachers on the professional and social life of the schools involved and on the language development of pupils and students. This influence will be referred to in the following as a NET effect.
5*1 Tracing the professional and social effects
As indicated above the NET effect was traced in a variety of ways. The impact of NETs at the school level was measured by questionnaire surveys in a quasi-experimental design which attempted to trace developing attitudes among all the key players including the students involved and their parents. These were designed to supplement the main avenue of investigation of these aspects of the scheme which was conducted by means of in-depth qualitative investigation in seven schools using case study methods. The case study data generated from the investigation is considered to be the most reliable and authentic since it represents a more direct source of information as to the effects of the scheme at the school level.
5.2 Measuring Language Gain
The NET effect on language proficiency and attitude to language was traced by means of investigation on experimental groups made up of primary and secondary school students taugjit by NETs matched with control groups from schools not employing NETs or from classes not taught by NETs.
As we have seen in section 3 above, part of the rationale for employing NETs in the local schools is to improve the quality of language teaching. This enhanced quality language teaching is expected to result in a greater improvement in language proficiency among the students taugfct by the NETs, compared to those students taugfrt solely by local teachers.
In order to explore this expectation, it is necessary to make comparisons between local Engjish teachers and NETs in terms of the results achieved by the students they teach. We can translate
20
this into a testable hypothesis, i.e. that students taught by NETs will perform better in certain aspects of language use than students of a similar ability who are taught by local teachers.
The aspects of language use hypothesised to be the most likely focus of this language improvement will be related to special qualities and abilities of NETs which distinguish them from local teachers. It is not possible to generalise with any accuracy about the qualities and abilities which local teachers possess, given the great range of abilities, experience and educational background which local teachers may have. Moreover a similar range of backgrounds and experience can be found among the NETs recruited for the scheme. However, one clear distinguishing feature stems from the feet that English is the native language of the NETs.. Another distinguishing feature of the majority of NETs, is that they do not speak the first language of their students. These characteristics imply that the presence of the NETs will encourage students to interact more in English, there being no other choice if students are going to communicate with the NET. The presence of the NETs will also expose students to a form of spoken English which represents a native standard as distinct from a local Hong Kong standard.
The job duties of the NETs as advised by the Education Department in a circular to schools (Education Department 1997b) focus on the potential of the NET in developing oral skills in the students. It is also clear from the Education Department's celebration of the success of the scheme in the booklet entitled 4TMet-working" (Education Department, 2000), that the main focus of NET teaching activities is on oral skill development. This concentration on oral skills, is not advised to the exclusion of other skills; indeed the first case study in Net-working focuses on an integrated skills approach. However there is sufficient evidence in the documents to lead to an expectation that any efifects of the NET would be concentrated in the area of oral competence.
Based on these distinguishing characteristics of the NETs and their expected teaching and learning focus in the schools, a series of hypotheses was drawn up to guide the evaluation of the impact of NETs on student Ability in the four language skills of reading, writing, speaking and listening.
5.2.1 Hypothesised Language Gains 5.2.1.1 Reading and Writing
Given the expectation that NETs would establish an English Corner in the school which would generate extensive reading activities, it might be expected that improvements in reading ability would result However, this wouM apply to aE students-th^
those not, since the English Comer is an extra-oirricular resource. Any effect of the English corner in terms of improved reading ability or attitude to reading would be felt across all users of the resource and not confined to students in classes taught by the NET. The major emphasis required of NETs, as listed in the guidelines issued to schools (Education Department, 1997b), was on the practice of oral skills. Assuming that the guidelines were followed, there would be little evidence to support a hypothesised gain in reading ability, as well as in the related skill of writing.
It was therefore hypothesised that there would not be any significant differences in the performance of students taught by NETs and those taught by local teachers on assessments of reading and the related skill of writing defined in tarns of the skills normally taught, practised and assessed at the levels in question.
5*2*1,2 Speaking
For speaking, given the emphasis in the job description of the NETs (Educational Department 1997b), it was hypothesised that there were likely to be differences in the performance of students taught by NETs and those taught by local teachers on assessments of spoken ability defined in terms of confidence, fluency, obtrusiveness of pronunciation errors, effectiveness of communication strategies and grammatical accuracy in the spoken English skills normally taugjit, practised and assessed at the levels in question.
5*2.13 Listening
Again, given the expected emphasis of NET teaching on oral-aural skill development, for listening, it was hypothesised that there were likely to be differences in the performance of students taught by native speakers of English over pupils taught by local teachers in the ability to understand natural spoken English, where natural spoken English is defined as a variety of English with phonological features characteristic of first language varieties, but in which speed, lexis and vernacular usage are controlled to the level of the language learner.
It was further hypothesised that as compared to students taught by local teachers, students taugjit by native speakers of English would exhibit:
a) greater confidence in processing natural spoken English, indicated by less hesitation in responding to commands and instructions;
b) higher levels of accuracy in identifying the intended meaning of spoken utterances delivered in natural spoken English.
5.2.1.4 Attitude
The role of affective factors in language learning has been well documented in the literature.
Gardner's seminal publication, Social psychology and second language learning: the role of attitudes and motivation, (Gardner, 1985; and also Gardner, 1972) crystallised the issues and provided the impetus for a plethora of studies exploring the relationship between attitudes to the language and the effectiveness of language learning. The research provides evidence of a strong positive relationship between the effectiveness of language learning and positive attitudes to the language in question (Oiler et al, 1972; Pierson & Fu» 19S2; Gardner &
Madntyre 1991, 1993; Baker, 1992; Crew, 1994, 1995; Mantte^romley, 199 These attitudes may be related to intrinsic or extrinsic aspects of motivation (Sadtono, 1991; Williams,
&Burden, 1997; Ehman & Dom^ 1998; Dornyei, 2000).
Based on the attitudinal and motivational literature, hypotheses were formulated with respect to language attitude which guided the evaluation team in the design of an instrument (based on existing instruments (British Council, 1989; Hepburn, 1991) to explore the developing attitude to English and to English speakers which could be attributed to exposure to a NET.
It was hypothesised that student attitudes towards English would b e enhanced b y p ositive experiences of exposure to native speakers of the language. It was likely therefore that there would be differences in attitudes towards English among students who were exposed to a NET
22
or taught b y aNET and those who were not This enhanced attitude was hypothesised to manifest itself in an increase in their willingness to expose themselves to more English in the form of reading, watching and listening to more English. In addition, if the scheme has an effect on enhancing proficiency in English, these forms of the language would he made more accessible to students.
Being exposed to a NET was hypothesised to bring the important realisation among students that English is a medium of communication, rather than just a school subject. This may result in greater willingness to use English in communicating with friends and greater confidence in communicating with native speakers of English generally.
In order to test these hypotheses it was decided to employ existing assessments supplemented by commercially available tests and/or specially developed instruments as necessary.
5*2.2 Existing Instruments
5.2.2.1 Language Assessments
All primary school students and junior secondary school students are expected to take standardised annual tests of Chinese, English and Mathematics developed by the Educational Research Section of the Education Department. For English, these Hong Kong Attainment Tests comprise assessments of reading, listening and writing. The tests are designed for various purposes including the assessment of achievement in English at each school level, the diagnosis of strengths and weaknesses in the subject "for guidance and counselling59 and the identification of less able pupils for remedial treatment. (Hong Kong Government 1997a;
1997b). The evaluation of the EELTPS employed Hong Kong Attainment Tests as the prime source of data regarding improvement in student language proficiency resulting from the scheme (British Council, 1989).
With the exception of the most recently developed tests for the primary sector, Hong Kong Attainment Tests (HKATs) do not include an assessment of spoken English ability. In addition, examination of samples of HKATs revealed that the assessment of listening and writing was somewhat limited in relation to the language proficiency gain hypothesised to result from exposure to native-speaking teachers as opposed to local teachers. It was therefore considered essential to identify or develop tests to supplement the information which would be provided by the HKATs. Similar considerations had motivated the EELTPS evaluators to develop supplementary tests of listening and speaking. Unfortunately, however, those tests were not available for use in the present project Section 6 below describes the instruments which were developed specially for the current study.
5.2.2.2 Attitudinal Questionnaires
It was not possible to identify an existing questionnaire tailored to the precise needs of the evaluation of the NET scheme and able to address the gains in language attitude and motivation hypothesised to result from the scheme. It was necessary therefore for the evaluation team to design a new instrument In doing so however, reference was made to an instrument designed for two earlier local studies. One was the instrument developed for the evaluation of the EELTPS by the British Council which included semantic differential items eliciting
perceptions of personality traits of speakers of English (British Council, 1989), The second was an instrument developed for a study exploring the relationship between motivation, attitude and effective language learning strategies (Hepburn, 1991).
53 Sampling Procedures
The research instruments were applied to two samples of primary pupils and secondary school students taught by native English teachers. These two samples are known as the first and second cohorts in the following discussion. The two cohorts are distinguished by the length of exposure to the NET as follows:
First Cohort - Secondary school students and primary school pupils taught by a NET in the period September 1998 to July 2000;
Second Cohort - Secondary school students and primary school pupils taught by a NET in the period September 1999 to July 2000.
The measures taken to ensure that the first and second cohort samples were representative of the population are described below.
53.1 Selection of Secondary Schools
53.1.1 Sampling Criteria (A) Fundamental Criteria
Multi-stage stratified random sampling was employed to develop the sample o f secondary schools for the project using the following criteria:
School district;
Medium of instruction;
Gender balance;
Funding type;
Religious affiliation;
Level of ability of students in the school.
The last oiterion, level of ability of students in the school, was used to group schools into three levels of student ability - Low, Medium and HfigjL This was based on the relative proportions of students in the school falling into different levels of ability as indicated by the Planning and Research Section of the Education Department of the HKSAR
(B) NET Deployment
Another important variable was the deployment pattern of the NETs in terms of the type of teaching they were expected to undertake in each school context In the absence of central information on NET deployment patterns, the project team undertook a survey to obtain a better understanding of the general pattern of NET deployment. Simple questionnaires were faxed to all secondary schools employing NETs (n = 388) requesting information about the mode of deployment of the NET* One hundred and seventy-eight questionnaires were returned, a return rate of 46%. Although the return rate was not high, the responses showed a clear and
24
consistent pattern with three major modes of deployment clearly identified: Oral-only, Whole class and Split class,
• In the Oral-only mode, the NET was deployed usually to teach at least one lesson per week or per cycle to all classes in the school.
• In the Whole class mode, the NET was assigned up to four whole classes and asked to teach all the English lessons to those classes.
• In the Split class mode, the NET was assigned parts of up to four classes and asked to teach all the English lessons to those students.
Split Class Deployment Patterns
The patterns of deployment in the split class mode were varied with at least seven possible variations as follows:
1. One class is split at random (i.e. not according to the ability of the pupils) and the NET is deployed to teach half the class for the whole term or year, while the other half of the class is taugjht by a local teacher.
2. One class is split according to the ability of the pupils and the NET is deployed to teach the higjier ability half of the class for the whole term or year, while the other half of the class is taught by a local teacher.
3. All classes at a particular level are split at random (Le, not according to the ability of the pupils) and the NET is deployed to teach half of each class for the whole tenn or year, while the other half of each class is taught by a local teacher.
4. All classes at a particular level are split according to the ability of the pupils and the NET is deployed to teach the higher Ability half of each class for the whole year, while the other halves of the classes are taught by local teachers.
5. One class is split at random (Le. not according to the ability of the pupils) and the NET is deployed to teach the higher ability half of the class for part of the year (e.g. alternate cycles or weeks), while the other half of the class is taught by a local teacher.
6. One class is split according to the ability of the pupils and the NET is deployed to teach half the class for part of the year (e.g, alternate cycles or weeks), while the other half of the class is taught by a local teacher.
7. A class is taught by both the local teacher and the NET in a team-teaching mode.
Typical Deployment Pattern
The range of variations in split class teaching created a complicating factor which was compounded by the feet that in both the Whole and Split class modes, NETs could also teach in the Oral-only mode for different levels of class. A typical deployment pattern might be:
2 F3 whole or split classes;
1 F4 whole or split class-
Oral lessons for all Fl and F2 pupils; and Extra-curricular activities (EGA).
In this case, any NET effect in the school would be felt in varying degrees by selected F3 and F4 students (tiu^ by all Fl and F2 students (through oral only teaching) and by the students in the whole school (through EGA)*
