Case Studies

3.6.1 Sampling Procedures

The procedure for the selection of case study schools and the use of survey data to inform the design of case studies is summarised here. Schools were selected for case studies based on several criteria. Since the findings from the case studies are not meant to be used to make generalisations to all schools in Hong Kong but instead to provide a deeper understanding of specific cases, we selected them purposefully and not randomly and sought out a diversity of school types and contexts.

By means of a question at the beginning of the School Principals’ Survey, Principals were invited to express willingness to be involved in the qualitative phase of the evaluation as case study schools. Nineteen indications of willingness to participate were received. These schools were prioritised based on selection criteria designed to create a sub-sample of schools which, while being broadly representative in terms of school types and contexts, could provide the Evaluation Team with a deeper understanding of specific cases in those contexts.

The case study phase of the evaluation involves 8 case study schools. Criteria to select these 8 schools from the 19 schools willing to participate took three factors into account: (i) representativeness in terms of size, location, student ability, medium of instruction, funding basis and whether the school was single sex or co-educational; (ii) NET characteristics and deployment pattern; and (iii) qualitative analysis of views expressed in completing the survey.

In terms of size and student ability, the team looked for balance, with at least two large, two medium and two small schools and at least two schools from each of the three bands. At least two schools should be included from each of the four regions: Hong Kong, Kowloon, the New Territories East, and New Territories West. A balance of aided and Government schools should be included, with a representative number of schools having different Medium of Instruction arrangements.

125 In terms of NET characteristics, the team sought a balance of genders. Number of years’

teaching experience in total, and number of years in the current school, as well as age, were also taken into consideration. Finally, deployment modes were looked at to ensure that a suitable balance of oral only, full-class only and mixed deployment modes was included.

Finally, responses provided by NETs, Principals, local English teachers and English Panel Chairs to open-ended survey questions were examined to identify schools where stakeholders perceived the operation of the Scheme positively or negatively and schools where there were inconsistencies in the positivity or negativity of views between different stakeholders.

Through this selection process, 8 schools were identified as potential case study schools. The other 11 schools were ranked as alternatives to be approached if some of the first eight declined to participate.

3.6.2 Final Selection of Case Study Schools

On 13 September 2016, the eight schools initially identified as potential case study schools were contacted by email and later in the week a follow up phone call was made to explain the study and to ask if the school would agree to participate. Three schools confirmed their participation, three declined and two were slow to reply. Six of the remaining eleven schools were contacted in the following week when it was suspected that the two schools which had not yet replied were unlikely to accept the invitation to participate. On 29 September 2016, three of the second batch of six schools declined the invitation and the three other schools asked for an extension to the case study period. Another four schools were contacted in an attempt to find schools willing to adhere to the proposed schedule, but all of these declined the invitation.

On 27 October 2016, one of the slow responders from the initial eight schools accepted the invitation, giving a total of seven schools which had agreed to participate in the study with the proviso, for three of them, that data collection would be extended until the end of January 2017.

In order to create a better balance of contexts in one district, one more school was included on January 2017, which was identified with the help of the EDB. Table 48 below provides a summary of the characteristics of the final selection of eight case study schools, all of which were co-educational schools.

Table 48. Summary of Characteristics of Case Study Schools

Size Student

Ability

Main MoI

NET gender

NET deployment Survey responses 1 4 classes /level

56 teachers

High English M 3 FC Positive

2 4 classes /level 52 teachers

High Chinese M 1 FC + Oral Positive

3 4 classes /level 54 teachers

High Chinese M Oral & Language Arts Neutral

4 4 classes /level 60 teachers

Low Chinese M Oral Positive

126 5 4 classes /level

55 teachers

High English M 2 FC + Oral Negative

6 4 classes /level 55 teachers

Low Chinese F Oral & Reading Positive

7 4 classes /level 55 teachers

Medium Chinese F 1 FC + Oral Neutral

8 2-3 classes /level 34 teachers

Low Chinese F 1 FC & Oral Negative

‘FC’ here indicates that the NET was teaching all the English lessons; ‘Oral’ indicates that the NET was assigned to teach lessons focussing on speaking skills.

3.6.3 Limitations

Two issues arose in the process of selecting schools for case study. Details of the issues, the means taken to address them, and the resulting consequences are provided below.

1. Schools declining invitation

Four of the initially selected eight case study schools and seven of the alternatives declined to participate, citing as reasons that the schools were very busy and had been preoccupied at the beginning of school year. Some local English teachers thought that the evaluation would increase their workload and were not willing to participate. School Principals were respectful of their local English teachers’ decision and declined although they had indicated their willingness to join the case study when responding to the survey. Some schools indicated that they had manpower shortages that might hinder them from full participation in the case study phase. Other schools declined to participate because the EPC, NET or SP had changed in the new school year. As participation was voluntary, the Research Team no longer contacted or negotiated with these schools after their withdrawal. The EDB helped to liaise with one more school which had not joined the survey and an extension was granted, so that a total of eight case study schools was finally achieved.

2. Difficulty in scheduling observations and interviews in the beginning of the school year Four of the schools mentioned in the above paragraph expressed concerns that they would have staffing difficulties at the beginning of the school year, and that the School Principals did not want to increase teachers’ workload by asking them to attend interviews and focus group discussions. They suggested the interviews be conducted between November and the beginning of the second semester. The Research Team and the NET Section of the EDB agreed to this suggestion.

One case study school stated that they would be able to arrange the meeting observation, the focus group of local English teachers and the interviews starting between March and April 2017. Owing to the aforementioned postponement, at least three interviews, two focus groups and four observations in total were conducted in April to June 2017. This delay in data collection also delayed data analysis.

127 3.6.4 Schedule of Case Studies

For each school, seven data-collection activities were planned to be conducted (i.e. School Principal interview, NET interview, English Panel Chair interview, local English teacher focus group, school observation (including NET shadowing), class observation, meeting observation). A summary of these interviews and the observations is provided in Table 49.

In total 56 data-collection activities in eight selected schools were planned to be completed by 31 October 2016, but due to the extensions explained above, data gathering continued until the end of June 2017. All interviews were audio recorded in the sessions with participants’

informed consent. In cases where participants declined to be recorded, copious notes were taken during and written up soon after the interview to document the interview. After the data collection was completed, interviews were transcribed and prepared for analysis in NVivo.

Table 49. Summary of Data Collection Activities in the Case Studies Data

collection activities in each school

One 1-hour recorded interview with the School Principal One 1-hour recorded interview with the NET

One 1-hour recorded interview with the English Panel Chair One 1-hour recorded focus group interview with 3-6 local English teachers who completed the online survey

One observation of a lesson taught by the NET

One observation of an English Panel meeting involving the NET Shadowing of the NET during one school day and observation of the English environment of the school

3.6.5 Methodology of Data Processing and Analysis

Recorded interviews and focus group discussions conducted as part of the case studies were transcribed verbatim in their totality and double-checked to ensure accuracy. Member checking was provided – that is researchers informed participants that transcripts would be emailed to them at their request for checking, and changes would be made if requested.

Notes and comments from the observation schedules used in the 24 observations were typed into a database for analysis. Analysis of the qualitative data involved listening to each of the recorded interviews and focus group discussions, reading the transcripts and field notes of observations, reviewing the stakeholders’ responses from their online surveys, and coding the data multiple times.

To be more specific, the transcripts, surveys, and observation notes relating to school level stakeholders were analysed in stages in which ‘paradigmatic analytic procedures’ were used to produce taxonomies and categories out of the common elements across the database for the analysis. At least two researchers (and sometimes three when divergent views were found) analysed the data from each case study. This allowed for triangulation and helped to monitor inter-rater reliability. The researchers approached the coding of the data case by case, or one school at a time, reading the three sets of field notes from the observations, studying the four interview transcripts often while listening to the recorded interviews, and reviewing the specific

128 responses from the online survey of the four stakeholders (School Principals, NETs, local English teachers and EPCs) when available. This first reading (and listening) was for a general idea of the various issues involved in this school, and for noting the themes that emerged. At this stage, the goal was to get a sense of the impact of the ENET Scheme at the school from multiple perspectives (School Principal, local English teachers, NETs and EPCs) as well as to note what was observed in the multiple visits to the school.

The next stage involved a closer reading of the 12 (or more) documents for each school (three observation schedules, four interview transcripts, four survey responses, and other artefacts such as meeting agendas and teaching schedules). A team of two researchers (PI, CoIs, and PM) was assigned to analyse each case study. Each researcher coded the 12 or more documents and individually wrote a short narrative summarising their findings, provided a title for the school that encapsulates their impressions of the findings, and made a list of their codes and the supporting interview, focus group and field note quotes under each one. They then associated the codes and corresponding quotes with one or more of the six research questions.

Finally, they made a note of any outliers and negative cases.

3.7 Qualitative Results and Findings