Consultancy Study on the Effectiveness of the Provision of Quality Kindergarten Education in Hong Kong under the Kindergarten Education Policy
Technical Report
Prepared by
Nirmala Rao, Carrie Lau, Stephanie Chan and Ben Richards with
Diana Lee, John Bacon-Shone and Patrick Ip
The University of Hong Kong February 2022
1 Contents
List of Tables ... 3
List of Figures ... 4
List of Appendices ... 6
Introduction ... 7
Background ... 7
Research questions ... 8
Sampling and participants ... 9
Study 1 ... 9
Study 2 ... 12
Information of participating KGs, principals, teachers and parents ... 15
Measures ... 16
Early Childhood Environment Rating Scale-Revised ... 16
Early Childhood Environment Rating Scale-Extension ... 16
Measure of Early Learning Environment ... 17
Sustained Shared Thinking and Emotional Well-being Scale ... 17
Principal Questionnaire ... 17
Teacher Questionnaire ... 17
Parent Questionnaire... 18
Principal Interview ... 18
Teacher Interview. ... 18
Parent Interview ... 18
Procedures ... 18
Classroom observations ... 18
Questionnaires and interviews ... 19
Analysis plan ... 19
Analysis of classroom observation data ... 20
Analysis of questionnaire data ... 22
Analysis of interview data ... 22
Findings and discussion ... 24
Overview of classroom observations ... 24
Domain scores ... 24
Mean scores on each scale overall, by class ... 24
Observation factor scores across waves ... 25
Theme 1. Improved funding and subsidy ... 28
1. 1. Grants received by KGs ... 28
1. 2. Usage of grants ... 28
1. 3. Summary and discussion ... 30
Theme 2: Improved TP ratio ... 32
2. 1. TP ratio and classroom quality ... 32
2. 2. Teacher-child interactions ... 32
2. 3. Summary and discussion ... 33
Theme 3: Strengthened support for professional development of teachers and principals . 34 3. 1. Teachers’ confidence and self-efficacy ... 34
3. 2. Provision of professional development activities for teachers and principals... 37
3. 3. Professional development and classroom quality ... 42
2
3. 4. Application of learning from professional development activities ... 43
3. 5. Support for new teachers ... 44
3. 6. Schools’ practical support to encourage teachers’ participation in professional development activities ... 47
3. 7. Summary and discussion ... 47
Theme 4: Revised guide to the pre-primary curriculum ... 49
4. 1. Principals’ and teachers’ perceptions towards the KECG ... 49
4. 2. Teachers’ and parents’ attitudes towards learning through play ... 49
4. 3. Schools’ policy and practice on learning through play ... 53
4. 4. Support for teachers on the implementation of the KECG ... 56
4. 5. Summary and discussion ... 56
Theme 5: Increased monitoring and quality assurance ... 58
5. 1. Involving teachers in decision-making ... 58
5. 2. Staff morale and stability of the teaching team ... 59
5. 3. Summary and discussion ... 59
Theme 6: Strengthened support for students with diverse needs ... 60
6. 1. Catering for students with diverse needs in classrooms ... 60
6. 2. Supporting students with special needs or at risk of developmental delay ... 63
6. 3. Supporting Non-Chinese speaking (NCS) students ... 68
6. 4. Summary and discussion ... 73
Theme 7: Strengthened parent engagement and education ... 75
7. 1. Home-school communication ... 75
7. 2. School-based parent education and involvement ... 81
7. 3. Home-based involvement ... 84
7. 4. Parent-teacher associations/ Parent groups... 84
7. 5. Summary and discussion ... 86
School factors conducive to the quality of KG education ... 87
Continuous professional development policy of schools ... 87
Teacher engagement, participation, and experiences ... 87
Parent involvement ... 88
General discussion and conclusions... 89
Limitations ... 95
Key Findings and Recommendations ... 96
3 List of Tables
Table 1. Number of participating KGs in Study 1 by district, KG size, OPRS and NCS Grant
status ... 10
Table 2. Total number of observations, questionnaires, and interviews in Pre-policy phase, Wave 1, Wave 2, and Wave 3 ... 15
Table 3. Information of the participating KGs in Study 1 and Study 2 ... 15
Table 4. Demographic information of participants in Study 1 and Study 2 ... 16
Table 5. Retained factors, domains with high loadings, and alphas ... 21
Table 6. Mean observation scores by rating scales, phases, and class levels (ECERS-E, ECERS-R and SSTEW are on a 7-point scale; MELE is on a 4-point scale) of classes observed in 25 KGs (Pre-policy n=15, Wave 1 n=50, Wave 2 n=50, Wave 3 n=44) ... 25
Table 7. Classroom observation scores on domains and items related to teacher-child interactions in Wave 1, Wave 2, and Wave 3 ... 33
Table 8. Polyserial correlations between hours of professional development expected for teachers (as reported by principals) and class observation scores, by wave... 42
Table 9. Polyserial correlations between KGs’ professional development arrangements (as reported by teachers across three waves) and overall class observation scores in Wave 3 ... 43
Table 10. Classroom observation scores on domains and items related to learning through play in Pre-policy phase, Wave 1, Wave 2, and Wave 3 ... 55
Table 11. Classroom observation scores on domains and items related to catering for students with diverse needs in Pre-policy phase, Wave 1, Wave 2, and Wave 3 ... 62
4 List of Figures
Figure 1. Theory of Change ... 8
Figure 2. Implemented research design and number of participants for Study 1 ... 11
Figure 3. Implemented research design for Study 2 ... 13
Figure 4. Factor scores across waves ... 25
Figure 5. Figure showing estimated scores of 4 factors across phases ... 27
Figure 6. Perceived impact of the KG policy in the Pre-policy phase ... 30
Figure 7. Teacher’s confidence towards their ability in teaching, by wave (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 35
Figure 8. Teacher’s self-efficacy, by wave (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 36
Figure 9. Frequency of professional development activities provided to the teacher (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 37
Figure 10. Priority of the school arrangement of professional development activities for teachers as reported by principals (Principal questionnaire; Wave 1: n=121; Wave 2: n=114; Wave 3: n=106)... 38
Figure 11. Teachers’ beliefs towards the school’s arrangements of the professional development activities (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 41
Figure 12. Extra support for new teachers from teacher’s perspective (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 45
Figure 13. Extra support for new teachers from principal’s perspective (Principal questionnaire; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 45
Figure 14. Teachers’ beliefs towards children’s learning (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 51
Figure 15. Parents’ beliefs towards play and learning (Parent questionnaire; Wave 1 n=879; Wave 2 n=798; Wave 3 n=431) ... 52
Figure 16. Support for students with special needs or at risk of developmental delay as reported by teachers (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 64
Figure 17. Support for students with special needs or at risk of developmental delay as reported by principals (Principal questionnaire; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 65
Figure 18. Parents’ perception on the usefulness of the support for children with special needs (Parent questionnaire) ... 66
Figure 19. Types of support for NCS students provided by kindergartens as reported by teachers (Teacher questionnaire; Wave 1 n=528; Wave 2 n=538; Wave 3 n=397) 69 Figure 20. Types of support for NCS students provided by kindergartens as reported by principals (Principal questionnaire; Wave 1 n=48; Wave 2 n=50; Wave 3 n=61) . 70 Figure 21. Parents’ perception on the usefulness of the support among NCS children receiving support (Parent questionnaire) ... 70
Figure 22. Methods of contacting parents about children’s learning and development (Teacher questionnaire: Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313 and Principal questionnaire: Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 76
Figure 23. Home-school partnership for parents: methods of contact with their child’s school (Parent questionnaire; Wave 1 n=879; Wave 2 n=798; Wave 3 n=431). ... 77
Figure 24. Teachers’ communication with parents (Teacher questionnaire: Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 79
5
Figure 25. Principals’ communication with parents (Principal questionnaire; Wave 1 n=121;
Wave 2 n=114; Wave 3 n=106) ... 80 Figure 26. Topics covered at workshops for parents (Parent questionnaire; Wave 1 n=879;
Wave 2 n=798; Wave 3 n=431) ... 81 Figure 27. Schools’ provision of parent involvement activities (Principal questionnaire; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 83 Figure 28. Frequency of parent workshops/ seminar (reported in questionnaires by principals
of 25 KGs participating in Study 2) ... 83 Figure 29. Home-based activities (Parent questionnaire; Wave 1 n=879; Wave 2 n=798;
Wave 3 n=431) ... 85 Figure 30. Correlations between average classroom quality and hours of professional
development expected reported by principals ... 87
6 List of Appendices
Appendix A: Item lists of observation measures and questionnaires ... 97
Appendix A1: Early Childhood Environment Rating Scale-Revised (ECERS-R) Item List ... 97
Appendix A2: Early Childhood Environmental Rating Scales – Extension (ECERS-E) Item List ... 99
Appendix A3: Measure of Early Learning Environmental (MELE) Item List ... 100
Appendix A4: Sustained Shared Thinking and Emotional Well-being Scale (SSTEW) Item List ... 102
Appendix A5: Principal Questionnaire Item List ... 103
Appendix A6: Teacher Questionnaire Item List ... 105
Appendix A7: Parent Questionnaire Item List ... 106
Appendix B: Additional tables and figures ... 107
Table B1. Factor analysis (principal component factors) of 18 ECERS-E, SSTEW, and MELE domains (n=159) ... 107
Table B2. Rotated 5-factor solution (quartimax rotation; n=159; loadings > .45 are highlighted for ease of interpretation) ... 108
Table B3. Classroom observation scores by domain ... 109
Table B4. ECERS-R scores by domain for Pre-policy and Wave 3 (K3 only) ... 113
Table B5. Overall mean classroom observation scores (ECERS-E, SSTEW, MELE) by wave... 114
Figure B1. Grants received by KGs (Principal questionnaires; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 115
Figure B2. Changes reported by principals following the introduction of the KECG (Principal questionnaire; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 116
Figure B3. Teachers’ perceptions of their teaching experience in school (Teacher questionnaire; Wave 1 n=1522; Wave 2 n=1446; Wave 3 n=1313) ... 117
Figure B4. Principals’ perceptions on their teaching experience in school (Principal questionnaire; Wave 1 n=121; Wave 2 n=114; Wave 3 n=106) ... 118
7 Introduction
Background
The Government of the Hong Kong Special Administrative Region commissioned The Faculty of Education, The University of Hong Kong to conduct a study on the effectiveness of the provision of quality kindergarten (KG) education under the KG education policy (KG policy) that was implemented in Hong Kong starting from the 2017/18 school year.
This study considers whether the quality of KG education has changed as a result of the implementation of the KG policy, and has 3 main objectives: (i) to examine the effectiveness of the provision of quality KG education in Hong Kong under the KG policy; (ii) to identify good practices and areas for improvement for KGs; and (iii) to investigate school factors that are conducive to the development of quality KG education.
This Final Report provides a summary of findings from the Pre-policy phase (from June to September 2017), Wave 1 (from November 2017 to September 2018), Wave 2 (from April 2019 to September 2019), and Wave 3 (from June 2020 to November 2020). The data were collected through (i) classroom observations; (ii) principal, teacher and parent questionnaires;
and (iii) principal, teacher, and parent interviews.
It is important for any evaluation study to have an articulated theory of change. In this instance, we were interested in how and why the KG policy was expected to lead to the desired change, i.e. improved quality in KG education in Hong Kong. The KG policy has led to improved funding and subsidy, improved teacher-to-pupil (TP) ratio, strengthened support for professional development of teachers and principals, implementation of the revised guide to the pre-primary curriculum, increased monitoring and quality assurance, strengthened support for students with diverse needs, and strengthened parent engagement and education. These outputs were, in turn, assumed to lead to positive outcomes as shown below in Figure 1.
8 Figure 1. Theory of Change
Research questions
There are 6 key research questions relevant to the KG policy set to be answered in the Final Report. The section on “General discussion and conclusions” of this report is structured around these 6 key research questions:
1. What are the impacts on learning and teaching of the students (e.g. in curriculum planning, teaching methods, and students’ engagement in learning activities)?
2. What are the impacts on catering for students’ diverse needs (e.g. teachers’ understanding of their specific needs, support to their learning, collaboration with relevant experts/
external organisations, and teachers’ training in this regard)?
3. What are the impacts on school management and organisation (e.g. transparency, holistic planning in resource deployment, and school culture and atmosphere)?
4. What are the impacts on teachers’ professional development including school policy relating to teachers’ development (e.g. staffing structure/ hierarchy)?
5. What are the impacts on parents’ engagement (e.g. more diverse communication channels, and promotion of parent education)?
6. What are the school factors that are conducive to the development of quality KG education?
9 Sampling and participants
Study 1
The sample of the study comprised the target sample of 100 KGs, plus an oversample of an additional 20 to compensate for anticipated non-response and attrition. In Wave 1, KGs were randomly sampled from a complete list of all KGs participating in the KG education scheme (Scheme). The probability of a KG being selected was weighted to the number of teachers in each KG to ensure that the sample represented teachers at KGs in Hong Kong. The sample was stratified by district to ensure geographical representation. Stratification ensured that a minimum of 5 KGs from each geographical district were included in the sample, with larger numbers in more populous districts.
Random sampling with replacement1 was used to ensure numbers of KGs meeting the selection criteria by school size and additional support to cater for students’ diverse needs, specifically additional grant for support to non-Chinese speaking (NCS) students (NCS Grant) and participation in the On-site Pre-school Rehabilitation Services2 (OPRS). Sampling with replacement was used such that, where possible, at least 1 small KG, at least 1 KG joining the OPRS, and at least 1 KG receiving the NCS Grant, were included in the sample in each district as far as possible. KGs were divided into large and small KGs, with “small” being defined as below the 25th percentile of all KGs covered by the Scheme in terms of the total number of enrolled students. Based on this, small KGs in this study refer to KGs with less than 89 enrolled students (irrespective of half-day or whole-day enrolment).
In Wave 1, 260 KGs were contacted, and a total of 121 KGs agreed to participate in the study, giving an overall acceptance rate of 46.5%. After the initial sample was constructed, resampling following rejection or non-response took place in 26 rounds. Table 1 below shows the composition of the final sample by district, and with the numbers of small KGs, KGs joining OPRS and KGs receiving the NCS Grant in each district of our sample established in Wave 1, Wave 2 and Wave 3. As shown in Table 1, in Wave 1, 1 district did not meet the minimum small/large KG criterion, 4 districts did not meet the OPRS criterion, and 3 districts did not meet the NCS Grant criterion. These cases arose mainly because of uneven distribution of KGs with these characteristics across districts. In these cases, there were no KGs present in the relevant district that met the criteria and had accepted participation in the study.
1 Random sampling with replacement means that schools were sampled randomly in each district, but if selection of the randomly chosen school implied that the minimum criteria for school size, NCS Grant status, and OPRS participation for that district were not met, the sampled school was unselected and a different school from that district was randomly sampled in its place. If necessary, this procedure would be repeated until the minimum criteria were met.
2 The Pilot Scheme on On-site Pre-school Rehabilitation Services was launched by the Government in 2015 and was later regularised in the 2018/19 school year. Inter-disciplinary service teams from the non-governmental organisations provide on-site services for pre-school children with mild disabilities and studying at the participating kindergartens and kindergarten-cum-child care centres. Inter-disciplinary service teams also render support services to teachers/child care workers and parents/carers.
10
Table 1. Number of participating KGs in Study 1 by district, KG size, OPRS and NCS Grant status
District Total KGs Small KGs Receive support
from OPRS Receive NCS Grant3 Wave
1 Wave 2 Wave
3 Wave
1 Wave
2 Wave
3 Wave 1 Wave
2 Wave
3 Wave 1 Wave
2 Wave Central and 3
Western 6 5 6 1 0 1 0 5 6 0 0 3
Eastern 7 7 6 3 3 2 1 7 6 1 2 3
Islands 6 6 5 3 3 3 0 6 5 2 2 4
Kowloon City 7 7 7 1 1 1 1 7 7 3 3 3
Kwun Tong 7 7 5 4 4 3 1 7 5 2 1 2
Kwai Tsing 6 6 6 2 2 2 1 6 6 2 1 5
North 7 7 7 1 1 1 1 7 7 0 0 2
Sai Kung 7 5 5 2 1 2 1 5 5 2 0 2
Southern 6 6 6 1 1 1 1 6 6 1 1 5
Sham Shui Po 8 7 6 1 0 1 0 7 6 4 4 4
Sha Tin 7 6 6 2 2 2 3 6 6 1 1 1
Tuen Mun 7 7 7 1 1 1 1 7 7 4 5 5
Tai Po 7 7 6 0 0 0 1 7 6 0 2 4
Tsuen Wan 5 5 5 1 1 1 0 5 5 1 1 3
Wan Chai 6 5 4 4 3 2 1 5 4 4 3 3
Wong Tai Sin 8 8 8 3 3 3 3 8 8 1 1 5
Yuen Long 7 6 6 1 1 1 3 6 6 1 1 2
Yau Tsim Mong 7 7 7 2 2 2 3 7 7 3 3 4
Total 121 114 108 33 29 29 22 114 108 32 31 60
In Wave 1, there were 121 KGs participating in the study. In Wave 2, 7 out of 121 KGs were excluded from the analysis. These include 1 KG that withdrew from the study and 6 KGs that did not respond to us. Among the 114 participating KGs, 9 KGs had a new principal in Wave 2 (2018/19 school year). Out of the remaining 120 KGs (excluding 1 KG that had withdrawn in Wave 2), 12 KGs were excluded from Wave 3 (3 KGs withdrew from the study and 9 KGs did not respond to us). In Wave 3, among the 108 participating KGs, 11 KGs had a new principal (2019/20 school year). Within each participating KG, the principal and all registered teachers with a full-time employment contract were invited to complete questionnaires. In Wave 1 and Wave 2, all principals from the participating KGs completed their questionnaires;
and in Wave 3, principals from 106 out of 108 KGs completed the questionnaires. The response rates4 of teachers completing questionnaires for the three waves were 84.89% (1522 teachers), 82.11% (1446 teachers), and 74.65% (1313 teachers) respectively. Among these respondents, 85 principals and 510 teachers completed questionnaires in all three waves. Attrition is quite common in longitudinal studies. We are unable to determine the exact reasons of why teachers/
3 During the course of the study, the terms of the NCS Grant were changed: in 2017/18 (Wave 1) and 2018/19 (Wave 2), a fixed sum of the grant was provided to KGs that had admitted at least 8 NCS students. From 2019/20 (Wave 3) onwards, a 5-tiered grant is provided to KGs based on the number of NCS students admitted (1 to 4 students, 5 to 7 students, 8 to 15 students, 16 to 30 students, and 31 or above students).
4 The response rate was calculated by dividing the number of teacher participants by the number of teachers reported in the principals’ questionnaires in each wave.
11
principals dropped out of the study, but some teachers/ principals may have left the school. In order to increase the likelihood that 100 KGs participated in all three waves of the study, we recruited 20 KGs in addition to our original target of 100 KGs. Hence, the response rate of principals and teachers for the three waves of our study from over 100 KGs is considered very satisfactory 5. Figure 2 presents the implemented research design and number of participants for Study 1.
Figure 2. Implemented research design and number of participants6 for Study 1
5 In a major study targeting households with children aged 6 or under in the United States, the response rate ranged between 57.8% and 78.7%, depending on the part of the survey in question (McPhee et al., 2015). McPhee, C., Bielick, S., Masterton, M., Flores, L., Parmer, R., Amchin, S., Stern, S., and McGowan, H. (2015). National Household Education Surveys Program of 2012: Data File User’s Manual (NCES 2015-030). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.
6 We collected 121, 114, and 106 principal questionnaires from 120, 113, and 106 principals in Wave 1, Wave 2, and Wave 3 respectively. This was because 2 of the participating KGs had the same principal in both Wave 1 and Wave 2.
12 Study 2
Figure 3 presents the implemented research design for Study 2. The sample of 25 KGs was drawn from the initial sample of 100 KGs used for Study 1 (i.e. the KGs before the oversample), with the probability of selection being in proportion to the number of students in the KG, making it representative of children in KGs in Hong Kong. To ensure that these 25 KGs are representative of the situation in Hong Kong, as with Study 1, the sample was stratified by district, and random sampling with replacement was used to ensure representation of KGs by NCS Grant status, OPRS status, and school size.
To understand the general situation before the implementation of the KG policy, 15 K3 classes in these 25 KGs were observed in the Pre-policy phase, with classroom observation data collected using the Early Childhood Environment Rating Scale-Revised (ECERS-R), the Early Childhood Environment Rating Scale-Extension (ECERS-E), the Sustained Shared Thinking and Emotional Well-being Scale (SSTEW), and the Measurement of Early Learning Environment (MELE) tools. At the Wave 1 phase, 25 K1 and 25 K2 classes from 25 KGs were observed. At the Wave 2 phase, we observed 25 K1 and 25 K3 classes from the same 25 KGs observed in Wave 1. In Wave 3, our original design was to follow the K2 (the same K1 classes observed in Wave 2) and K3 classes (the same K1 classes in Wave 1). However, owing to suspension of face-to-face classes due to the COVID-19 pandemic, we were unable to observe the K2 classes within the 2019/20 school year and only observed the K3 classes in the 2019/20 school year. Instead, we observed the 25 K2 classes in the first term of the 2020/21 school year (not observed in previous waves). Although we intended to observe the K3 classes from all 25 KGs, we only observed 19 K3 classes in Wave 3 because of suspension of face-to-face classes or different class resumption arrangements under the COVID-19 pandemic. While this deviated from our original research design, it should be noted that the observed 19 KGs and the 6 KGs that we had not observed were similar in terms of school size (mean number of students: 19 observed KGs = 201.9; 6 excluded KGs = 202.2), and proportion of students with special needs (19 observed KGs = 6.3%, 6 excluded KGs = 5.4%). Because of the similar school characteristics, we are confident that the decrease in sample size did not impact the findings.
In each wave, while the plan was to observe the same classes that we observed in previous waves (e.g. the K3 classes observed in Wave 3 ideally should be the K2 classes we observed in Wave 2), the observed classes may or may not be of the same group of children observed in previous waves in their respective class levels. This was because students were sometimes dispersed in different classes if KGs that had more than 1 class within each class level7. Classroom observation data in three waves were collected using the ECERS-E, SSTEW, and MELE tools whereas 4 domains of the ECERS-R were also used for K3 classes in Wave 3.
7 While we tried to follow the same class of students as much as possible, in cases where it was not possible to follow the same classes because of the arrangement of individual KGs across class levels, we used random sampling to the maximum extent in our sampling such that error variance may be balanced out.
13 Figure 3. Implemented research design for Study 2
14
In both Wave 1 and Wave 2, parents of all children of the classes where observations took place were invited to complete a parent questionnaire. In Wave 1, a total of 879 parents (421 K1, 434 K2, 24 missing data on class level) completed the questionnaires; the response rate was 73.43%. In Wave 2, a total of 798 parents (395 K1, 354 K3, 49 missing data on class level) completed the questionnaires; the response rate was 83.20%. In Wave 3, we invited the parents of the K2 and K3 children in our original design in the 2019/20 school year to complete the questionnaires. We observed the K2 classes in the 2020/21 school year instead of the 2019/20 school year as originally planned because of suspension of face-to-face classes. Therefore, the K2 parents were not the parents of the observed classes in Wave 3. Furthermore, we were not always able to follow the same group of children in each wave, hence, some parents who had previously participated were not invited to complete the questionnaire. In Wave 2, among the 354 K3 parents, 146 parents also completed the questionnaires in Wave 1. Among the 431 parents in Wave 3 (154 K2 parents, 268 K3 parents, 9 missing on class level), 78 K2 and 122 K3 parents also completed in previous waves when their children were in K1 classes (Wave 1 for K3 parents, and Wave 2 for K2 parents).
In Wave 1 and Wave 2, (i) 1 teacher from each observed classroom was interviewed (a total of 50 teacher interviews); (ii) each KG principal was interviewed (a total of 24 principal interviews conducted with 24 principals as 1 of the principals was serving 2 out of the 25 KGs);
and (iii) 2 parents from each KG, 1 from each observed class, were either randomly selected or selected through purposive sampling (in the case when we were able to invite the same parent who had participated previously) for interview in Wave 2 (a total of 50 parent interviews8, see Table 2). In Wave 3, since we only observed K3 classes during our data collection stage for the interviews (2019/20 school year), instead of interviewing the teachers and parents of the observed K2 classes, we followed the teachers and parents who had been previously interviewed in Wave 1. However, where this was not possible, we randomly selected a teacher from the KG and a parent from the same K2 class as the parent we intended to interview.
Random selection of parents for interview was implemented by: (a) the teacher creating a numbered list of all children in their class; (b) the research team randomly selecting a number from the list; and (c) the teacher approaching the parent of the child corresponding to that number for interview. If written informed consent was not obtained for the interview from the selected parent, another child would be randomly selected with the same procedures until a parent agreed to participate.
8 For cases where the same parents were not interviewed in previous waves, we used random sampling to the maximum extent in our sampling such that error variance may be balanced out.
15
Table 2. Total number of observations, questionnaires, and interviews in Pre-policy phase, Wave 1, Wave 2, and Wave 3
Study 1
Data type Pre-policy Wave 1 Wave 2 Wave 3
Principal
questionnaires9 25 (25 KGs)
121 (121 KGs) 114 (114 KGs) 106 (106 KGs) Teacher questionnaires N/A 1522 (121 KGs) 1446 (114 KGs) 1313 (108 KGs)
Study 2
Data type Pre-policy Wave 1 Wave 2 Wave 3
Classroom observations 15 (15 KGs) 50 (25 KGs) 50 (25 KGs) 4410 (25 KGs) Parent questionnaires N/A 879 (25 KGs) 798 (25 KGs) 431 (25 KGs) Principal interviews 24 (25 KGs) 24 (25 KGs) 24 (25 KGs) 24 (25 KGs) Teacher interviews N/A 50 (25 KGs) 50 (25 KGs) 50 (25 KGs)
Parent interviews N/A 50 (25 KGs) 50 (25 KGs) 50 (25 KGs)
Information of participating KGs, principals, teachers and parents
Table 3 presents the information of the participating KGs and Table 4 presents demographic information of the participating principals, teachers and parents.
Table 3. Information of the participating KGs in Study 1 and Study 2
Study 1 Study 2
Wave 1 Wave 2 Wave 3 Wave 1 Wave 2 Wave 3
n 121 114 106 25 25 25
Percentage of KGs that offer
half-day (HD) programme 57% 58% 54% 68% 75% 67%
Percentage of KGs that offer whole-day (WD)
programme
49% 52% 51% 48% 63% 58%
Percentage of KGs that offer long whole-day (LWD) programme
41% 37% 37% 40% 29% 29%
Percentage of KGs that offer both HD and WD
programmes
45% 45% 43% 44% 54% 50%
9 We collected 121, 114, and 106 principal questionnaires from 120, 113, and 106 principals in Wave 1, Wave 2, and Wave 3 respectively, and conducted principal interviews with 24 principals of 25 KGs. This was because 2 of the participating KGs had the same principal.
10 We intended to conduct classroom observations at 25 K2 and 25 K3 classes in Wave 3. However, we were unable to visit 6 K3 classes in Wave 3 because of the suspension of face-to-face classes or different class resumption arrangements under the COVID-19 pandemic.
16
Table 4. Demographic information of participants in Study 1 and Study 2
Respondents Gender Wave 1 Wave 2 Wave 3
Principals n 120 113 106
Females 118 106 100
Male 1 1 1
Not indicated 1 6 5
Teachers n 1522 1446 1313
Females 1447 1371 1254
Male 25 22 26
Not indicated 50 53 33
Parents n 879 798 431
Mothers 707 605 342
Fathers 126 122 70
Non-parent guardians 27 18 8
Not indicated 19 49 11
Measures
Measures including classroom observation rating scales, questionnaires and interview protocols for principals, teachers, and parents were used in the Pre-policy, Wave 1, Wave 2 and Wave 3 phases. Appendix A includes information about the measures. With the exception of ECERS-R which was used in the Pre-policy and Wave 3 phases for K3 classes only, Wave 1, Wave 2, and Wave 3 (K2 classes) used the same set of measures (i.e. ECERS-E, SSTEW and MELE). A description of each measure is presented below:
Early Childhood Environment Rating Scale-Revised (see Appendix A1). ECERS-R11 measures the overall quality of KG classrooms. The ECERS-R consists of 43 items and has 7 domains: (a) Space and Furnishings; (b) Personal Care Routines; (c) Language Reasoning; (d) Activities; (e) Interaction; (f) Programme Structure; and (g) Parents and Staff. Each item is scored on a 7-point scale with the following descriptors: 1 (inadequate), 3 (minimal), 5 (good) and 7 (excellent). The item“Nap/rest” (item 11) was dropped for consistency as there were half-day and whole-day Scheme-KGs participating in this study. Only 4 domains (i.e. Space and Furnishings, Personal Care Routines, Activities, and Programme Structure) were scored in Wave 3. We have not included all domains in ECERS-R because the other domains were tapped in the measures we used (i.e., ECERS-E, SSTEW, MELE).
Early Childhood Environment Rating Scale-Extension (see Appendix A2). ECERS-E12 assesses the curricular aspects of quality. It consists of 15 items and contains 4 domains: (a) Literacy, (b) Mathematics, (c) Science and the Environment, and (d) Diversity. Each item is scored on a 7-point scale with the following descriptors: 1 (inadequate), 3 (minimal), 5 (good)
11 Harms, T., Clifford, R. M., & Cryer, D. (2005). Early Childhood Environment Rating Scale - Revised Edition.
New York, NY: Teachers College Press.
12 Sylva, K., Siraj-Blatchford, I., & Taggart, B. (2010). Assessing quality in the early years: Early Childhood Environment Rating Scale-Extension (ECERS-E): Four curricular subscales, rev. 4th ed. New York: Teachers’
College Press.
17
and 7 (excellent) The item, “Sounds in Words” (item 4) was dropped to be in line with the Chinese medium of instruction of KGs in Hong Kong13.
Measure of Early Learning Environment (see Appendix A3). MELE (MELQO, 2016)14 tool is used to measure global and domain-specific quality of stimulation in KG classrooms. MELE consists of 50 items and measures the following 9 domains: (a) Physical Environment; (b) Interaction; (c) Inclusiveness; (d) Teaching/Learning: Overview; (e) Teaching/Learning:
Language and Literacy; (f) Teaching/Learning: Numbers and Numeracy; (g) Teaching/Learning: Nature and Science; (h) Teaching/Learning: Group activities; and (i) Teaching/Learning: Free-choice Indoor Activities. Before administering MELE, the contextual relevance of the scale was discussed, and 10 items from the Physical Environment, Safety and Hygiene domain15, Interaction, and Language and Literacy were removed. These items pertained to school cleanliness and safety (e.g. availability of drinking water;
handwashing facilities; covered classroom space), which were basic requirements for school registration of KGs and so unlikely to vary. We also dropped 2 items as we found that in the Hong Kong context, KGs were unlikely to vary on these items: (i) children’s waiting time (KGs in Hong Kong had fairly structured routines and children rarely had to wait for long periods) and (ii) children’s use of writing implements (writing implements were readily available in Hong Kong KG classrooms). Each item is scored on a 4-point scale, with 1 indicating low quality and 4 high quality.
Sustained Shared Thinking and Emotional Well-being Scale (See Appendix A4). SSTEW16 measures the quality of practices that support children’s skills in sustained shared thinking, emotional well-being, the development of strong relationships, effective communication and self-regulation. The scale consists of 14 items and has 5 domains: (a) Building Trust, Confidence and Independence; (b) Social and Emotional Well-being; (c) Supporting and Extending Language and Communication; (d) Supporting Learning and Critical Thinking; and (e) Assessing Learning and Language. Each item is scored on a 7-point scale with the following descriptors: 1 (inadequate), 3 (minimal), 5 (good) and 7 (excellent).
Principal Questionnaire (See Appendix A5). This self-developed questionnaire taps into the following: (1) School Information (e.g. number of students and staff); (2) Principal’s and Teachers’ Professional Development; (3) Home-school Partnership; (4) Catering for Diverse Needs; (5) School Curriculum; (6) Principals’ Beliefs and School Perceptions; and (7) Principal Background Information (e.g. years of work experience, gender and age).
Teacher Questionnaire (See Appendix A6). This self-developed questionnaire includes questions on the following: (1) Teaching Duties and Practices; (2) Home-school Partnership;
(3) Teachers’ Professional Development; (4) Teachers’ Beliefs and School Perceptions; and (5) Background Information (e.g. years of work experience, gender and age).
13 This is because the grapheme (character) for Chinese represents a morpheme (unit of meaning) and not a phoneme (unit of sound) as in the case of the English language. The ECERS-E was developed for English language programmes.
14 Measuring Early Learning Quality and Outcomes (MELQO) Project. (2016). Measure of Early Learning Environments (MELE) Module. UNESCO. Washington, DC.
15 Since all items from the Safety and Hygiene domain were removed, the domain was not included in the measure used in this study.
16 Siraj, I., Kingston, D. & Melhuish, E. (2015). Assessing Quality in Early Childhood Education and Care:
Sustained Shared Thinking and Emotional Well-being (SSTEW) Scale for 2-5-year-olds Provision. London, United Kingdom: Trentham Books.
18
Parent Questionnaire (See Appendix A7). This self-developed questionnaire includes questions on (1) Home-school Partnership; (2) Parent-child Activities; (3) Views towards Your Child’s School and (4) Student Background Information, including student age, gender, and family demographics (e.g. parental education).
Principal Interview. Principals were interviewed to elaborate on catering for students’ diverse needs, home-school partnership, school governance, curriculum development and lesson design, and teacher and principal professional development. Additional follow-up questions within each topic were added in Wave 3 as requested by the Education Bureau (EDB) Consultancy Team. We also added 2 questions on the support for students with diverse needs and communication with parents under the COVID-19 pandemic.
Teacher Interview. Full-time registered teachers were interviewed to elaborate on catering for students’ diverse needs, home-school partnership, school governance, curriculum development and lesson design, and teacher professional development. Additional follow-up questions within each topic and 2 new topics (changes in resource usage after KG joined the Scheme and school culture) were added in Wave 3 as requested by the EDB Consultancy Team. We also added 2 questions on the support for students with diverse needs and communication with parents under the COVID-19 pandemic.
Parent Interview. Parents were interviewed to discern their views and perspectives on the KGs (e.g. curriculum, support for learner diversity). They were also asked to provide information on their involvement and engagement with their children’s learning and development at home and school. In Wave 3, a question on communication with the school during the COVID-19 pandemic was added.
Procedures
Classroom observations
Class observations were conducted between June and July 2017 (Pre-policy phase), between November 2017 and January 2018 (Wave 1), and between April and May 2019 (Wave 2). In Wave 3, classroom observations were conducted from June to July 2020 for K3 classes, and from October to November 2020 for K2 classes.
The procedures were identical in all phases. A team of 2 assessors observed each KG class.
Half-day sessions were observed for 3 hours and whole-day sessions were observed for 6 hours, excluding naps and mealtimes. In Wave 3, all observed classes were half-day sessions because of the territory-wide class resumption arrangements under the COVID-19 pandemic.
Undergraduate and postgraduate students and graduates of Early Childhood Education degree programmes were trained to use ECERS-R, ECERS-E, SSTEW and MELE through a series of training workshops that lasted for 2 days, including 2 additional days of in-field practices in KGs. Observers were required to attain inter-rater reliability of at least 85% with an expert assessor, denoted as the Gold Standard, before collecting field data. Further, inter-observer reliability was calculated by examining agreement between the assessors and the Gold Standard in 3 of the 25 KGs observed in each wave. In Wave 3, all assessors had been previously trained and had the experience of participating in the previous rounds of class observation. These assessors had attended a refresher training before data collection.
19 Questionnaires and interviews
In Wave 1, we delivered the principal and teacher questionnaires to 121 participating KGs between June and August 2018. In Wave 2, we contacted the 121 KGs between March and September 2019, and the questionnaires were distributed to 115 KGs between June and August 2019.
As reported in the section “Sampling and participants”, data from 7 out of 121 KGs (Wave 2) and 12 out of 120 KGs (Wave 3) were excluded from the analysis as the KGs had either withdrawn from the study or had not returned the questionnaires. Each KG was given 2 weeks to complete and return the questionnaires, but we accommodated the individual requests and schedule of certain KGs, especially in Wave 3 during the COVID-19 pandemic. The principal questionnaires and teacher questionnaires had been collected by September 2018 (Wave 1), September 2019 (Wave 2), and October 2020 (Wave 3).
For the 25 KGs that participated in Study 2, trained interviewers conducted the interviews with principals, teachers, and parents. Interviews were conducted with principals, teachers, and parents from the 25 KGs from June to July 2018 (Wave 1), from May to June 2019 (Wave 2), and from July to August 2020 (Wave 3). In the Pre-policy, Wave 1 and Wave 2 phases, all interviews were conducted face-to-face in KGs. In Wave 3, we adopted a flexible approach under the COVID-19 pandemic: all interviews with parents and teachers were conducted over the phone while interviews with principals were conducted via face-to-face meetings, video conferences or over the phone depending on interviewees’ preferences.
Analysis plan
The analysis was conducted in 3 parts: (i) quantitative analysis of classroom observation data;
(ii) quantitative analysis of principal, teacher and parent questionnaire data; and (iii) qualitative analysis of principal, teacher and parent interview data. The findings presented in this report were the overall situations found in the first 3 years since the implementation of the KG policy as well as the features analysed on the situations across Wave 1, Wave 2, and Wave 3 of the study. The quantitative analysis therefore involved reporting descriptive statistics of data collected from classroom observations and questionnaires. It did not use sampling weights because it was intended to report on observation scores and questionnaire responses within this study’s sample of KGs, rather than to make inferences about all KGs in Hong Kong.
Techniques such as regression were also used to examine relations between key variables.
The findings presented in the main body of this report are structured around the 7 key themes identified in the Theory of Change (Figure 1, p.8):
1. Improved funding and subsidy 2. Improved TP ratio
3. Strengthened support for professional development of teachers and principals 4. Revised guide to the pre-primary curriculum
5. Increased monitoring and quality assurance
6. Strengthened support for students with diverse needs 7. Strengthened parent engagement and education
This structure was designed to directly focus on the analysis of the different outputs that were expected from the KG policy and would enable the analysis to make reference to any
20
combination of observation, questionnaire and interview findings as appropriate, thereby taking advantage of the multiple sources of data collection provided by this study and maximising the potential for triangulation of findings from different sources.
Analysis of classroom observation data
The analysis plan for the classroom observation data was to first produce descriptive statistics (including means and standard deviations) of the observations from each wave, broken down by class levels (K1, K2, and K3), and then compare against the descriptive statistics from previous waves. Statistics were calculated both by domains and for an average of all items in the ECERS-E, SSTEW and MELE scales.
Subsequent to previous waves, we conducted some initial exploratory analysis of the factors underlying the classroom observation scales. After the completion of all waves of data collection, we have a unique opportunity to analyse the factor structure of all 3 scales with a substantially increased sample size compared to the exploratory analyses conducted previously.
Sufficiently large sample size is highly important for establishing the reliability and validity of factor analysis, as documented in many methodological papers (e.g. Mundfrom et al., 200517), so we feel strongly that including cases from all study waves is of the utmost importance to maximise the methodological rigour of the factor analysis findings. Although the factor analysis from previous waves was informative in an exploratory sense, we can now specify the factors in the dataset in a more rigorous way with the advantage of the larger sample.
The classroom observation component of the study provides a unique opportunity to examine the psychometric properties of 3 different scales at the same time – ECERS-E, MELE, and SSTEW – which were developed to focus on the measurement of preschool quality internationally. This offers much more detail and nuance than studies using only one observation scale. This was a key rationale for including several scales as part of the study design.
Because there are certain universally agreed upon elements of preschool classroom quality (including teacher-child interactions), it was expected that there are items on the 3 scales that we chose to use were similar and correlated with each other. Factor analysis enables us to examine the common elements of the scales and can allow us to group the similar items together in a way that identifies the distinct constructs being measured. Factor analysis deals with the overlap between items a statistically appropriate way.
One great advantage of using factor analysis with this dataset is that it is possible to identify dimensions across all 3 scales, which can be interpreted substantively in a way that is more intuitive and theoretically meaningful than simply taking the average of one or other of the scales. Another advantage is that factor analysis can identify the dimensionality of the scale and ensure that there is no over interpretation based only on items or subscales. We can take advantage of the unique opportunity this dataset provides to understand the different dimensions seen across these 3 scales combined.
In summary, we would use the factor analysis based on the data from all waves of the study because:
17 Mundfrom, D. J., Shaw, D. G., & Ke, T. L. (2005). Minimum sample size recommendations for conducting factor analyses. International Journal of Testing, 5(2), 159-168.
21
1. The additional data collection during Wave 3 provides a larger sample size;
2. Factor analysis of 3 different and overlapping scales measuring preschool classroom quality gives us more nuanced information than most other analyses;
3. The scales were expected to overlap and factor analysis is an appropriate statistical technique to deal with this and identify distinct dimensions; and
4. The grouping of several items together for each dimension is psychometrically more robust than focusing on one specific item or a small subscale of items.
Factor analysis (principal component factors) was conducted on the domain scores (18 domains) from all observations across all 4 phases (Pre-policy, Wave 1, Wave 2, and Wave 3; n=159).
The factor analysis generated 5 factors with eigenvalues of greater than one (see Table B1 and B2 in Appendix B). Eventually, only Factor 1 to Factor 4 (see Table 5) were retained because Factor 5 had low reliability (α=.39), and also had an eigenvalue of only slightly greater than 1 (1.03). Comparisons across waves were made in terms of the scores on the 4 factors. We also analysed certain domains and items pertaining to the KG policy measures and hypothesised impacts. These findings are reported under the relevant themes in this report.
Table 5. Retained factors, domains with high loadings, and alphas
Factor Alpha Domains with high loadings Factor name18 Factor 1 0.82 All SSTEW domains; MELE Interaction;
MELE Teaching and Learning: Overview Supporting
socioemotional and cognitive development Factor 2 0.73 ECERS-E Literacy, Mathematics, Diversity;
MELE Physical Environment,
Teaching/Learning: Free-choice Indoor Activities
Learning environment, catering for learner diversity and free- choice indoor activities Factor 3 0.58 ECERS-E Science and Environment; MELE
Teaching/Learning: Nature and Science Nature and living Factor 4 0.59 MELE Interaction, Inclusiveness,
Teaching/Learning: Language and Literacy, Teaching/Learning: Group Activities
Inclusiveness, group activities and teacher- child interaction Since the MELE measure (4-point scale) was on a different scale than the other measures (7- point scale), all variables were rescaled to have a maximum score of 10. Mean factor scores (scale range 0 to 10) were calculated across each of the waves. A multilevel repeated measures model was run for each of the 4 factors using Stata’s “mixed” command, with each factor score as the dependent variable and wave as the independent variable. This type of model is able to take into account the nesting of observations within schools, as well as the longitudinal nature of the data across waves, to test for statistically significant changes. One of the advantages of mixed models (compared to, for instance, repeated measures ANOVA models) is that they do not require equal numbers of observations per wave, and can therefore take advantage of all
18 Factor names are adopted with reference to the domains of classroom observation tools (ECERS-E, MELE, SSTEW, ECERS-R) concerned as well as the guiding principles and relevant learning areas, such as Nature and living, of the Kindergarten Education Curriculum Guide.
22
data despite the differing numbers of observations across waves. The model was then run again with wave squared as the independent variable to create a quadratic model, and plots of estimated factor scores across waves were created.
Factor 1 (Supporting socioemotional and cognitive development) includes all 5 domains on the SSTEW measure (building trust, confidence and independence; social and emotional well- being; supporting and extending language and communication; supporting learning and critical thinking; and assessing learning and language), and 2 MELE domains (interaction; and teaching and learning: overview). The factor covers mainly aspects of interactional quality of practices that builds children’s social and emotional well-being, relationship development, and skills in sustained shared thinking. Factor 2 and Factor 3 include domains from both ECERS- E and MELE. Factor 2 (Learning environment, catering for learner diversity and free-choice indoor activities) includes domains that are related to the classroom environment, and curricular quality that promote literacy, mathematics, and learner diversity, as well as the quality of free-choice indoor activities. Factor 3 (Nature and living) covers classroom quality on nature, science, and the environment. Factor 4 includes qualities that support interaction, inclusiveness, language and literacy, and group activities captured by the MELE scale.
To study the relationship between school factors and classroom quality, we also explored the associations of key variables (variables related to continuous professional development policy of schools, teacher engagement, participation and experiences as well as parent involvement) and examined the observation scores and changes in observation scores. Findings are reported under the section “School factors conducive to the quality of KG education” in this report.
Analysis of questionnaire data
Both inferential and descriptive analyses were conducted for analysing the questionnaire data across waves. In the analyses, the percentages of participants selecting each option in each question are reported. Even though each questionnaire was analysed separately, we compared responses from respondents (i.e. teachers, principals, and parents) to the same question. A comparison was made in the questionnaire responses between three waves in the themes examined in this study. The longitudinal nature of the dataset was examined to measure changes in questionnaire responses over time using latent growth curve modelling, based on the subset of respondents with repeated measures across waves.
Analysis of interview data
The principal, teacher, and parent interviews were analysed qualitatively. The analyses focused on enabling the understanding of the diverse range of practices and comparing the practices across waves, under the KG policy and the processes by which staff and parents were responding to the KG policy. We followed a two-stage thematic analysis approach (Fereday
& Muir-Cochrane, 2006)19. Data were analysed using NVivo Version 12 (QSR International, 2018)20. In Wave 1, we first identified themes deductively from interview transcripts based on a codebook of themes we had developed a priori based on our underlying theory (Theory of Change) and the literature. Broadly speaking, the themes fall under the 7 major themes, namely,
19 Fereday, J. and E. Muir-Cochrane (2006). Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods 5(1): 1-
20 QSR International (2018). NVivo, Version 12. QSR International. 11.
23
improved funding and subsidy, improved TP ratio, strengthened support for professional development of teachers and principals, revised guide to the pre-primary curriculum, increased monitoring and quality assurance, strengthened support for students with diverse needs, and strengthened parent engagement and education. This was followed by a second round of coding, in which we identified themes inductively from the data, to include observations that were not covered in the scope of our codebook. In Waves 2 and 3, we used the same codebook of themes developed in Wave 1 as the first round of coding. In the second round of coding, again themes were identified inductively from the data that were not covered in our Wave 1 codebook.
24 Findings and discussion
Overview of classroom observations
In presenting an overview of classroom observation results, observation data in all three waves were included in the analysis. In addition, a comparison is made between the Pre-policy K3 classes and the K3 classes in Wave 3. It should be noted that we observed different classes in Wave 1 (K1 and K2 classes), Wave 2 (K1 and K3 classes), and Wave 3 (K2 and K3 classes).
One would expect that KGs would prepare children in K3 classes for the transition to primary school and teachers would be responsive to maturational changes in children. The study’s design to observe different class levels was to ensure that it is possible to identify differences in the impact of KG policy between the years, and to allow us to make comparisons across class levels over time.
Domain scores
Statistics were produced for the Pre-policy, Wave 1, Wave 2, and Wave 3 classroom observation results. Tables B3 and B4 (presented in Appendix B) show the number of observations, means, standard deviations, minimums and maximums for each domain of ECERS-E, ECERS-R (only used for K3 in the Pre-policy phase and Wave 3), SSTEW, and MELE. Statistics are shown for all observations combined, and then individually for Wave 1 K1 classes, Wave 1 K2 classes, Wave 2 K1 classes, Wave 2 K3 classes, Wave 3 K2 classes and Wave 3 K3 classes. ECERS-R (used for the Pre-policy phase only), ECERS-E, and SSTEW were measured on a 7-point scale, whilst MELE was measured on a 4-point scale.
Domain scores can be interpreted as an average of the items within that individual domain, with higher numbers indicating greater classroom quality. The ECERS-R, ECERS-E and SSTEW scales have a maximum score of 7. A score of 1 is considered to reflect inadequate practice while a score of 7 is considered to reflect excellent practice21; and the MELE scale scores quality on a 4-point scale, from 1 (low) to 4 (high). Table B3 (in Appendix B) shows the classroom observation scores for ECERS-E, SSTEW and MELE by domain. Overall, classes had the highest ECERS-E mean score on the Literacy domain (3.39) and lowest on the Science and Environment domain (1.60). The highest mean score for SSTEW was on the Supporting and extending language and communication domain (4.72), and the lowest was on the Supporting learning and critical thinking domain (2.53). The highest MELE mean score was on the Interaction domain (3.59) and the lowest was on the Teaching/Learning: Nature and Science domain (1.97).
Mean scores on each scale overall, by class
Table B5 (in Appendix B) shows the mean scores on all items from each of the classroom observation measures, calculated by taking the average item score for ECERS-E, ECERS-R (only for the Pre-policy phase and Wave 3), SSTEW, and MELE. Results are shown for all
21 Researchers have considered scores below 3 as “poor”; scores between 3.00 and 4.99 “average”; and scores above 5 as “good”. The ECERS-R, ECERS-E and SSTEW scales were normed in Euro-American contexts, which have less structure and deploy different pedagogical approaches than kindergarten classrooms in Hong Kong.
Hence, interpretation of the absolute scores of the scales is not meaningful. For example, preschools in different countries may score differently, but this does not indicate the countries with lower scores have lower quality. We consider the absolute ECERS-R, ECERS-E and SSTEW scores in our study to reflect average classroom quality.
That stated, the focus of this study is to examine change in kindergarten quality after the implementation of the KG education policy and not on the absolute ratings per se.
25
observations overall, and also broken down by class level in each wave (Wave 1 K1, Wave 1 K2, Wave 2 K1, and Wave 2 K3, Wave 3 K2 and Wave 3 K3). As presented in Table 6, amongst the mean scores for ECERS-E, the one in Wave 2 K3 classes (2.81) was the highest, and the one in Wave 3 K3 classes (1.93) was the lowest. Mean SSTEW score in the Wave 3 K2 classes (3.97) was the highest and the one in Pre-policy K3 classes (3.10) was the lowest.
Mean MELE score in Wave 2 K3 classes (3.27) was the highest and the one in Pre-policy K3 classes (2.73) was the lowest. Differences in K1 class scores across waves were tested for statistical significance using t-tests.
Table 6. Mean observation scores by rating scales, phases, and class levels (ECERS-E, ECERS- R and SSTEW are on a 7-point scale; MELE is on a 4-point scale) of classes observed in 25 KGs (Pre-policy n=15, Wave 1 n=50, Wave 2 n=50, Wave 3 n=44)
Pre-policy Wave 1 Wave 2 Wave 3
K3 K1 K2 K1 K3 K2 K3
ECERS-E 2.40 2.19 2.30 2.54 2.81 2.72 1.93
ECERS-R22 3.72 N.A. N.A. N.A. N.A. N.A. 2.58
SSTEW 3.10 3.32 3.47 3.71 3.73 3.97 3.13
MELE 2.73 2.89 2.96 3.08 3.27 3.13 2.81
Observation factor scores across waves
Figure 4 shows the mean factor scores (scale range 0 to 10) across each of the waves, for Factor 1 to Factor 4 and estimated factor scores based on a quadratic model. Across waves, classrooms had higher scores on Factor 4 (Inclusive, group activities and teacher-child interaction), and lower scores on Factor 3 (Nature and living). It implies the relative strength in the interactional quality and promotion of inclusiveness as compared with quality in terms of nature and science captured by the 3 scales we used in the study.
Figure 4. Factor scores across waves
22 The mean scores for ECERS-R are the average scores of the 4 ECERS-R domains common across Pre-policy and Wave 3 (i.e. Space and Furnishings, Personal Care Routines, Activities, and Programme Structure).
01 23 45 67 89 10
FACTOR 1: Supporting socioemotional and cognitive development
FACTOR 2: Learning environment, catering for learner diversity and
free-choice indoor activities
FACTOR 3: Nature and
living FACTOR 4:
Inclusiveness, group activities and teacher-
child interaction
Mean score (range 0 to 10)
PREPOLICY WAVE 1 WAVE 2 WAVE 3
26
Comparing across waves, scores for Factor 1 (Supporting socioemotional and cognitive development), Factor 2 (Learning environment, catering for learner diversity and free-choice indoor activities), and Factor 4 (Inclusiveness, group activities and teacher-child interaction) increased progressively across waves between Pre-policy and Wave 2, and then decreased slightly during Wave 3. Factor 3 (Nature and living) scores were highest during Wave 1, before decreasing slightly during Wave 2 and then more substantially during Wave 3.
Next, a multilevel repeated measures model was run for each of the 4 factors using Stata’s
“mixed” command, with each factor score as the dependent variable and stage as the independent variable. This type of model is able to take into account the nesting of observations within schools, as well as the longitudinal nature of the data across waves.
Multilevel repeated measures models were used once more, but this time using a quadratic model. The results for each factor are shown in Figure 5, and confirm the pattern seen for Figure 4 whereby for factors 1, 2, and 4 there was an increase in scores across the Pre-policy to Wave 2 phases, and a decrease during Wave 3. The findings suggest that classroom quality on these factors concerned improved over time up to Wave 2, and lowered in Wave 3. Since there were restrictions in place in classrooms under the COVID-19 pandemic, we are uncertain whether the changes in Wave 3 are accurate reflections of the quality under the KG policy.
27
Figure 5. Figure showing estimated scores of 4 factors across phases
Estimated Factor 1 (Supporting socioemotional and cognitive development) scores across waves
(quadratic model)
Estimated Factor 2 (Learning environment, catering for learner diversity and free-choice indoor activities) scores across waves (quadratic
model)
Estimated Factor 3 (Nature and living) scores
across waves (quadratic model) Estimated Factor 4 (Inclusiveness, group activities and teacher-child interaction) scores
across waves (quadratic model)
In the following sections, we present findings and discussion from the perspectives of the 7 outputs of the KG policy using the Theory of Change.
