Quality Assurance Division Education Bureau
An Independent Study led by Professor John MacBeath University of Cambridge
The Impact Study on the
Effectiveness of External School Review in Enhancing School Improvement through School Self-evaluation in Hong Kong
(The Impact Study on the Effectiveness of External School Review)
Final Report
THE IMPACT STUDY ON THE EFFECTIVENESS OF
EXTERNAL SCHOOL REVIEW IN ENHANCING SCHOOL IMPROVEMENT THROUGH SCHOOL SELF-EVALVATION IN HONG KONG
(THE IMPACT STUDY ON THE EFFECTIVENESS OF EXTERNAL SCHOOL REVIEW)
FINAL REPORT
ABSTRACT
As the School Development and Accountability (SDA) framework has unfolded during the first cycle of implementation, it has had a powerful and largely positive impact on participating schools. It has put self- evaluation centre stage and lent a sense of urgency to improvement and accountability. There is considerable cause for satisfaction and self- congratulation for what has so far been achieved. In this respect Hong Kong is in the forefront of the move from ‘top-down’ inspection, to the
‘sequential’ model of school self-evaluation (SSE) / external school review (ESR) adopted by the Education Bureau (EDB), now widely seen as the most appropriate form of school development and accountability in the international arena.
In these early stages of a changing model, there remain issues to be tackled, most significantly achieving the balance between improvement and accountability. The overall positive experiences of 724 schools conducting ESR in the first SDA cycle provides a foundation to build on, while the issues of concern raised by school personnel provide an agenda for the future. The challenge is now to provide support for schools to embed self-evaluation as an integral aspect of school and classroom life and to infuse staff’s thinking and practice. Supporting that with ongoing professional development, and keeping the structure and nature of ESR under constant review and refinement is the immediate priority.
EXECUTIVE SUMMARY BACKGROUND
1. The Impact Study on the Effectiveness of ESR in Enhancing School Improvement through SSE in Hong Kong (The Impact Study), an independent study led by Professor John MacBeath of the University of Cambridge from 2003/04, was to evaluate the implementation of the SDA framework with the following objectives:
To evaluate
• the implementation of ESR in the first SDA cycle (2003/04 to 2007/08);
• the impact of ESR in enhancing continuous school improvement through SSE; and
• Hong Kong SSE and ESR practice in relation to other international initiatives.
2. The Impact Study has followed the implementation of ESR since 2003/04, collecting quantitative and qualitative data from five successive cohorts of schools. The data come primarily from school personnel themselves both through interviews and cross-school focus groups with key stakeholders and by means of anonymised questionnaires. The ground covered by questionnaires included purposes and procedures of ESR, schools’ preparedness for ESR, openness and transparency of the ESR process, issues of teachers’ workload and stress, effectiveness of ESR, sustainability of SSE, and the impact of SSE and ESR.
3. Feedback has been continuously collected from various anonymised questionnaires to teachers and principals in 648 ESR schools (including primary, secondary and special) from 2003/04 to April 2008 with over 42,000 teachers/
principals and 203 School Improvement Teams (SIT), with an overall response rate of 70%. The richness of the data makes the study one of the most thorough and comprehensive of its kind undertaken by any government body.
OVERALL IMPACT OF SSE & ESR
4. The implementation of SSE and ESR as complementary processes has served as a significant catalyst to change and school improvement. As a key objective of the Impact Study, the impact of ESR in validating and supporting SSE, which in turn facilitates continuous school improvement, could be categorized into the following 5 main areas:
Giving impetus to nurturing the culture of SSE
5. There is consistent evidence that ESR has played a significant role in helping schools to develop a more reflective culture, and acting as a catalyst for continuous school improvement. Post-ESR questionnaire to teachers and principals from 2003/04 to 2007/08 reveal an increasing positive response to ESR as helping school staff to identify strengths and areas for improvement and to plan future goals accordingly.
Promoting the use of data and evidence as a basis for SSE
6. There is now lessening apprehension about the use of data, greater self-scrutiny, reflection and collegial dialogue. SSE tools provided by EDB, have helped schools to make the transition from an ‘impressionistic’ to an ‘evidence-based’
approach to evaluate school performance.
Creating a greater sense of openness, transparency and collaboration within schools
7. Obtaining the views of a range of stakeholders has encouraged schools to view their practice from differing perspectives, combined with a more distributed leadership in which policy development, at least in some schools, is now becoming more widely shared, particularly through the involvement of SIT.
Enhancing a sense of ownership and team spirit
8. SSE is credited with giving school staff a greater stake in school improvement and a sense of ownership, improved team spirit, and heightened visibility of schools’ accomplishments, both affirming good practice and lending a more critical edge to SSE. Following ESR there is evidence of increased opportunities for teaching staff to work together, across classrooms and departments, sharing practice more openly and assuming greater responsibility for improving practice.
Creating a positive impact on learning and teaching
9. With greater transparency of classroom teaching, peer observation and collaborative lesson planning have given impetus to lessons becoming more engaging and student-centred. ESR recommendations have provided support for school leaders to put in place structural supports and opportunities for reflection, dialogue, and shared pedagogy.
RECOMMENDATIONS
Owning the improvement process and enhancing internal accountability
10. As schools have become more comfortable with the SDA framework and have grasped the benefits of SSE, it has brought with it a growing sense of accountability — to students, colleagues, parents, the school management committees and school sponsoring bodies. This is what is referred to as ‘internal accountability’, the necessary precursor of external accountability. In other words, when there is a strong shared commitment to SSE within the school staff, there is a greater sense of ownership and enhanced confidence in preparing for ESR. While the evidence shows that this is well developed in some schools, it is embryonic in others. The evidence also shows that ownership and enhanced confidence are fostered when there are professional discussions about learning experiences, opportunities and outcomes, and a consistent drive to focus attention on priorities for the young people that schools serve. The introduction of the streamlined and re-organised Performance Indicators (PI), the Planning- Implementation-Evaluation (P-I-E) cycle and provision of focus questions are designed to help schools adopt an inquiry-based approach, but it is crucial that this does not become mechanistic or inhibit spontaneous and creative approaches to SSE. There is a lesson to be learned from international experiences in this respect.
Capacity building across the school system
11. It is vital that EDB continues to build on what has been achieved, highlighting how schools have benefited, particularly in the area of teacher morale and job satisfaction. Capacity building is a long term proposition, and requires vigilance as to the sustainability of SSE following the ESR dip. It needs to be allied to cultivation of a new generation of teachers committed to SSE and equipped with skills, tools and continuing professional development opportunities.
12. Enhancing the role and expertise of external reviewers is a continuing challenge.
Variation in quality, skill and preparedness of external reviewers points to a need for judicious selection, in-depth induction, monitoring and ongoing mentoring/coaching together with an up-to-date information flow and formative feedback loops.
An enhanced role for school improvement teams
13. As School Improvement Teams (SIT) can hold the key to internal accountability and embedding of SSE, membership requires a cross-section of staff with high credibility among their colleagues, scope to exercise initiative and creativity, ownership and a shared vision as to how SSE can feed into school improvement.
Senior leaders need to deploy SIT creatively, giving new teachers as well as experienced staff opportunities to support their peers in school-wide improvement. In this respect the “Online Interactive Resource on Enhanced School Improvement through SSE and ESR” (Online Interactive Resource) of the EDB Website could be used as a reference point and a knowledge source, to be used systematically and critically in order for SIT to reflect on their role and the potential scope of their work. As it develops in the future, the Online Interactive Resource continues to provide a platform for school leaders, SIT, teachers and other stakeholders to share their experience in school improvement, and help to address their concerns and priorities.
Strengthening post-ESR support
14. A key focus for the SDA framework is to ensure that at system and school levels, there is an enhanced ability to cope with and initiate change for continuous improvement. Progressive embedding of SSE relies on continuing support and challenge — from critical friends, other principals, outstanding SIT members in other schools, from exemplary classroom teachers and from creative use of sources such as the Online Interactive Resource. There is a key role for the Regional Education Officers (REO), in particular in the support and challenge they provide for low and mid-performing schools. There needs to be scope for them to play an enhanced role in advising schools in priority setting taking into consideration the ESR recommendations. EDB should strengthen post-ESR support, facilitating collegial networking, providing opportunities for school leaders, SIT and teachers to learn from, and with, their peers.
Student engagement in SSE: raising the priority
15. Progress has been made in grasping the power of student voice but it is still seen as a relatively low priority in schools. There is considerable scope for schools to grasp the potential of engaging students in SSE, both for students themselves as learners and for schools as learning organisations. The brilliant examples of this in the current version of the Online Interactive Resource should be complemented and further enriched. Formal opportunities for staff discussion and planning around exemplars of good practice could be built into ongoing professional development. It should be encouraged by review teams and others who work in a development relationship with schools.
Creating and sustaining a self-evaluation climate
16. Teachers’ workload has remained a contentious issue, with much of the concern focused specifically on documentation. That it was mentioned much less from 2005/06 onwards is a signal that the message from the Quality Assurance Division (QAD) about reducing the plethora of paper has been getting through.
EDB continues to place emphasis on the feedback loop through which it is able to respond positively to teachers’ and schools’ concerns about pressure and workload associated with change and reform. The key SSE tools including the Performance Indicators (PI), Key Performance Measures (KPM) and Stakeholder Survey (SHS), have been revised, based on the principles of simplification, re- organisation and refinement. The aims of revision are to help schools review their work in a more effective and focused manner, so as to alleviate teachers’
workload. It is critical, however, that SSE is not simply associated with workload as this works counter to it being recognised as integral to the day-to-day work of teachers, not an event or an onerous extra.
17. School leaders at every level will play a vital role in reinforcing this message and creating a climate friendly to SSE and innovation while ensuring that funding and support are deployed to best effect.
Embedding SSE
18. The ultimate purpose of the SDA framework is to embed SSE into the thinking and practice of teachers in schools. It is a long term goal but to which a range of stakeholders can contribute. This includes:
• The School Management Committee (SMC) / Incorporated Management Committee (IMC) who need to be informed, keep a watching brief on school development, and offer both support and challenge to school staff to strive for continuous school improvement.
• Senior leaders who keep the SMC / IMC up-to-date on school development, lead their colleagues by example, and keep SSE continuously to the fore in people’s thinking and practice.
• Middle managers who act as intermediaries between senior leaders and school staff, encouraging teachers within their departments to step outside of their subject to adopt a commitment to whole-school improvement.
• School Improvement Teams who share leadership, take the initiative in supporting their colleagues and assume responsibility for the successful embedding of SSE practice.
• Teachers who are the ultimate gatekeepers and champions of SSE, through promoting continuing reflection and critique on the quality of learning and teaching in their classrooms and beyond, through more evidence-based approaches.
• Parents who are the first and most important educator, have a responsibility to take every opportunity to maintain a liaison with teachers in a joint commitment to support their children’s learning.
• Students who will only become effective lifelong learners when they are self- evaluators, play a role in constructive critique of school life and contribute to school improvement.
A diversified mode of review
19. The biggest challenge for the future will come in the form of a more diversified mode of review for schools at different stages of their development. There are important lessons to be learned both from the first cycle of SDA and from international experience. Lessons learned over the last few years must feed productively into the planning of the second cycle, so that ESR becomes more school-specific and focused, taking as its starting point the improvements suggested in the first cycle and schools’ stated priority areas for development.
20. While there may be opposition to perceptions of unequal treatment, the principle of wise stewardship of public finance review according to school needs, will have to be recognised and accepted as both just and pragmatic. Reinforcing this message will be helped by enlisting allies and advocates from schools and school sponsoring bodies so as to create a tipping point of professional opinion.
21. With regard to classroom observation, there is a need for more rigorous briefing for ESR teams, better information for teachers as to the purpose and criteria of lesson observation and what it is reasonable to expect by way of post-lesson feedback. The very purpose and nature of lesson observation also need to be open to critical discussion and review.
1. INTRODUCTION
Objectives of the Impact Study
1.1 The SDA framework in its first cycle of implementation (from 2003/04 to 2007/08) aims to:
• promote the implementation of a systematic and vigorous SSE mechanism in schools;
• benefit schools through external school review (ESR) by complementing the schools’ self-evaluation (SSE);
• promote the use of readily accessible data on KPM and SHS, and evidence as a basis for robust SSE to facilitate professional capacity building;
• create a greater sense of openness, transparency and accountability within schools and provide the education system and the public with information on school performance; and
• focus the school community on improving learning outcomes.
1.2 To provide an external measure and to draw on international experience for reviewing the mechanisms and effectiveness of ESR, EDB commissioned the evaluation as an independent study, led by Professor John MacBeath, University of Cambridge. The Impact Study on the effectiveness of ESR in enhancing school improvement through SSE in Hong Kong (the Impact Study) was conducted with the following objectives to evaluate:
• implementation of ESR in the first SDA cycle (2003/04 to 2007/08);
• impact of ESR in enhancing continuous school improvement through SSE;
and
• Hong Kong SSE and ESR practice in relation to other international initiatives.
Research methodology
1.3 The Impact Study has followed the implementation of ESR since 2003/04, collecting data from five successive cohorts of schools. Thus it is able to pinpoint trends over time, giving the study a longitudinal element, and to assess the response of school personnel to progressive changes made by EDB.
1.4 The Impact Study has drawn on both quantitative and qualitative approaches to produce its findings. The data come primarily from school personnel
themselves both through interviews and cross-school focus groups with key stakeholders and by means of anonymised questionnaires. The latter had a high rate of return. In addition to structured items, the survey allows space for open-ended comments and respondents could express strong views, both positive and negative. The ground covered by questionnaires included purposes and procedures of ESR, schools’ preparedness for ESR, openness and transparency of the ESR process, issues of teachers’ workload and stress, effectiveness of ESR, sustainability of SSE, and the impact of SSE and ESR.
The data
From ESR schools
1.5 Evaluation of the implementation of ESR has been conducted continuously, with feedback collected from various questionnaires to teachers and principals in 648 ESR schools (including primary, secondary and special) from 2003/04 to April 2008 with over 42,000 teachers/ principals and 203 SIT, with an overall response rate of 70%. The richness of the data makes the study one of the most thorough and comprehensive of its kind undertaken by any government body.
• The Post-ESR Questionnaire1 on SSE/ESR was administered to all teachers and principals in primary, secondary and special schools after the issue of the draft ESR report. Feedback was collected from schools which had undergone ESR from 2003–08, with a response rate of about 81%.
The fact that all responses were anonymised ensures that teachers could feel free to express their views about the actual situation of the schools.
• The Post-ESR Review Questionnaire2 was issued to all teachers and principals in the following school year after ESR to collect views about the impact of ESR on school development and self-evaluation. Feedback was received from respondents of 597 schools which had undergone ESR from 2003–07, with a response rate of about 57%.
• The Questionnaire for SDA framework3 was designed to collect views from the school improvement teams (SIT) of schools which had undergone ESR, on how they perceive their role and ESR’s impact on SSE and school development. Feedback was collected from 203 SITs of schools which undergone ESR from 2003-06, with a response rate of about 82%.
1 The Post-ESR Questionnaire is a source of data on the conduct and impact of SSE/ESR which comes from teachers and principals who have experienced SSE and ESR.
2 The Post-ESR Review Questionnaire is issued to all teachers and principals from schools which had undergone ESR and is a reliable source of data on views about SSE after ESR.
3 The Questionnaire for SDA framework is a survey to collect both quantitative and qualitative data on views of SIT about specific school changes after ESR, and whether and how SSE has been embedded in daily school practice.
• Field observation of the ESR process in 20 schools and case studies in 16 schools which had undergone ESR in 2003/04 and 2004/05.
• 17 cross-school focus groups interviews with key stakeholders including School Sponsoring Bodies, SMC, Principals, SIT, basic rank teachers, parents and students.
From ESR teams
• 1,074 and 207 questionnaire responses from ESR team members and external reviewers respectively with corresponding response rates of 33%
and 43%. The relatively lower response rates might be attributed to the fact that the same group of Quality Assurance Division (QAD) reviewers may serve on many ESR teams, and some did not feel it necessary to return the questionnaire after each ESR unless they had new points to make.
1.6 While such large scale data are helpful from a policy maker’s viewpoint, amalgamating data from 648 schools into one set of data actually conceals important variations from one school to the next and from one group of staff to the next within schools. Background data on school types and teacher characteristics have allowed a disaggregation of perspectives so as to produce a more fine-grained picture of where the issues lie. The more we disaggregate perspectives by school type (primary, secondary and special) and by status (principals, middle managers and teachers) of those who completed the questionnaires, the more complex and fine grained the picture becomes.
2. BUILDING ON PHASES I AND II IMPACT STUDY Achievements and concerns
2.1 Phase I Impact Study evaluated the implementation of ESR in 99 schools undergoing the exercise in 2003/04. With reference to the objectives of implementing the first cycle of the SDA framework, there was clear evidence of a deepening understanding of the purposes of ESR and SSE for continuous school improvement; promoting the use of data and evidence as a basis for SSE; developing a more systematic and informed approach to SSE in schools;
creating a greater sense of openness and transparency within schools for stakeholders; and helping schools to identify their strengths and areas for improvement, especially in the areas of learning and teaching.
2.2 Phase II Impact Study, with more data analysed from 99 and 139 schools undergone ESR in 2003/04 and 2004/05 respectively, identified the following achievements and concerns:
• A deepening understanding and heightened confidence of school staff in relation to SSE and ESR;
• Classroom teaching becoming more engaging, student-centred, and open and receptive to student voice;
• A welcome for the insights of ESR teams and setting of clear agenda for improvement after the review;
• The enhanced skills of ESR teams in conducting review;
• Sharing of thinking and practice by teachers beyond the classroom in a whole-school dialogue;
• A growing concern for and willingness to engage with evidence to move from impressionistic evaluation of quality and performance to a more systematic, rigorous and informed approach to assessing practice; and
• Concerns about teachers’ workload and stress.
Modifying ESR procedures to address teachers’ concerns
2.3 At each stage of the Impact Study, it has created dialogue within EDB, with QAD staff, with schools invited to seminars and workshops, etc. Taking the review of ESR procedures as a formative and collaborative exercise, EDB regularly made reference to the findings of the study together with the views collected from various internal and external mechanisms to streamline the ESR procedures and reduce the amount of documentation required. In July 2005, significant modifications were made to the implementation requirements when there were further signs of over-preparation by some schools for SSE and ESR and diversion from a self-reflecting process to a reporting process. The key changes made include:
• bringing to a stop ratings by schools and the ESR teams on the 14 PI areas;
• withholding the uploading of ESR reports to the EDB website for public access having regard to schools’ concern that ESR reports could be selectively reported in the media to the detriment of the schools’
reputation;
• confining the number of school documents prepared for ESR to three, i.e., School Self-Assessment (SSA) Report on the 14 PI areas — the formal SSA report prepared prior to ESR, KPM and Stakeholder Survey (SHS) findings; and
• recommending that the SSA report should not exceed 20 pages.
2.4 The series of responsive measures also included the development of an e- Platform for SDA (ESDA) to enhance systematic data management, an on-line data collection tool and the revised version of SHS.
3. KEY FINDINGS OF PHASE III IMPACT STUDY
3.1 Phase I and Phase II Impact Study reports, as a composite, have concluded that with self-evaluation centre stage in school improvement and accountability, the primary challenge for the future is to provide support for schools to embed self-evaluation as an integral aspect of school and classroom life, infusing the thinking and practice of teachers and school leaders. This implies ongoing professional development, keeping the structure and nature of ESR under constant review and continually refining and enhancing the relationship between internal SSE and external school review.
3.2 With the Impact Study straddling over the years of the implementation of the first cycle of the SDA framework, Phase III of the study builds on its two predecessors (of Phases I and II) to make some concluding remarks about the developing story of SSE and ESR in Hong Kong schools, by gauging schools’
response to, and rating of, two key aspects: the ESR procedures, and the way the ESR teams work. In addition, the study looks at the perceived impact of SSE and ESR on school management and teaching and learning from those schools which went through ESR. The findings not only provide a ‘state-of- the-art’ review of the impact on schools but also illustrate ways in which school staff have responded over the course of the period from 2003/04 to 2007/08.
The effectiveness of ESR
3.3 Responses of school personnel to the question “How effectively do the ESR procedures address issues of accountability, openness and transparency?”
were very positive, with a marked increase of views between 2003/04 with 59% agreement and 2007/08 with 68% of school staff agreeing that the ESR process was open and transparent.
3.4 Overall and across all five cohorts, a majority of school staff expressed satisfaction when asked “How satisfied are you with the ESR process?” The satisfaction rating has risen from 59% in 2003/04 to 63% in 2007/08. Among the positive comments were many both appreciative and complimentary, for example:
My impression of ESR has completely changed. That was a very professional and meaningful assessment. ( primary teacher)
3.5 The brevity of the review process emanated, it was claimed by a few, in a cursory judgement, failing to take into account the context in which schools worked, using a ‘common yardstick’. Yet to the question “How satisfied are schools with the range of activities included in ESR?” there is a positive response by school staff, from around 60% agreement in 2003/04 to 71%
agreement in 2007/08.
3.6 There was a revisiting of comments with regard to lesson observation:
teachers’ claim of being unaware of criteria used, and lack of individual feedback following the lesson observed. The perceived stress and dissatisfaction associated with lesson observation were seen as exacerbating the overall pressure of the ESR visit.
3.7 While write-in comments may invite expression by people frustrated by the lack of opportunities to express their views elsewhere and may come from schools with poor leadership there is, however, enough substance in these comments for them to deserve serious consideration. There continue to be some voices with write-in comments from about 1,260 responses (3.0% of the overall respondents). Other than some respondents expressing appreciation for ESR teams’ professionalism, the comments are skewed significantly to the negative. Key issues were:
• inadequate time given to evaluating the school performance during the ESR process;
• inadequate time for school’s preparation and response to the draft report;
• workload and stress related to documentation and lesson observation;
• nature and process of lesson observation, and its lack of feedback to individual teachers;
• validity and objectivity of team judgements based on school context; and
• lack of concrete suggestions for improvement.
3.8 The inter-relationship between review time, opportunities for feedback and validity of judgements has to be understood as in the context of review.
Concern over subjective judgement made within a short time span are virtually inevitable but have to be weighed against the comments that ESR teams were perceptive and accurate in the quality of their judgements. These issues are particularly acute in relation to lesson observation. This is always a sensitive issue and raises questions about the basis on which judgements are made.
Teachers’ concerns were primarily about the amount of time given to observation, the nature of conclusions reached as a consequence to the overall effectiveness of learning and teaching, and the lack of oral feedback to individual teachers which followed. While these issues were raised by a small minority in written comments, they were also a subject of comment in focus group and face-to-face meetings, and point to an issue which will be of continuing relevance as the ESR process develops and grows over time.
How schools rate the work of the review teams
3.9 The second major aspect of the findings covers how school personnel rate the work of ESR teams. Consistently over five cohorts, external reviewers have been rated as sincere and friendly, with an increase from 76% in agreement in
2003/04 to 81% in 2007/08. The same trend is evident in response to the statement “The external reviewers were professional in their work.” The agreement rating has risen from 69% in 2003/04 to 76% in 2007/08. This points strongly to ESR teams becoming more sensitive, approachable and informed about the process and pitfalls of ESR.
3.10 Schools’ response to the question “How accurate was ESR’s evaluation of schools’ performance?” again are getting positive. A clear indicator that the scope and quality of ESR teams’ work was being seen progressively in a more positive light with an agreement rating rising from 58% to 70% between 2003/04 and 2007/08.
Enhancing school improvement through SSE and SSA
3.11 The items which refer to the SSA are also highly positively rated. Included for the first time in the 2006/07 questionnaire survey, two statements receive a positive response:
“Through school self-assessment I have a better understanding of the overall performance of the school”: 85% of staff agreed/strongly agreed.
“School self-assessment has enhanced the professional exchange among school staff on school improvement”: 78% of staff agreed/strongly agreed.
3.12 The Questionnaire for SDA framework was administered to schools having undergone ESR in 2003/04, 2004/05 & 2005/06 to invite the SIT to describe the extent to which their school fits the descriptors of seven key factors that are likely to distinguish schools with strongly embedded SSE:
• A strong shared belief that SSE is an essential ingredient of school improvement;
• SSE seen not as an event but as a continuing process in which evidence is sought and used critically and creatively;
• A routine established for teachers and students to evaluate the quality of classroom teaching and to explore alternative strategies which enhance learning;
• Teachers welcome their colleagues’ support and critique through classroom observation and other forums in which teachers exchange and review their practice;
• In the light of continuous feedback from staff, students and parents, school structures being revised to create time and opportunities for reflection and dialogue;
• Strategies in place which ensure that as teachers and senior leaders leave, SSE remains a core commitment of the school; and
• Evidence being gathered in a systematic way so that the school is open and welcoming of ESR.
3.13 SIT members are, overall, very positive about the value and impact of SSE/ESR with primary schools apparently more able to create a coherent whole-school approach than their secondary colleagues. A closer read suggests that there is greater room for doubt when it comes to the integration and embedding of SSE into the daily life of schools and classrooms and into teachers’ habits of thought. This casts some doubt on potential sustainability and suggests that the continuing challenge is for school leaders and SIT to support the use of tools and evidence for SSE on a more systematic basis.
3.14 Systematic evidence gathering, ongoing documentation and data management are widely seen by SITs as integral to an evidence-based approach. There is also virtually 100 percent consensus that SSE is a continuous process, and that it is evidence-based and implies the possession and use of SSE tools.
3.15 The embedding of collaborative lesson planning through structured time, together with peer observation, receives very strong support from teachers (close to 100% in 2003 – 06). Lesson observation as part of teacher appraisal is also viewed positively (over 84% in 2003 – 06), though seen as less attractive than peer evaluation (96% in 2003/04, 93% in 2004/05 and 97% in 2005/06).
3.16 Empowerment and delegation mechanisms for distributed leadership appear to be most strongly seen as a key element of sustainability (around 95%
agreement in three cohorts), followed by succession planning of senior leadership (around 85% in both 2003/04 & 2004/05 cohorts and 87% in 2005/06). There is greater room for doubt, however, when it comes to a sense of ownership of SSE among teachers (72–73% in both 2003/04 & 2004/05 cohorts and 80% in 2005/06).
3.17 Ongoing dialogue on school improvement is generally seen as important but there is a considerable measure of disagreement as to who is involved in that process. There is virtual unanimity on teachers’ active engagement in dialogue on school priorities (over 94% in 2003 – 06) while teacher dialogue on school values falls to around 84% in three cohorts. However, when it comes to students’ active dialogue on values, agreement plummets — to 20%
in 2003/04 and 25% in both 2004/05 & 2005/06; and even further as regards student voice on school priorities. While parents are seen as having a stronger dialogic role with regard to priorities and values, it is still a minority who respond in positive terms.
The appropriateness of the refined ESR procedures
3.18 One of the objectives of the Phase III Impact study has been to evaluate the implementation of ESR in schools from 2005/06 onwards in terms of the appropriateness of the refined ESR procedures in addressing schools’ concerns
about teachers’ workload and stress, and the quality of the ESR team’s work in validating SSE.
3.19 Phase III Impact Study explores the following issues after ESR procedures had been modified.
How did schools find the modified requirements for ESR? To what extent are the modified requirements and procedures able to address schools’ concerns?
3.20 From the feedback collected, schools generally welcome the modification of implementation requirements and the various support measures introduced for SSE and ESR. Teachers’ concern about workload has been addressed to a large extent. The clearest evidence of improvement over time, however, is in relation to the pre-ESR visit. By 2007/08, agreement (including those agreeing and strongly agreeing) on the informative value of the pre-ESR visit has risen to 86% from 77% in 2003/04, while on the question “The pre-ESR visit addressed our queries and concerns.” there is a highly significant increase from 53% to 81%. This rise is particularly significant for two reasons.
Firstly, because it occurs only in the last year when procedures had been modified. Secondly, because it is less about information than addressing emotional issues and teachers’ anxiety. Over the years, teachers’ ratings on the item “ESR did not exert much pressure on me” had been steady from 2003/04 to 2005/06, but perception has become more positive in the 2007/08 school year. Those who agreed or strongly agreed increased from a previous average of about 23% to 32% in the latest survey. The percentage of teachers who strongly disagreed with this statement, reduced from around 18% in the previous years to 13% in 2007/08.
3.21 Stress and workload tend to be inter-related and are a common theme in many countries in which demands on schools are on the increase. In this respect effective and visionary leadership hold the key. While SSE and the prospect of ESR were often seen as adding to both teachers’ workload and stress, it was in preparation for ESR that the quality of leadership was either affirmed or exposed. Where there was strong and self-confident leadership, the process was managed with minimum disruption, minimal anxiety and minimal addition to workload. Where leadership was weak, anxiety and disruption to routine could run through the school, with a principal’s own anxiety and lack of confidence infecting staff and inducing over-preparation.
4. OVERALL IMPACT OF SSE AND ESR
4.1 The implementation of SSE and ESR as complementary processes has served as a significant catalyst to change and school improvement. As a key objective of the Impact Study, the impact of ESR in validating and supporting SSE, which in turn facilitates continuous school improvement, could be categorized into the following 5 main areas:
Giving impetus to nurturing the culture of SSE
4.2 Taken together there is consistent evidence from these various data sources that ESR has played a significant role in helping schools to develop a more reflective culture, and acting as a catalyst for continuous school improvement through the conduct of rigorous and systematic SSE. Following ESR many schools have engaged in a critical review of their school-wide planning and its alignment with programme plans of subject panels, taking a keener account of school-wide objectives and educational vision. There has been a significant shift in some schools for staff to be more rigorous in self-criticism and to be supportively critical of colleagues’ practice.
4.3 As identified from the Post-ESR questionnaire to teachers, over the five years from 2003/04 to 2007/08:
• On the new item in the 2006/07 and 2007/08 questionnaire survey, around 80% of the teachers in both cohorts agreed that “ESR helps me reflect on the effectiveness of my school work”.
• “The ESR has accurately identified the strengths of our school and the areas for improvement”, there is an increase from 58% in 2003/04 to 70%
in 2007/08.
• In five cohorts, there is a consistent consensus with over 71% agreeing that “The ESR has helped our school devise future goals and development plans”.
4.4 In the Post-ESR Review Questionnaire for teachers, over the four years from 2003/04 to 2006/07, there is a consistent consensus with around 71% agreeing that “The school has progressively put in place a systematic approach to SSE”, and around 64% agreeing that “ESR has helped to promote continuous development through SSE.”
Promoting the use of data and evidence as a basis for SSE
4.5 Apprehension about the use of data, strongly associated with numbers and arcane statistics, appears to be lessening. Engaging in SSE and going through the SSA process has required greater self-scrutiny, reflection and collegial dialogue. SSE tools provided by EDB, including the Schools Value-Added Information System (SVAIS), Assessment Programme for Affective and Social Outcomes (APASO), KPM and SHS, and school-based questionnaires have been used by schools to adopt a data-driven approach to SSE. Schools have therefore increasingly made the transition from an ‘impressionistic’ to an
‘evidence-based’ approach to evaluate school performance. According to the findings of the Questionnaire for SDA framework, 92% of the SIT in 2003 – 2006 agreed that “As a school we have moved from subjective evaluation to a more evidence-based approach”.
Creating a greater sense of openness, transparency and collaboration within schools
4.6 The inclusion of a range of stakeholders (including teachers, students and parents) in the SSE process has encouraged schools to view their practice from differing perspectives. In the findings of the Post-ESR Review Questionnaire for teachers, over the years around two thirds of the respondents in primary, secondary and special schools were in agreement that “The school considers different stakeholders’ views when formulating school policies.” Channels are provided to give stakeholders access to information on school performance and to encourage teachers’ participation in formulating school plans and major concerns. There is evidence of a greater willingness to appreciate differing perspectives and priorities, and where teachers had previously been content to leave decision-making to middle and senior managers, there is now evidence of heightened willingness to assume responsibility and to exercise personal and shared authority. SSE/ESR has provided impetus to a more distributed leadership in which policy development, at least in some schools, is now more widely shared, particularly through the involvement of SIT.
Enhancing a sense of ownership and team spirit
4.7 SSE was credited with giving staff a greater stake in school improvement and a sense of ownership. This, in turn, had improved team spirit, making achievements more visible and creating a feeling of pride in what had been accomplished together. ESR was credited with affirming what staff had done.
It was said in one school that within the visit, morale had visibly increased and that conversations among staff were more frequent, more informed, more self- critical, and leading to more effective implementation of school policies.
After ESR, SSE would focus on the major concerns. We had more thorough discussion and staff have built up a greater sense of ownership and involvement. (secondary school principal)
4.8 There was evidence that following ESR, there were increased opportunities for staff to work together, across classrooms and across departmental boundaries.
As one panel member claimed, ‘Before ESR, it was a one panel thing, but it becomes an across-panel matter after ESR.’ While in the past, only the middle managers had an overview of school performance, it was said, now all staff had a clear understanding of the performance of the school.
Creating a positive impact on learning and teaching
4.9 Schools are increasingly aware of the need to improve student learning with evidence so that classroom teaching is now more open to evaluation by fellow teachers and students, resulting in a greater sense of professional trust and critique. Classroom teaching has been more engaging, student-centred and open to critical appraisal by other staff, senior management and students.
While it cannot be claimed that change in pedagogy is solely attributable to SSE/ESR, self-evaluation has played a part in helping teachers to be more
open with their colleagues’ critique. Practices such as peer observation and collaborative lesson planning are seen to be given significant impetus, with ESR recommendations helping school leaders to put in place the structural support to create time and opportunities for reflection and dialogue, such as flexible timetabling and “structured time for collaborative lesson planning”.
In the Questionnaire for SDA framework, on the item “SSE and ESR have together made a positive impact on the quality of learning and teaching in this school”, the level of agreement among SIT in all schools over the three years from 2003/04 to 2005/06 was as high as 97%.
5. INTERNATIONAL PERSPECTIVES
5.1 Quality assurance systems around the world are in a state of continuous evolution. This is because no country has yet found the ideal balance between internal and external evaluation of school quality and effectiveness. At two extremes are the views that only a rigorous external and objective system can provide an authentic picture of how good a school is, and that schools themselves are the only body which has the self-knowledge to evaluate itself.
Most commentators and policy bodies now believe that the ideal is a combination of the two and that external review works best when there is well- developed SSE in place and that effective SSE is bolstered by strong external support. For example, a 2004 study conducted by the Standing International Conference on Inspection (SICI) in Europe found that:
The school visits conducted as part of the project have shown that self- evaluation is most effective in countries that have the strongest external support to the process and thus have created a culture and climate for effective school self-evaluation.
5.2 In this respect Hong Kong is in the forefront of the move from ‘top-down’
inspection, to the ‘sequential’ model of SSE/ESR adopted by EDB, now widely seen as the most appropriate form of school development and accountability. There are close parallels to the New Relationship with Schools in England, where the focus of inspection is on the robustness of the school’s ability and self-confidence to tell its own story. This does not happen overnight and requires a continuing commitment by local authorities, regional education offices and central governments, to encourage and support schools to build capacity.
5.3 The history of self-evaluation in many countries shows that self-evaluation has often grown from the bottom up, individual schools, or clusters of schools having initiated their own process, typically through partnership with a university, school district or commune. Examples of this could be found in almost any country (e.g. England, Germany, Australia, Singapore, Thailand, Hong Kong) before system-wide adoption of SSE. These self-initiated approaches often had vitality and drive because they were ‘owned’ by schools themselves but also suffered because there was no wider networking or systemic support. The lesson for governments is to provide that systemic
support, to nurture those home grown SSE mechanisms while enhancing their efforts with facilitative (not mandated) frameworks and provision of appropriate SSE tools and strategies. School improvement is essentially an integral process in the work of teachers and school leaders when there is a learning culture. It is in the process of building that learning culture that SSE takes root and ESR plays a supportive and challenging role.
5.4 In the change from inspection or QA systems, the essential purpose of school review is often unclear to teachers and, therefore a lot of ground learning has to take place to alleviate anxiety and misconception. The onus falls on policy makers, on frameworks and structures and on review teams themselves to make preparation for review as positive and minimally disruptive as possible.
5.5 It will be critical in the immediate and long term future to stay abreast of development elsewhere, observing the precept – “Watch, don’t copy!”. It will be of equally high priority to respond positively and with resolution to increasing pressures both internally within Hong Kong as well as from globalisation imperatives. It will be critical to keep the dialogue open but to stand by the evidence, continuing to build the case with the support of key spokespersons and powerful advocates of SSE, from schools and governing bodies. Equally vital is to continue to deploy the leading-edge practitioners in reviewing, support, coaching and mentoring roles.
6. CONCLUSIONS AND RECOMMENDATIONS
Owning the improvement process and enhancing internal accountability
6.1 As schools have become more comfortable with the SDA framework and have grasped the benefits of SSE, it has brought with it a growing sense of accountability — students, colleagues, parents, the school management committees and school sponsoring bodies. This is what is referred to as
‘internal accountability’, the necessary precursor of external accountability.
6.2 Internal accountability describes the conditions in a school that precede and shape the responses of schools to pressure that originates in policies outside the organisation. The level or degree of internal accountability is measured by the degree of convergence among what individuals say they are responsible for (responsibility), what people say the organisation is responsible for (expectations), and the internal mechanism and processes by which people literally account for their work (accountability structures). With strong internal accountability, schools are likely to be more responsive to external pressure for performance. The concept of ‘inquiry-based accountability’ provides the entry point for professional discussions about learning experiences, opportunities and outcomes, directing people’s attention to priorities for the young people that schools serve.
6.3 In other words, when there is a strong shared commitment to SSE within the school staff, there is a greater sense of ownership and enhanced confidence in
preparing for ESR. While the evidence shows that this is well developed in some schools, it is embryonic in others. The introduction of streamlined and re- organised PI, the Planning-Implementation-Evaluation (P-I-E) cycle and provision of focus questions are designed to help schools adopt an inquiry-based approach but it is crucial that this does not become mechanistic or inhibit spontaneous and creative approaches to SSE. There is a lesson to be learned from international experiences where the introduction of prescriptive frameworks has created greater compliance and reduced diversity and school- led initiatives.
Capacity building across the school system
6.4 It is now widely recognised that system-wide reform relies on there being a resilience and a capacity at every level; so change is not resisted but welcomed because people, individually and collectively, have the understanding, the ability and the tools to effect change. It is vital, therefore, that EDB continues to build on what has been achieved, highlighting how schools have benefited, particularly in the area of teacher morale and job satisfaction. Capacity building is a long term proposition, and requires vigilance as to the sustainability of SSE following the ESR dip. It needs to be allied to cultivation of a new generation of teachers committed to SSE and equipped with skills, tools and continuing professional development opportunities. In parallel, liaison with teacher educators in tertiary institutes on pre-service teacher training is crucial in preparing teachers for a system in transition.
6.5 Enhancing the role and expertise of external reviewers is a continuing challenge.
Variation in quality, skill and preparedness of external reviewers points to a need for judicious selection, in-depth induction, monitoring and ongoing mentoring/coaching together with an up-to-date information flow and formative feedback loops.
6.6 Consideration also needs to be given to expectations of, and pressure on, ESR teams and team leaders in particular. It is easy to underestimate the stress they are under and the high stakes nature of their work, as well as the potential deleterious effects on the quality of their work and its sustainability. The issue of work/life balance needs to be revisited at every level of policy development.
6.7 Professional development programmes will continue to play an important role in relation to innovation such as the New Senior Secondary Curriculum.
Workshops for those who hold the levers for school improvement — senior and middle leaders and SIT and school personnel seconded to ESR teams — will continue to play a vital part in territory-wide capacity building. Experience gained as seconded school personnel and external reviewers enhances leadership capacity both within their own schools and in strengthening networked learning and communities of practice.
An enhanced role for school improvement teams
6.8 SIT are a signal strength of the Hong Kong approach and, when selected and well supported by senior leaders, hold the key to internal accountability and embedding of SSE. Experience shows that when membership comprises a cross-section of staff with high credibility among their colleagues, their influence is most likely to pay off in tangible school improvement. This happens when they have scope to exercise initiative and creativity, have a genuine sense of ownership and a shared vision as to how SSE can feed into school improvement. Senior leaders need to deploy SIT creatively, giving new teachers as well as experienced staff opportunities to support their peers in school-wide improvement.
6.9 This has time and resource implications and senior leaders need the perspicacity and courage to share leadership, provide space and support for SIT to take the initiative, and take risks together with a strong sense of accountability to their colleagues. While there may be cultural resistance to a more democratic approach, practice in leading-edge schools shows that it can be implemented, evaluated and be shown to enhance practice. In this respect the Online Interactive Resource of the EDB Website could be used as a reference point and a knowledge source, to be used systematically and critically in order for SIT to reflect on their role and the potential scope of their work. As it develops in the future, the Online Interactive Resource continues to provide a platform for school leaders, SIT, teachers and other stakeholders to share their experience in school improvement and help to address their concerns and priorities. Evidence from the Impact Study indicates the following:
• Membership of the team covers a cross-section of staff with high credibility among their colleagues.
• The School Improvement Team enjoys scope to exercise initiative and creativity.
• There is a willingness and capability to ask hard questions and instil an ethos of accountability.
• Teamwork exceeds and synergises the professional capacities of all its members.
• Initiative and ownership create confidence and shared leadership throughout the team.
• There is a vision as to what SSE can achieve and how it can feed into school improvement.
Strengthening post-ESR support
6.10 Research has shown that it takes some years for change to reach the tipping point into reformed practice. It is well established that following inspection or
external review, there is often a period of recovery and return to routine. It is important that the momentum for self-evaluation is maintained, a progressive embedding of SSE in the day-to-day life of schools and classrooms. This can prove difficult for schools to do this for themselves and most schools need for continuing support and challenge both from school leadership and from external sources.
6.11 A key focus for the SDA framework, therefore, is to ensure that at system and school levels, there is an enhanced ability to cope with and initiate change for continuous improvement. Progressive embedding of SSE will rely on continuing support and challenge — from critical friends, other principals, outstanding SIT members in other schools, from exemplary classroom teachers and from creative use of sources such as the Online Interactive Resource. There is a key role for Regional Education Officers (REO), in particular in the support and challenge they provide for low and mid- performing schools. There needs to be scope for them to play an enhanced role in advising schools in priority setting taking into consideration the ESR recommendations. EDB should strengthen post-ESR support, facilitating collegial networking, and providing opportunities for school leaders, SIT and teachers to learn from, and with, their peers.
Student engagement in SSE: raising the priority
6.12 It is only in the last decade or so that countries around the world have begun to grasp the importance of students playing a central role in SSE and school improvement. The evidence from the Impact Study shows that progress has been made in grasping the power and potential of student voice. However, in many schools this is still seen as a relatively low priority. This is explained in part by perceptions that students may have little of worth to say, or because they may be too challenging, or it may stem from a school’s lack of knowledge as to how to go about it and what strategies to use the data. In some cases ‘voice’ and ‘participation’ may be addressed but restricting the scope of what students are allowed to say or do.
6.13 There remains considerable latitude for schools to grasp the potential of engaging students in SSE and to be more adventurous in learning from students. This has multiple benefits — for students themselves as learners and for schools as learning organisations. The brilliant examples of this in the current version of the Online Interactive Resource should be complemented and further enriched.
6.14 Formal opportunities for staff discussion and planning around exemplars of good practice, whether as exemplified by the Online Interactive Resource, whether in face-to-face collegial exchange or in focused school visits, could be built into ongoing professional development. It should be encouraged by review teams and others who work in a development relationship with schools.
Creating and sustaining a self-evaluation climate
6.15 SSE is effective, owned by teachers and participated in by students when there is a climate conducive to reflection, inquiry and shared analysis of practice.
As long as it is perceived as a ‘burden’, adding to workload, it will not be engaged in with enthusiasm and goodwill. The finding that workload has been a contentious issue has a lot to do with such perceptions and the association of SSE with documentation. The time and pressure in preparing for ESR may be seen as a strong indicator of a school’s failure to maintain data and documentation over time, or to embed a robust SSE culture. That it was mentioned much less from 2005/06 onwards is perhaps a signal that the message from QAD about reducing the plethora of paper has been getting through.
6.16 EDB continues to place emphasis on the feedback loop through which it is able to respond positively to teachers’ and schools’ concerns about pressure and workload associated with change and reform. The key SSE tools including the PI, KPM and SHS have been revised, based on the principles of simplification, re-organisation and refinement. The aims of revision are to help schools review their work in a more effective and focused manner, so as to alleviate teachers’ workload. It is critical, however, that the perceived connection between SSE and workload is removed as this hampers efforts to have recognised as integral to the day-to-day work of teachers, not an event or an onerous extra.
6.17 School leaders at every level will play a vital role in reinforcing this message and creating a climate friendly to self-evaluation and innovation while ensuring that funding and support are deployed to best effect.
Embedding SSE
6.18 Each of the above recommendations are essentially directed at a common goal — embedding SSE into the thinking and practice of teachers in Hong Kong schools together with a commitment to intelligent accountability. No educational system is able to reach such a goal by didactic or top-down mandate. It is a long term goal but one to which a range of stakeholders can contribute. This includes:
The School Management Committee / Incorporated Management Committee who need to be informed, keep a watching brief on school development and offer both support and challenge to school staff to strive for continuous school improvement.
Senior leaders who keep the SMC / IMC up-to-date on school development lead their colleagues by example, and keep SSE continuously to the fore in people’s thinking and practice.
Middle managers who act as intermediaries between senior leaders and school staff, encouraging teachers within their departments to step
outside of their subject to adopt a commitment to whole-school improvement.
School Improvement Teams who share leadership, take the initiative in supporting their colleagues and assume responsibility for the successful embedding of SSE practice.
Teachers who are the ultimate gatekeepers and champions of SSE, through promoting continuing reflection and critique on the quality of learning and teaching in their classrooms and beyond, through more evidence-based approaches.
Parents who are the first and most important educator, have a responsibility to take every opportunity to maintain a liaison with teachers in a joint commitment to support their children’s learning.
Students who will only become effective lifelong learners when they are self-evaluators, play a role in constructive critique of school life and contribute to school improvement.
6.19 Each of these different stakeholder groups plays its part in the overall thrust of school improvement but not in isolation. Their roles are interlocking, complementary and often overlapping. They are hallmarks of the learning organisation, a place in which everyone learns and everyone contributes to making the school a better place, not only for children but for all of those who work or visit there.
A diversified mode of review
6.20 One of the biggest challenges for the future in respect of SDA will come in the form of a more diversified mode of review for schools at different stages of their development. There are important lessons to be learned both from the first cycle of SDA and from international experience. Lessons learned over the last few years must feed productively into the second cycle, so that ESR becomes more school-specific and focused, taking as its starting point the improvements suggested in the first cycle and schools’ stated priority areas for development.
6.21 While there may be opposition to what may be perceived as unequal treatment, the principle of wise stewardship of public finance and exercising each review according to school needs will have to be recognised and accepted as both just and pragmatic. Reinforcing this message will be helped by enlisting allies and advocates from schools and school sponsoring bodies so as to create a tipping point of professional opinion.
6.22 Concerns about classroom observation also need to be the subject of dialogue between schools and policy makers as its purpose and protocols continue to remain unclear to many teachers and cause some resentment. There would appear to be a need for more rigorous briefing for ESR teams, better
information for teachers as to the purpose and criteria of lesson observation, and what it is reasonable to expect (and not expect) by way of post-lesson feedback. The very purpose and nature of lesson observation also need to be open to critical discussion and review. Consideration also needs to be given to exploring what alternative procedures might be more effective in reviewing the overall effectiveness of learning and teaching, while addressing teachers’
expectation of individual feedback in the future.
