Investigation of the Mechanism of the Hong Kong Diploma of Secondary Education Examination Question Paper Setting of the History Subject:
Major Findings and Recommendations from the Education Bureau Task Force
In the 2020 Hong Kong Diploma of Secondary Education (HKDSE) Examination, one of the questions in the History subject has aroused great controversy in society.
In Question 2(c) of Paper 1 (“Question 2(c)”), candidates were required to answer whether they agreed with the quoted statement “Japan did more good than harm to China in the period 1900-45” using information provided in the Question as well as their own knowledge. There was great concern and dispute in the society on the appropriateness of the Question. The Education Bureau (EDB) issued a statement regarding the matter on 14 May 20201, and requested the Hong Kong Examinations and Assessment Authority (HKEAA) to investigate the matter, including conducting a review of the question-setting mechanism to ensure fairness, impartiality and credibility of the HKDSE Examination. The Secretary for Education affirmed in the press briefing on 15 May 2020 that the EDB would review the existing mechanism with a view to ensuring the sustained quality of the HKDSE Examination and examination questions.
To deliberate on the appropriateness of this Question, two special meetings were held by the HKEAA Council on 18 and 21 May 2020. After considering the professional views on the examination and assessment of the History subject, the related curriculum aims and objectives as set out in the History Curriculum and Assessment Guide (Secondary 4 – 6), and the interests of the candidates, the HKEAA Council made a decision to invalidate Question 2(c). An internal review team was also set up within the HKEAA to examine, among others, the mechanism of setting and quality-assuring questions for the 2020 HKDSE History Examination and how far the process was in compliance with the established guidelines. The HKEAA Internal Review Report was submitted to the EDB on 23 June 2020.
A Task Force on Hong Kong Diploma of Secondary Education Examination:
Investigation of the Problematic History Question and the Mechanism of Examination Paper Setting (“Task Force”) was set up under the EDB in June 2020 to conduct an investigation based on the HKEAA’s internal review. In the process of the review, the Task Force (a) studied the HKEAA Internal Review Report;
(b) examined records/documents/files relevant to the development of the 2020 HKDSE History Examination question papers as made available to the Task Force by the HKEAA and the examination personnel2 concerned; (c) interviewed staff
1 EDB statement : www.info.gov.hk/gia/general/202005/14/P2020051400838.htm?fontSize=1
2 Examination personnel involved in the 2020 HKDSE History Examination question paper development
members of the HKEAA and the examination personnel involved in the question paper development and moderation process; and (d) studied the responses of follow- up questionnaires issued to the abovementioned personnel.
From the review by the Task Force, it was noted that the HKEAA had put in place certain quality assurance measures for all stages of the question paper development and had provided its staff members with a comprehensive set of internal rules and regulations. These measures serve to ensure that the Moderation Committee for each subject will draw on the collective responsibility from its members in the question paper development process, and safeguard against having the process dominated by one or a few individuals. The effective implementation of these quality assurance measures, however, requires all personnel concerned to carry out their designated duties appropriately in strict compliance with relevant rules and regulations. Yet, the Task Force observed that: (a) among the causes of the incident, an HKEAA officer responsible for the question paper development process did not comply with some of the HKEAA rules and regulations, and did not adhere to the quality assurance measures in the development of the 2020 HKDSE History Examination question papers; and (b) there were inadequacies in the existing quality assurance measures that might have hampered the monitoring of the question paper development process.
The Task Force’s major findings are summarised as follows:
1. Pre-setting
According to the HKEAA rules and regulations for HKDSE Examination question paper development, all pre-setting meetings should be chaired by the Chief Examiner through secure means in secure venues, with the discussion confined to providing the Setter with directions on setting the questions whereby all source materials and questions drafted by the Setter should be provided at the moderation stage for the Moderation Committee’s information or discussion.
For the 2020 HKDSE History Examination, however, the HKEAA officer concerned neither informed nor involved the Chief Examiner in the pre-setting meetings, but conducted the meetings with only the Setter. Some of the discussions on the sets of source materials and drafts of Question 2(c) were conducted at inappropriate locations including public places or through non- secure means, and no records of these meetings or discussions were kept for the
process included the Moderation Committee comprising one Chief Examiner, one Setter, five Moderators (these seven members are outside subject experts appointed by the HKEAA), and one HKEAA Manager – Assessment Development (the only HKEAA officer). After the moderation process, the finalised question papers were checked by three Proofreaders and two Assessors to ensure that the question papers were free from errors.
Moderation Committee’s information. The scope of discussion in the pre- setting meetings also went beyond the directions for question-setting but delved into concrete discussion and screening of the sets of source materials with drafts of Question 2(c). In addition, the unused source materials and draft questions were not provided to the Moderation Committee for information or discussion.
2. Moderation
According to the HKEAA rules and regulations for HKDSE Examination question paper development, all moderation meetings should be chaired by the Chief Examiner and attended by Moderators, who together should take collective responsibility for the moderation process by contributing their views and discussing the questions. Relevant HKEAA internal guidelines should be provided to Moderation Committee members to draw their attention to sensitivity issues before or during the moderation process. The marking scheme should also be prepared in conjunction with the draft questions to act as a yardstick of the assessment foci of the questions in the moderation process.
For the 2020 HKDSE History Examination, four of the moderation meetings were conducted only between the HKEAA officer concerned and the Setter; and one specific Moderator did not attend any moderation meeting at all. These imply that not all Moderating Committee members have actively participated in the whole question paper development process.
Regarding the provision of relevant documents during the moderation process, the HKEAA officer concerned did not provide Moderation Committee members with the HKEAA guidelines on handling sensitive issues, nor was reference made to these guidelines during the entire moderation process. Furthermore, the marking scheme was not provided in conjunction with the draft questions at the beginning but at a late stage of the moderation process, and did not address the core requirement of Question 2(c), i.e. no marking criteria were provided for assessing candidates’ holistic judgement of the quoted statement on whether
“Japan did more good than harm to China”, rendering it an incomplete marking scheme. The Task Force is of the view that the guidelines could have alerted members to deliberate from the angles of sensitivity, and a more detailed marking scheme with specific marking criteria on candidates’ holistic judgement of the quoted statement could have alerted Moderation Committee members to the values implied in Question 2(c) and whether these were in alignment with the curriculum aims and objectives stipulated in the History Curriculum and Assessment Guide (Secondary 4 – 6).
The non-compliance with the HKEAA rules and regulations by the HKEAA officer concerned not only rendered it difficult for the HKEAA to draw on the
collective responsibility from all members of the History Moderation Committee but also greatly weakened the intended functions of the HKEAA rules and regulations.
3. Checking and Proofreading
The Task Force found that comments on Question 2(c), provided by a Moderator and a Proofreader respectively during the checking and proofreading stages, were not brought to the attention of the Moderation Committee or Chief Examiner for information or consideration as required by the HKEAA rules and regulations. Taking collective responsibility for the development of the question paper, the Moderation Committee should have been provided with these comments and feedback for information and consideration before endorsement of the final version of the question paper. However, it was the HKEAA officer concerned who made the decision to disregard these late stage comments which could be critical.
4. Other Processes Related to Question Paper Development
There is room for improvement in the nomination mechanism of Moderation Committee members in terms of the nomination and selection processes. For example, nomination invitations may not have effectively reached the potential candidates in the school and tertiary sectors.
In light of these findings, the Task Force makes the following recommendations to the HKEAA to enhance the monitoring of compliance with quality assurance measures through strengthening the involvement of senior staff members of the HKEAA, providing all Moderation Committee members with information on all meetings (including pre-setting meetings and moderation meetings) as well as all feedback received at the different stages of the question paper development process to prevent similar incidents, thereby ensuring the sustained quality of the HKDSE Examination and examination questions. The recommendations are to:
1. strengthen the involvement of senior staff members of the HKEAA in monitoring the quality assurance measures through introducing a new reporting channel for all Moderation Committee members to relay their feedback regarding the entire question paper development process to one or more designated senior staff members of the HKEAA, who in turn should review and report any unresolved divergent views / disputes documented in the records of meetings of the Moderation Committee to a designated committee under the HKEAA Council, which comprises senior staff members of the HKEAA and experts nominated by the EDB, for timely handling and follow up;
2. enhance the guidelines in the HKEAA internal manual for question paper development work governing the conduct of all meetings in the question paper development process (including pre-setting meetings and moderation meetings), such that all Moderation Committee members would be informed of every meeting to safeguard the principle of collective responsibility for the question paper development process;
3. ensure proper documentation and reporting of meetings throughout the entire question paper development process, including the compilation of summaries of moderation meetings for record-keeping purposes and the logging of crucial issues discussed and decisions made, to be endorsed by the Moderation Committee;
4. ensure that the feedback gathered in the entire question paper development process is appropriately provided to the Moderation Committee for attention and consideration in a concluding meeting, in which the final version of the question papers should be provided for collective endorsement and signing off;
5. enhance the nomination process of Moderation Committee members and expand the pool of potential candidates from the school and tertiary sectors, require concrete evidence of qualifications and suitability of the nominated candidates with clear justifications for selection and rejection, as well as appoint curriculum officers from the EDB as ex-officio members;
6. strengthen the professional development for the staff members of the HKEAA on the latest knowledge and technology in setting high quality questions and examinations, and the pre-service briefing for Moderation Committee members and other question paper development personnel to familiarise them with their roles and functions, as well as the procedures of each step of the question paper development process so that all personnel concerned would be able to offer professional views, especially on the alignment between curriculum and assessment;
7. strike a balance between confidentiality and transparency of the question paper development process by strengthening communication between the Moderation Committee and relevant committees under the Curriculum Development Council and the HKEAA, such as the Subject Committee, to ensure better alignment between curriculum and assessment; and
8. examine the relevance of the findings and recommendations and consider whether they are applicable to other HKDSE Category A subjects3 , and collaborate with the EDB to formulate and implement relevant improvement measures promptly and effectively.
November 2020
3 HKDSE Category A subjects refer to the senior secondary subjects recommended by the Curriculum Development Council (CDC) for implementation in local secondary schools. A list of the subjects can be found in the Secondary Education Curriculum Guide (CDC, 2017), Booklet 2, pages 29 to 31. Hyperlink:
www.edb.gov.hk/attachment/en/curriculum-
development/renewal/Guides/SECG%20booklet%202_en_20180831.pdf.
