High-stakes OSCE International Workshop高標準OSCE國際工作坊手冊

(1)

Preface

As physicians’ clinical skills cannot be measured solely by written

examination, a National objective structured clinical examination (OSCE)

has been considered necessary as a part of medical licensure examination.

By the end of 2008, about 20 medical centers/ hospitals in Taiwan

announced that OSCE has been a regular clinical examination for their

trainees. However, there is little consensus about how to implement a

high-stake, large scale OSCE.

The Medical Council of Canada (MCC) has administered an OSCE for

the license to practice medicine since 1992. The high stake, large scale

OSCE is to test physicians’ skills of history taking, physical examination,

and communication. The examination results with psychometric evidence

indicate that a full-scale national administration of an OSCE model for

licensure is feasible in Canada.

Professor, Sydney Marla Smee is currently the Manager of MCCQE Part

II, Evaluation Bureau Medical Council of Canada (1990-Present). She is

an internationally recognized expert in implementing a high stake OSCE.

The workshop in Taiwan lead by Dr. Smee is to facilitate the

establishment of a Taiwanese model of high stake OSCE.

(2)

 Agenda --- 3

 Speaker’s Curriculum Vitae --- 5

 Instructor Assistant’s Curriculum Vitae --- 10

 Moderator’s Curriculum Vitae --- 12

 PowerPoint (Day 1) --- 13

 PowerPoint (Day 2) --- 32

 Related Articles

 The Use of Standardized Patient Assessments for

Certification and Licensure Decisions ---

 The Comparison of Physician Examiners and

Trained Assessors in a High-Stakes OSCE Setting ----



Physician Scores on a National Clinical Skills

Examination as Predictors of Complaints to Medical

Regulatory Authorities

--- Participants ---

37

45

49

59

(3)

Day1

Agenda

High- Stakes OSCE (I): Administrative Issues and Related Topics

Date: January 7, 2010 (Thursday)

Place:

臺北醫學大學醫學綜合大樓前棟 4 樓-誠樸廳

台北市信義區 110 吳興街 250 號

Instructor:

Dr. Sydney M. Smee

Instructor Assistant: Dr. Charity T.C. Tsai

Time

Activity

Moderator

08:30-08:45

Registration

08:45-09:00

Opening Remarks

Minister Chaur-Shin Yung 楊朝祥部長

President Wen-Ta Chiu 邱文達校長

Director Chong-Liang Shi 石崇良處長

Prof. Chi-Wan

Lai

賴其萬教授

09:00-09:30

Presentation: Update on Plans for a Taiwan

OSCE

09:30-10:30

Presentation: High Stakes Multi-site OSCE:

Key Processes and Design Issues

10:30-10:45

Break

10:45-12:00

Group Activity: Five Groups- Create an

OSCE Design

12:00-13:00

LUNCH

13:00-14:30

Group Activity: Group Reports & Discussion

14:30-14:45

Break

14:45-15:15

Group Activity: Large Group Exercise-

Examination Day Scheduling

15:15-16:15

Group Activity: Five Groups- Required

Resources - Human and Other

16:15-17:00

Group Activity: Group Reports

(4)

Day2

Agenda

High- Stakes OSCE (I): Administrative Issues and Related Topics

Date: January 8, 2010 (Friday)

Place:

臺北醫學大學醫學綜合大樓前棟 4 樓-誠樸廳

台北市信義區 110 吳興街 250 號

Instructor: Dr. Sydney M. Smee

Instructor Assistant: Dr. Charity T.C. Tsai

Time

Activity

08:30-09:00

Check-in: Questions/Answers

09:00-10:30

Presentation & Exercise: Costing the OSCE

10:30-10:45

Break

10:45-11:15

Presentation: Training Issues- Examiners, SPs and Staff

11:15-12:00

Presentation: Production Timeline-Critical Milestones

12:00-13:00

LUNCH

13:00-14:30

Presentation: Hot Topics: Examiner Qualifications, Incidents,

Re-scores and Appeals

14:30-14:45

Break

14:45-17:00

Wrap-Up - Next Steps

17:00

Closing Remarks 閉幕致詞

(5)

Speaker’s Curriculum Vitae

D

r

.

S

y

d

n

e

y

M

.

S

m

e

Higher Education

2007

Doctor of Philosophy

University of Ottawa

Major: Education

Minor: Measurement and Evaluation

1994

Master of Education

Ontario Institute of Studies in Education

University of Toronto

Major: Adult Education

1982

Bachelor of Arts

McMaster University

Major: Political Science

Professional Positions

1990-Present

Manager, MCCQE Part II

Evaluation Bureau, Medical Council of Canada

1987 - 1990

Coordinator, Volunteer Services

Casey House Hospice, Toronto

1986-1988

Consultant, Standardized Patient Program Development

University of Massachusetts, University of Toronto, McMaster

University

1984 - 1985

Coordinator, Patient Instructor Program

(6)

Editorial Activities

Ad Hoc Reviewer:

Advances in Health Sciences: 2001, 2007

Medical Education: 2001 – 2002, 2005, 2009.

Publications

Boulet, J.R., Smee, S.M., Dillon, G.F., and Gimpel, J.R. (2009).The use of

standardized patient assessments for certification and licensure

decisions. Simulation in Health Care 4:1 Spring.

Smee, SM. (2008). High Stakes OSCE scoring: Station-specific rating scales versus

checklists. Paper presented at the 13

th

Ottawa Conference on Medical Education:

Melbourne, Australia.

Smee, SM. (2008). Impact of judgmental weights for OSCE checklist items on station

pass marks. Paper presented at the 13

th

Ottawa Conference on Medical

Education: Melbourne, Australia.

Boursicot, KA, Smee, SM, & Paterson, J. (2008). Ten years of monitoring test

security in graduation level OSCEs. Paper presented at the 13

th

Ottawa

Conference on Medical Education: Melbourne, Australia.

Wood, TJ & Smee, S. (2008). Does editing an OSCE station after an examination

improve its performance on subsequent examinations? Paper presented at the

2008 annual meeting of the Association of Medical Educators of Europe

(AMEE): Prague, Czech Republic.

Wood, TJ, Smee, SM, Bartman, I, & Blackmore, DE. (2008) Do two different

processes for limiting false positive errors add to the quality of the pass/fail

decision on a high stakes examination? Paper presented to the annual meeting

on Research in Medical Education (RIME): San Antonio, USA.

Tamblyn, R, Abrahamowicz, M, Dauphinee, D, Wenghofer, E, Jacques, A, Klass, D,

Smee, S, Blackmore, D, Winslade, N, Girard, N, Du Berger, R, Bartman, I,

Buckeridge, D, & Hanley, J. (2007). Physician Scores on a National Clinical

Skills Examination as Predictors of Complaints to Medical Regulatory

Authorities. Journal of the American Medical Association, 298, 993-1001.

(7)

Humphrey-Murto, S, Smee, SM, Touchie, C, Wood, TJ, & Blackmore, DE. (2005). A

comparison of physician examiners and trained assessors in a high-stakes OSCE

setting. Academic Medicine, 80, s59-s62.

Boursicot, KAM & Smee, SM. (2004). Setting standards for a finals Objective

Structured Clinical Examination (OSCE): Comparing the borderline group

method with an Angoff approach. Paper presented at the 10

th

Ottawa

International Conference on Medical Education, Ottawa, Canada.

Smee, SM. (2003). ABC of learning and teaching in medicine: Skill-based assessment.

British Medical Journal, 326, 703-706.

Smee, SM, Dauphinee, WD, Blackmore, DE, Rothman, AI, Reznick, R, & Des

Marchais, J. (2003). A sequenced OSCE for licensure: Administrative issues,

results and myths. Advances in Health Sciences Education: Theory and Practice,

8, 223-236.

Birtwhistle, R, Blackmore, DE, Smee, SM, & Wood, T. (2002). Does specialty play a

role when physicians are used as examiners in a nationally administered OSCE?

Paper presented at the 9th Ottawa International Conference on Medical

Education Capetown, South Africa.

Blackmore, DE & Smee, SM. (2002). Weighted vs. unweighted OSCE checklists. In

Paper presented at the 9th Ottawa International Conference on Medical

Education Capetown, South Africa.

Smee, SM & Blackmore, DE. (2002). Setting standards for an objective structured

clinical examination: The borderline group method gains ground on Angoff.

Medical Education, 35, 1009-1010.

Smee, SM & Blackmore, DE. (2002). Authors' reply: Setting standards for an

objective structured clinical examination: The borderline group method gains

ground on Angoff. Medical Education, 36, 388-389.

Smee, SM & Blackmore, DE. (2001). Commentary - Setting standards for an

objective structured clinical examination: The borderline group method gains

ground on Angoff. Medical Education, 35, 1009-1010.

Dauphinee, WD, Boulais, AP, Smee, SM, Rothman, AI, Reznick, R, & Blackmore,

DE. (2000). Examination results of the Licentiate of the Medical Council of

(8)

Proceedings of the Eighth International Ottawa Conference - Evolving

Assessment: Protecting the Human Dimension (pp. 92-98). Philadelphia:

National Board of Medical Examiners.

Dauphinee, WD, Blackmore, DE, Smee, SM, Rothman, AI, Des Marchais, J, &

Reznick, RK. (2000). Adaptive testing: A report on the results and myths arising

from the use of a sequenced OSCE for national licensure. In D. E. Melnick (Ed.),

Proceedings of the Eighth Ottawa International Conference - Evolving

Assessment: Protecting the Human Dimension (pp. 241-246). Philadelphia:

National Board of Medical Examiners.

Smee, SM, Blackmore, DE, Rothman, AI, Reznick, RK, & Dauphinee, WD. (2000).

Pioneering a sequenced OSCE for the Medical Council of Canada: An

administrative overview. In D. E. Melnick (Ed.), Proceedings of the Eighth

International Ottawa Conference - Evolving Assessment: Protecting the Human

Dimension (pp. 234-240). Philadelphia: National Board of Medical Examiners.

Poldre, P, Smee, SM, Reznick, RK, Blackmore, DE, Birtwhistle, R, Blouin, D,

Chalmers, A, Galway, B, Hodges, B, MacFadyen, J, & Spady, D. (1999). The

experience of thousands: The post-examination OSCE station review process of

the Medical Council of Canada. In D. E. Melnick (Ed.), Evolving assessment:

Protecting the human dimension (CD-ROM) Philadelphia: National Board of

Medical Examiners.

Dauphinee, WD, Blackmore, DE, Smee, SM, Rothman, A. I, & Reznick, RK. (1997).

Using the judgments of physician examiners in setting the standards for a

national multi-center high stakes OSCE. Advances in Health Sciences Education:

Theory and Practice, 2, 201-211.

Dauphinee, WD, Blackmore, DE, Smee, SM, Rothman, AI, & Reznick, RK. (1997).

Optimizing the input of physician examiners in setting standards for a large

scale OSCE: Experience with Part II of the Qualifying Examination of the

Medical Council of Canada. In A. J. J. A. Scherpbier, C. P. M. van der Vleuten,

J. J. Rethans, & A. F. W. van der Steeg (Eds.), Advances in Medical Education:

Proceedings of the Seventh Ottawa International Conference on Medical

Education (pp. 656-658). Dordrecht: Kluwer Academic Publishers.

(9)

Medical Education (pp. 458-461). Dordrecht: Kluwer Academic Publishers.

Smee, SM. & Sumawong, V. (1997). Advancing the use of standardized patients: A

workshop for the consortium of Thai medical schools. In AJJA. Scherpbier,

CPM van der Vleuten, JJ Rethans, & AFW van der Steeg (Eds.), Advances in

Medical Education (pp. 714-716). Dordrecht: Kluwer Academic Publishers.

Smee, SM. & Blackmore, DE. (1997). Preparing physician examiners for a high

stakes, multi-site OSCE. In AJJA Scherpbier, CPM. van der Vleuten, JJ Rethans,

& AFW van der Steeg (Eds.), Advances in Medical Education (pp. 462-469).

Dordrecht: Kluwer Academic Publishers.

Reznick, RK, Blackmore, DE, Dauphinee, WD, Rothman, AI, & Smee, SM. (1996).

Large-scale high-stakes testing with an OSCE: Report from the Medical Council

of Canada. Academic Medicine, S71, 19-21.

Smee, SM. (1994). Medical Education Clinic: Using SPs for teaching and evaluation.

Adult Education Quarterly, 6, 9-10.

Reznick, RK, Blackmore, DE, Cohen, R, Baumber, JS, Rothman, AI, Smee, SM,

Chalmers, A, Poldre, P, Birtwhistle, R, Walsh, P, Spady, D, & Bérard, MJ.

(1993). An objective structured clinical examination for the licentiate of the

Medical Council of Canada: From research to reality. Academic Medicine, S68,

4-6.

Reznick, RK, Smee, SM, Baumber, JS, Cohen, R, Rothman, AI, Blackmore, DE, &

Bérard, M. (1993). Guidelines for estimating the real cost of an objective

structured clinical examination. Academic Medicine, 68, 513-517.

Reznick, RK, Smee, SM, Rothman, AI, Chalmers, A, Swanson, DB, Dufresne, L,

Lacombe, G, Baumber, J, Poldre, P, Levasseur, L, Cohen, R, Mendez, J, &

Bérard, M. (1992). An objective structured clinical examination for the licentiate:

Report of the pilot project of the Medical Council of Canada. Academic

Medicine, 48, 487-494.

Stillman, PL, Swanson, DB, Smee, SM, Stillman, AE, & Ebert, TH. (1986).

Assessing the clinical skills of residents with standardized patients. Annals of

Internal Medicine, 105, 762-771.

(10)

China Medical University

University of Calgary, MSc in Medical Education

University of Calgary, PhD in Medical Education

Present Academic and Administrative approintment:

Director, Department of Pediatrics, Taipei

Medical University WanFang Hospital

Vice Director, Faculty of Medicine, Taipei

Medical University

Vice Director, Education and Research, Taipei

Medical University WanFang Hospital

Associate professor, Taipei Medical University

Instructor Assistant’s Curriculum Vitae

Charity TC Tsai, MD, PhD

Publication (in recent 5 years)

1. Tsuen-Chiuan Tsai, Using children as standardized patients for assessing clinical

competence in pediatrics. Arch Dis Child 89 (12): 1117-1120, Dec. 2004

2. Tsuen-Chiuan Tsai, M.D., Peter H. Harasym, Ph.D., Cheri Nijssen-Jordan, and

Greg Powell, Learning gains derived from a high fidelity simulation in

emergency department J Formos Med Assoc 105 (1):94-98, 2006

3. JD Tsai, FU Huang, CC Lin, TC Tsai, HC Lee, and JC Sheu. Intermittent

hydronephrosis secondary to ureteropelvic junction obstruction: clinical and

imaging features. Pediatrics. 2006 Jan;117(1):139-46.

4. Tsai TC, Harasym PH. Challenges of pediatric residency education in Taiwan.

Acta Pediatrica Sinica. 47(1):3-6, 2006

5. Tsuen-Chiuan Tsai . Psychosocial effects on caregivers for children in Taiwan on

chronic peritoneal dialysis. Kidney Int. 2006 Dec;70(11):1983-7

6. Tsuen-Chiuan Tsai . University of Washington 家醫科及臨床技能中心參訪. J

Med Education. Jan. 10(1): 86-88, 2006

7. Tsuen-Chiuan Tsai,

1

Pei-Jung Chang,

2

Shin-Yuan Fang

,

2

Chyi-Her Lin

3.

A

Mannequin-based Simulation on Teaching Emergent Crisis Care. J Med

Education, 10(2): 115-125, 2006

8. Sheu JC, Koh CC, Chang PY, Wang NL, Tsai JD, Tsai TC. Ureteropelvic

junction obstruction in children: 10 years' experience in one institution.

Pediatr Surg Int. 2006 Jun;22(6):519-23

9. Tsai YC, Tsai TC, Tsaf JD, Huang FY, Lin CC, Sheu JC. Clinical analysis of

chronic peritoneal dialysis related peritonitis in children. Pediatr Neonatol. 2006

(11)

Yeh, Jing-Jane Tsai, Yin-Fan Chang. Analysis of OSCE results: experience in

National Cheng Kung University Medical College. J Med Education 10

(4):313-23, 2006

11. Huang DTN, Tsai TC, Huang FY, Tsai JW, Chiu NC, Lin CC. Clinical

differentiation of acute pyelonephritis from lower urinary tract infection in

children. Journal of microbiology, immunology and infection. J Microbiol

Immunol Infect. 2007;40:513-517

12. Tsai TC. Resistance to educational change: management and communication.

Pediatr Neonatol.48:3-6, 2007

13. Tsai TC, Lin CH, Harasym PH, Violato C. Students' perception on medical

professionalism: the psychometric perspective. Med Teach. 2007

Mar;29(2-3):128-34.

14. Peter H. Harasym, Tsuen-Chiuan Tsai, and Payman Hemmati. Current trends in

developing medical students’ critical thinking abilities. Kaohsiung J Med Sci

July 2008. 24 (7) 341-354

15. Tsuen-Chiuan Tsai. The Use of Medical Cognition in Medical Curriculum

Reform in Taiwan. Pediatr Neonatol 2008;49(3):53−57

16. 蔡淳娟、邱文達、王先震、連吉時、粟發滿、郭雲鼎、徐明義.The use of portfolio

in internship clinical education. J of Med Edu 12(1): 8-19.2008.

17. Lee MD, Lin CC, Huang FY, Tsai TC, Huang CT, Tsai JD. Screening young

children with a first febrile urinary tract infection for high-grade vesicoureteral

reflux with renal ultrasound scanning and technetium-99m-labeled

dimercaptosuccinic acid scanning. J Pediatr. 2009 Jun;154(6):797-802.

18. 顏如娟,蔡淳娟,郭耿南,張殷瑞,陳泰宏.台灣醫師人力需求之探討.投稿台灣公

共衛生雜誌 2009/07

19. Tsuen-Chiuan Tsai, Peter H. Harasym, Sylvain Coderre, Kevin McLaughlin, &

Tyrone Donnon. Assessing ethical problem solving by reasoning rather than

decision making. Med Edu 2009: 43: 1188–1197

20. Ju-Chuan Yen,Tsuen-Chiuan Tsai, Min-Huei Hsu, Kung-Jiang Chang, Du-Jian

Tsai, Wei-Hua Lee. The attitudes toward disclosure of medical errors: the

perspectives of Taiwanese with different occupational backgrounds. Submit to

The American Journal of Bioethics (UAJB-2009-0218) 2009/04/14

21. Tsuen-Chiuan Tsai, Peter H. Harasym. An Ethical Reasoning Model:

Contributions to Medical Education. (Submitted to Med Educ in 2009/09)

22. 蔡淳娟,林其和,劉克明. 台灣各界對醫學系學制變革可行性的看法. 投稿醫

(12)

Moderator’s Curriculum Vitae

Professor Chao Hsiang Yang (楊朝祥部長)

Present Positions

Minister, Ministry of Examinations

Professor Wen-Ta Chiu (邱文達校長)

Present Positions

Professor and President, Taipei Medical University

Professor Chong-Liang Shi (石崇良處長)

Present Positions

Director of Medical Affairs, Department of Health

Professor Chi-Wan Lai (賴其萬教授)

Present Positions

Executive Secretary, Medical Education Committee, Ministry of Education

CEO, Taiwan Medical Accreditation Council

(13)

Date: 07 January, 2010 High- Stakes OSCE (I): Administrative Issues and Related Topics

Sydney M Smee, Ph. D.

Manager, MCCQE Part II

Evaluation Bureau, Medical Council of Canada

Presented at Taipei Medical University, WanFang Hospital by the Taiwan Association of Medical Education, January 2010

1

My Background

• Standardized Patient at McMaster University with Dr.

Howard Barrows and Gayle Gliva-McConvey

• Coordinator for Patient Instructor Program at University

of Massachusetts for Dr. Paula Stillman

• Did other things….

• Standardized Patient Training Coordinator for the pilot

project for the Medical Council of Canada’s new

Qualifying Examination Part II in 1992

• Now – Manager for the MCCQE Part II

• Keep this in mind - I am strongly influenced by long time

commitment to standardized patients and OSCEs

2

(14)

3

What is the Part II?

• (Part I assesses knowledge and clinical

decision-making at end of medical

school – computer testing / MCQs)

• Part II is an OSCE that assesses clinical

skills after 12 months of post-graduate

clinical training

• Multi-site, administered twice per day

• Timed circuit of 12 stations

• Patient-based

• Physician-scored

• Prerequisite for licensure in Canada

since 1993

4

Why do we have the Part II?

• Requested by the Medical Licensing Authorities

because they were facing:

▫ Increasing number of complaints, often based on a

physician’s communication skills.

▫ Need to be publicly accountable; e.g., reports that not

all trainees were being assessed in a clinical setting.

▫ Obligation to audit the training of all medical

graduates seeking licensure in Canada.

(15)

5

Content of Part II

• Multidisciplinary, patient-based cases

▫ Some have a written component based directly on the

patient problem

• Common or acute presenting problems

▫ Some problems include legal and ethical issues

• Assesses skills:

▫ History taking

▫ Physical examination skills

▫ Counseling / patient education skills

▫ Patient management ability

Part II - Fall 2009

• 12 case OSCE with 2 pilot cases

▫ 5+5 minute couplets (patient + written components)

▫ 10 minute case (sometimes there was an oral component)

• OSCE was run twice per day

• 16 university-based sites (at teaching hospitals)

• 2,644 test takers assessed in two days

• Most common site model ran two tracks and

administered the OSCE twice in one day – assessed

128 test takers

-▫ 44 clinic rooms

- 50 to 60 standardized patients

▫ 44 physician examiners

- 16 to 20 staff people

(16)

Goals for Workshop

• Provide overview of OSCE processes

• Set specifications for an OSCE

• Identify critical tasks and timelines

• Estimate costs

• Review training needs

• Discuss “hot topics”

• Specify next steps in the process

7

Six OSCE Processes for Multi-site Model

• Initiation

▫ Governance

▫ Terms of Reference ▫ Design and scope

• Content Development

▫ Create

▫ Edit and Format ▫ Approve

• Site Development

▫ Determine specifications ▫ Recruit key staff

▫ Provide training

• Production (pre-OSCE)

▫ Establish supply needs ▫ Create and produce materials ▫ Ship

• Processing (post-OSCE)

▫ Receive ▫ Enter scores ▫ Analyze data

• Registration and Reporting

▫ Eligibility criteria

▫ Document requirements ▫ Results and appeals

(17)

Initiation

Part II Terms of Reference:

• Collaborative agreement

• Schools provide office space, clinic

space (for OSCE days), equipment, and name the Chief Examiner

• MCC provides funds, OSCE

materials, training and supervision

• Design & Scope

• Part II has changed over time

• Two days / 20 stations / 5 sites

• One day / sequenced / 12 sites

• One day / 14 stations / 15 sites

• One day / 12 stations / 16 sites

• Two days/ 12 stations / 17 sites

• This workshop:

• Design and define an OSCE…

9

MCC Part II Governance Model

Content Development

• MCC content is created centrally

▫ Multidisciplinary committee with members balanced across

• Medical specialty

• Language (French – English)

• Geography

• Gender

▫ Members are active clinicians with faculty appointments

▫ All have a keen interest in medical education

▫ Faculty from the medical schools work with the committee

to create and review cases

• MCC staff provide support:

▫ Plan meetings

▫ Psychometric and logistical advice

▫ Edit, format and produce OSCE materials

• More on case writing at the next workshop….

(18)

Site Development

(It’s all about the people.)

• Determine site specifications

▫ Clinic Room requirements ▫ Large rooms for orientation ▫ Office space, furniture and computers, telephones, etc. ▫ Exam day staffing needs ▫ Parking and catering

• Recruit key site staff

▫ Site Manager ▫ Chief Examiner ▫ SP Trainer

• Provide training and

supervision

▫ Job descriptions ▫ Manuals ▫ Central meetings ▫ Telephone support ▫ Site visits

▫ Exam day supervision ▫ SP training materials

 Case materials  Videos

 Internet resources  Diagrams

 Training for Trainers

11

Production

• Establish supply needs

▫ Signage, props, registration materials, incident reporting

▫ Exam sheets

▫ Orientation materials for site staff, examiners, and test takers

 Print, video, other? (e.g., booklets and lab values)

▫ Identify security challenges and solutions

• Create and produce exam materials

▫ What software?

▫ How are you collecting score data?

▫ What about other data? Feedback from examiners?

• Ship the materials to the site(s)

▫ Shipment deadline is a major milestone

(19)

Processing

• Receive (after the examination)

▫ Verify return of materials is complete

• Enter scores

▫ What process depends on software used

▫ Ensure format of data allows for quality assurance and for

score reporting

• Analyze data

▫ Item analysis of cases

▫ Reliability of OSCE

▫ Review of examiners and exam day reports

▫ Review of results

13

Registration and Reporting

• Eligibility criteria

▫ What are the criteria and how will this be communicated to test takers?

• Document requirements

▫ How will you verify eligibility?

• Results

▫ Who has access to the results?

▫ How will they be reported?

• Appeals

▫ Define conflict of interest and other reasons for appeals

▫ Determine process for appeals

 Consider different levels; e.g., rescores and feedback calls

(20)

....Making many decisions

Presented at Taipei Medical University, WanFang Hospital by the Taiwan

Association of Medical Education, January 2010

15

16 Define performance standard Which clinical skills? Who assesses? What clinical presentations? Length of stations? How many stations?

(21)

Define a standard –

MCCQE Part II:

Acceptably competent to enter independent

medical practice in Canada

17

Blueprint criteria = Validity

Body Systems Clinical Tasks

History Taking Physical Exam Procedural Patient Education ?? Cardiac Respiratory X Gastrointestinal Reproductive X X ??

18

Define the criteria for selecting cases for

each test form

(22)

MCCQE Part II Blueprint Criteria

19

DOMAIN Counseling/Education 2

History 4 (or 5)

Management / Acute Care 2

Physical exam 4 (or 3)

Combined History / Physical

(2 – if possible)

Patient Interaction Rating scales are integrated within most stations

DISCIPLINE Medicine 3

Ob/Gyn 2

Pediatrics 2

Psychiatry 2

Surgery 3

NOTE Review problems to ensure balance across body systems CLEO content must comprise a minimum of 10% of overall content.

CLEO is integrated into cases with oral questions and patient interaction rating scale items

HISTORY of USE No more than 3 stations with high exposure (as a guide) New cases in each blueprint (no specific numbers assigned)

Item Total Score

Correlation Minimum: 0.20 (AS A GUIDE) GENDER as balanced as possible

AGE GROUPS Elderly / Adult / Adolescent / Young children One or more cases representing each age group

Patient problems are common or critical; each case is linked to the MCC Objectives

Clinical Skills and Presentations

• Some things are more “OSCE-able” than others

• Some cases cost more to administer

▫ Multi-patient cases and manikins may be a challenge

• Large-scale, multi-site OSCE have some limitations

▫ Must have the same resources at all sites

• Focus on skills and content best assessed by OSCE

▫ Content that can be assessed more cost effectively in

other formats (like multiple choice examinations)

should not be included in an OSCE

(23)

Long versus short cases / Reliability

• Long cases (>12 minute cases)

▫ Assess complete encounters (e.g., Hx + Physical)

▫ Fewer cases and examiners per test taker

▫ Long testing time required to achieve reliable scores

▫ Can assess performance on more complex problems

• Short cases (5 -12 minute cases)

▫ Assess samples of performance



Suggests assessment of clinical judgment

▫ More cases and more examiners per test taker

increases reliability

• Mix of long and short cases

▫ More complex OSCE design (but still do-able)

21

What is your preference for an OSCE?

1. Fewer longer cases

2. More shorter cases

3. Mix of short and long cases

(24)

What do you think the minimum

number of cases is for a valid OSCE?

1. Six

2. Eight

3. Ten

4. Twelve

5. Fourteen

6. Sixteen

7. Eighteen

23

Which of the following do you think

should be assessed in an OSCE?

1. History taking

2. Physical examination skills

3. Patient education ability

4. Managing acute problems

5. Managing non-acute problems

6. Responses to ethical issues

7. Interpretation of investigations

8. Procedural skills

9. Written tasks (e.g., admission orders)

10. Other

(25)

Group Task for After Break

• Five Groups – One Reporter for each group

• Start designing an OSCE for assessing ....

• Specify the clinical skills you want assessed

▫ Define the range of clinical presentations by

discipline and/or body system (or other criteria)

▫ Determine how many patient cases

▫ Specify the time limit for cases

• Group reports after lunch…

25

Group Reports

(26)

Exam Day Scheduling

• To cost an OSCE you need a design and an exam

day schedule (to start)

• Large portion of the costs are people

▫ Need to specify what is expected of the people

• Once you cost a design, you will likely revise it…

27

28

Activity Time Chief Examiner Manager SP Trainer Timers Hall Staff Group Leaders Caterers

Set-up site 6:00

Register SPs /Staff 6:15 Register / Orient Examiners 6:30 Register / Orient AM Test takers 6:45 Move people to stations 7:00

1 Start OSCE 7:15

Break in OSCE? 7:30

2 End OSCE 7:45

Lunch Break? 8:00

Register / Orient PM Test takers 8:15

3 Start OSCE 8:30 Break in OSCE? 8:45 4 End OSCE 9:00 Clean-up Site 9:15 9:30 9:45 10:00 10:15 10:30 10:45 11:00 11:15 11:30 11:45 12:00 12:15 12:30 12:45 13:00 13:15 13:30 13:45 14:00 14:15 14:30 14:45 15:00 15:15

(27)

Tasks

• Central Tasks

▫ Communicate with stakeholders: Test takers, medical faculties, ?? ▫ Establish registration

processes

▫ Create and produce OSCE materials

▫ Develop protocols for training everyone ▫ Visit sites

▫ Run central meetings ▫ Process results ▫ Report results

• Site Tasks

▫ Plan layout of OSCE stations at the site

▫ Book clinics and orientation rooms

▫ Recruit SPs and examiners ▫ Train SPs

▫ Recruit staff- train them too ▫ Plan out the OSCE day in

great detail ▫ Run the OSCE ▫ Ship everything back

29

Production Timeline - Milestones

10. Report Results 9. Complete QA on Scores 8.Exam

Day 7. Ship to site(s) mark sheets6. Print

5. Training meetings 4. Recruit staff at exam site 3. Letter of Agreement with site 2. Finalize case content 1. Finalize OSCE design

30

Work backwards……

Production processes overlap with site development

processes…..

(28)

2. Finalize case content 1. Finalize OSCE design

31

5. Training meetings 4. Recruit site personnel

3. Agreement with sites

3. Letter of Agreement signed

▫ Funds forwarded

▫ Chief Examiner appointed

4. Recruit site personnel

▫ Manager and SP Trainer

▫ Examiners and Standardized Patients ▫ Site staff for timing, registration etc.

5. Training meetings

▫ Centrally – site team(s) and core staff ▫ Not too early and not too late

▫ Balance input from sites with central control….

(29)

8. Exam Day 7. Ship to site

6. Print mark sheets

▫ Significant deadline – no more changes without pain ▫ Production takes careful, detailed planning

▫ Attend to security issues/ inventory tracking etc.

7. Ship to sites

▫ Central control is “gone” – it’s in the hands of the site team ▫ Increasing focus on support

8. Exam day

▫ Oversight

▫ Incident reporting

▫ What can go wrong, will go wrong – be prepared to learn

33

10. Report Results 9. Complete QA on Scores

9. Complete quality assurance on scores ▫ Verify the scores are accurately recorded ▫ Assess how the cases performed (item analysis)

▫ Assess how the examiners performed (any hawks or doves?) ▫ Assess how the OSCE performed (reliability)

▫ Take into account exam day incident reports

10. Report results ▫ Who gets what? ▫ Scores? Pass/Fail? ▫ Feedback?

(30)

Required Resources

Central Resources

• Staff Positions

▫ How many? ▫ What qualifications?

• Case Writers

▫ How many?

▫ How many meetings?

• Office(s)

▫ Existing or new?

• Equipment and software

Site Resources

• Staff positions

▫ How many? ▫ What qualifications?

• Office(s)

▫ New or existing?

• SP training room(s)

• Equipment and software

• Challenges?

▫ What else do you need to know?

35

Five Groups – Tasks to be assigned

• Group 1:

• Group 2:

• Group 3:

• Group 4:

• Group 5:

(31)

Group Reports

(32)

(33)

Date: 08 January, 2010 High- Stakes OSCE (I): Administrative Issues and Related Topics

38

A large group experiment with an Excel workbook

Training Issues

• Chief Examiners & Managers

▫ Critical roles in preparing for the OSCE at the site level ▫ At the beginning, these

people can help create

protocols and forms to ensure the OSCE is standardized ▫ Benefit of multi-site OSCE is

working with others, receiving training

▫ Cost of multi-site OSCE is loss of local autonomy

• Examiners

▫ Orient to the OSCE

▫ Training may be delivered locally by site staff OR centrally via web-based technology

▫ When and how to orient examiners is a compromise between the ideal and practical

▫ Local support and direction to site-specific issues is always needed

(34)

40

Training Issues

• Standardized Patients

▫ Longer cases require more training

▫ More complex cases require more training (e.g.,

psychiatric affect, multiple physical symptoms)

▫ Good training involves practicing with someone acting as a candidate and getting feedback

▫ Training should happen close to the OSCE

• Exam Day Staff

▫ Need to understand the OSCE and why its important

▫ Need to know their specific tasks (e.g. timing, collecting mark sheets, directing people)

▫ Need to know what to do when something goes wrong (and something always does) ▫ Training based on centrally

developed materials and given locally leads to fair testing

(35)

Hot Topics

• Examiner qualifications

• Incidents

• Re-scores and Appeals

42

Examiner Qualifications

• Formal MCC Criteria:

▫ Must have the Licentiate of the Medical Council of Canada

(LMCC).

▫ Should be 3 years post-LMCC.

▫ Should have at least 3 years in independent practice.

▫ Cannot be residents or fellows. Examiners must hold an

unrestricted license and currently be practicing medicine.

▫ Should have the ability and stamina for the task (e.g.,

hearing loss is a serious handicap).

▫ May be community physicians.

• Informally, Chief Examiners interview any examiners

who are not known

(36)

Incident Reports – What can go wrong?

• Conflicts of interest

• Examiner errors

• Candidate is inappropriate

• Timing errors

• Illness (SPs, test takers, examiners)

• Missing materials

• Missing SP

• Missing examiner

• Flooding toilet, fire alarms, parades,

strikes…

44

What can be done?

• With a pass/fail examination the incidents that

are most critical are those that impact test takers

who are borderline failures

▫ Is incident significant?

▫ Delete the station?

▫ Invalidate the examination?



MCC will not assume a pass standing

(37)

Re-scores and Appeals:

Risk Management

• Have policies and procedures in place for

answering post-OSCE questions and complaints

▫ Follow-up t0 selected letters and incident reports

▫ Providing feedback in exam reports or by request

▫ Rescores to verify results

▫ Appeal process for complaints



Process for low level issues, more formal for serious

issues

46

Sydney M Smee [email protected] www.mcc.ca

(38)

Review Article

The Use of Standardized Patient Assessments for Certification

and Licensure Decisions

John R. Boulet, PhD; Sydney M. Smee, PhD; Gerard F. Dillon, PhD; John R. Gimpel, DO

Although standardized patients have been employed for formative assessment for over 40 years, their use in high-stakes medical licensure examinations has been a relatively recent phenomenon. As part of the medical licensure process in the United States and Canada, the clinical skills of medical students, medical school graduates, and residents are evaluated in a simulated clinical environment. All of the evaluations attempt to provide the public with some assurance that the person who achieves a passing score has the knowledge and/or requisite skills to provide safe and effective medical services. Although the various standardized patient-based licensure examinations differ somewhat in terms of purpose, content, and scope, they share many common-alities. More important, given the extensive research that was conducted to support these testing initiatives, combined with their success in promoting educational activities and in identifying individuals with clinical skills deficiencies, they provide a framework for validating new simulation modalities and extending simulation-based assessment into other areas.

(Sim Healthcare 4:35– 42, 2009)

Key Words: Licensure, Certification, Simulation, Standardized patient, Simulated patient,

OSCE

T

here are many types of simulations that are currently being used to assess healthcare professionals.1– 4 _{In both Canada}

and the United States (US), many of these simulation modal-ities, including multiple choice questions, part-task trainers, and computer-based case simulations, have been used as part of the examination process used to certify and license physi-cians.1,5,6 _{These simulation-based examinations, which can}

vary somewhat in terms of purpose and focus, all attempt to provide the public with some assurance that the person who achieves a passing score has the knowledge and/or requisite skills to provide safe and effective medical services, either independently or under supervision. Here, as with any sim-ulation-based assessment, the structure, content, fidelity, and difficulty of the modeled exercises, combined with the scores, will determine what inferences one can make about the indi-vidual test taker.

From a simulation perspective, the use of standardized patients (SPs) for certification and licensure decisions has been a relatively recent phenomenon.7_{Historically, SP-based}

assessments were implemented as part of formative

evalua-tion activities.8 –10_{Individuals were trained to portray specific}

patient conditions, allowing medical students to practice their clinical skills and receive immediate feedback concern-ing strengths and weaknesses. In the 1980s, with an increased emphasis on evaluating what medical trainees could do, as opposed to what they knew, various organizations started research programs aimed at determining how assessments employing SPs could be structured to make valid skills-based proficiency decisions. Over the next two decades, the end result of these research activities was the implementation of a number of high-stakes assessments all aimed at measuring abilities in key clinical skills domains. Although these re-search efforts required extensive resources, they were suc-cessful in identifying the specific conditions and structures that are needed to produce defensible scores and decisions for multistation, performance-based, simulation activities.11–17

The introduction of SP-based certification and licensure examinations in medicine was a monumental achievement. Although other high-stakes simulation-based assessments have been developed and used in other professions, the logis-tical, economical, and psychometric challenges associated with national multistation clinical skills assessments were staggering.18,19_{Organizations that built these assessments all}

had to address concerns regarding test content (eg, types of scenarios to model), test administration models (eg, fixed versus temporary sites; number, timing and sequencing sta-tions), measurement rubrics (eg, holistic or analytic), eligi-bility requirements, scoring models (eg, compensatory or

From the Foundation for Advancement of International Medical Education and Research (J.R.B.), Philadelphia, PA; Medical Council of Canada (S.M.S.), Ottawa, ON, Canada; National Board of Medical Examiners (G.F.D.), Philadelphia, PA; and National Board of Osteopathic Medical Examiners (J.R.G.), Conshohocken, PA.

Reprints: John R. Boulet, PhD, Foundation for Advancement of International Medical Education and Research, 3624 Market Street, Philadelphia, PA 19104 (e-mail: [email protected]).

(39)

clinical skills as part of certification/licensure process,20_each

of these organizations was able to produce a high-quality simulation-based assessment that was appropriate for their particular needs. In doing so, many lessons were learned, the most important being that simulation-based summative as-sessment of clinical skills was viable, even with large examinee populations, differing testing purposes, and varying exami-nation administration protocols.

PURPOSE

The purpose of this article was to describe and contrast the Clinical Skills Assessment (CSA) programs that are employed in Canada and the US as part of the certification and licensure process for physicians. These assessments include the Medi-cal Council of Canada (MCC) Qualifying Examination Part II (MCCQE Part II),21_{the United States Medical Licensing}

Examination (USMLE) Step 2 Clinical Skills (USMLE Step 2 CS),22 _{and the National Board of Osteopathic Medical}

Examiners (NBOME) Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE).23 _{To better understand the}

USMLE Step 2 CS, a brief overview of the Educational Com-mission for Foreign Medical Graduates (ECFMG) CSA is also provided.24_{The CSA was used to assess the clinical skills of}

international medical graduates (IMGs) before the introduc-tion of USMLE Step 2 CS. Following this overview, a brief synthesis of the similarities and differences in the assessments and assessment programs is provided. With these distinc-tions in mind, and knowing the success and scope of the individual testing programs, it is possible to envision where summative simulation-based assessment activities could be enhanced, applied in other areas, and used for the evaluation of nonphysician healthcare professionals.

ASSESSMENT OF CLINICAL SKILLS

In general terms, clinical skills refer to information gath-ering and communication skills, applied during the patient encounter, that help to establish an accurate diagnosis and support high-quality treatment. Within the medical educa-tion and practice community, these skills have long been recognized as essential to patient care. Several organizations, including those responsible for the accreditation of under-graduate and under-graduate medical education (GME) programs, have included clinical skills among the competencies deemed important to the education and assessment of practicing phy-sicians.25–27_{As a result, it is not surprising that considerable}

efforts have been made to develop, and subsequently defend, testing methods than can be used to reliably and validly mea-sure these skills.

STANDARDIZED PATIENTS

SPs, often referred to as simulated patients or pro-grammed patients, are people who have been trained to ac-curately portray the role of a patient with a specific medical condition or conditions. The term “standardized” refers to the fact that the person is specifically trained to model the

who interview the same SP with the same presenting com-plaint will receive, on questioning, the same patient history. The physical findings relevant to the case, either real or sim-ulated, need to be stable and, for a given modeled scenario, they must not vary from one SP to another.

LARGE-SCALE SP EXAMINATIONS

Medical Council of Canada Qualifying Examination Part II

Since 1912, the MCC has been setting an examination that is a prerequisite for medical licensure in Canada; the Licenti-ate of the MCC is granted to those who successfully complete it. In 1992, the MCC added the Qualifying Examination Part II (MCCQE Part II) to the assessment sequence. Initially the MCCQE Part II was a 20-station Objective Structured Clini-cal Examination (OSCE).7,28_{Although the use of OSCEs is}

now commonplace throughout the world, implementing a national summative, performance-based, assessment based on a series of SP encounters was, at the time, unprecedented. The impetus for implementing the MCCQE Part II came largely from the licensing authorities. In the late 1980s, be-cause of the number and nature of related complaints that they received each year, members of these authorities began calling for an assessment of clinical and communication skills. The existing paper-and-pencil test of medical knowl-edge and problem solving (MCC Qualifying Examination Part I—MCCQE Part I) was not sufficient to address the emergent belief that candidates for medical licensure should be assessed more broadly.

To qualify for the MCCQE Part II, candidates must have completed successfully 12 months of postgraduate clinical training and passed the MCCQE Part I, currently a computer-adaptive test of knowledge and clinical decision-making. The number of candidates who qualify for the MCCQE Part II continues to grow. In 1992, 401 candidates took the exami-nation. In 2007, 3481 candidates completed this assessment, a more than eightfold increase.

As the measurement qualities of the MCCQE Part II be-came better understood, the number of stations was reduced from 20 to 14, and is now set at 12. This reduction in station length could be attributed to evolving test development pro-cesses, allowing for a more efficient and appropriate targeting of test content to examinee ability. Each station is based on a clinical problem presented by a SP; scoring is completed by physicians who observe from within the room. Checklists and rating scales are used to generate the station scores. At this time, the MCCQE Part II is comprised of eight 10-minute encounters with a SP and six couplet stations that include a 5-minute encounter with a SP followed by a 5-minute written component (Two of the stations in the as-sessment, including one of the couplets, are used for pilot testing purposes). Four domains are assessed based on com-mon presenting problems: history-taking skills, physical ex-amination skills, patient management, and doctor-patient interactions. Patient safety issues and professionalism are also evaluated.

(40)

ing poor performance in one station may be compensated by superior performance in another, the overall pass/fail deci-sion is based on a conjunctive standard; candidates must pass both by total score (the sum of their station scores) and by the number of stations passed.

Results from the MCCQE Part II are reported as a stan-dard score (mean ⫽ 500, standard deviation ⫽ 100). The examination is criterion-referenced, with the individual sta-tion pass marks set using the borderline group method.29

Candidates receive a bar graph indicating their performance in each of four domains relative to the mean score for their testing cohort. The four domains are data gathering (from history taking and physical examination tasks), patient inter-action (from rating scale items across stations), problem-solving and decision-making (based on certain stations; eg, acute care of trauma and the written work from the couplet stations), and legal, ethical, and organizational issues (which comprises a minimum of 10% of the total score). More ex-tensive feedback is provided to those candidates who are un-successful; specifically, they are told which stations they failed and are provided with a more extensive description of the four domains.

To balance accessibility and costs, a multisite, fixed test form model with two administrations per year is employed. In the spring, one test form is administered twice over 1 day at 10 university sites across Canada. At most sites, the examina-tion runs in two or more parallel tracks. In the fall, there are two test forms, one for each of 2 days of testing, and the examination runs at 16 sites. In spring, over 500 SPs are trained to simulate the patient problems. Twice that many are recruited for the fall. Ensuring that the SPs present their problems consistently and with sufficient fidelity for valid testing is critical. Each site has its own trainers who recruit and prepare the SPs according to the protocols developed centrally. Training videos, meetings with MCC staff, consul-tation with supervising physicians, along with telephone sup-port are all part of a process aimed at ensuring the SPs are ready for the examination.

Like all large-scale testing programs, there have been some administrative challenges. Developing feasible, psychometri-cally sound cases (simulated scenarios) is an ongoing task and takes considerable time and effort. Because the MCCQE Part II is a national examination, the scoring instruments and the supporting materials for SP training are developed centrally by a multidisciplinary test committee. Cases range from those requiring relatively little simulation (eg, history of diarrhea) to those where the SP must accurately simulate specific pa-tient presentations (eg, shortness of breath, decreased con-sciousness, pain, anxiety).

The MCC is continuously assessing different aspects of the MCCQE Part II. Numerous research studies suggest that both valid and reliable competency decisions are being made.30 –32 _{Most recently, the predictive validity of the}

MCCQE Part II was investigated by looking at the relation-ship between MCCQE Part I and Part II scores and complaint

decision-making component from the MCCQE Part I were predictors for complaints.

Educational Commission for Foreign Medical Graduates Clinical Skills Assessment

Based on several years of extensive research and consulta-tion with the MCC, the ECFMG CSA was instituted in July 1998.34,35_{This 11 station clinical skills examination was}

de-veloped to evaluate whether graduates of international med-ical schools (IMGs) possessed the skills necessary to enter supervised GME programs in the US. Successful completion of this examination became one of the required elements for ECFMG certification. Initially, the assessment was offered at one fixed site in Philadelphia, Pennsylvania. In 2002, in col-laboration with the National Board of Medical Examiners, a second testing site was constructed in Atlanta, Georgia. Be-tween 1998 and 2004, 43,624 IMGs were tested (37,930 first-time takers) in a total of 372,674 simulated clinical encoun-ters. During this time, numerous studies were published, several providing evidence to support the validity of the as-sessment scores.36 –38_{Of particular note, research was}

con-ducted to show that SP and physician evaluations of clinical skills were comparable.39 _{In 2004, administration of the}

ECFMG CSA ceased. Instead, IMGs were required to take and pass USMLE Step 2 CS (described below), a similar sim-ulation-based assessment that was developed to measure the clinical skills of American allopathic medical students and graduates. The USMLE Step 2 CS examination is part of the USMLE sequence (There are three “Steps” to the USMLE. Step 1 is intended to assess whether the examinee under-stands and can apply important concepts of the sciences basic to the practice of medicine. Step 2 focuses on the examinee’s knowledge, skills, and understanding of clinical science es-sential for provision of patient care “under supervision”— typically the point that medical school graduates begin their postgraduate education and experience. Step 3 is intended to assess whether the examinee can apply medical knowledge and understanding of biomedical and clinical science essen-tial for the unsupervised, independent practice of medicine.) To qualify for a medical license to practice in the US, gradu-ates of MD-granting schools in the US and gradugradu-ates of med-ical schools located outside the US must take and pass all components of USMLE.

United States Medical Licensing Examination Step 2 Clinical Skills

From the time that introduction of the USMLE program was first proposed in the late 1980s, it was the intent of the National Board of Medical Examiners and the Federation of State Medical Boards (the organizations that sponsor USMLE) to include clinical skills among the areas assessed as part of the examination program supporting the US medical licensing system. After many years of development, this goal became a reality in June 2004 when USMLE Step 2 CS was administered for the first time.40_{At this point, the previously}

existing Step 2 examination, a 1-day, computer-based multi-ple choice questionnaire test, was renamed the Step 2 Clinical

High-stakes OSCE International Workshop高標準OSCE國際工作坊手冊

Preface

As physicians’ clinical skills cannot be measured solely by written

examination, a National objective structured clinical examination (OSCE)

has been considered necessary as a part of medical licensure examination.

By the end of 2008, about 20 medical centers/ hospitals in Taiwan

announced that OSCE has been a regular clinical examination for their

trainees. However, there is little consensus about how to implement a

high-stake, large scale OSCE.

The Medical Council of Canada (MCC) has administered an OSCE for

the license to practice medicine since 1992. The high stake, large scale

OSCE is to test physicians’ skills of history taking, physical examination,

and communication. The examination results with psychometric evidence

indicate that a full-scale national administration of an OSCE model for

licensure is feasible in Canada.

Professor, Sydney Marla Smee is currently the Manager of MCCQE Part

II, Evaluation Bureau Medical Council of Canada (1990-Present). She is

an internationally recognized expert in implementing a high stake OSCE.

The workshop in Taiwan lead by Dr. Smee is to facilitate the

establishment of a Taiwanese model of high stake OSCE.

Contents

 Agenda --- 3

 Speaker’s Curriculum Vitae --- 5

 Instructor Assistant’s Curriculum Vitae --- 10

 Moderator’s Curriculum Vitae --- 12

 PowerPoint (Day 1) --- 13

 PowerPoint (Day 2) --- 32

 Related Articles

 The Use of Standardized Patient Assessments for

Certification and Licensure Decisions ---

 The Comparison of Physician Examiners and

Trained Assessors in a High-Stakes OSCE Setting ----



Physician Scores on a National Clinical Skills

Examination as Predictors of Complaints to Medical

Regulatory Authorities

--- Participants ---

37

45

49

59

Day1

Agenda

High- Stakes OSCE (I): Administrative Issues and Related Topics

Date: January 7, 2010 (Thursday)

Place:

臺北醫學大學 醫學綜合大樓前棟 4 樓-誠樸廳

台北市信義區 110 吳興街 250 號

Instructor:

Dr. Sydney M. Smee

Instructor Assistant: Dr. Charity T.C. Tsai

Time

Activity

Moderator

08:30-08:45

Registration

08:45-09:00

Opening Remarks

Minister Chaur-Shin Yung 楊朝祥部長

President Wen-Ta Chiu 邱文達校長

Director Chong-Liang Shi 石崇良處長

Prof. Chi-Wan

Lai

賴其萬教授

09:00-09:30

Presentation: Update on Plans for a Taiwan

OSCE

09:30-10:30

Presentation: High Stakes Multi-site OSCE:

Key Processes and Design Issues

10:30-10:45

Break

10:45-12:00

Group Activity: Five Groups- Create an

OSCE Design

12:00-13:00

LUNCH

13:00-14:30

Group Activity: Group Reports & Discussion

14:30-14:45

臺北醫學大學醫學綜合大樓前棟 4 樓-誠樸廳

臺北醫學大學醫學綜合大樓前棟 4 樓-誠樸廳