Composite reliability of workplace-based assessment for international medical graduates.

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. Practising doctors should, however, be assessed on their performance rather than their competency, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing the performance of IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each cohort was assessed at their workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 Mini-Clinical Examination Exercises (mini-CEX) and 1020 multisource feedback (MSF) sessions. PARTICIPANTS: 103 male and 39 female candidates based in urban and rural hospitals of the Hunter New England Health region, from 28 countries (Africa, Asia, Europe, South America, South Pacific). MAIN OUTCOME MEASURES: The reliability of the three WBA tools; the composite reliability of the tools as a group. RESULTS: The composite reliability of our WBA toolbox program was good: the composite reliability coefficient for five CBDs and 12 mini-CEX was 0.895 (standard error of measurement, 0.138). When the six MSF results were included, the composite reliability coefficient was 0.899 (standard error of measurement, 0.125). CONCLUSIONS: WBA is a reliable method for assessing IMGs when multiple tools and assessors are used over a period of time. This form of assessment meets the criteria for "good assessment" (reliability ≥ 0.8) and can be applied in other settings.

A systematic approach to workplace-based assessment for international medical graduates.

OBJECTIVES: To test whether a summative workplace-based assessment (WBA) is feasible and acceptable for international medical graduates (IMGs). DESIGN, SETTING AND PARTICIPANTS: A 6-month trial with 27 IMGs from teaching hospitals in Newcastle, Australia. IMGs were assessed by 65 trained assessors from different disciplines, using blueprinted, preset criteria. MAIN OUTCOME MEASURES: Mini-clinical evaluation exercises, case-based discussions, in-training assessments and multisource feedback. At the end of the trial, assessors and candidates gave feedback. RESULTS: All IMGs were successful at the end of the assessment. The format was well received and acceptable to the candidates and assessors. CONCLUSIONS: WBA is feasible and acceptable to assessors and candidates for assessment of IMGs, but it is intensive in use of resources and time.

Longitudinal Outcome of Programmatic Assessment of International Medical Graduates.

INTRODUCTION: Australia depends on international medical graduates (IMGs) to meet workforce shortages. The current standard assessment for IMGs is by clinical examination in observed structured clinical encounter (OSCE) format lasting 200 minutes. There are concerns about adequateness of this assessment as it does not test the qualities required to practice in a new country. We introduced a programmatic performance-based assessment for IMGs to prepare them to meet these challenges. The workplace-based assessment (WBA) program involves six-month longitudinal programmatic assessments comprising of 12 mini-clinical evaluation exercises (Mini-CEX), five case-based discussions (CBD), two in-training assessments (ITAs) and two sets of multisource feedback (MSF) assessments. We assessed 254 IMGs since 2010. We conducted a survey to evaluate the satisfaction with the program and the outcomes of these doctors. METHODS: We surveyed 254 candidates from 2010 to 2020. The survey used "SelectSurvey" tool with 12 questions and free-text comments. All candidates were sent the survey link to their last registered mobile phone using "Telstra Instant Messaging Service". We analysed the data using Microsoft "Excel". RESULTS: We received 153 (60%) responses. Amongst them, 141 (92%) candidates did not require further supervised practice for general registration and 129 (84%) candidates hold general/specialist registration. The candidates found the program useful and felt well supported. They appreciated real patient encounters. The feedback with positive critiquing was helpful in improving their clinical practice. The negative themes were program costs and frustration with the length of the program. CONCLUSION: Upon completion of the WBA program and obtaining the AMC certificate, most of the doctors were able to gain general registration. Seventy-eight (50%) candidates chose to continue their careers within the local area with 124 (80%) of them within the state. Our survey shows a comprehensive assessment program with immediate constructive feedback produces competent doctors to fill the medical workforce shortages.

Workplace-based assessment; learner and assessor perspectives.

OBJECTIVE: To examine the acceptability and educational impact of the workplace-based assessment program for international medical graduates on candidates and assessors. METHOD: A grounded theory-based qualitative analysis of the experiences of 17 candidates and eleven assessors using focus groups, interviews, and surveys. RESULTS: Both candidates and assessors identified positive opportunities for improved performance of international medical graduates. Their integration into the workforce was facilitated by improved communication and peer acceptance, from ongoing multifaceted feedback and time to practice skills. CONCLUSION: This study showed a high level of acceptability of the Newcastle workplace-based assessment program among candidates and assessors.

The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates.

OBJECTIVE: To evaluate the feasibility, reliability and acceptability of the mini clinical evaluation exercise (mini-CEX) for performance assessment among international medical graduates (IMGs). DESIGN, SETTING AND PARTICIPANTS: Observational study of 209 patient encounters involving 28 IMGs and 35 examiners at three metropolitan teaching hospitals in New South Wales, Victoria and Queensland, September-December 2006. MAIN OUTCOME MEASURES: The reliability of the mini-CEX was estimated using generalisability (G) analysis, and its acceptability was evaluated by a written survey of the examiners and IMGs. RESULTS: The G coefficient for eight encounters was 0.88, suggesting that the reliability of the mini-CEX was 0.90 for 10 encounters. Almost half of the IMGs (7/16) and most examiners (14/18) were satisfied with the mini-CEX as a learning tool. Most of the IMGs and examiners enjoyed the immediate feedback, which is a strong component of the tool. CONCLUSION: The mini-CEX is a reliable tool for performance assessment of IMGs, and is acceptable to and well received by both learners and supervisors.