Composite reliability of workplace-based assessment of international medical graduates.

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. The performance rather than the competency of practising doctors should, however, be assessed, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each was assessed in the workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 mini-clinical examination exercises (mini-CEX), and 1020 multi-source feedback (MSF) assessments. PARTICIPANTS: 103 male and 39 female candidates from 28 countries (Africa, Asia, Europe, South America, South Pacific) in urban and rural hospitals of the Hunter New England Health region. MAIN OUTCOME MEASURES: The composite reliability across the three WBA tools, expressed as the standard error of measurement (SEM). RESULTS: In our WBA program, a combination of five CBD and 12 mini-CEX assessments achieved an SEM of 0.33, greater than the threshold 0.26 of a scale point. Adding six MSF results to the assessment package reduced the SEM to 0.24, which is adequately precise. CONCLUSIONS: Combining data from different WBA assessment instruments achieves acceptable reliability for assessing IMGs, provided that the panel of WBA assessment types are carefully selected and the assessors are calibrated.

Composite reliability of workplace-based assessment for international medical graduates.

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. Practising doctors should, however, be assessed on their performance rather than their competency, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing the performance of IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each cohort was assessed at their workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 Mini-Clinical Examination Exercises (mini-CEX) and 1020 multisource feedback (MSF) sessions. PARTICIPANTS: 103 male and 39 female candidates based in urban and rural hospitals of the Hunter New England Health region, from 28 countries (Africa, Asia, Europe, South America, South Pacific). MAIN OUTCOME MEASURES: The reliability of the three WBA tools; the composite reliability of the tools as a group. RESULTS: The composite reliability of our WBA toolbox program was good: the composite reliability coefficient for five CBDs and 12 mini-CEX was 0.895 (standard error of measurement, 0.138). When the six MSF results were included, the composite reliability coefficient was 0.899 (standard error of measurement, 0.125). CONCLUSIONS: WBA is a reliable method for assessing IMGs when multiple tools and assessors are used over a period of time. This form of assessment meets the criteria for "good assessment" (reliability ≥ 0.8) and can be applied in other settings.

Workplace-based assessment; learner and assessor perspectives.

OBJECTIVE: To examine the acceptability and educational impact of the workplace-based assessment program for international medical graduates on candidates and assessors. METHOD: A grounded theory-based qualitative analysis of the experiences of 17 candidates and eleven assessors using focus groups, interviews, and surveys. RESULTS: Both candidates and assessors identified positive opportunities for improved performance of international medical graduates. Their integration into the workforce was facilitated by improved communication and peer acceptance, from ongoing multifaceted feedback and time to practice skills. CONCLUSION: This study showed a high level of acceptability of the Newcastle workplace-based assessment program among candidates and assessors.

Workplace-based assessment for international medical graduates: at what cost?

OBJECTIVE: To estimate the cost of resources required to deliver a program to assess international medical graduates (IMGs) in Newcastle, Australia, known as the Workplace Based Assessment (WBA) Program. DESIGN AND SETTING: A costing study to identify and evaluate the resources required and the overheads of delivering the program for a cohort of 15 IMGs, based on costs in 2012. MAIN OUTCOME MEASURES: Labour-related costs. RESULTS: The total cost in 2012 for delivering the program to a typical cohort of 15 candidates was $243,384. This equated to an average of $16,226 per IMG. After allowing for the fees paid by IMGs, the WBA Program had a deficit of $153,384, or $10,226 per candidate, which represents the contribution made by the health system. CONCLUSION: The cost per candidate to the health system of this intensive WBA program for IMGs is small.

Interns are from Venus, consultants are from Mars: differential perception among clinicians.

OBJECTIVE: To test for the presence of sex-based differences in perception (the notion that men and women "think" differently, and that differences in perception are biologically based) among healthcare professionals. DESIGN: Prospective survey. SETTING AND PARTICIPANTS: 90 medical personnel at a tertiary care hospital in Newcastle, NSW. INTERVENTION: Healthcare professionals were shown two pictures that could be interpreted as depicting either a young or an old person, and a word that could be seen as geometric shapes. MAIN OUTCOME MEASURES: The effects of sex, age, seniority, and specialisation in relation to the first impression of the image, the ability to change one's perception, and the speed of perception. RESULTS: Contrary to popular opinion, male physicians were more likely to perceive the older figures, and just as likely as women to be able to change their perception. Surgeons and junior staff were more likely to see, as well as being faster to form, an impression requiring abstract thought, and were more able to change their perceptions. CONCLUSIONS: Traditional sex stereotypes do not apply to medical personnel, but other age-based stereotypes, and professional rivalries (medical versus surgical) may have some empiric basis.