International Medical Graduates and the Physician Workforce.

Importance: Physician shortages and the geographic maldistribution of general and specialist physicians impair health care delivery and worsen health inequity in the US. International medical graduates (IMGs) represent a potential solution given their ready supply. Observations: Despite extensive clinical experience, evidence of competence, and willingness to practice in underserved communities, IMGs experience multiple barriers to entry in the US, including the immigration process, the pathways available for certification and licensing, and institutional reluctance to consider non-US-trained candidates. International medical graduates applying to postgraduate training programs compare favorably with US-trained candidates in terms of clinical experience, prior formal postgraduate training, and research, but have higher application withdrawal rates and significantly lower residency and fellowship match rates, a disparity that may be exacerbated by the recent elimination of objective performance metrics, such as the US Medical Licensing Examination Step 1 score. Once legally in the US, IMGs encounter additional obstacles to board eligibility, research funding, and career progression. Conclusions and Relevance: International medical graduates offer a viable and available solution to bridge the domestic physician supply gap, while improving workforce diversity and meaningfully addressing the public health implications of geographic maldistribution of general and specialist physicians, without disrupting existing physician stature and salaries. The US remains unable to integrate IMGs until systematic policy changes at the national level are implemented.

Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.

BACKGROUND: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. OBJECTIVE: In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education. METHODS: We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses. RESULTS: A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5's performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non-English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5's (P=.03) and GPT-4's (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT's accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs. CONCLUSIONS: GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education. TRIAL REGISTRATION: PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=506687.

From admissions to licensure: education data associations from a multi-centre undergraduate medical education collaboration.

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements.

DO seniors and IMGs have lower match probabilities than MD seniors after adjusting for specialty choice and USMLE Step 1 score.

CONTEXT: It is unknown if US residency applicants of different educational backgrounds (US allopathic [MD], Doctor of Osteopathic Medicine [DO], and international medical graduates [IMG]) but comparable academic performance have similar match success. OBJECTIVES: Our objective was to compare match probabilities between applicant types after adjusting for specialty choice and United States Medical Licensing Examination (USMLE) Step 1 scores. METHODS: We performed a secondary analysis of published data in National Resident Matching Program (NRMP) reports from 2016, 2018, 2020, and 2022 for US MD seniors, DO seniors, and IMGs (US citizens and non-US citizens). We examined the 10 specialties with the most available spots in 2022. Average marginal effects from a multiple variable logistic regression model were utilized to estimate each non-MD senior applicant type's probability of matching into their preferred specialty compared to MD seniors adjusting for specialty choice, Step 1 score, and match year. RESULTS: Each non-MD applicant type had a lower adjusted percent difference in matching to their preferred specialty than MD seniors, -7.1 % (95 % confidence interval [CI], -11.3 to -2.9) for DO seniors, -45.6 % (-50.6 to -40.5) for US IMGs, and -56.6 % (-61.5 to -51.6) for non-US IMGs. Similarly, each non-MD applicant type had a lower adjusted percent difference in matching than MD seniors across almost all Step 1 score ranges, except for DO seniors with Step 1 scores <200 (-2.0 % [-9.5 to 5.5]). CONCLUSIONS: After adjusting for specialty choice, Step 1 score, and match year, non-US MD applicants had lower probabilities of matching into their preferred specialties than their US MD colleagues.

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match.

OBJECTIVE: To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM). MATERIALS AND METHODS: Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit. RESULTS: A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P < .001). As cutoff score increases, percentage of URM applicants decreases. CONCLUSION: The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

US Medical Licensing Exam Performance and American Board of Surgery Qualifying and Certifying Examinations.

BACKGROUND: Program directors use US Medical Licensing Exam (USMLE) scores as criteria for ranking applicants. First-time pass rates of the American Board of Surgery (ABS) Qualifying (QE) and Certifying (CE) Examinations have become important measures of residency program quality. USMLE Step 1 will become pass/fail in 2022. STUDY DESIGN: American Board of Surgery QE and CE success rates were assessed considering multiple characteristics of highly ranked (top 20) applicants to 22 general surgery programs in 2011. Chi-square, t-test, Wilcoxon Rank sum, linear and logistic regression were used, as appropriate. RESULTS: The QE and CE first attempt pass rates were 96% (235/244) and 86% (190/221), respectively. QE/CE success was not significantly associated with sex, race, research experience, or publications. Alpha Omega Alpha (AΩA) status was associated with success on the index CE (98% vs 83%; p = 0.008). Step 1 and Step 2 Clinical Knowledge (CK) scores of surgeons who passed QE on their first attempt were higher than scores of those who failed (Step 1: 233 vs 218; p = 0.016); (Step 2CK: 244 vs 228, p = 0.009). For every 10-point increase in Step 1 and 2CK scores, the odds of passing CE on the first attempt increased 1.5 times (95% CI 1.12, 2.0; p = 0.006) and 1.5 times (95% CI 1.11, 2.02, p = 0.008), respectively. For every 10-point increase in Steps 1 and 2CK scores, the odds of passing the QE on the first attempt increased 1.85 times (95% CI 1.11, 3.09; p = 0.018) and 1.86 times (95% CI 1.14, 3.06, p = 0.013), respectively. CONCLUSIONS: USMLE Step 1 and Step 2 CK examination scores correlate with American Board of Surgery QE and CE performance and success. The USMLE decision to transition Step 1 to a pass/fail examination will require program directors to identify other factors that predict ABS performance for ranking applicants.

Multiple United States Medical Licensing Examination Attempts and the Estimated Risk of Disciplinary Actions Among Graduates of U.S. and Canadian Medical Schools.

PURPOSE: The United States Medical Licensing Examination (USMLE) recently announced 2 policy changes: shifting from numeric score reporting on the Step 1 examination to pass/fail reporting and limiting examinees to 4 attempts for each Step component. In light of these policies, exam measures other than scores, such as the number of examination attempts, are of interest. Attempt limit policies are intended to ensure minimum standards of physician competency, yet little research has explored how Step attempts relate to physician practice outcomes. This study examined the relationship between USMLE attempts and the likelihood of receiving disciplinary actions from state medical boards. METHOD: The sample population was 219,018 graduates from U.S. and Canadian MD-granting medical schools who passed all USMLE Step examinations by 2011 and obtained a medical license in the United States, using data from the NBME and the Federation of State Medical Boards. Logistic regressions estimated how attempts on Steps 1, 2 Clinical Knowledge (CK), and 3 examinations influenced the likelihood of receiving disciplinary actions by 2018, while accounting for physician characteristics. RESULTS: A total of 3,399 physicians (2%) received at least 1 disciplinary action. Additional attempts needed to pass Steps 1, 2 CK, and 3 were associated with an increased likelihood of receiving disciplinary actions (odds ratio [OR]: 1.07, 95% confidence interval [CI]: 1.01, 1.13; OR: 1.09, 95% CI: 1.03, 1.16; OR: 1.11, 95% CI: 1.04, 1.17, respectively), after accounting for other factors. CONCLUSIONS: Physicians who took multiple attempts to pass Steps 1, 2 CK, and 3 were associated with higher estimated likelihood of receiving disciplinary actions. This study offers support for licensure and practice standards to account for physicians' USMLE attempts. The relatively small effect sizes, however, caution policy makers from placing sole emphasis on this relationship.

Admissions Is Not Enough: The Racial Achievement Gap in Medical Education.

The achievement gap is a disparity in academic and standardized test performance that exists between White and underrepresented minority (URM) students that begins as early as preschool and worsens as students progress through the educational system. Medical education is not immune to this inequality. URM medical students are more likely to experience delayed graduation and course failure, even after accounting for science grade point average and Medical College Admission Test performance. Moreover, URM students are more likely to earn lower scores on licensing examinations, which can have a significant impact on their career trajectory, including specialty choice and residency competitiveness. After the release of preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS) and public commentary on these recommendations, the National Board of Medical Examiners and Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition from a 3-digit numeric score to pass/fail scoring. Given that another of InCUS's recommendations was to "minimize racial demographic differences that exist in USMLE performance," it is paramount to consider the impact of this scoring change on URM medical students specifically. Holistic admissions are a step in the right direction of acknowledging that URM students often travel a further distance to reach medical school. However, when residency programs emphasize USMLE performance (or any standardized test score) despite persistent test score gaps, medical education contributes to the disproportionate harm URM students face and bolsters segregation across medical specialties. This Perspective provides a brief explanation of the achievement gap, its psychological consequences, and its consequences in medical education; discusses the potential effect of the Step 1 scoring change on URM medical students; and provides a review of strategies to redress this disparity.

Are USMLE Scores Valid Measures for Chief Resident Selection?

BACKGROUND: The US Medical Licensing Examination (USMLE) Step 1 and Step 2 scores are often used to inform a variety of secondary medical career decisions, such as residency selection, despite the lack of validity evidence supporting their use in these contexts. OBJECTIVE: We compared USMLE scores between non-chief residents (non-CRs) and chief residents (CRs), selected based on performance during training, at a US academic medical center that sponsors a variety of graduate medical education programs. METHODS: This was a retrospective cohort study of residents' USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores from 2015 to 2020. The authors used archived data to compare USMLE Step 1 and Step 2 CK scores between non-CR residents in each of the eligible programs and their CRs during the 6-year study period. RESULTS: Thirteen programs enrolled a total of 1334 non-CRs and 211 CRs over the study period. There were no significant differences overall between non-CRs and CRs average USMLE Step 1 (239.81 ± 14.35 versus 240.86 ± 14.31; P = .32) or Step 2 scores (251.06 ± 13.80 versus 252.51 ± 14.21; P = .16). CONCLUSIONS: There was no link between USMLE Step 1 and Step 2 CK scores and CR selection across multiple clinical specialties over a 6-year period. Reliance on USMLE Step 1 and 2 scores to predict success in residency as measured by CR selection is not recommended.

How is increased selectivity of medical school admissions associated with physicians' career choice? A Japanese experience.

BACKGROUND: During the long-lasting economic stagnation, the popularity of medical school has dramatically increased among pre-medical students in Japan. This is primarily due to the belief that medicine is generally a recession-proof career. As a result, pre-medical students today who want to enter medical school have to pass a more rigorous entrance examination than that in the 1980s. This paper explores the association between the selectivity of medical school admissions and graduates' later career choices. METHODS: A unique continuous measure of the selectivity of medical school admissions from 1980 to 2017, which is defined as the deviation value of medical schools, was merged with cross-sectional data of 122 990 physicians aged 35 to 55 years. The association between the deviation value of medical schools and various measures of physicians' career choices was explored by logistic and ordinary least square regression models. Graduates from medical schools in which the deviation value was less than 55 were compared with those from more competitive medical schools, after controlling for fixed effects for the medical school attended by binary variables. RESULTS: From 1980 to 2017, the average deviation value increased from 58.3 to 66.3, indicating a large increase in admission selectivity. Empirical results suggest that increasing selectivity of a medical school is associated with graduates having a higher probability of choosing a career in an acute hospital as well as having a lower probability of opening their own clinic and choosing a career in primary health care. Graduating from a highly competitive medical school (i.e., deviation value of more than 65) significantly increases the probability of working at typical acute hospitals such as so-called 7:1 hospitals (OR 1.665 2, 95%CI 1.444 0-1.920 4) and decreases the probability of working at primary care facilities (OR 0.602 6, 95%CI 0.441 2-0.823 0). It is also associated with graduates having a higher probability of becoming medical board certified (OR 1.294 6, 95%CI 1.108 8-1.511 4). CONCLUSION: Overall, this paper concludes that increased selectivity of medical school admissions predicts a higher quality of physicians in their own specialty, but at the same time, it is associated with a lower supply of physicians who go into primary care.