Making the grade: licensing examination performance by medical school accreditation status.

BACKGROUND: Accreditation systems strive to ensure the quality of undergraduate (basic) medical education and encourage ongoing improvements. Despite increasing global emphasis on quality assurance activities, there is limited research linking accreditation of medical education to improved student and graduate outcomes. The purpose of this study is to compare the United States Medical Licensing Examination® (USMLE®) performance of students and graduates who attended international medical schools accredited by an agency recognized by the World Federation of Medical Education (WFME) to individuals who attended schools that did not meet this criterion. METHODS: During the 2018-2020 study period, 39,650 individuals seeking Educational Commission for Foreign Medical Graduates® (ECFMG®) certification took one or more USMLE examinations. We cross-tabulated USMLE performance (first-attempt pass/fail result) and medical school accreditation status. RESULTS: Individuals seeking ECFMG certification who attended international medical schools accredited by an agency recognized by WFME had higher or comparable USMLE first-attempt pass rates compared to individuals who attended medical schools that did not meet this criterion. CONCLUSIONS: Implementing and maintaining meaningful accreditation systems requires substantial resources. These results provide important positive evidence that external evaluation of educational programs is associated, on average, with better educational outcomes, including in the domains of basic science, clinical knowledge, and clinical skills performance.

Comparison of Bedside and Video-Based Capillary Refill Time Assessment in Children.

OBJECTIVES: Capillary refill time (CRT) to assess peripheral perfusion in children with suspected shock may be subject to poor reproducibility. Our objectives were to compare video-based and bedside CRT assessment using a standardized protocol and evaluate interrater and intrarater consistency of video-based CRT (VB-CRT) assessment. We hypothesized that measurement errors associated with raters would be low for both standardized bedside CRT and VB-CRT as well as VB-CRT across raters. METHODS: Ninety-nine children (aged 1-12 y) had 5 consecutive bedside CRT assessments by an experienced critical care clinician following a standardized protocol. Each CRT assessment was video recorded on a black background. Thirty video clips (10 with bedside CRT < 1 s, 10 with CRT 1-2 s, and 10 with CRT > 2 s) were randomly selected and presented to 10 clinicians twice in randomized order. They were instructed to push a button when they visualized release of compression and completion of a capillary refill. The correlation and absolute difference between bedside and VB-CRT were assessed. Consistency across raters and within each rater was analyzed using the intraclass correlation coefficient (ICC). A Generalizability study was performed to evaluate sources of variation. RESULTS: We found moderate agreement between bedside and VB-CRT observations (r = 0.65; P < 0.001). The VB-CRT values were shorter by 0.17 s (95% confidence interval, 0.09-0.25; P < 0.001) on average compared with bedside CRT. There was moderate agreement in VB-CRT across raters (ICC = 0.61). Consistency of repeated VB-CRT within each rater was moderate (ICC = 0.71). Generalizability study revealed the source of largest variance was from individual patient video clips (57%), followed by interaction of the VB-CRT reviewer and patient video clip (10.7%). CONCLUSIONS: Bedside and VB-CRT observations showed moderate consistency. Using video-based assessment, moderate consistency was also observed across raters and within each rater. Further investigation to standardize and automate CRT measurement is warranted.

International Medical Graduates in the United States Psychiatry Workforce.

OBJECTIVE: This study describes the supply, distribution, and characteristics of international medical graduate (IMG) psychiatrists who provide services in the USA. METHODS: Cross-sectional study design, using descriptive statistics based on combined data from the American Medical Association (2020 Physician Masterfile) and the Educational Commission for Foreign Medical Graduates. RESULTS: International medical graduates continue to make significant contributions to the US physician workforce. As a group, they represent 29% of active psychiatrists in the USA, compared to 23% in all other medical specialties. Many IMG psychiatrists were US citizens who obtained their medical degrees outside the USA or Canada, often in the Caribbean. In some states (i.e., Florida, New Jersey), over 40% of active psychiatrists are IMGs. Over 30% of IMG psychiatrists graduated from medical schools in India and Pakistan. CONCLUSIONS: This study provides an overview of the psychiatric workforce in the USA, quantifying the specific contribution of IMGs. Several factors, including immigration policies, continued expansion of US medical schools, and the number of available residency positions, could impact the flow of IMGs to the US. Longitudinal studies are needed to better understand the implications for workforce composition and distribution, and their potential impact on the care of psychiatric patients.

Competence in Decision Making: Setting Performance Standards for Critical Care.

BACKGROUND: Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees. METHODS: Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards. RESULTS: For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario. CONCLUSIONS: Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

What we measure and what we should measure in medical education.

CONTEXT: As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society. OBJECTIVES: We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role. METHODS: We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies. DISCUSSION: As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.

Considerations that will determine if competency-based assessment is a sustainable innovation.

Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA's further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public's health.

Supply, distribution and characteristics of international medical graduates in family medicine in the United States: a cross-sectional study.

BACKGROUND: To describe the supply, distribution, and characteristics of international medical graduates (IMGs) in family medicine who provide patient care in the U.S. METHODS: A cross-sectional study design, using descriptive statistics on combined data from the Educational Commission for Foreign Medical Graduates and the American Medical Association, including medical school attended, country of medical school, and citizenship when entering medical school. RESULTS: In total, 118,817 physicians in family medicine were identified, with IMGs representing 23.8% (n = 28,227) of the U.S. patient care workforce. Of all 9579 residents in family medicine, 36.0% (n = 3452) are IMGS. In total, 35.9% of IMGs attended medical school in the Caribbean (n = 10,136); 19.9% in South-Central Asia (n = 5607) and 9.1% in South-Eastern Asia (n = 2565). The most common countries of medical school training were Dominica, Mexico, and Sint Maarten. Of all IMGs in family medicine who attended medical school in the Caribbean, 74.5% were U.S. citizens. In total, 40.5% of all IMGs in family medicine held U.S. citizenship at entry to medical school. IMGs comprise almost 40% of the family medicine workforce in Florida, New Jersey and New York. CONCLUSIONS: IMGs play an important role in the U.S. family medicine workforce. Many IMGs are U.S. citizens who studied abroad and then returned to the U.S. for graduate training. Given the shortage of family physicians, and the large number of IMGs in graduate training programs, IMGs will continue to play a role in the U.S. physician workforce for some time to come. Many factors, including the supply of residency training positions, could eventually restrict the number of IMGs entering the U.S., including those contributing to family practice.

Simulation-Based Assessment of Critical Care "Front-Line" Providers.

OBJECTIVES: Develop a standardized simulation method to assess clinical skills of ICU providers. DESIGN: Simulation assessment. SETTING: Simulation laboratory. SUBJECTS: Residents, Critical Care Medicine fellows, acute care nurse practitioner students. INTERVENTIONS: Performance scoring in scenarios from multiple Critical Care Medicine competency domains. MEASUREMENTS AND MAIN RESULTS: Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively. CONCLUSIONS: The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Rescuing the Clinical Breast Examination: Advances in Classifying Technique and Assessing Physician Competency.

OBJECTIVE: Develop new performance evaluation standards for the clinical breast examination (CBE). SUMMARY BACKGROUND DATA: There are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy. METHODS: This performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy. RESULTS: Accuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001). CONCLUSIONS: Our results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

The Simulation-Based Assessment of Pediatric Rapid Response Teams.

OBJECTIVE: To create scenarios of simulated decompensating pediatric patients to train pediatric rapid response teams (RRTs) and to determine whether the scenario scores provide a valid assessment of RRT performance with the hypothesis that RRTs led by intensivists-in-training would be better prepared to manage the scenarios than teams led by nurse practitioners. STUDY DESIGN: A set of 10 simulated scenarios was designed for the training and assessment of pediatric RRTs. Pediatric RRTs, comprising a pediatric intensive care unit (PICU) registered nurse and respiratory therapist, led by a PICU intensivist-in-training or a pediatric nurse practitioner, managed 7 simulated acutely decompensating patients. Two raters evaluated the scenario performances and psychometric analyses of the scenarios were performed. RESULTS: The teams readily managed scenarios such as supraventricular tachycardia and opioid overdose but had difficulty with more complicated scenarios such as aortic coarctation or head injury. The management of any particular scenario was reasonably predictive of overall team performance. The teams led by the PICU intensivists-in-training outperformed the teams led by the pediatric nurse practitioners. CONCLUSIONS: Simulation provides a method for RRTs to develop decision-making skills in managing decompensating pediatric patients. The multiple scenario assessment provided a moderately reliable team score. The greater scores achieved by PICU intensivist-in-training-led teams provides some evidence to support the validity of the assessment.

A comparison of physician emigration from Africa to the United States of America between 2005 and 2015.

BACKGROUND: Migration of health professionals has been a cause for global concern, in particular migration from African countries with a high disease burden and already fragile health systems. An estimated one fifth of African-born physicians are working in high-income countries. Lack of good data makes it difficult to determine what constitutes "African" physicians, as most studies do not distinguish between their country of citizenship and country of training. Thus, the real extent of migration from African countries to the United States (US) remains unclear. This paper quantifies where African migrant physicians come from, where they were educated, and how these trends have changed over time. METHODS: We combined data from the Educational Commission for Foreign Medical Graduates with the 2005 and 2015 American Medical Association Physician Masterfiles. Using a repeated cross-sectional study design, we reviewed the available data, including medical school attended, country of medical school, and citizenship when entering medical school. RESULTS: The outflow of African-educated physicians to the US has increased over the past 10 years, from 10 684 in 2005 to 13 584 in 2015 (27.1% increase). This represents 5.9% of all international medical graduates in the US workforce in 2015. The number of African-educated physicians who graduated from medical schools in sub-Saharan countries was 2014 in 2005 and 8150 in 2015 (304.6% increase). We found four distinct categorizations of African-trained physicians migrating to the US: (1) citizens from an African country who attended medical school in their own country (86.2%, n = 11,697); (2) citizens from an African country who attended medical school in another African country (2.3%, n = 317); (3) US citizens who attended medical school in an African country (4.0%, n = 537); (4) citizens from a country outside Africa, and other than the United States, who attended medical school in an African country (7.5%, n = 1013). Overall, six schools in Africa provided half of all African-educated physicians. CONCLUSIONS: The number of African-educated physicians in the US has increased over the past 10 years. We have distinguished four migration patterns, based on citizenship and country of medical school. The majority of African graduates come to the US from relatively few countries, and from a limited number of medical schools. A proportion are not citizens of the country where they attended medical school, highlighting the internationalization of medical education.

Patients of doctors further from medical school graduation have poorer outcomes.

CONTEXT: There is an apparent contradiction between the findings of studies indicating that patient outcomes are better when physicians have a greater volume of practice and those that find outcomes to be worse with increased time since training, which implies greater volume. OBJECTIVES: This study was designed to estimate the adjusted relationships between physicians' characteristics, including recent practice volume and time since medical school graduation, and patient outcomes. METHODS: This is a retrospective observational study based on all Pennsylvania hospitalisations over 7 years for acute myocardial infarction, congestive heart failure, gastrointestinal haemorrhage, hip fracture and pneumonia. It refers to 694 020 hospitalisations in 184 hospitals attended by 5280 internists and family physicians. Patient severity of illness at admission and in-hospital mortality, hospital location and volume, and the physician's recent practice volume, time since medical school graduation, board certification, and citizenship or medical school location were analysed. RESULTS: After adjustment, recent practice volume did not have a statistically significant association with in-hospital mortality for all of the conditions combined. By contrast, each decade since graduation from medical school was associated with a 4.5% increase in relative risk for patient mortality. CONCLUSIONS: Recent practice volume does not mitigate the increase in patient mortality associated with physicians' time since medical school graduation. These findings underscore the need to finds ways to support and encourage learning.

Simulation-based assessment to identify critical gaps in safe anesthesia resident performance.

BACKGROUND: Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty. METHODS: An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys. RESULTS: Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice. CONCLUSION: The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.

Overview of the world's medical schools: an update.

CONTEXT: That few data are available on the characteristics of medical schools or on trends within medical education internationally constitutes a major challenge when developing strategies to address physician workforce shortages. Quality and up-to-date information is needed to improve health and education policy planning. METHODS: We used publicly available data from the International Medical Education Directory and Avicenna Directories, and an internal education programme database to gather data on medical education provision worldwide. We sent a semi-structured questionnaire to a selection of 346 medical schools, of which 218 (63%) in 81 different countries or territories replied. We contacted ministries of health, national agencies for accreditation or similar bodies to clarify inconsistencies among sources. We identified key informants to obtain country-level specific information. Descriptive statistics were used to analyse current medical school data by country. RESULTS: There are about 2600 medical schools worldwide. The countries with the largest numbers of schools are India (n = 304), Brazil (n = 182), the USA (n = 173), China (n = 147) and Pakistan (n = 86). One-third of all medical schools are located in five countries and nearly half are located in 10 countries. Of 207 independent states, 24 have no medical school and 50 have only one. Regionally, numbers of citizens per school differ: the Caribbean region has one school per 0.6 million population; the Americas and Oceania each have one school per 1.2 million population; Europe has one school per 1.8 million population; Asia has one school per 3.5 million population, and Africa has one school per 5.0 million population. In 2012, on average, there were 181 graduates per medical school. CONCLUSIONS: The total number and distribution of medical schools around the world are not well matched with existing physician numbers and distribution. The collection and aggregation of medical school data are complex and would benefit from better information on local recognition processes. Longitudinal comparisons are difficult and subject to several sources of error. The consistency and quality of medical school data need to be improved to accurately document potential supply; one example of such an advancement is the World Directory of Medical Schools.

Examining changes in certification/licensure requirements and the international medical graduate examinee pool.

Changes in certification requirements and examinee characteristics are likely to influence the validity of the evidence associated with interpretations made based on test data. We examined whether changes in Educational Commission for Foreign Medical Graduates (ECFMG) certification requirements over time were associated with changes in internal medicine (IM) residency program director ratings and certification examination scores. Comparisons were made between physicians who were ECFMG-certified before and after the Clinical Skills Assessment (CSA) requirement. A multivariate analysis of covariance was conducted to examine the differences in program director ratings based on CSA cohort and whether the examinees emigrated for undergraduate medical education (national vs. international students). A univariate analysis of covariance was conducted to examine differences in scores from the American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. For both analyses, United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores were used as covariates. Results indicate that, of those certified by ECFMG between 1993 and 1997, 17 % (n = 1,775) left their country of citizenship for undergraduate medical education. In contrast, 38 % (n = 1,874) of those certified between 1999 and 2003 were international students. After adjustment by covariates, the main effect of cohort membership on the program director ratings was statistically significant (Wilks' λ = 0.99, F 5, 15391 = 19.9, P < 0.001). However, the strength of the relationship between cohort group and the ratings was weak (η = 0.01). The main effect of migration status was statistically significant and weak (Wilks' λ = 0.98, F 5,15391 = 45.3, P < 0.01; η = 0.02). Differences in ABIM Internal Medicine Certification Examination scores based on whether or not CSA were required was statistically significant, although the magnitude of the association between these variables was very small. The findings suggest that the implementation of an additional evaluation of skills (e.g., history-taking, physical examination) as a prerequisite to postgraduate medical education (residency) provides some additional, relevant data to those who select ECFMG-certified residents.

A validity study of COMLEX-USA Level 3 with the new test design.

CONTEXT: The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores. OBJECTIVES: The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework. METHODS: Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE). RESULTS: All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure. CONCLUSIONS: This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series.