CATALYST: Piloting a Longitudinal Assessment and Learning Program for Board Recertification and Continuous Professional Development.

Context: As a proposed alternative to the traditional recertification examination, CATALYST is a longitudinal formative assessment platform created on cognitive learning principles. CATALYST was designed by the National Board of Osteopathic Medical Examiners to encourage more complex and durable practice-relevant learning and demonstration of ongoing competencies. Objective: To investigate the value of the CATALYST platform using board diplomates' subjective feedback and comparison of performance on CATALYST questions with performance on board examinations. Methods: Diplomates from 3 osteopathic specialty boards (the American Osteopathic Board of Internal Medicine, the American Osteopathic Board of Pediatrics, and the American Osteopathic Board of Obstetricians and Gynecologists) participated in this pilot study. Over the course of 16 weeks, participants were provided 2 questions per week via the CATALYST platform. An evaluation survey was emailed at the end of the study period to collect participants' feedback. Survey results and correlations of CATALYST performance with past or upcoming board examination scores were analyzed. Results: A total of 196 diplomates completed the surveys, with 95% reporting that participation in the platform would help them stay current in their specialties and 91% reporting that participation would help them provide better care to their patients. For the AOBIM, a significant correlation was found between the number of CATALYST questions answered correctly and performance on the board examination (r=0.51, P<.001). The correlations found for the AOBP and AOBOG were not significant (r=0.197, P=.296, and r=0.370, P=.075, respectively). Conclusion: The CATALYST platform could offer valuable contributions to the board recertification process and to patient safety. Further investigations are being conducted on a new user-friendly platform.

2019 United States Osteopathic Medical Regulatory Summit: Consensus, Recommendations, and Next Steps in Defining Osteopathic Distinctiveness.

BACKGROUND: Osteopathic distinctiveness is a result of professional education, identity formation, training, credentialing, and qualifications. With the advancement of a single graduate medical education (GME) accreditation system and the continued growth of the osteopathic medical profession, osteopathic distinctiveness and professional identity are seen as lacking clarity and pose a challenge. SUMMIT: To achieve consensus on a succinct definition of osteopathic distinctiveness and to identify steps to more clearly define and advance that distinctiveness, particularly in professional self-regulation, a representative group of osteopathic medical students, residents, physicians, and members of the licensing, GME, and undergraduate medical education (UME) communities convened the 2019 United States Osteopathic Medical Regulatory Summit in February 2019. Key features of osteopathic distinctiveness were discussed. Growth in the profession; changes in health care delivery, technology, and demographics within the profession and patient communities; and associated challenges and opportunities for osteopathic medical practice and patients were considered. CONSENSUS: Osteopathic medicine is a distinctive practice that brings unique, added value to patients, the public, and the health care community at large. A universal definition and common understanding of that distinctiveness is lacking. Efforts to unify messaging that defines osteopathic distinctiveness, to align the distinctive elements of osteopathic medical education and professional self-regulation across a continuum, and to advance research on care and educational program outcomes are critical to the future of the osteopathic medical profession. RECOMMENDATIONS: (1) Convene a task force of groups represented at the Summit to develop a succinct and consistent message defining osteopathic distinctiveness. (2) Demonstrate uniqueness of the profession through research demonstrating efficacy of care and patient outcomes, adding to the public good. (3) Harmonize GME and UME by beginning to align entrustable professional activities with UME milestones. (4) Convene representatives from osteopathic specialty colleges and certification boards to define curricular elements across GME, certification, and osteopathic continuous certification. (5) Build on the Project in Osteopathic Medical Education and Empathy study.

An Investigation of the Relationship Between COMLEX-USA Licensure Examination Performance and State Licensing Board Disciplinary Actions.

PURPOSE: Passing the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) serves as a licensing requirement, yet there is limited understanding between this high-stakes exam and performance outcomes. This study examined the relationship between COMLEX-USA scores and disciplinary actions received by osteopathic physicians. METHOD: Data for osteopathic physicians (N = 26,383) who graduated from medical school between 2004 and 2013 were analyzed using multinomial logistic regression to assess the relationship between COMLEX-USA scores and placement into one of 3 disciplinary action categories relative to no action received, controlling for years in practice and gender. RESULTS: Less than 1% of physicians in this study (n = 187) had a disciplinary action(s). Controlling for all COMLEX-USA levels, years in practice, and gender, higher Level 3 scores were associated with significant decreased odds for all action categories: revoked licensed (odds ratio [OR] = 0.51, 95% confidence interval [CI] 0.36, 0.72; P < .001), imposed limitations to practice (OR = 0.59, 95% CI 0.41, 0.84; P < .01), and other action imposed (OR = 0.48, 95% CI 0.33, 0.69; P < .001), relative to not receiving an action. In these same models, higher Level 2 Performance Evaluation Biomedical/Biomechanical Domain scores decreased the odds for an action that revoked a license (OR = 0.75, 95% CI 0.58, 0.98; P < .05) and imposed limitations to practice (OR = 0.64, 95% CI 0.49, 0.84; P < .001). CONCLUSIONS: These findings provide evidence that the COMLEX-USA delivers useful information regarding the likelihood of a practitioner receiving state board disciplinary actions.

Association Between Performance on COMLEX-USA and the American College of Osteopathic Family Physicians In-Service Examination.

BACKGROUND: The primary goal of residency programs is to select and educate qualified candidates to become competent physicians. Program directors often use performance on licensure examinations to evaluate the ability of candidates during the resident application process. The American College of Osteopathic Family Physicians (ACOFP) administers an in-service examination (ISE) to residents annually. There are few prior studies of the relationship between the Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series and formative assessments of residents in training. OBJECTIVE: We explored the relationship between performance on COMLEX-USA and the ACOFP in-service examination to offer support on the use of licensing examinations in resident selection. METHODS: In 2016, performance data from the COMLEX-USA and the ISE were matched for 3 resident cohorts (2011-2013, inclusive; N = 1384). Correlations were calculated to examine the relationship between COMLEX-USA and ISE scores. Multiple linear regression models were used to determine if performance on COMLEX-USA significantly predicted third-year ISE (ISE-3) scores. RESULTS: Findings indicated that correlations among performance on COMLEX-USA and ISE were statistically significant (all P < .001), and there was strong intercorrelation between COMLEX-USA Level 3 and ISE-1 performance (r = 0.57, P < .001). Performance on the COMLEX-USA Levels 1 and 2-Cognitive Examination significantly predicted performance on the ISE-3 (F(2,1381) = 228.8, P < .001). CONCLUSIONS: The results support using COMLEX-USA as a part of resident selection in family medicine. Additionally, program directors may use performance on COMLEX-USA to predict success on the ISE-3.

Evidence-Based Redesign of the COMLEX-USA Series.

To ensure that the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) reflects the evolving practice of osteopathic medicine, the National Board of Osteopathic Medical Examiners has developed new content and format specifications for an enhanced, competency-based examination program to be implemented with COMLEX-USA Level 3 in 2018. This article summarizes the evidence-based design processes that served as the foundation for blueprint development and the evidence supporting its validity. An overview is provided of the blueprint's 2 dimensions: Competency Domains and Clinical Presentations. The authors focus on the evidence that supports interpretation of test scores for the primary and intended purpose of COMLEX-USA, which is osteopathic physician licensure. Important secondary uses and the educational and catalytic effect of assessments are also described. This article concludes with the National Board of Osteopathic Medical Examiners' plans to ensure that the COMLEX-USA series remains current and meets the needs of its stakeholders-the patients who seek care from osteopathic physicians.

The Use of COMLEX-USA and USMLE for Residency Applicant Selection.

BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) and the United States Medical Licensing Examination (USMLE) are recognized by all state medical licensing boards in the United States, and the Federation of State Medical Boards has supported the validity of both examinations for medical licensure. Many osteopathic medical students take both examinations. OBJECTIVE: The purpose of this study was to investigate performance on COMLEX-USA Level 1 and USMLE Step 1 of students from colleges of osteopathic medicine where the majority of students took both examinations. METHODS: Data were collected on the entering classes of 2010 and 2011. Relationships between the COMLEX-USA Level 1 and the USMLE Step 1 were quantified using Pearson correlations. The correlation between outcomes on the 2 examinations was evaluated using the phi coefficient. A contingency table was constructed to look at first-attempt outcomes (pass/fail). RESULTS: Data for 2010 and 2011 were collected from 3 osteopathic medical schools, with 795 of 914 students (87%) taking both examinations. The correlation between first-attempt COMLEX-USA Level 1 and USMLE Step 1 scores was statistically significant across and within all 3 schools. The overall correlation was r(795) = 0.84 (P < .001). Pass/fail status on the 2 examinations was moderately correlated (φ = 0.39, P < .01). CONCLUSIONS: Our study found a strong association between COMLEX Level 1 and USMLE Step 1 performance. Additional studies to accurately compare scores on these examinations are warranted.

Preparation Strategies of Osteopathic Medical Students for the COMLEX-USA Level 2-PE.

CONTEXT: Since 2002, osteopathic medical schools have made curricular changes to further enhance the clinical skills of their students, to prepare them for residency training, and to pass the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE). OBJECTIVE: To report how students at osteopathic medical schools prepare for the COMLEX-USA Level 2-PE, and to investigate the effect of these techniques on examination performance. METHODS: A standardized survey was given to students before the beginning of their examination to assess the preparation of osteopathic medical students for the COMLEX-USA Level 2-PE, such as coursework, orientation materials, and standardized patient (SP) encounters. Surveys that were completed by first-time test takers during the 2013-2014 and 2014-2015 test cycles were included in this study. RESULTS: Of 9120 surveys administered, 8733 were completed, achieving a response rate of 95.8%. Of those 8733 respondents, 8706 students (99.7%) reported having SP encounters during the first and second year of medical school, and 7379 (84%) reported having at least 1 SP encounter in years 3 and 4. Of 8733 students, 6079 (70%) reported receiving feedback from an osteopathic physician on their SP encounters, and 6049 (69%) and 6253 (72%) reported having viewed the COMLEX-USA Level 2-PE orientation video online and having read the examination's orientation guide, respectively. The largest difference in preparation between students who passed the COMLEX-USA Level 2-PE and students who did not was a prerequisite SP examination at their school, with 5574 students (68.9%) who passed reporting having participated compared with 364 students (56.5%) who failed. None of the differences in clinical skills training and test preparation was associated with statistically significant differences in pass or fail status. CONCLUSION: Osteopathic medical students use a variety of methods to enhance their clinical skills in preparation for the COMLEX-USA Level 2-PE, with universal use of SP programs since the COMLEX-USA Level 2-PE was implemented in 2004. Educators should continue to foster ways to develop students' clinical skills that reflect new advances in education and assessment to ensure that future osteopathic physicians can demonstrate competency in fundamental clinical skills before beginning postgraduate training.

Relationship between COMLEX-USA scores and performance on the American Osteopathic Board of Emergency Medicine Part I certifying examination.

CONTEXT: Few studies have investigated how well scores from the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) series predict resident outcomes, such as performance on board certification examinations. OBJECTIVES: To determine how well COMLEX-USA predicts performance on the American Osteopathic Board of Emergency Medicine (AOBEM) Part I certification examination. METHODS: The target study population was first-time examinees who took AOBEM Part I in 2011 and 2012 with matched performances on COMLEX-USA Level 1, Level 2-Cognitive Evaluation (CE), and Level 3. Pearson correlations were computed between AOBEM Part I first-attempt scores and COMLEX-USA performances to measure the association between these examinations. Stepwise linear regression analysis was conducted to predict AOBEM Part I scores by the 3 COMLEX-USA scores. An independent t test was conducted to compare mean COMLEX-USA performances between candidates who passed and who failed AOBEM Part I, and a stepwise logistic regression analysis was used to predict the log-odds of passing AOBEM Part I on the basis of COMLEX-USA scores. RESULTS: Scores from AOBEM Part I had the highest correlation with COMLEX-USA Level 3 scores (.57) and slightly lower correlation with COMLEX-USA Level 2-CE scores (.53). The lowest correlation was between AOBEM Part I and COMLEX-USA Level 1 scores (.47). According to the stepwise regression model, COMLEX-USA Level 1 and Level 2-CE scores, which residency programs often use as selection criteria, together explained 30% of variance in AOBEM Part I scores. Adding Level 3 scores explained 37% of variance. The independent t test indicated that the 397 examinees passing AOBEM Part I performed significantly better than the 54 examinees failing AOBEM Part I in all 3 COMLEX-USA levels (P<.001 for all 3 levels). The logistic regression model showed that COMLEX-USA Level 1 and Level 3 scores predicted the log-odds of passing AOBEM Part I (P=.03 and P<.001, respectively). CONCLUSION: The present study empirically supported the predictive and discriminant validities of the COMLEX-USA series in relation to the AOBEM Part I certification examination. Although residency programs may use COMLEX-USA Level 1 and Level 2-CE scores as partial criteria in selecting residents, Level 3 scores, though typically not available at the time of application, are actually the most statistically related to performances on AOBEM Part I.

Using standardized patients to assess the communication skills of graduating physicians for the comprehensive osteopathic medical licensing examination (COMLEX) level 2-performance evaluation (level 2-PE).

BACKGROUND: Standardized patients can be trained to assess the communication and interpersonal skills of medical students and graduates. PURPOSE: The purpose of this study is to present data to support the psychometric adequacy of the communication ratings provided by standardized patients. METHODS: Using the data from testing of 3, 450 examinees over a 1-year period, a number of psychometric analyses were undertaken. These included a variance component analysis, the calculation of various validity coefficients, the comparison of communication ratings for select examinee cohorts and case characteristics, and the investigation of some potential sources of score invalidity. RESULTS: Communication skills scores are moderately correlated to other competencies (knowledge, skills) and may be influenced by candidate characteristics such as gender and English language proficiency. They are not dependant on the age of the examinees, the clinical case content, or the gender of the standardized patients. CONCLUSIONS: For a multistation assessment, a reasonably precise and valid estimate of a candidate's communication ability can be obtained from trained standardized patients.

Getting "beyond the barriers" in reforming osteopathic medical education.

As the healthcare needs of the United States change, some leaders at colleges of osteopathic medicine and osteopathic graduate medical education programs have embraced one very important and timeless goal: to prepare future physicians to meet society's health needs. These medical educators have made significant strides toward moving "beyond the barriers" to effect curricular reform and quality improvement at their institutions. Some of the barriers to osteopathic medical education reform are addressed in this article, which recommends allowing curricular evolution and faculty development; expanding clinical learning and teaching; breaking down departmental walls; integrating osteopathic principles and practice; reevaluating admission requirements of colleges of osteopathic medicine; and eradicating the unspoken and, ironically, often detrimental culture of medicine, which can be contrary to compassionate patient care and healing.

Standardized patients and mechanical simulators in teaching and assessment at colleges of osteopathic medicine.

CONTEXT: A 2001 survey of 19 colleges of osteopathic medicine (COMs) revealed that standardized patient programs (SPPs) are increasingly used in osteopathic medical education. However, no new data have been published since. OBJECTIVES: To evaluate current SPP and mechanical simulator use at COMs compared with previous survey results. METHODS: In 2005, an electronic survey regarding the use of SPPs (eg, staffing, facilities) and mechanical simulators in the teaching and assessment of students' clinical skills was sent to the deans of the 23 fully accredited COMs and branch campuses. RESULTS: Responses were received from all 23 COMs for a 100% response rate. According to survey results, 19 COMs (87%) had active SPPs, 2 COMs (9%) reported that SPPs were in development, and the remaining 2 COMs (9%) used students as patients. In comparison, only 12 COMs (63%) in 2001 had active SPPs. Results indicated an increased use of standardized patients for assessment, particularly in physician-patient communication, osteopathic manipulative medicine, and osteopathic manipulative treatment. In addition, 12 COMs (52%) reported using mechanical simulators in the teaching or assessment of clinical skills. CONCLUSION: From 2001 to 2005, the use of SPPs and mechanical simulators at COMs increased substantially.

Relationships between scores on the COMLEX-USA Level 2-Performance Evaluation and selected school-based performance measures.

The Comprehensive Osteopathic Medical Licensing Examination USA Level 2 Performance Evaluation (COMLEX-USA Level 2-PE) is a national multistation performance examination designed to examine students' osteopathic clinical skills. The current study examines the relationship between achievement levels on the COMLEX-USA Level 2-PE and selected school-related variables for the class of 2005 at the West Virginia School of Osteopathic Medicine in Lewisburg, WVa (N=70). Significant (P<.01) correlations between the COMLEX-USA Level 2-PE summary performance and selected academic achievement measures include: weighted Physical Diagnosis grade, 0.41; weighted year 1 and year 2 Osteopathic Principles and Practice grade, 0.37: overall year 2 grade point average, 0.42; the objective structured clinical evaluation (OSCE) Physical Examination score, 0.40; and the OSCE Total Station score, 0.33. While further research is needed, the current study found modest but notable relationships between school-generated academic variables and performance on the COMLEX-USA Level 2-PE, and therefore supports the validity of the COMLEX-USA Level 2-PE examination for assessing the clinical skills of future osteopathic physicians.

Survey on the clinical skills of osteopathic medical students.

As part of the standard-setting methods used by the National Board of Osteopathic Medical Examiners for its Comprehensive Osteopathic Medical Licensing Examination clinical skills performance evaluation (COMLEX-USA Level 2-PE), a self-administered survey was distributed electronically and by mail to deans of colleges of osteopathic medicine, directors of graduate medical education programs, osteopathic medical students, and experts chosen demographically to represent osteopathic physicians in the United States. Groups were asked to rate fourth-year osteopathic medical students and interns on their clinical skills and acceptable pass rates and expected pass rates on the COMLEX-USA Level 2-PE. The surveys were not used systematically to compute the passing standards but to provide additional support for their validity. The viewpoints of the deans differed from those of the students, osteopathic graduate medical education program directors, and experts regarding clinical skills proficiencies and acceptable pass rates. However, all of the groups agreed that, on average, some students and interns do not have adequate clinical skills. These results provide additional support for requiring acceptable performance on a comprehensive clinical skills examination before admission to osteopathic graduate medical education programs.