Evaluating Performance of ChatGPT on MKSAP Cardiology Board Review Questions.

Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing tool created by OpenAI. Much of the discussion regarding artificial intelligence (AI) in medicine is the ability of the language to enhance medical practice, improve efficiency and decrease errors. The objective of this study was to analyze the ability of ChatGPT to answer board-style cardiovascular medicine questions by using the Medical Knowledge Self-Assessment Program (MKSAP).The study evaluated the performance of ChatGPT (versions 3.5 and 4), alongside internal medicine residents and internal medicine and cardiology attendings, in answering 98 multiple-choice questions (MCQs) from the Cardiovascular Medicine Chapter of MKSAP. ChatGPT-4 demonstrated an accuracy of 74.5 %, comparable to internal medicine (IM) intern (63.3 %), senior resident (63.3 %), internal medicine attending physician (62.2 %), and ChatGPT-3.5 (64.3 %) but significantly lower than cardiology attending physician (85.7 %). Subcategory analysis revealed no statistical difference between ChatGPT and physicians, except in valvular heart disease where cardiology attending outperformed ChatGPT (p = 0.031) for version 3.5, and for heart failure (p = 0.046) where ChatGPT-4 outperformed senior resident. While ChatGPT shows promise in certain subcategories, in order to establish AI as a reliable educational tool for medical professionals, performance of ChatGPT will likely need to surpass the accuracy of instructors, ideally achieving the near-perfect score on posed questions.

Evaluating Board Certification Within the Aesthetic Marketplace.

BACKGROUND: Unlike most health care sectors, patients can select an aesthetic surgery provider without considering insurance coverage. Patients therefore must be able to make informed choices regarding provider selection. Surgeon qualifications are part of the data patients evaluate in their decision making. To characterize the provider landscape that patients face, this study compares the certification requirements of various boards within the aesthetic marketplace. METHODS: Four boards were identified for analysis based on a Google search of "board of plastic surgery": the American Board of Plastic Surgery (ABPS), the American Board of Cosmetic Surgery (ABCS), the American Board of Facial Plastic and Reconstructive Surgery (ABFPRS), and the American Board of Facial Cosmetic Surgery (ABFCS). Information on certification requirements was obtained from each board's official website. RESULTS: ABPS requires that diplomates complete an Accreditation Council for Graduate Medical Education (ACGME)-accredited plastic surgery residency, pass a written and oral examination that includes a case collection, and meet continual standards to maintain certification. ABCS and ABFCS both require an American Academy of Cosmetic Surgery (AACS) cosmetic surgery fellowship and passage of a written and oral examination. Neither board has case collection or continuing certification requirements. ABFPRS requires completion of either an ACGME-accredited otolaryngology or plastic surgery residency. Its examination process includes written and oral components as well as a case log. ABFPRS has enacted continuing certification requirements for diplomates credentialed in 2001 and later. ABPS is the only board that is a member of the American Board of Medical Specialties (ABMS). CONCLUSIONS: ABPS stands apart as the only board within the aesthetic marketplace with rigorous standards for precertification training, demonstrating competency through examinations and case logs, and maintaining certification. Being an ABMS member board also contributes to ABPS being the preeminent organization for identifying physicians who practice safe, effective aesthetic surgery.

Associations of Internal Medicine Residency Milestone Ratings and Certification Examination Scores With Patient Outcomes.

Importance: Despite its importance to medical education and competency assessment for internal medicine trainees, evidence about the relationship between physicians' milestone residency ratings or the American Board of Internal Medicine's initial certification examination and their hospitalized patients' outcomes is sparse. Objective: To examine the association between physicians' milestone ratings and certification examination scores and hospital outcomes for their patients. Design, Setting, and Participants: Retrospective cohort analyses of 6898 hospitalists completing training in 2016 to 2018 and caring for Medicare fee-for-service beneficiaries during hospitalizations in 2017 to 2019 at US hospitals. Main Outcomes and Measures: Primary outcome measures included 7-day mortality and readmission rates. Thirty-day mortality and readmission rates, length of stay, and subspecialist consultation frequency were also assessed. Analyses accounted for hospital fixed effects and adjusted for patient characteristics, physician years of experience, and year. Exposures: Certification examination score quartile and milestone ratings, including an overall core competency rating measure equaling the mean of the end of residency milestone subcompetency ratings categorized as low, medium, or high, and a knowledge core competency measure categorized similarly. Results: Among 455 120 hospitalizations, median patient age was 79 years (IQR, 73-86 years), 56.5% of patients were female, 1.9% were Asian, 9.8% were Black, 4.6% were Hispanic, and 81.9% were White. The 7-day mortality and readmission rates were 3.5% (95% CI, 3.4%-3.6%) and 5.6% (95% CI, 5.5%-5.6%), respectively, and were 8.8% (95% CI, 8.7%-8.9%) and 16.6% (95% CI, 16.5%-16.7%) for mortality and readmission at 30 days. Mean length of stay and number of specialty consultations were 3.6 days (95% CI, 3.6-3.6 days) and 1.01 (95% CI, 1.00-1.03), respectively. A high vs low overall or knowledge milestone core competency rating was associated with none of the outcome measures assessed. For example, a high vs low overall core competency rating was associated with a nonsignificant 2.7% increase in 7-day mortality rates (95% CI, -5.2% to 10.6%; P = .51). In contrast, top vs bottom examination score quartile was associated with a significant 8.0% reduction in 7-day mortality rates (95% CI, -13.0% to -3.1%; P = .002) and a 9.3% reduction in 7-day readmission rates (95% CI, -13.0% to -5.7%; P < .001). For 30-day mortality, this association was -3.5% (95% CI, -6.7% to -0.4%; P = .03). Top vs bottom examination score quartile was associated with 2.4% more consultations (95% CI, 0.8%-3.9%; P < .003) but was not associated with length of stay or 30-day readmission rates. Conclusions and Relevance: Among newly trained hospitalists, certification examination score, but not residency milestone ratings, was associated with improved outcomes among hospitalized Medicare beneficiaries.

Large Language Model Performance on Practice Epilepsy Board Examinations.

This diagnostic study examines whether large language models are able to pass practice licensing examinations for epilepsy.

Maintenance of Certification (MOC)-Benefit vs Burden.

This Viewpoint examines the demands of maintenance of certification (MOC) requirements from the ABIM on balance with the projected benefits to quality of patient care.

A Coronavirus Disease 2019 Pandemic Pivot: Development of the American Board of Anesthesiology's Virtual APPLIED Examination.

In 2020, the coronavirus disease 2019 (COVID-19) pandemic interrupted the administration of the APPLIED Examination, the final part of the American Board of Anesthesiology (ABA) staged examination system for initial certification. In response, the ABA developed, piloted, and implemented an Internet-based "virtual" form of the examination to allow administration of both components of the APPLIED Exam (Standardized Oral Examination and Objective Structured Clinical Examination) when it was impractical and unsafe for candidates and examiners to travel and have in-person interactions. This article describes the development of the ABA virtual APPLIED Examination, including its rationale, examination format, technology infrastructure, candidate communication, and examiner training. Although the logistics are formidable, we report a methodology for successfully introducing a large-scale, high-stakes, 2-element, remote examination that replicates previously validated assessments.

Correlation Between Neurosurgical Residency Written Board Scores and Case Logs.

OBJECTIVE: There are few objective measures for evaluating individual performance throughout surgical residency. Two commonly used objective measures are the case log numbers and written board examination scores. The objective of this study was to investigate possible correlations between these measures. METHODS: We conducted a retrospective review of the American Board of Neurological Surgery (ABNS) written board scores and the Accreditation Council for Graduate Medical Education case logs of 27 recent alumni from neurologic surgery residency training programs at The Ohio State Wexner Medical Center and the University of Nebraska Medical Center. RESULTS: The number of spine cases logged was significantly correlated with the ABNS written examination performance in univariate linear regression (r2 = 0.182, P = 0.0265). However, case numbers from all other neurosurgical subspecialties did not significantly correlate with ABNS written board performance (P > 0.1). CONCLUSIONS: Identifying which objective measures correlate most closely with resident education could help optimize the structure of residency training programs. We believe that early exposure to focused aspects of neurosurgery helps the young resident learn quickly and efficiently and ultimately score highly on standardized examinations. Therefore program directors may want to ensure focused exposure during the early years of residency, with particular attention to worthwhile rotations in spine neurosurgery.

Continuing Medical Education and Lifelong Learning.

Continuing medical education is an ongoing process to educate clinicians and provide patients with up-to-date, evidence-based care. Since its inception, the maintenance of certification (MOC) program has changed dramatically. This article reviews the development of MOC and its integration with the 6 core competencies, including the practice-based learning and improvement cycle. The concept of lifelong learning is discussed, with specific focus on different methods for surgeons to engage in learning, including simulation, coaching, and communities of practice. In addition, the future of MOC in continuous professional development is reviewed.

Construct Validation of the American Board of Anesthesiology's APPLIED Examination for Initial Certification.

BACKGROUND: The American Board of Anesthesiology administers the APPLIED Examination as a part of initial certification, which as of 2018 includes 2 components-the Standardized Oral Examination (SOE) and the Objective Structured Clinical Examination (OSCE). The goal of this study is to investigate the measurement construct(s) of the APPLIED Examination to assess whether the SOE and the OSCE measure distinct constructs (ie, factors). METHODS: Exploratory item factor analysis of candidates' performance ratings was used to determine the number of constructs, and confirmatory item factor analysis to estimate factor loadings within each construct and correlation(s) between the constructs. RESULTS: In exploratory item factor analysis, the log-likelihood ratio test and Akaike information criterion index favored the 3-factor model, with factors reflecting the SOE, OSCE Communication and Professionalism, and OSCE Technical Skills. The Bayesian information criterion index favored the 2-factor model, with factors reflecting the SOE and the OSCE. In confirmatory item factor analysis, both models suggest moderate correlation between the SOE factor and the OSCE factor; the correlation was 0.49 (95% confidence interval [CI], 0.42-0.55) for the 3-factor model and 0.61 (95% CI, 0.54-0.64) for the 2-factor model. The factor loadings were lower for Technical Skills stations of the OSCE (ranging from 0.11 to 0.25) compared with those of the SOE and Communication and Professionalism stations of the OSCE (ranging from 0.36 to 0.50). CONCLUSIONS: The analyses provide evidence that the SOE and the OSCE measure distinct constructs, supporting the rationale for administering both components of the APPLIED Examination for initial certification in anesthesiology.

The American Board of Pathology's 2020 Continuing Certification Program.

CONTEXT.­: Certification by the American Board of Pathology (ABPath) is a valued credential that serves patients, families, and the public and improves patient care. The ABPath establishes professional and educational standards and assesses the knowledge of candidates for initial certification in pathology. Diplomates certified in 2006 and thereafter are required to participate in Continuing Certification (CC; formerly Maintenance of Certification) in order to maintain certification. OBJECTIVE.­: To inform and update the pathology community on the history of board certification, the requirements for CC, ABPath CertLink, changes to the CC program, and ABPath compliance with recommendations from the American Board of Medical Specialties Vision Commission; and to demonstrate the value of CC participation for diplomates with non-time-limited certification. DATA SOURCES.­: This review uses ABPath archived minutes of the CC Committee and the Board of Trustees, the ABPath CC Booklet of Information, the collective knowledge of the ABPath staff and trustees, and the American Board of Medical Specialties 2018-2019 Board Certification Report. CONCLUSIONS.­: The ABPath continues to update the CC program to make it more relevant and meaningful and less burdensome for diplomates. Adding ABPath CertLink to the program has been a significant enhancement for the assessment of medical knowledge and has been well received by diplomates.

Do Milestone Ratings Predict Physical Medicine and Rehabilitation Board Certification Examination Scores?

ABSTRACT: The Accreditation Council of Graduate Medical Education developed the Milestones to assist training programs in assessing resident physicians in the context of their participation in Accreditation Council of Graduate Medical Education-accredited training programs. Biannual assessments are done over a resident's entire training period to define the trajectory in achieving specialty-specific competencies. As part of its process of initial certification, the American Board of Physical Medicine and Rehabilitation requires successful completion of two examinations administered approximately 9 mos apart. The Part I Examination measures a single dimensional construct, physical medicine and rehabilitation medical knowledge, whereas Part II assesses the application of medical and physiatric knowledge to multiple domains, including data acquisition, problem solving, patient management, systems-based practice, and interpersonal and communication skills through specific patient case scenarios. This study aimed to investigate the validity of the Milestones by demonstrating its association with performance in the American Board of Physical Medicine and Rehabilitation certifying examinations. A cohort of 233 physical medicine and rehabilitation trainees in 3-yr residency programs (postgraduate year 2 entry) in the United States from academic years 2014-2016, who also took the American Board of Physical Medicine and Rehabilitation Parts I and II certifying examinations between 2016 and 2018, were included in the study. Milestones ratings in four distinct observation periods were correlated with scores in the American Board of Physical Medicine and Rehabilitation Parts I and II Examinations. Milestones ratings of medical knowledge (but not patient care, professionalism, problem-based learning, interpersonal and communication skills, and systems-based practice) predicted performance in subsequent Part I American Board of Physical Medicine and Rehabilitation Examination, but none of the Milestone ratings correlated with Part II Examination scaled scores.