A Look at Demographics and Transition to Virtual Assessments: An Analysis of Bias in the American Board of Surgery General Surgery Certifying Exams.

OBJECTIVE: The goals of this study were (1) to assess if examiner ratings in the American Board of Surgery (ABS) General Surgery Cetifying Exam (CE) are biased based on the gender, race, and ethnicity of the candidate or the examiners, and (2) if the format of delivering of the exams, in-person or virtual, affects how examiners rate candidates. DESIGN: We included every candidate-examiner combination for first time takers of the general surgery oral exam. Total scores and pass/fail outcomes based on the 4 scores given by examiners to candidates were analyzed using multilevel models, with candidates as random effects. Explanatory variables included the gender, race, and ethnicity of candidates and examiners, and the format of the exam (in-person or virtual). Candidates' first attempt scores on the ABS General Surgery Qualifying Exam (QE) were also included in the models to control for the baseline knowledge of the candidate. Three sets of models were evaluated for each demographic variable (gender, race, ethnicity) due to missingness in data. p-values and coefficients of determination R2 were used to quantify the statistical and practical significance of the model coefficients (an existent relationship between the explored variables on CE scores was considered statistically and practically significant if the p-value was lower than 0.01 and R2 higher than 1%). PARTICIPANTS: All first-time takers of the American Board of Surgery General Surgery Certifying Exam from 2016 to 2022 that had demographic data, and the examiners that participated in those exams. RESULTS: The number of candidates/examiners for the 3 sets of models was 8665/514 (gender), 5906/465 (race), and 4678/295 (ethnicity). The demographic variables, format of the exam, or their interactions were not found to significantly relate to examiner-candidate ratings or pass/fail outcomes. The only variable that was significantly related to CE scores was candidates' QE scores, which was added to the models as a measure of candidates' initial knowledge; this held for all models for total scores (F[1,8659] = 1069.89, p-value < 0.01, R2 = 5% [gender models], F(1,5696.3) = 589.13, p-value < 0.01, R2 = 5% [race models], F(1,4459.5) = 278.33, p-value < 0.01, R2 = 5% [ethnicity models]), and pass/fail outcomes (CI = 1.61-1.73, p-value < 0.01, R2 = 3% [gender models], CI = 1.67-1.85, p-value < 0.01, R2 = 3% [race models], CI = 2.17-2.90, p-value < 0.01, R2 = 3% [ethnicity models]). CONCLUSIONS: This study shows that there is not a relationship between candidate and examiner gender, race, or ethnicity, and exam outcomes based on statistical models looking at examiner-candidate ratings and pass/fail outcomes. In addition, the delivery of the certifying exam in a virtual format appears to have no statistical impact on outcomes compared to in-person delivery. This suggests that the ABS is performing well in both demographic bias and virtual space.

When the Boards Fail Us-Structural Bias in Certification Examinations.

This essay shines a light on structural bias inherent to the board certification examination process, sharing the author's experience preparing and sitting for the examination while contending with co-occurring challenging life events.

2023-2024 American Board of Orthodontics practice analysis study: Part 1.

As a specialty board, the American Board of Orthodontics (ABO) serves to protect the public and the orthodontic specialty by certifying orthodontists. The demonstration of commitment to lifelong learning and self-improvement is critical to achieving the highest level of patient care. The ABO completed a practice analysis study in 2023 to ensure all examinations represent current assessments of proficiency in orthodontics at a level of quality that satisfies professional expectations. The practice analysis is essential to providing a demonstrable relationship between the examination content and orthodontic practice and provides a critical foundation for ABO's examination programs.

Preliminary Development of a Program ABSITE Dashboard (PAD) to Guide Curriculum Innovation.

PURPOSE: Medical Knowledge for general surgery residents' is assessed by the American Board of Surgery In- Training Examination (ABSITE). ASBITE score reports contain many metrics residency directors can utilize to assess resident progress and perform program evaluation. The purpose of this study was to develop a framework to evaluate program effectiveness in teaching specific subtest and subtopic areas of the ABSITE, using ABSITE score reports as an indicator. The aim is to demonstrate the identification of topic areas of weakness in program-wide performance on the ABSITE to guide proposed modification of the general surgery residency program curriculum, and to initiate development of a data visualizing dashboard to communicate these metrics. METHODS: A single institution retrospective study was performed utilizing ABSITE score reports from general surgery residents at a large academic training program from 2017 to 2020. ABSITE performance metrics from 320 unique records were entered into a database; statistical analysis for linear trends and variance were conducted for standard scores, subtest standard scores, and incorrect subtest topics. Deviation from national average scores were calculated by subtracting the national average score from each subtest score for each trainee. Data were displayed as medians or proportions and are displayed to optimize visualization as a proof-of-concept for the development of a program dashboard. RESULTS: Trends and variance in general surgery program and cohort performance on various elements of the ABSITE were visualized using figures and tables that represent a prototype for a program dashboard. Figure A1 demonstrates one example, in which a heatmap displays the median deviation from national average scores for each subtest by program year. Boxplots show the distribution of the deviation from national average, range for national average scores, and the recorded scores for each subtest by program year. Trends in median deviation from the national average scores are displayed for each program year paneled by subtest or for each exam year paneled by cohort. Median change in overall test scores from one program year to another in a cohort is visualized as a table. Bar graphs show the most often missed topics across all program years and heatmaps were generated showing the proportion of times each topic was missed for each subtest and exam year. CONCLUSIONS: We demonstrate use of ABSITE reports to identify specific thematic areas of opportunities for curriculum modification and innovation as an element of program evaluation. In this study we demonstrate, through data analysis and visualization, feasibility for the creation of a Program ABSITE Dashboard (PAD) that enhances the use of ABSITE reports for formative program evaluation and can guide modifications to surgery program curriculum and educational practices.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

INTRODUCTION: Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. METHODS: Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. RESULTS: After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. DISCUSSION: This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

How Does the Sequence of the American Board of Surgery Examinations Impact Pass/Fail Outcomes?

OBJECTIVE: Historically, the American Board of Surgery required surgeons to pass the qualifying examination (QE) before taking the certifying examination (CE). However, in the 2020-2021 academic year, with mitigating circumstances related to COVID-19, the ABS removed this sequencing requirement to facilitate the certification process for those candidates who were negatively impacted by a QE delivery failure. This decoupling of the traditional order of exam delivery has provided a natural comparator to the traditional route and an analysis of the impact of examination sequencing on candidate performance. METHODS: All candidates who applied for the canceled July 2020 QE were allowed to take the CE before passing the QE. The sample was then reduced to include only first-time candidates to ensure comparable groups for performance outcomes. Logistic regression was used to analyze the relationship between the order of taking the QE and the CE, controlling for other examination performance, international medical graduate status, and gender. RESULTS: Only first-time candidates who took both examinations were compared (n=947). Examination sequence was not a significant predictor of QE pass/fail outcomes, OR=0.54; 95% CI, 0.19-1.61, P =0.26. However, examination sequence was a significant predictor of CE pass/fail outcomes, OR=2.54; 95% CI, 1.46-4.68, P =0.002. CONCLUSIONS: This important study suggests that preparation for the QE increases the probability of passing the CE and provides evidence that knowledge may be foundational for clinical judgment. The ABS will consider these findings for examination sequencing moving forward.

Maintenance of Certification (MOC)-Benefit vs Burden.

This Viewpoint examines the demands of maintenance of certification (MOC) requirements from the ABIM on balance with the projected benefits to quality of patient care.

Performance of a Large Language Model on Practice Questions for the Neonatal Board Examination.

This Diagnostic/Prognostic Study evaluates the performance of a large language model in generating answers to practice questions for the neonatal-perinatal board examination.

The Role of Specialty Certification in Career-Long Competence.

Across the medical profession there is broad acceptance of the critical role of continuing medical education (CME) in enabling physicians to adapt to both new information and evolving expectations within the profession. In the presence of widespread participation in CME, some have attempted to question, discredit, or marginalize the role of ongoing lifelong assessment of physician knowledge and skills through specialty continuing certification, advocating instead for a participatory standard based only on engagement with CME. This essay outlines the limitations of physician self-evaluation and clarifies the need for external assessments. Certification boards' role is to set specialty-specific standards for competence, assess to those standards, and assure the public that certified physicians are adequately maintaining their skills and abilities; doing so credibly necessarily requires, in part, independent assessments of physician competence. In these contexts, the specialty boards are taking approaches to identify performance gaps and leverage intrinsic motivation to facilitate physician engagement in targeted learning. Specialty board continuing certification plays a unique role, distinct from and complementary to the CME enterprise. Calls to eliminate continuing certification requirements beyond self-directed CME are contradictory to the evidence and fail the profession and the public.

Cognitive perspectives on maintaining physicians' medical expertise: I. Reimagining Maintenance of Certification to promote lifelong learning.

Until recently, physicians in the USA who were board-certified in a specialty needed to take a summative test every 6-10 years. However, the 24 Member Boards of the American Board of Medical Specialties are in the process of switching toward much more frequent assessments, which we refer to as longitudinal assessment. The goal of longitudinal assessments is to provide formative feedback to physicians to help them learn content they do not know as well as serve an evaluation for board certification. We present five articles collectively covering the science behind this change, the likely outcomes, and some open questions. This initial article introduces the context behind this change. This article also discusses various forms of lifelong learning opportunities that can help physicians stay current, including longitudinal assessment, and the pros and cons of each.

Revisiting the relationship between examination pass rate and eligibility pathway among candidates for pharmacist board certification.

BACKGROUND: Pharmacists with specialized knowledge and skills are able to seek board certification from the Board of Pharmacy Specialties (BPS). In 2018, BPS conducted research to evaluate the relationship between eligibility pathways (i.e., completion of a postgraduate year 2 [PGY-2] specialty residency, completion of a postgraduate year 1 [PGY-1] residency plus 2 years of practice experience, or 4 years of practice experience) and certification examination pass rate. The study found statistically significant differences in pass rate based on eligibility criteria across all 6 specialties studied. There was a trend for higher pass rates in cohorts of examinees eligible for board certification based on completion of postgraduate residency training. OBJECTIVES: This study aimed to compare examination pass rates among different eligibility cohorts for board certification in recognized pharmacy practice specialties and compare contemporary findings with previously published findings. DESIGN: This cross-sectional study was conducted on data retrieved from BPS certification applications and examination administration records. SETTING AND PARTICIPANTS: Examinees in the United States and Canada for 9 BPS certification programs. OUTCOME MEASURES: A chi-square analysis was used to identify whether there were differences in pass rate among eligibility pathway cohorts. RESULTS: A total of 14,894 examinees met inclusion criteria; 6312 (42.4%) of the 14,894 examinees analyzed were eligible via practice experience, 5768 (38.7%) were eligible via PGY-1 completion, and 2814 (18.9%) were eligible via PGY-2 completion. A statistically significant difference was found for the relationship between pass rate and eligibility pathway for 8 of 9 BPS certification examinations analyzed (significant: ambulatory care pharmacy, critical care pharmacy, cardiology pharmacy, infectious diseases pharmacy, oncology pharmacy, psychiatric pharmacy, pediatric pharmacy, pharmacotherapy; not significant: solid organ transplantation pharmacy). Post hoc analyses showed that, in most cases, the PGY-2 eligibility cohort outperformed the PGY-1 eligibility cohort, which in turn outperformed the practice experience pathway cohort. CONCLUSION: Analysis of contemporary BPS certification examination administration data yields statistically significant differences among the pass rates by eligibility pathway, replicating previous findings and expanding the scope of the analysis.