The Anesthesiology Milestones 2.0: An Improved Competency-Based Assessment for Residency Training.

The evolution of medical education, from a time-based to a competency-based platform, began nearly 30 years ago and continues to slowly take shape. The development of valid and reproducible assessment tools is the first step. Medical educators across specialties acknowledge the challenges and remain motivated to develop a relevant, generalizable, and measurable system. The Accreditation Council for Graduate Medical Education (ACGME) remains committed to its responsibility to the public by assuring that the process and outcome of graduate medical education in the nation's residency programs produce competent, safe, and compassionate doctors. The Milestones Project is the ACGME's current strategy in the evolution to a competency-based system, which allows each specialty to develop its own set of subcompetencies and 5-level progression, or milestones, along a continuum of novice to expert. The education community has now had nearly 5 years of experience with these rubrics. While not perfect, Milestones 1.0 provided important foundational information and insights. The first iteration of the Anesthesiology Milestones highlighted some mismatch between subcompetencies and current and future clinical practices. They have also highlighted challenges with assessment and evaluation of learners, and the need for faculty development tools. Committed to an iterative process, the ACGME assembled representatives from stakeholder groups within the Anesthesiology community to develop the second generation of Milestones. This special article describes the foundational data from Milestones 1.0 that was useful in the development process of Milestones 2.0, the rationale behind the important changes, and the additional tools made available with this iteration.

First-Year Results of the American Board of Anesthesiology's Objective Structured Clinical Examination for Initial Certification.

In 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate an Objective Structured Clinical Examination (OSCE) into its initial certification examination system. Previously, the ABA's staged examination system consisted of 2 written examinations (the BASIC and ADVANCED examinations) and the Standardized Oral Examination (SOE). The OSCE and the existing SOE are now 2 separate components of the APPLIED Examination. This report presents the results of the first-year OSCE administration. A total of 1410 candidates took both the OSCE and the SOE in 2018. Candidate performance approximated a normal distribution for both the OSCE and the SOE, and was not associated with the timing of the examination, including day of the week, morning versus afternoon session, and order of the OSCE and the SOE. Practice-based Learning and Improvement was the most difficult station, while Application of Ultrasonography was the least difficult. The correlation coefficient between SOE and OSCE scores was 0.35 ([95% confidence interval {CI}, 0.30-0.39]; P < .001). Scores for the written ADVANCED Examination were modestly correlated with scores for the SOE (r = 0.29 [95% CI, 0.25-0.34]; P < .001) and the OSCE (r = 0.15 [95% CI, 0.10-0.20]; P < .001). Most of the candidates who failed the SOE passed the OSCE, and most of the candidates who failed the OSCE passed the SOE. Of the 1410 candidates, 77 (5.5%) failed the OSCE, 155 (11.0%) failed the SOE, and 25 (1.8%) failed both. Thus, 207 (14.7%) failed at least 1 component of the APPLIED Examination. Adding an OSCE to a board certification examination system is feasible. Preliminary evidence indicates that the OSCE measures aspects of candidate abilities distinct from those measured by other examinations used for initial board certification.

Development of an Objective Structured Clinical Examination as a Component of Assessment for Initial Board Certification in Anesthesiology.

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA's introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually "do" in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations-competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA's journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

The American Board of Anesthesiology's Standardized Oral Examination for Initial Board Certification.

The American Board of Anesthesiology (ABA) has been administering an oral examination as part of its initial certification process since 1939. Among the 24 member boards of the American Board of Medical Specialties, 13 other boards also require passing an oral examination for physicians to become certified in their specialties. However, the methods used to develop, administer, and score these examinations have not been published. The purpose of this report is to describe the history and evolution of the anesthesiology Standardized Oral Examination, its current examination development and administration, the psychometric model and scoring, physician examiner training and auditing, and validity evidence. The many-facet Rasch model is the analytic method used to convert examiner ratings into scaled scores for candidates and takes into account how difficult grader examiners are and the difficulty of the examination tasks. Validity evidence of the oral examination includes that it measures aspects of clinical performance not accounted for by written certifying examinations, and that passing the oral examination is associated with a decreased risk of subsequent license actions against the anesthesiologist. Explaining the details of the Standardized Oral Examination provides transparency about this component of initial certification in anesthesiology.