Development and Pilot Testing of a Programmatic System for Competency Assessment in US Anesthesiology Residency Training.

BACKGROUND: In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments. METHODS: Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution's clinical competency committee (CCC). RESULTS: New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation. CONCLUSIONS: A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.

A Consensus-Driven Revision of the Accreditation Council for Graduate Medical Education Case Log System: Pediatric Anesthesiology Fellowship Education.

BACKGROUND: Clinical experiences, quantified by case logs, are an integral part of pediatric anesthesiology fellowship programs. Accreditation of pediatric anesthesiology fellowships by the Accreditation Council of Graduate Medical Education (ACGME) and establishment of case log reporting occurred in 1997 and 2009, respectively. The specialty has evolved since then, but the case log system remains largely unchanged. The Pediatric Anesthesiology Program Directors Association (PAPDA) embarked on the development of an evidence-based case log proposal through the efforts of a case log task force (CLTF). This proposal was part of a larger consensus-building process of the Society for Pediatric Anesthesia (SPA) Task Force for Pediatric Anesthesiology Graduate Medical Education. The primary aim of case log revision was to propose an evidence-based, consensus-driven update to the pediatric anesthesiology case log system. METHODS: This study was executed in 2 phases. The CLTF, composed of 10 program directors representing diverse pediatric anesthesiology fellowship programs across the country, utilized evidence-based literature to develop proposed new categories. After an approval vote by PAPDA membership, this proposal was included in the nationally representative, stakeholder-based Delphi process executed by the SPA Task Force on Graduate Medical Education. Thirty-seven participants engaged in this Delphi process, during which iterative rounds of surveys were used to select elements of the old and newly proposed case logs to create a final revision of categories and minimums for updated case logs. The Delphi methodology was used, with a two-thirds agreement as the threshold for inclusion. RESULTS: Participation in the Delphi process was robust, and consensus was almost completely achieved by round 2 of 3 survey rounds. Participants suggested that total case minimums should increase from 240 to 300 (300-370). Participants agreed (75.86%) that the current case logs targeted the right types of cases, but requirements were too low (82.75%). They also agreed (85.19%) that the case log system and minimums deserved an update, and that this should be used as part of a competency-based assessment in pediatric anesthesia fellowships (96%). Participants supported new categories and provided recommended minimum numbers. CONCLUSIONS: The pediatric anesthesiology case log system continues to have a place in the assessment of fellowship programs, but it requires an update. This Delphi process established broad support for new categories and benchmarked minimums to ensure the robustness of fellowship programs and to better prepare the pediatric anesthesiology workforce of the future for independent clinical practice.

The Anesthesiology Milestones 2.0: An Improved Competency-Based Assessment for Residency Training.

The evolution of medical education, from a time-based to a competency-based platform, began nearly 30 years ago and continues to slowly take shape. The development of valid and reproducible assessment tools is the first step. Medical educators across specialties acknowledge the challenges and remain motivated to develop a relevant, generalizable, and measurable system. The Accreditation Council for Graduate Medical Education (ACGME) remains committed to its responsibility to the public by assuring that the process and outcome of graduate medical education in the nation's residency programs produce competent, safe, and compassionate doctors. The Milestones Project is the ACGME's current strategy in the evolution to a competency-based system, which allows each specialty to develop its own set of subcompetencies and 5-level progression, or milestones, along a continuum of novice to expert. The education community has now had nearly 5 years of experience with these rubrics. While not perfect, Milestones 1.0 provided important foundational information and insights. The first iteration of the Anesthesiology Milestones highlighted some mismatch between subcompetencies and current and future clinical practices. They have also highlighted challenges with assessment and evaluation of learners, and the need for faculty development tools. Committed to an iterative process, the ACGME assembled representatives from stakeholder groups within the Anesthesiology community to develop the second generation of Milestones. This special article describes the foundational data from Milestones 1.0 that was useful in the development process of Milestones 2.0, the rationale behind the important changes, and the additional tools made available with this iteration.

Medical simulation utilization among pediatric anesthesiology fellowship programs.

BACKGROUND: Simulation-based education is a mainstay in education of pediatric anesthesiology trainees. Despite the known benefits, there is variability in its use and availability among various pediatric anesthesiology fellowship programs. AIM: The primary aim was to understand the current state of simulation-based education among pediatric anesthesiology fellowship programs and define barriers that impede the development of an effective simulation program. METHODS: This survey-based, observational study of simulation activities within United States-based pediatric anesthesiology fellowship programs was approved by the Institutional Review Boards (IRB) of the authors' institutions. A 35-question survey was developed in an iterative manner by simulation educators (AA, WW, DY) and a statistician familiar with survey-based research (AN) using research electronic data capture (REDCap) for tool development and data collation. Descriptive and thematic analyses were performed on the quantitative and qualitative responses in the survey, respectively, and were stratified with small, medium, and large fellowship programs. RESULTS: Forty-five of 60 (75%) fellowship programs responded to the survey. The presence of a dedicated simulation program director and number of simulation instructors was positively associated with the size of program and years in operation. Dedicated simulation support was variable across programs and was usually present within the larger programs. A positive association also existed for educational activities among all programs mostly based on size of program and years in operation. Protected time was the most commonly cited barrier to having a comprehensive and sustainable simulation program. There was general agreement for establishing a standardized and shared curriculum among fellowship programs. Approximately 70% of simulation programs had no formal simulation instructor training requirement. CONCLUSIONS: Simulation-based curricula are broadly offered by many fellowship programs. Improved collaboration locally, regionally, and nationally may improve educational opportunities for fellowship programs, particularly the small ones. These efforts may begin with the development of a standardized curriculum and formal instructor training programs.