Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process.

BACKGROUND AND OBJECTIVES: Entrustable professional activities (EPAs) will be used for initial certification by the American Board of Pediatrics by 2028. Less than half of pediatric fellowships currently use EPAs for assessment, yet all will need to adopt them. Our objectives were to identify facilitators and barriers to the implementation of EPAs to assess pediatric fellows and to determine fellowship program directors' (FPD) perceptions of EPAs and Milestones. METHODS: We conducted a survey of FPDs from 15 pediatric subspecialties. EPA users were asked about their implementation of EPAs, barriers encountered, and perceptions of EPAs. Nonusers were queried about deterrents to using EPAs. Both groups were asked about potential facilitators of implementation and their perceptions of Milestones. RESULTS: The response rate was 65% (575/883). Of these, 344 (59.8%) were EPA users and 231 (40.2%) were nonusers. Both groups indicated work burden as a barrier to implementation. Nonusers reported more barriers than users (mean [SD]: 7 [3.8] vs 5.8 [3.4], P < .001). Both groups identified training materials and premade assessment forms as facilitators to implementation. Users felt that EPAs were easier to understand than Milestones (89%) and better reflected what it meant to be a practicing subspecialty physician (90%). In contrast, nonusers felt that Milestones were easy to understand (57%) and reflected what it meant to be a practicing subspecialist (58%). CONCLUSIONS: Implementing EPA-based assessment will require a substantial investment by FPDs, facilitated by guidance and easily accessible resources provided by multiple organizations. Perceived barriers to be addressed include FPD time constraints, a need for additional assessment tools, and outcomes data.

Analysis of fellowship program director opinions of entrustable professional activities in adolescent medicine fellowship.

OBJECTIVES: This study aimed to explore the minimum entrustable professional activity (EPA) supervision levels at which pediatric fellowship program directors (FPDs) would be willing to graduate fellows and the levels deemed necessary for safe and effective practice for each of the common pediatric subspecialty and the four adolescent medicine-specific EPAs. METHODS: This cross-sectional study utilized survey data from pediatric FPDs in 2017. FPDs indicated the minimum level of supervision (LOS) for fellows at graduation and for safe and effective practice. RESULTS: 82 percent (23/28) of adolescent medicine FPDs completed the survey. For each EPA, there were differences (p<0.05) between LOS expected for graduation and for safe and effective practice. There was also variability in the level at which FPDs would graduate fellows. CONCLUSIONS: This study summarizes pediatric FPD opinions regarding the minimum levels of supervision required for fellows at the time of graduation as well as the levels deemed necessary for safe and effective practice. The difference between the minimum LOS at which FPDs would graduate a fellow and that deemed appropriate for safe and effective practice, along with variability in minimum LOS for graduation, highlight the need for clearer standards for fellowship graduation as well as more structured early career support for ongoing learning. These data highlight variability in FPD opinion regarding such expectations and both the need to better define desired training outcomes and potential need for post-graduation supervision in clinical practice.

Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment.

BACKGROUND: Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment. OBJECTIVES: We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME). METHODS: In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale. RESULTS: One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05). CONCLUSIONS: We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Clinical competency committee perceptions of entrustable professional activities and their value in assessing fellows: A qualitative study of pediatric subspecialty program directors.

OBJECTIVES: To examine the composition and processes of Clinical Competency Committees (CCCs) assigning entrustable professional activity (EPA) levels of supervision for pediatric subspecialty fellows and to examine fellowship program director (FPD) perspectives about using EPAs to determine fellows' graduation readiness. METHODS: A qualitative study was performed using one-on-one interviews with a purposeful sample of pediatric subspecialty FPDs to yield a thematic analysis. Semi-structured interview guides were used for participants who self-identified as EPA users or non-users. Inductive analysis and coding were performed on transcripts until theoretical sufficiency was attained. RESULTS: Twenty-eight FPDs were interviewed. There was significant variability in the composition and processes of CCCs across subspecialties. FPDs felt that CCCs intuitively understand what entrustment means, allowing for ease of application of level of supervision (LOS) scales and consensus. FPDs perceived that EPAs provided a global assessment of fellows and are one tool to determine graduation readiness. CONCLUSIONS: Although there was variability in the makeup and processes of CCCs across subspecialties, FPDs believe EPAs are intuitive and relatively easy to implement. Consensus can be reached easily using EPA-specific LOS scales focusing on entrustment. FPDs desire a better understanding of how EPAs should be used for graduation.

Pediatric Emergency Medicine Fellowship Program Directors' Viewpoint: Minimum Levels of Entrustment for Graduating Fellows and Practicing Physicians to Perform the Subspecialty's Professional Activities.

OBJECTIVES: Entrustable Professional Activities (EPAs) are essential tasks physicians perform within their professions. Entrustment levels that pediatric emergency medicine (PEM) fellowship program directors (FPDs) expect graduating fellows to achieve for PEM-specific and common pediatric subspecialty EPAs remain unreported. This study aims to determine minimum entrustment levels FPDs require fellows to achieve to graduate from fellowship and to compare FPD expectations for fellows versus practicing PEM physicians. METHODS: Secondary analysis of PEM-specific data from a national multispecialty cross-sectional survey of pediatric subspecialty FPDs. For 6 PEM-specific and 7 common pediatric subspecialty EPAs, PEM FPDs indicated (1) minimum entrustment levels fellows should achieve by training completion, (2) whether they would allow a fellow to graduate below these minimum levels, and (3) minimum levels for safe and effective practice by PEM physicians. Minimum levels were defined as the level that more than 80% of FPDs would not drop below. RESULTS: Sixty of 77 PEM FPDs (78%) completed the survey. Most respondents did not require fellows to achieve the highest level (level 5-no supervision) by graduation for any PEM-specific EPAs. The median level FPDs expected for practicing PEM physicians was 5 (trusted to perform without supervision) for EPAs 1 and 4 and level 4 (indirect supervision for complex cases) for the remaining PEM-specific EPAs. Minimum levels expected by FPDs for common subspecialty pediatric EPAs were lower for both groups. CONCLUSIONS: Most PEM FPDs indicated that they would graduate fellows before their achievement of the highest entrustment level for all EPAs. Most also indicated that they do not expect practicing PEM physicians to perform all EPAs without supervision. These findings indicate need for stakeholders to evaluate current structure and outcomes of PEM fellowship programs and for institutions and organizations to ensure adequate support in time and resources for ongoing learning for practicing PEM physicians.

Psychosis-Spectrum Screening and Assessment Within a College Counseling Center: A Pilot Study Exploring Feasibility and Clinical Need.

Evidence supports the use of brief psychosis-spectrum screening tools for identifying individuals at an increased risk of developing a psychotic disorder. Screening has not been well studied in general mental health settings that serve young adults in the age range associated with highest risk for psychosis. This study explored the feasibility of psychosis-risk screening and assessment among help-seeking students at a university counseling center. The PRIME Screen-Revised was administered to students at clinic intake. Participants who screened positively were offered a follow-up assessment using the Structured Interview for Psychosis-risk Syndromes (SIPS). At intake, 510 students completed the PRIME Screen-Revised, with 132 (25.9%) screening positive. Comprehensive psychosis-spectrum evaluations were completed with 38 participants, and 22 met criteria for a psychosis-spectrum disorder, representing 57.9% of this subsample. Findings suggest that psychosis-risk screening in a college clinic is a promising approach to identifying those at high risk for or in the early stages of psychosis.

Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activities Across Subspecialties.

OBJECTIVE: To determine the relationship between level of supervision (LOS) ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) with their associated subcompetency milestones across subspecialties and by fellowship training year. METHODS: Clinical Competency Committees (CCCs) in 14 pediatric subspecialties submitted LOS ratings for 6 Common Subspecialty EPAs and subcompetency milestone levels mapped to these EPAs. We examined associations between these subcompetency milestone levels and LOS ratings across subspecialty training year by fitting per-EPA linear mixed effects models, regressing LOS rating on milestone level and on training year. RESULTS: CCCs from 211 pediatric fellowship programs provided data for 369 first, 336 second, and 331 third year fellows. Mean subcompetency milestone levels increased similarly among subspecialties for most EPAs compared with the reference, Adolescent Medicine. Mean subcompetency milestones mapped to each EPA and mean EPA LOS ratings generally increased by training year across all subspecialties. CONCLUSIONS: Subcompetency milestones levels mapped to each Common Subspecialty EPA and the EPA LOS ratings increase similarly across subspecialties and by training year, providing validity evidence for using EPA LOS to assess pediatric subspecialty trainee performance. This study supports the development of tools to facilitated the CCC evaluation process across all pediatric subspecialties.

Continued Supervision for the Common Pediatric Subspecialty Entrustable Professional Activities May Be Needed Following Fellowship Graduation.

PURPOSE: Entrustable professional activities (EPAs) are one approach to competency-based medical education (CBME), and 7 EPAs have been developed that address content relevant for all pediatric subspecialties. However, it is not known what level of supervision fellowship program directors (FPDs) deem necessary for graduation. The Subspecialty Pediatrics Investigator Network (SPIN) investigated FPD perceptions of the minimum level of supervision required for a trainee to successfully graduate. METHOD: In 2017, SPIN surveyed all FPDs of accredited fellowships for 14 subspecialties. For each EPA, the minimum supervision level for graduation (ranging from observation only to unsupervised practice) was set such that no more than 20% of FPDs would accept a lower level. RESULTS: The survey response rate was 82% (660/802). The minimum supervision level for graduation varied across the 7 EPAs from 2 (direct) to 4 (indirect for complex cases), with significant differences between EPAs. The percentage of FPDs desiring a lower minimum supervision level ranged from 3% to 17%. Compared with the 4 nonclinical EPAs (quality improvement, management, lead within the profession, scholarship), the 3 clinical EPAs (consultation, handover, lead a team) had higher minimum supervision graduation levels (P < .001), with less likelihood that an FPD would graduate a learner below their minimum level (P < .001). CONCLUSIONS: Consensus among FPDs across all pediatric subspecialties demonstrates the potential need for ongoing supervision for graduates in all 7 common pediatric subspecialty EPAs after fellowship. As CBME programs are implemented, processes and infrastructure to support new graduates are important considerations for leaders.

Learner Levels of Supervision Across the Continuum of Pediatrics Training.

PURPOSE: To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship. METHOD: The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings. RESULTS: Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency. CONCLUSIONS: Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities.

Importance: Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective: To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants: Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions: Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures: Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results: Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance: This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities.

PURPOSE: Entrustable professional activities (EPAs) represent the routine and essential activities that physicians perform in practice. Although some level of supervision scales have been proposed, they have not been validated. In this study, the investigators created level of supervision scales for EPAs common to the pediatric subspecialties and then examined their validity in a study conducted by the Subspecialty Pediatrics Investigator Network (SPIN). METHOD: SPIN Steering Committee members used a modified Delphi process to develop unique scales for six of the seven common EPAs. The investigators sought validity evidence in a multisubspecialty study in which pediatric fellowship program directors and Clinical Competency Committees used the scales to evaluate fellows in fall 2014 and spring 2015. RESULTS: Separate scales for the six EPAs, each with five levels of progressive entrustment, were created. In both fall and spring, more than 300 fellows in each year of training from over 200 programs were assessed. In both periods and for each EPA, there was a progressive increase in entrustment levels, with second-year fellows rated higher than first-year fellows (P < .001) and third-year fellows rated higher than second-year fellows (P < .001). For each EPA, spring ratings were higher (P < .001) than those in the fall. Interrater reliability was high (Janson and Olsson's iota = 0.73). CONCLUSIONS: The supervision scales developed for these six common pediatric subspecialty EPAs demonstrated strong validity evidence for use in EPA-based assessment of pediatric fellows. They may also inform the development of scales in other specialties.

Essentials of PEM Fellowship Part 2: The Profession in Entrustable Professional Activities.

This article is the second in a 7-part series that aims to comprehensively describe the current state and future directions of pediatric emergency medicine (PEM) fellowship training from the essential requirements to considerations for successfully administering and managing a program to the careers that may be anticipated upon program completion. This article describes the development of PEM entrustable professional activities (EPAs) and the relationship of these EPAs with existing taxonomies of assessment and learning within PEM fellowship. It summarizes the field in concepts that can be taught and assessed, packaging the PEM subspecialty into EPAs.

Validation of the Pittsburgh Infant Brain Injury Score for Abusive Head Trauma.

BACKGROUND: Abusive head trauma is the leading cause of death from physical abuse. Misdiagnosis of abusive head trauma as well as other types of brain abnormalities in infants is common and contributes to increased morbidity and mortality. We previously derived the Pittsburgh Infant Brain Injury Score (PIBIS), a clinical prediction rule to assist physicians deciding which high-risk infants should undergo computed tomography of the head. METHODS: Well-appearing infants 30 to 364 days of age with temperature <38.3°C, no history of trauma, and a symptom associated with an increased risk of having a brain abnormality were eligible for enrollment in this prospective, multicenter clinical prediction rule validation. By using a predefined neuroimaging paradigm, subjects were classified as cases or controls. The sensitivity, specificity, and negative and positive predictive values of the rule for prediction of brain injury were calculated. RESULTS: A total of 1040 infants were enrolled: 214 cases and 826 controls. The 5-point PIBIS included abnormality on dermatologic examination (2 points), age ≥3.0 months (1 point), head circumference >85th percentile (1 point), and serum hemoglobin <11.2g/dL (1 point). At a score of 2, the sensitivity and specificity for abnormal neuroimaging was 93.3% (95% confidence interval 89.0%-96.3%) and 53% (95% confidence interval 49.3%-57.1%), respectively. CONCLUSIONS: Our data suggest that the PIBIS accurately identifies infants who would benefit from neuroimaging to evaluate for brain injury. An implementation analysis is needed before the PIBIS can be integrated into clinical practice.

A theoretical assessment of a thermal technique to measure acoustic power radiated by ultrasound transducers.

The parameters affecting the temperature rise in an insonified absorber are studied computationally. Finite-element and analytical solutions are obtained for the transient energy equation in a cylindrical absorber. When the ultrasound beam radius is less than the radius of the absorber, the temperature field is seen to be considerably more complex than when the absorber cross section is uniformly heated. Circumstances in which power predictions based upon uniform heating would result in appreciable error are identified. The rise time required to achieve equilibrium is studied as a function of operational parameters, including absorber geometry and thermal properties as well as ultrasound beamwidth and frequency. The rise time is seen to increase approximately as the square of the absorber length, while optimized temperature rise increases linearly with absorber length, demonstrating a tradeoff in ultrasound power determination via equilibrium temperature measurements: longer lengths produce higher sensitivity, but also longer times before measurements can be made. A transient technique that may bypass this tradeoff is suggested.