Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment.

BACKGROUND: Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment. OBJECTIVES: We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME). METHODS: In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale. RESULTS: One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05). CONCLUSIONS: We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activities Across Subspecialties.

OBJECTIVE: To determine the relationship between level of supervision (LOS) ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) with their associated subcompetency milestones across subspecialties and by fellowship training year. METHODS: Clinical Competency Committees (CCCs) in 14 pediatric subspecialties submitted LOS ratings for 6 Common Subspecialty EPAs and subcompetency milestone levels mapped to these EPAs. We examined associations between these subcompetency milestone levels and LOS ratings across subspecialty training year by fitting per-EPA linear mixed effects models, regressing LOS rating on milestone level and on training year. RESULTS: CCCs from 211 pediatric fellowship programs provided data for 369 first, 336 second, and 331 third year fellows. Mean subcompetency milestone levels increased similarly among subspecialties for most EPAs compared with the reference, Adolescent Medicine. Mean subcompetency milestones mapped to each EPA and mean EPA LOS ratings generally increased by training year across all subspecialties. CONCLUSIONS: Subcompetency milestones levels mapped to each Common Subspecialty EPA and the EPA LOS ratings increase similarly across subspecialties and by training year, providing validity evidence for using EPA LOS to assess pediatric subspecialty trainee performance. This study supports the development of tools to facilitated the CCC evaluation process across all pediatric subspecialties.

Achieving Entrustable Professional Activities During Fellowship.

BACKGROUND AND OBJECTIVES: Entrustable Professional Activities (EPAs) were developed to assess pediatric fellows. We previously showed that fellowship program directors (FPDs) may graduate fellows who still require supervision. How this compares with their expectations for entrustment of practicing subspecialists is unknown. METHODS: We surveyed US FPDs in 14 pediatric subspecialties through the Subspecialty Pediatrics Investigator Network between April and August 2017. For each of 7 common pediatric subspecialty EPAs, we compared the minimum level of supervision that FPDs required for graduation with the level they expected of subspecialists for safe and effective practice using the Friedman rank sum test and paired t test. We compared differences between subspecialties using linear regression. RESULTS: We collected data from 660 FPDs (response rate 82%). For all EPAs, FPDs did not require fellows to reach the level of entrustment for graduation that they expected of subspecialists to practice (P < .001). FPDs expected the least amount of supervision for the EPAs consultation and handovers. Mean differences between supervision levels for graduation and practice were smaller for clinical EPAs (consultation, handovers, lead a team) when compared with nonclinical EPAs (quality improvement, management, lead the profession and scholarship; P = .001) and were similar across nearly all subspecialties. CONCLUSIONS: Fellowship graduates may need continued development of clinical and nonclinical skills in their early practice period, underscoring a need for continued assessment and mentoring. Graduation readiness must be based on clear requirements, with alignment of FPD expectations and regulatory standards, to ensure quality care for patients.

Competencies, Milestones, and a Level of Supervision Scale for Entrustable Professional Activities for Scholarship.

PROBLEM: Scholarship is an important element of both undergraduate and graduate medical education, and scholarly activity is required for all pediatric fellows. However, despite the creation of entrustable professional activities (EPAs) for scholarship, the specific progressive levels of performance and the appropriate level of supervision for a given performance level have not been defined. The authors developed competencies and milestones for the scholarship EPA to provide a framework for assessment across the continuum; a level of supervision scale was also developed. APPROACH: The Vitae Researcher Development Framework served as a template to create the competencies and milestones for the scholarship EPA. Beginning in September 2015 and using a modified Delphi approach, three drafts were circulated to individuals with expertise in various types of scholarship until broad agreement was achieved. Then, in October 2016, the Steering Committee of the Subspecialty Pediatrics Investigator Network created a level of supervision scale, modeled after one it had previously developed. OUTCOMES: Eight competencies were identified as important in making entrustment decisions related to scholarship. For each competency, five milestone levels that span the continuum from novice to expert were created. A supervision scale with five progressive levels of entrustment was also created. NEXT STEPS: Next steps include a study to obtain validity evidence for the supervision scale and determine the correlation between milestone and supervision levels. These competencies, milestones, and supervision levels can potentially serve as a road map for trainees and junior faculty and also play a role in the assessment of physician-scientists.

A novel workplace-based assessment for competency-based decisions and learner feedback.

BACKGROUND: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. METHODS: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. RESULTS: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4-8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. CONCLUSIONS: Outcomes - including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment - provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities.

PURPOSE: Entrustable professional activities (EPAs) represent the routine and essential activities that physicians perform in practice. Although some level of supervision scales have been proposed, they have not been validated. In this study, the investigators created level of supervision scales for EPAs common to the pediatric subspecialties and then examined their validity in a study conducted by the Subspecialty Pediatrics Investigator Network (SPIN). METHOD: SPIN Steering Committee members used a modified Delphi process to develop unique scales for six of the seven common EPAs. The investigators sought validity evidence in a multisubspecialty study in which pediatric fellowship program directors and Clinical Competency Committees used the scales to evaluate fellows in fall 2014 and spring 2015. RESULTS: Separate scales for the six EPAs, each with five levels of progressive entrustment, were created. In both fall and spring, more than 300 fellows in each year of training from over 200 programs were assessed. In both periods and for each EPA, there was a progressive increase in entrustment levels, with second-year fellows rated higher than first-year fellows (P < .001) and third-year fellows rated higher than second-year fellows (P < .001). For each EPA, spring ratings were higher (P < .001) than those in the fall. Interrater reliability was high (Janson and Olsson's iota = 0.73). CONCLUSIONS: The supervision scales developed for these six common pediatric subspecialty EPAs demonstrated strong validity evidence for use in EPA-based assessment of pediatric fellows. They may also inform the development of scales in other specialties.

From Flexner to competencies: reflections on a decade and the journey ahead.

This article is a sequel to one published in 2002 only a few years after the initiation of the shift to competency-based medical education (CBME). The authors reflect on the major forces that have influenced the movement and tipped the balance toward widespread adoption of CBME in the United States, primarily in graduate medical education. These forces include regulatory bodies, international counterparts, and the general public. The authors highlight the most important lessons learned over the decade. These include (1) the need for standardization of language to develop a shared vision of the path ahead, (2) the power of direct observation in assessment, (3) the challenge of developing meaningful measures of performance, (4) desired outcomes as the starting point for curriculum development, (5) dependence on reflection in the development of expertise, (6) the need for exploiting the role of learners in their learning, and (7) competent clinical systems as the required learning environment for producing competent physicians.The authors speculate on why this most recent attempt to shift to CBME differs from previous aborted attempts. They conclude by explaining how the recent lessons learned inform the vision of what successful implementation of CBME would look like, and discussing the importance of milestones, entrustable professional activities, and an integrated, rather than a reductionist, approach to assessment of competence. The fundamental question at each step along the way in implementing CBME should be "How do we improve medical education to provide better care for patients?"

Perspective: Competencies, outcomes, and controversy--linking professional activities to competencies to improve resident education and practice.

Regulatory organizations have recently emphasized the importance of structuring graduate medical education around mastery of core competencies. The difficulty is that core competencies attempt to distill a range of professional behaviors into arguable abstractions. As such, competencies can be difficult to grasp for trainees and faculty, who see them as unrelated to the intricacies of daily patient care. In this article, the authors describe how two initiatives are converging in a way that should make competencies tangible and relevant. One initiative is based on the idea that competencies will be more meaningful if trainees understand specifically how they relate to important professional activities in their own specialty. The authors suggest that there is a dyadic relationship between competencies and major professional activities in pediatric medicine. They also suggest that these relationships should be discussed as part of the process by which trainees are entrusted to perform clinical activities without direct supervision. The other initiative proposes to construct narrative milestones that provide a picture of what progression toward mastery of core competencies might look like. Together, the authors argue, these two initiatives should illuminate the core competencies by providing relevant clinical context and valuable educational substance.

The Residency Review and Redesign in Pediatrics (R3P) Project: roots and branches.

The Residency Review and Redesign in Pediatrics Project was fortunate to be able to take advantage of careful thinking by others. In addition to pediatricians, we sought advice from medical educators across the spectrum of medicine, especially in internal and family medicine. Participants in the project concluded early on that top-down "redesign" of pediatric resident education was neither realistic nor appropriate. A better and more durable alternative is a formal process by which residency education can learn and evolve over time. By committing to that model, pediatrics would finally carry out the mandate of the 1978 Task Force on Pediatric Education.

From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills.

The Accreditation Council for Graduate Medical Education Outcome Project has shifted the focus of residents' education to competency-based outcomes of learning. The challenge of meaningful assessment of learner competence has stimulated interest in the Dreyfus and Dreyfus Model, a framework for assessing skill acquisition that describes developmental stages beginning with novice and progressing through advanced beginner, competent, proficient, expert, and master. Many educators have adopted this model, but no consensus about its adaptation to clinical medicine has been documented. In this article, the authors seek to integrate generally accepted knowledge and beliefs about how one learns to practice clinical medicine into a coherent developmental framework using the Dreyfus and Dreyfus model of skill acquisition. Using the general domain of patient care, the characteristics and skills of learners at each stage of development are translated into typical behaviors. A tangible picture of this model in real-world practice is provided through snapshots of typical learner performance at discrete moments in time along the developmental continuum. The Dreyfus and Dreyfus model is discussed in the context of other developmental models of assessment of learner competence. The limitations of the model, in particular the controversy around the behaviors of "experts," are discussed in light of other interpretations of expertise in the literature. Support for descriptive developmental models of assessment is presented in the context of a discussion of the deconstructing versus reconstructing of competencies.

Teaching residents systems-based practice through a hospital cost-reduction program: a "win-win" situation.

BACKGROUND: The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project challenges residency programs to teach and evaluate competence in 6 domains, including systems-based practice (SBP). One element of SBP is to "practice cost-effective health care ... that does not compromise quality." Institutional cost-reduction efforts included targeting laboratory fees by using point-of-care testing with an iSTAT machine rather than in the central laboratory. Predicted cost savings were 50,000 US dollars per month. Because residents are primary users of laboratory resources, the authors engaged them in the process and implementation of system redesign. DESCRIPTION: The residents identified barriers to use of the iSTAT and solutions to overcome those barriers. EVALUATION: As a result of resident solutions, use of the iSTATwent from 40% of potential iSTATtests being ordered on the iSTAT to 98%, with an estimated 549,780 US dollars per year in savings to the hospital. CONCLUSION: Involving residents in hospital cost-reduction efforts allows both teaching and evaluation of competence in SBP while realizing significant cost savings.