A Qualitative Analysis of Assessment and Evaluation Practices Within the Individualized Curriculum.

OBJECTIVE: The Accreditation Council of Graduate Medical Education requires an "individualized curriculum" (IC) in pediatric residency. A shared understanding across programs of methods to evaluate the IC is lacking. We explored pediatric program leaders' perceptions of assessment and evaluation within the IC to further understand and inform most useful practices. METHODS: We conducted a phenomenology study using semi-structured interviews to 1) determine what IC assessment and evaluation strategies are utilized in pediatric residency programs, and 2) explore program leaders' perceptions of the feasibility and value of assessment and evaluation in the IC. We recruited a purposive sample of leaders from 15 pediatric residency programs of various sizes and regions. Interviews were recorded and transcribed. Data were analyzed to produce themes. RESULTS: Three themes arose from our analysis: 1) Systematic assessment of the IC, though desired, is not robust in pediatric residency training; 2) Program differences present unique barriers and facilitators that impact the feasibility of assessment within the IC; and 3) Meaningfulness of assessment in the IC varies by stakeholder. Themes reflected perspectives on individual learner assessment and program evaluation. CONCLUSIONS: While systematic assessment and evaluation of the IC in pediatric residency training is lacking, program leaders desire feasible strategies that are meaningful to the resident, the program, and its leadership team. Leveraging current assessment and evaluation tools and aligning new assessment strategies could promote the integration of IC assessment with other assessments, minimizing burden. More structured IC assessment and evaluation could inform how to best achieve curricular goals of the IC.

Burnout and Association With Resident Performance as Assessed by Pediatric Milestones: An Exploratory Study.

INTRODUCTION: Prior work demonstrating that burnout is associated with decreased performance in medical trainees has relied on self-report and/or single-site studies. We explored the relationship between burnout status and Milestones-based scores in pediatric residents nationally. METHODS: In April to June 2016, we confidentially surveyed residents using the Maslach Burnout Inventory. Separately, programs submitted resident Milestones scores in June 2016. We examined the relationship between burnout and performance as assessed by Milestones scores for each domain of competence. We performed multivariate analysis to determine which components of burnout (depersonalization [DP], emotional exhaustion, and lack of personal accomplishment [PA]) were most impactful. RESULTS: About 1494 of 2368 (63%) residents at 32 programs completed the Maslach Burnout Inventory and had Milestones scores submitted. Residents who scored positive for burnout scored lower in all Milestones domains. Subgroup analysis demonstrated that this association was only significant (P < .05) in the post-graduate year 1 (PGY1) categorical pediatric cohort. In the PGY1 residents (n = 442), those positive for burnout had lower Milestones scores in patient care (PC) (2.78 vs 2.98), systems-based practice (2.69 vs 2.87), practice-based learning and improvement (2.77 vs 2.93), professionalism (3.09 vs 3.24), and interpersonal and communication skills (2.95 vs 3.12), but not medical knowledge. Multivariate analysis demonstrated that, in PGY1 residents, lower PC score was associated with lower PA and higher DP. CONCLUSIONS: Burnout is associated with decreased Milestones performance for pediatric PGY1 residents. DP and low PA were associated with lower PC scores in PGY1 residents. Future research should address whether strategies to mitigate burnout improve PGY1 performance.

Describing Overarching Curricular Goals for Individualized Education.

Phenomenon: The phenomenon of individualized education, an essential component of competency-based medical education, addresses individual learner needs while working toward standardized learning outcomes. One challenge with broadly implementing individualized education is the lack of a pragmatic operational definition. To formalize expectations for individualized education, the Accreditation Council of Graduate Medical Education in 2013 began requiring six months of individualized curriculum (IC) during pediatric residency; however, there is not a national standard of formal curricular goals for the IC as an educational entity. Examining and describing the overarching curricular goals of IC could provide a framework for discourse about and further study of individualized education in medicine across disciplines and the continuum of medical education. Thus, we aimed to describe the phenomenon of individualized education through the lens of the goals of the IC in pediatric residency in the United States. Approach: In 2017, a purposeful sample of Pediatric Residency leaders were recruited to represent a diverse sample of program sizes, regions of the country, and importance of the IC to the program leadership. They completed an online survey with open-ended questions describing formal and implicit goals of their program's IC. The authors analyzed responses initially using conventional content analysis, then investigated whether the themes for program goals aligned with any existing educational theory. The concepts and language aligned with the principles of self-determination theory (SDT); therefore, the IC goals were subsequently grouped using the SDT domains of relatedness, autonomy, and competence. A focus group with a subset of survey respondents was conducted for member checking and elaboration of concepts. Findings: Program leaders from a diverse sample of 36 programs participated in the survey and a subset of 11 programs participated in the focus group. The common goals across all programs are listed in parentheses and organized by domains of SDT: 1)Relatedness goals (engage in mentorship, select a career) cultivate resident's professional identity based on their desired future career path; 2)autonomy goals (create a learning plan, practice accountability) help residents plan their path; and fulfillment of their plans lead to 3)competence goals (develop targeted clinical skills and knowledge, ensure comprehensive exposure, address learning gaps), ensuring they have a comprehensive skill set for their chosen identity. Insights: This study provides a framework to describe the phenomenon of individualized education through the lens of IC curricular goals in pediatric residency. The goals for IC that emerged from this study serve as a pragmatic framework for implementation of individualized education. They provide a common language and structure to promote more rigorous and collaborative study of individualized education across programs, disciplines, and settings in medicine. They may function as a roadmap for learners to navigate educational activities and for programs to help shape the experiences of their learners and examine outcomes of individualized education in their programs. The framework can also help individual pediatric residency programs structure improvements to their IC. Disciplines beyond pediatrics may also use this framework to better structure elective experiences to capitalize on the benefits of individual education.

Pediatric Program Director Minimum Milestone Expectations Before Allowing Supervision of Others and Unsupervised Practice.

BACKGROUND: The Accreditation Council for Graduate Medical Education requires semiannual Milestone reporting on all residents. Milestone expectations of performance are unknown. OBJECTIVE: To determine pediatric program director (PD) minimum Milestone expectations for residents before being ready to supervise and before being ready to graduate. METHODS: Mixed methods survey of pediatric PDs on their programs' Milestone expectations before residents are ready to supervise and before they are ready to graduate, and in what ways PDs use Milestones to make supervision and graduation decisions. If programs had no established Milestone expectations, PDs indicated expectations they considered for use in their program. Mean minimum Milestone level expectations were adjusted for program size, region, and clustering of Milestone expectations by program were calculated for before supervise and before graduate. Free-text questions were analyzed using thematic analysis. RESULTS: The response rate was 56.8% (113 of 199). Most programs had no required minimum Milestone level before residents are ready to supervise (80%; 76 of 95) or ready to graduate (84%; 80 of 95). For readiness to supervise, minimum Milestone expectations PDs considered establishing for their program were highest for humanism (2.46; 95% confidence interval [CI], 2.21-2.71) and professionalization (2.37; 95% CI, 2.15-2.60). Minimum Milestone expectations for graduates were highest for help-seeking (3.14; 95% CI, 2.83-3.46). Main themes included the use of Milestones in combination with other information to assess learner performance and Milestones are not equally weighted when making advancement decisions. CONCLUSIONS: Most PDs have not established program minimum Milestones, but would vary such expectations according to competency.

Competent for Unsupervised Practice: Use of Pediatric Residency Training Milestones to Assess Readiness.

PURPOSE: To describe clinical skills progression during pediatric residency using the distribution of pediatric milestone assessments by subcompetency and year of training and to determine reasonable milestone expectations at time of graduation. METHOD: Multi-institutional cohort study of the milestones reported to the Accreditation Council for Graduate Medical Education for all 21 pediatric subcompetencies. Most subcompetencies were measured using five milestone levels (1 = novice, 2 = advanced beginner, 3 = competent, 4 = proficient, 5 = master); 3 subcompetencies had only four levels defined. RESULTS: Milestone assessments for 2,030 pediatric residents in 47 programs during academic year 2013-2014 were obtained. There was significant variation in end-of-year milestone ratings for residents within each level of training, which decreased as training level increased. Most (78.9%; 434/550) graduating third-year pediatric residents received a milestone rating of ≥ 3 in all 21 subcompetencies; fewer (21.1%; 116/550) received a rating of ≥ 4 in all subcompetencies. Across all training levels, professionalism and interpersonal communication skills were rated highest; quality improvement was rated lowest. CONCLUSIONS: Trainees entered residency with a wide range of skills. As they advanced, skill variability within a training level decreased. Most graduating pediatric residents were still advancing on the milestone continuum toward proficiency and mastery, and an expectation of milestone ratings ≥ 4 in all categories upon graduation is unrealistic; milestone ratings ≥ 3 upon graduation may be more realistic. Understanding current pediatric residents' and graduates' skills can help to identify key areas that should be specifically targeted during training.

The AAP Resilience in the Face of Grief and Loss Curriculum.

A career in pediatrics can bring great joy and satisfaction. It can also be challenging and lead some providers to manifest burnout and depression. A curriculum designed to help pediatric health providers acquire resilience and adaptive skills may be a key element in transforming times of anxiety and grief into rewarding professional experiences. The need for this curriculum was identified by the American Academy of Pediatrics Section on Medical Students, Residents and Fellowship Trainees. A working group of educators developed this curriculum to address the professional attitudes, knowledge, and skills essential to thrive despite the many stressors inevitable in clinical care. Fourteen modules incorporating adult learning theory were developed. The first 2 sections of the curriculum address the knowledge and skills to approach disclosure of life-altering diagnoses, and the second 2 sections focus on the provider's responses to difficult patient care experiences and their needs to develop strategies to maintain their own well-being. This curriculum addresses the intellectual and emotional characteristics patient care medical professionals need to provide high-quality, compassionate care while also addressing active and intentional ways to maintain personal wellness and resilience.

In Pursuit of Meaningful Use of Learning Goals in Residency: A Qualitative Study of Pediatric Residents.

PURPOSE: Medical education aims to equip physicians for lifelong learning, an objective supported by the conceptual framework of self-regulated learning (SRL). Learning goals have been used to develop SRL skills in learners across the medical education continuum. This study's purpose was to elicit residents' perspectives on learning goal use and to develop explanations suggesting how aspects of the learning environment may facilitate or hinder the meaningful use of learning goals in residency. METHOD: Resident focus groups and program director interviews were conducted in 2012-2013, audio-recorded, and transcribed. Programs were selected to maximize diversity of size, geographic location, type of program, and current use of learning goals. Data were analyzed using the constant comparative method associated with grounded theory. Further analysis compared themes frequently occurring together to strengthen the understanding of relationships between the themes. Through iterative discussions, investigators built a grounded theory. RESULTS: Ninety-five third-year residents and 12 program directors at 12 pediatric residency programs participated. The analysis identified 21 subthemes grouped into 5 themes: program support, faculty roles, goal characteristics and purposes, resident attributes, and accountability and goal follow-through. Review of relationships between the themes revealed a pyramid of support with program support as the foundation that facilitates the layers above it, leading to goal follow-through. CONCLUSIONS: Program support facilitates each step of the SRL process that leads to meaningful use of learning goals in residency. A strong foundation of program support should include attention to aspects of the implicit curriculum as well as the explicit curriculum.

Resident Self-Assessment and Learning Goal Development: Evaluation of Resident-Reported Competence and Future Goals.

OBJECTIVE: To determine incidence of learning goals by competency area and to assess which goals fall into competency areas with lower self-assessment scores. METHODS: Cross-sectional analysis of existing deidentified American Academy of Pediatrics' PediaLink individualized learning plan data for the academic year 2009-2010. Residents self-assessed competencies in the 6 Accreditation Council for Graduate Medical Education (ACGME) competency areas and wrote learning goals. Textual responses for goals were mapped to 6 ACGME competency areas, future practice, or personal attributes. Adjusted mean differences and associations were estimated using multiple linear and logistic regression. RESULTS: A total of 2254 residents reported 6078 goals. Residents self-assessed their systems-based practice (51.8) and medical knowledge (53.0) competencies lowest and professionalism (68.9) and interpersonal and communication skills (62.2) highest. Residents were most likely to identify goals involving medical knowledge (70.5%) and patient care (50.5%) and least likely to write goals on systems-based practice (11.0%) and professionalism (6.9%). In logistic regression analysis adjusting for postgraduate year (PGY), gender, and degree type (MD/DO), resident-reported goal area showed no association with the learner's relative self-assessment score for that competency area. In the conditional logistic regression analysis, with each learner serving as his or her own control, senior residents (PGY2/3+s) who rated themselves relatively lower in a competency area were more likely to write a learning goal in that area than were PGY1s. CONCLUSIONS: Senior residents appear to develop better skills and/or motivation to explicitly turn self-assessed learning gaps into learning goals, suggesting that individualized learning plans may help improve self-regulated learning during residency.

The pediatrics milestones: initial evidence for their use as learning road maps for residents.

OBJECTIVE: As the next step in competency-based medical education, the Pediatrics Milestone Project seeks to provide a learner-centered approach to training and assessment. To help accomplish this goal, this study sought to determine how pediatric residents understand, interpret, and respond to the Pediatrics Milestones. METHODS: Cognitive interviews with 48 pediatric residents from all training levels at 2 training programs were conducted. Each participant reviewed one Pediatrics Milestone document (PMD). Eight total Pediatrics Milestones, chosen for their range of complexity, length, competency domain, and primary author, were included in this study. Six residents, 2 from each year of residency training, reviewed each PMD. Interviews were transcribed and coded using inductive methods, and codes were grouped into themes that emerged. RESULTS: Four major themes emerged through coding and analysis: 1) the participants' degree of understanding of the PMDs is sufficient, often deep; 2) the etiology of participants' understanding is rooted in their experiences; 3) there are qualities of the PMD that may contribute to or detract from understanding; and 4) participants apply their understanding by noting the PMD describes a developmental progression that can provide a road map for learning. Additionally, we learned that residents are generally comfortable being placed in the middle of a series of developmental milestones. Two minor themes focusing on interest and practicality were also identified. CONCLUSIONS: This study provides initial evidence for the Pediatrics Milestones as learner-centered documents that can be used for orientation, education, formative feedback, and, ultimately, assessment.

Self-assessment on the competencies and reported improvement priorities for pediatrics residents.

BACKGROUND: Self-assessment and self-directed learning are essential to becoming an effective physician. OBJECTIVE: To identify factors associated with resident self-assessment on the competencies, and to determine whether residents chose areas of self-assessed relative weakness as areas for improvement in their Individualized Learning Plan (ILP). METHODS: We performed a cross-sectional analysis of the American Academy of Pediatrics' PediaLink ILP database. Pediatrics residents self-assessed their competency in the 6 Accreditation Council for Graduate Medical Education competencies using a color-coded slider scale with end anchors "novice" and "proficient" (0-100), and then chose at least 1 competency to improve. Multivariate regression explored the relationship between overall confidence in core competencies, sex, level of training, and degree (MD or DO) status. Correlation examined whether residents chose to improve competencies in which they rated themselves as lower. RESULTS: A total of 4167 residents completed an ILP in academic year 2009-2010, with residents' ratings improving from advanced beginner (48 on a 0-100 scale) in postgraduate year-1 residents (PGY-1s) to competent (75) in PGY-3s. Residents rated themselves as most competent in professionalism (mean, 75.3) and least competent in medical knowledge (mean, 55.8) and systems-based practice (mean, 55.2). In the adjusted regression model, residents' competency ratings increased by level of training and whether they were men. In PGY-3s, there was no difference between men and women. Residents selected areas for improvement that correlated to competencies where they had rated themselves lower (P < .01). CONCLUSION: Residents' self-assessment of their competencies increased by level of training, although residents rated themselves as least competent in medical knowledge and systems-based practice, even as PGY-3s. Residents tended to choose subcompetencies, which they rated as lower to focus on improving.