A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States.

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Impact of a Regional Grant Program Through the Lens of Social Cognitive Career Theory: A Mixed-Method Evaluation.

PURPOSE: Evaluations of educational grant programs have focused on research productivity, with few examining impacts on grantees or effective program characteristics. This evaluation examined the regional grant program sponsored by Group on Educational Affairs to examine if and how grantees' careers were affected by funding, and if these experiences aligned with program goals. METHOD: In this concurrent, mixed-methods theory-driven evaluation, quantitative and qualitative data were analyzed independently and then integrated to examine complementarity. Quantitative data examined differences among 4 geographic regions and included proposal and grantee characteristics abstracted from administrative records of 52 funded proposals from 2010-2015 grant cycles. Qualitative data from 23 interviews conducted from 2018 to 2019 explored the impact on grantees, with Social Cognitive Career Theory (SCCT) serving as a framework for deductive thematic analysis. To facilitate integration of findings, quantitative data were layered onto each interview to permit exploration of associations between the 2 data types. RESULTS: Although significant regional differences existed in project length and amount of funding, there were few regional differences in grantee experiences. Despite small funding amounts, grants were perceived as career launching pads. The SCCT framework accounted for grantee experiences, including researcher identity formation and subsequent research, but did not capture collaboration phenomena. Integration of the 2 data types identified experience patterns unique to different groups of grantees (e.g., more or less research experience). The diversity among grantees suggests that clarification of program goals and stronger alignment with criteria for funding may be warranted. CONCLUSIONS: This evaluation illuminates why small educational grant programs may or may not impact interest and productivity in research. Implications exist for funders, including clarifying program goals and providing support for less experienced grantees. Future research should explore grantee subsets (e.g., underrepresented in medicine) to further identify what fosters or inhibits careers of medical education scholars.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

Applying Conceptual and Theoretical Frameworks to Health Professions Education Research: An Introductory Workshop.

Introduction: Literature suggests that the quality and rigor of health professions education (HPE) research can be elevated if the research is anchored in existing theories and frameworks. This critical skill is difficult for novice researchers to master. We created a workshop to introduce the practical application of theories and frameworks to HPE research. Methods: We conducted two 60- to 75-minute workshops, one in 2019 at an in-person national conference and another in 2021 during an online national education conference. After a brief role-play introduction, participants applied a relevant theory to a case scenario in small groups, led by facilitators with expertise in HPE research. The workshop concluded with a presentation on applying the lessons learned when preparing a scholarly manuscript. We conducted a postworkshop survey to measure self-reported achievement of objectives. Results: Fifty-five individuals participated in the in-person workshop, and approximately 150 people completed the online workshop. Sixty participants (30%) completed the postworkshop survey across both workshops. As a result of participating in the workshop, 80% of participants (32) indicated they could distinguish between frameworks and theories, and 86% (32) could apply a conceptual or theoretical framework to a research question. Strengths of the workshop included the small-group activity, access to expert facilitators, and the materials provided. Discussion: The workshop has been well received by participants and fills a gap in the existing resources available to HPE researchers and mentors. It can be replicated in multiple settings to model the application of conceptual and theoretical frameworks to HPE research.

A Systematic Approach to Provide Feedback to Presenters at Virtual and Face-to-Face Professional Meetings.

Introduction: To promote their personal and professional growth, medical educators need practical, actionable feedback on their scholarship, as well as guidance for documenting their scholarship in educator portfolios. We offer a framework and resources to provide formative and summative feedback to faculty, administrators, and/or learners delivering an oral presentation at a face-to-face or virtual health professions education meeting. Methods: In 2014, the leadership of the Central Group on Educational Affairs (CGEA) meeting planning committee developed and piloted a process to provide individuals with formative and summative feedback on their oral CGEA research presentations at face-to-face meetings and create a transparent process for determining the Best Presentation Award. The feedback process was implemented for 7 years until revised in 2021 for the CGEA's first virtual meeting. Past and present meeting organizers conducted four focus groups in 2021 with presenters and peer reviewers via Zoom. Transcripts were analyzed for major themes using conventional content analysis. Results: To date, 102 presentation assessments have been conducted, including formative and summative assessments. Sixty-two volunteer assessors have participated, 19 (31%) of whom served for more than 2 years. Focus groups identified best practices and suggestions to improve the feedback process. Discussion: This resource offers a feasible, systematic process to provide individuals with formative feedback on presentations at professional conferences, promote a community of practice for personal and professional development, and create a transparent process for determining a Best Presentation Award. Participants valued providing and receiving feedback and recommended implementation at other professional meetings.

What Is an Innovation Article? A Systematic Overview of Innovation in Health Professions Education Journals.

PURPOSE: Innovation articles have their own submission category and guidelines in health professions education (HPE) journals, which suggests innovation might be a unique genre of scholarship. Yet, the requirements for innovation submissions vary among journals, suggesting ambiguity about the core content of this type of scholarship. To reduce this ambiguity, the researchers conducted a systematic overview to identify key features of innovation articles and evaluate their consistency in use across journals. Findings from this review may have implications for further development of innovation scholarship within HPE. METHOD: In this systematic overview, conducted in 2020, the researchers identified 13 HPE journals with innovation-type articles and used content analysis to identify key features from author guidelines and publications describing what editors look for in innovation articles. The researchers then audited a sample of 39 innovation articles (3/journal) published in 2019 to determine presence and consistency of 12 innovation features within and across HPE journals. Audit findings informed the researchers' evaluation of innovation as a genre in HPE. RESULTS: Findings show variability of innovation feature presence within and across journals. On average, articles included 7.8 of the 12 innovation features (SD 2.1, range 3-11). The most common features were description of: how the innovation was implemented (92%), a problem (90%), what was new or novel (79%), and data or outcomes (77%). On average, 5.5 (SD 1.5) out of 12 innovation features were consistently used in articles within each journal. CONCLUSIONS: The authors identified common features of innovation article types based on journal guidelines, but there was variability in presence and consistency of these features, suggesting HPE innovations are in an emerging state of genre development. The authors discuss potential reasons for variability within this article type and highlight the need for further discussion among authors, editors, and reviewers to improve clarity.

Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship.

Introduction: Education scholarship requires peer reviewers. For novice scholars, reviewing is an important developmental activity that cultivates deeper participation in the scholarship community. Yet getting started with reviewing is challenging for those not involved with the educational scholarship community. Beyond scientific expertise, reviewers must have a mentoring mindset, skills in providing constructive feedback, and knowledge of common ethical challenges associated with producing and evaluating scholarship. Methods: Our workshop introduced novice health professions educators to peer reviewing. It included four stimulus presentations about the peer reviewer's mindset and skills, followed by reinforcement activities. The workshop could be adapted to variously sized groups. An 8:1 ratio of participants to facilitators was ideal for activities. Topics covered included considerations before accepting an invitation, the review process, the good citizen approach to reviewing, and reviewer ethics. The session concluded with suggestions for continued development of peer reviewer competencies. The workshop was evaluated using a custom survey. Results: Throughout 2019 and early 2020, 58 health professions educators and trainees participated in the workshop across varied venues. Evaluations were obtained from 33 participants (57%). Nearly all rated the workshop as high quality and valuable to peer reviewer preparation. Most (26 of 33; 75%) gained confidence about their qualifications to serve as reviewers. Eighty percent (28 of 33) believed they could recognize ethical dilemmas. Discussion: This workshop provided a springboard for peer reviewing health professions education scholarship. Participants generally praised the experience for introducing them to the world of peer review and preparing them for it.

"What do you think about nephrology?" A national survey of internal medicine residents.

BACKGROUND: Interest in nephrology has been declining among internal medicine residents but the reasons behind this observation are not well characterized. Our objective was to evaluate factors influencing residents' choice of subspecialty. METHODS: This is a mixed-method QUAL-QUAN design study that used the results of our previously published qualitative analysis on residents' perception of nephrology to create and pilot a questionnaire of 60 questions. The final questionnaire was distributed to 26 programs across the United States and a total of 1992 residents. We calculated response rates and tabulated participant characteristics and percentage of participant responses. We categorized choice of fellowship into 2 medical categories (Highly Sought After vs. Less Sought After) and fitted a logistic regression model of choosing a highly vs. less sought after fellowship. RESULTS: Four hundred fifteen out of 1992 (21%) US residents responded to the survey. Of the 268 residents planning to pursue fellowship training, 67 (25%) selected a less sought after fellowship. Female sex was associated with significantly higher odds of selecting a less sought after fellowship (OR = 2.64, 95% CI: 1.47, 4.74). Major factors deterring residents from pursuing nephrology were perception of inadequate financial compensation, broad scope of clinical practice and complexity of patient population. We observed a decline in exposure to nephrology during the clinical years of medical school with only 35.4% of respondents rotating in nephrology versus 76.8% in residency. The quality of nephrology education was rated less positively during clinical medical school years (median of 50 on a 0-100 point scale) compared to the pre-clinical years (median 60) and residency (median 75). CONCLUSION: Our study attempts to explain the declining interest in nephrology. Results suggest potential targets for improvement: diversified trainee exposure, sub-specialization of nephrology, and increased involvement of nephrologists in the education of trainees.

Best Practices for Building and Supporting Effective ACGME-Mandated Program Evaluation Committees.

Introduction: The Accreditation Council for Graduate Medical Education (ACGME) mandates that residency training programs form program evaluation committees (PECs) to monitor program delivery and outcomes, generate annual program evaluations (APEs), facilitate strategic planning, and implement continuous quality improvement projects. Though PECs provide essential documentation to position programs for successful accreditation decisions, few resources exist in the literature for PEC members. Methods: Employing Kern's model for curriculum development, we conducted a needs assessment in 2016 that resulted in adding a 2-hour workshop on building and supporting effective PECs to a certificate program for residency program directors. The workshop used a flipped classroom model with prework readings and guiding questions to familiarize participants with ACGME requirements for PECs and APEs. Several activities helped participants identify best practices for PECs and discuss authentic examples of mission statements, APEs, and action plans. Results: From 2017 to 2019, we offered this workshop on three different occasions to a total of 42 participants (34 residency program directors/associate program directors and eight program coordinators). In 2019, 14 participants completed a web-based evaluation after the session. All agreed or strongly agreed that the workshop met the learning objectives, utilized interactive teaching methods, included useful APE examples, and provided valuable resources. Discussion: This workshop addresses a gap in the literature by helping program directors identify best practices for PECs. The APE template and workshop examples can be adjusted to fit the needs of individual institutions.

Effectiveness of a Leadership Development Course for Chief Residents: A Longitudinal Evaluation.

BACKGROUND: Nonclinical skills (eg, self-regulation, team leadership, conflict resolution) are essential for success as a chief resident (CR). The literature on programs teaching these skills reports few if any effectiveness outcomes. OBJECTIVE: We reported the outcomes of a leadership course for CRs using participants' self-reported outcomes and assessments from their program directors (PDs). METHODS: A 2-day curriculum focused on emotional intelligence competencies, including self-awareness, self-management, social awareness, and relationship management. We used a logic model to align 2017-2018 curriculum with targeted outcomes. Questionnaires before and after the course assessed short-term and intermediate outcomes for the participants and PD interviews evaluated observed changes in CRs' performance attributable to the course. RESULTS: A total of 74 residents participated in the course, and 65% and 59% responded to the post-course and follow-up questionnaires, respectively. Over 95% of respondents indicated developing leadership knowledge and skills and connecting with new CRs in the post-course questionnaire. During follow-up, CRs reported applying concepts learned during chief residency, using tools to address conflict, engaging in quality and patient safety projects, and continuing to interact with other participants. The relationships between reported outcomes and participants' gender/prior leadership training were not significant (P > .05), with small to medium effect sizes (0.01-0.32). All 14 PDs offered positive appraisal of the CRs, but we could not specifically attribute this growth to the course. CONCLUSIONS: Participation in this CR leadership development course was associated with enhancement and application of leadership competencies in immediate and intermediate time frames.

Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals.

Background: Despite recommendations from survey scientists, surveys appear to be utilized in medical education without the critical step of pretesting prior to survey launch. Pretesting helps ensure respondents understand questions as survey developers intended and that items and response options are relevant to respondents and adequately address constructs, topics, issues or problems. While psychometric testing is important in assessing aspects of question quality and item performance, it cannot discern how respondents, based upon their lived experiences, interpret the questions we pose.Aim: This audit study explored whether authors of medical education journal articles within audited journals reported pretesting survey instruments during survey development, as recommended by survey scientists and established guidelines/standards for survey instrument development.Methods: Five national and international medical education journals publishing survey articles from Jan. 2014 - Dec. 2015 were audited to determine whether authors reported pretesting during survey development. All abstracts within all issues of these journals were initially reviewed. Two hundred fifty-one articles met inclusion criteria using a protocol piloted and revised prior to use.Results: The number of survey articles published per journal ranged from 11 to 106. Of 251 audited articles, 181 (72.11%) described using a new instrument without pretesting, while 17 (6.77%) described using a new instrument where items were pretested. Fifty-three (21.12%) articles described using pre-existing instruments; of these, no articles (0%) reported pretesting existing survey instruments prior to use.Conclusions: Findings from this audit study indicate that reported survey pretesting appears to be lower than that reported in healthcare journals. This is concerning, as results of survey studies and evaluation projects are used to inform educational practices, guide future research, and influence policy and program development. Findings apply to both survey developers and faculty across a range of fields, including evaluation and medical education research.

A Professionalism Curricular Model to Promote Transformative Learning Among Residents.

BACKGROUND: Using the frameworks of transformational learning and situated learning theory, we developed a technology-enhanced professionalism curricular model to build a learning community aimed at promoting residents' self-reflection and self-awareness. The RAPR model had 4 components: (1) Recognize: elicit awareness; (2) Appreciate: question assumptions and take multiple perspectives; (3) Practice: try new/changed perspectives; and (4) Reflect: articulate implications of transformed views on future actions. OBJECTIVE: The authors explored the acceptability and practicality of the RAPR model in teaching professionalism in a residency setting, including how residents and faculty perceive the model, how well residents carry out the curricular activities, and whether these activities support transformational learning. METHODS: A convenience sample of 52 postgraduate years 1 through 3 internal medicine residents participated in the 10-hour curriculum over 4 weeks. A constructivist approach guided the thematic analysis of residents' written reflections, which were a required curricular task. RESULTS: A total of 94% (49 of 52) of residents participated in 2 implementation periods (January and March 2015). Findings suggested that RAPR has the potential to foster professionalism transformation in 3 domains: (1) attitudinal, with participants reporting they viewed professionalism in a more positive light and felt more empathetic toward patients; (2) behavioral, with residents indicating their ability to listen to patients increased; and (3) cognitive, with residents indicating the discussions improved their ability to reflect, and this helped them create meaning from experiences. CONCLUSIONS: Our findings suggest that RAPR offers an acceptable and practical strategy to teach professionalism to residents.

Medical student resilience strategies: A content analysis of medical students' portfolios.

INTRODUCTION: Stress and burnout among medical students is a well-recognized concern. A student's ability to employ resilience strategies to self-regulate behaviour is critical to the student's future career as a physician. METHODS: We retrospectively reviewed a sampling of year 1, 2 and 5 portfolio essays focused on the Personal Development competency and performance milestones, written by 49 students from three different classes in a 5-year programme devoted to training physician investigators. Two medical educators used a framework established by Jensen and colleagues (2008) to identify the nature and prevalence of various resilience strategies (valuing the physician role, self-awareness, personal arena, professional arena, professional support and personal support) medical students reported in portfolio essays. RESULTS: All students documented at least one strategy in their essays each year. In all years, the most commonly documented strategies were in the personal arena (95.7% of year 1, 98% of year 2 and 87.8% of year 5 portfolios). The least frequently documented strategy in all years was professional support (42.8% of year 1, 38.8% of year 2, and 28.6% of year 5 portfolios). Year 5 portfolios discussed personal support strategies (79.6%) more frequently than year 1 (53.1%) and year 2 (59.2%) portfolios. DISCUSSION: The results suggest that medical students can identify stressors and articulate resilience strategies that can be employed to potentially address them.

The Learning Environment Counts: Longitudinal Qualitative Analysis of Study Strategies Adopted by First-Year Medical Students in a Competency-Based Educational Program.

PURPOSE: The move toward competency-based education will require medical schools and postgraduate training programs to restructure learning environments to motivate trainees to take personal ownership for learning. This qualitative study explores how medical students select and implement study strategies while enrolled in a unique, nontraditional program that emphasizes reflection on performance and competence rather than relying on high-stakes examinations or grades to motivate students to learn and excel. METHOD: Fourteen first-year medical students volunteered to participate in three, 45-minute interviews (42 overall) scheduled three months apart during 2013-2014. Two medical educators used structured interview guides to solicit students' previous assessment experiences, preferred learning strategies, and performance monitoring processes. Interviews were digitally recorded and transcribed verbatim. Participants confirmed accuracy of transcripts. Researchers independently read transcripts and met regularly to discuss transcripts and judge when themes achieved saturation. RESULTS: Medical students can adopt an assessment for learning mind-set with faculty guidance and implement appropriate study strategies for mastery-learning demands. Though students developed new strategies at different rates during the year, they all eventually identified study and performance monitoring strategies to meet learning needs. Students who had diverse learning experiences in college embraced mastery-based study strategies sooner than peers after recognizing that the learning environment did not reward performance-based strategies. CONCLUSIONS: Medical students can take ownership for their learning and implement specific strategies to regulate behavior when learning environments contain building blocks emphasized in self-determination theory. Findings should generalize to educational programs seeking strategies to design learning environments that promote self-regulated learning.