"What do you think about nephrology?" A national survey of internal medicine residents.

BACKGROUND: Interest in nephrology has been declining among internal medicine residents but the reasons behind this observation are not well characterized. Our objective was to evaluate factors influencing residents' choice of subspecialty. METHODS: This is a mixed-method QUAL-QUAN design study that used the results of our previously published qualitative analysis on residents' perception of nephrology to create and pilot a questionnaire of 60 questions. The final questionnaire was distributed to 26 programs across the United States and a total of 1992 residents. We calculated response rates and tabulated participant characteristics and percentage of participant responses. We categorized choice of fellowship into 2 medical categories (Highly Sought After vs. Less Sought After) and fitted a logistic regression model of choosing a highly vs. less sought after fellowship. RESULTS: Four hundred fifteen out of 1992 (21%) US residents responded to the survey. Of the 268 residents planning to pursue fellowship training, 67 (25%) selected a less sought after fellowship. Female sex was associated with significantly higher odds of selecting a less sought after fellowship (OR = 2.64, 95% CI: 1.47, 4.74). Major factors deterring residents from pursuing nephrology were perception of inadequate financial compensation, broad scope of clinical practice and complexity of patient population. We observed a decline in exposure to nephrology during the clinical years of medical school with only 35.4% of respondents rotating in nephrology versus 76.8% in residency. The quality of nephrology education was rated less positively during clinical medical school years (median of 50 on a 0-100 point scale) compared to the pre-clinical years (median 60) and residency (median 75). CONCLUSION: Our study attempts to explain the declining interest in nephrology. Results suggest potential targets for improvement: diversified trainee exposure, sub-specialization of nephrology, and increased involvement of nephrologists in the education of trainees.

Time to Loosen the Apron Strings: Cohort-based Evaluation of a Learner-driven Remediation Model at One Medical School.

BACKGROUND: Remediation in the era of competency-based assessment demands a model that empowers students to improve performance. AIM: To examine a remediation model where students, rather than faculty, develop remedial plans to improve performance. SETTING/PARTICIPANTS: Private medical school, 177 medical students. PROGRAM DESCRIPTION: A promotion committee uses student-generated portfolios and faculty referrals to identify struggling students, and has them develop formal remediation plans with personal reflections, improvement strategies, and performance evidence. Students submit reports to document progress until formally released from remediation by the promotion committee. PROGRAM EVALUATION: Participants included 177 students from six classes (2009-2014). Twenty-six were placed in remediation, with more referrals occurring during Years 1 or 2 (n = 20, 76 %). Unprofessional behavior represented the most common reason for referral in Years 3-5. Remedial students did not differ from classmates (n = 151) on baseline characteristics (Age, Gender, US citizenship, MCAT) or willingness to recommend their medical school to future students (p < 0.05). Two remedial students did not graduate and three did not pass USLME licensure exams on first attempt. Most remedial students (92 %) generated appropriate plans to address performance deficits. DISCUSSION: Students can successfully design remedial interventions. This learner-driven remediation model promotes greater autonomy and reinforces self-regulated learning.

Association of research self-efficacy with medical student career interests, specialization, and scholarship: a case study.

This study used variables proposed in social cognitive career theory (SCCT) to focus the evaluation of a research curriculum at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University (CCLCM). Eight cohorts of CCLCM medical students completed a web-based version of the six-scale Clinical Research Appraisal Inventory-Short Form (CRAI-SF) items at matriculation (n = 128) or graduation (n = 111) during 2009-2013. Parametric statistics were used to compare CRAI-SF scales to domains proposed in SCCT: trainees' characteristics (gender, training level, advanced degree), career interests, career intentions (medical specialty), and performance (peer-reviewed publications and required thesis topic). A number of lessons emerged in using theory to frame the evaluation of a complex educational program. Graduates rated their research self-efficacy significantly higher on all six CRAI-SF scales with large effect sizes (>.90) on five scales (Conceptualizing a Study, Study Design and Analysis, Responsible Research Conduct, Collaborating with Others, and Reporting a Study). Women and men did not have significantly different scores on CRAI-SF scales (p > .05), suggesting that the research program provides adequate supports for women students. Most thesis projects addressed clinical (36.9 %, n = 41) or translational (34.2 %, n = 38) research topics. The CRAI-SF discriminated between medical school matriculates and graduates, suggesting that research self-efficacy increases with mastery experiences. No significant relationships occurred between CRAI-SF scores and graduates' thesis topics or chosen clinical specialty. Correlations demonstrated significant relationships between graduates' perceptions of research self-efficacy and their interest in clinical research careers.

"Making Space for More People, More Perspectives, and More Ideas": How Medical Education Journal Editors View Their Role in Capacity Building.

PURPOSE: Medical education journal editors work to improve scholarly rigor and expand access to scholarship. How editors conceptualize these dual roles is unknown and holds implications for the kinds of scholarship that are valued and made visible through publication. The authors applied the concept of capacity building to examine how medical education journal editors conceptualize and operationalize capacity building and to identify the contextual factors that support or constrain these efforts. METHOD: Using a reflexive thematic approach, the authors interviewed 13 editors of 11 medical education journals between February 2022 and March 2023. Data collection and analyses were performed iteratively. Interview transcripts were coded by 2 authors to generate initial codes, which were then refined by the research team until final themes were created. RESULTS: Editors viewed capacity building "making space" for people and perspectives. Making space for people included efforts to support editors and reviewers, offer feedback to authors, and engage with readers. Making space for perspectives included efforts to promote discussion on diversity and inclusion and to introduce ideas perceived to advance the field. How editors made space was shaped by a journal's ecosystem, including local features unique to each journal (e.g., aims, scope, readership) and system-level factors that influenced all journals (e.g., English as the language of science, academic promotion and tenure). CONCLUSIONS: Although medical education journal editors' capacity-building work will inevitably guide future scholars and scholarship, their work should be considered in the context of a larger ecosystem, including such features as institutional academies for scholarship and promotion and tenure policies. Improving rigor and expanding access will require clarifying how editors' approach and understand capacity building to advance as a field. Future work should clarify how individual journals' capacity-building efforts can be facilitated and how journals can collectively reimagine spaces for knowledge-building conversations.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

Impact of a Regional Grant Program Through the Lens of Social Cognitive Career Theory: A Mixed-Method Evaluation.

PURPOSE: Evaluations of educational grant programs have focused on research productivity, with few examining impacts on grantees or effective program characteristics. This evaluation examined the regional grant program sponsored by Group on Educational Affairs to examine if and how grantees' careers were affected by funding, and if these experiences aligned with program goals. METHOD: In this concurrent, mixed-methods theory-driven evaluation, quantitative and qualitative data were analyzed independently and then integrated to examine complementarity. Quantitative data examined differences among 4 geographic regions and included proposal and grantee characteristics abstracted from administrative records of 52 funded proposals from 2010-2015 grant cycles. Qualitative data from 23 interviews conducted from 2018 to 2019 explored the impact on grantees, with Social Cognitive Career Theory (SCCT) serving as a framework for deductive thematic analysis. To facilitate integration of findings, quantitative data were layered onto each interview to permit exploration of associations between the 2 data types. RESULTS: Although significant regional differences existed in project length and amount of funding, there were few regional differences in grantee experiences. Despite small funding amounts, grants were perceived as career launching pads. The SCCT framework accounted for grantee experiences, including researcher identity formation and subsequent research, but did not capture collaboration phenomena. Integration of the 2 data types identified experience patterns unique to different groups of grantees (e.g., more or less research experience). The diversity among grantees suggests that clarification of program goals and stronger alignment with criteria for funding may be warranted. CONCLUSIONS: This evaluation illuminates why small educational grant programs may or may not impact interest and productivity in research. Implications exist for funders, including clarifying program goals and providing support for less experienced grantees. Future research should explore grantee subsets (e.g., underrepresented in medicine) to further identify what fosters or inhibits careers of medical education scholars.

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States.

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Evidence within a portfolio-based assessment program: what do medical students select to document their performance?

BACKGROUND: Decisions about performance in programs of assessment that provide an array of assessment evidence require judgments about the quality of different pieces of assessment data to determine which combination of data points best represent a trainee's overall performance. AIM: In this article, we examine the nature of evidence selected by first-year medical students to include in a portfolio used to make promotion decisions. METHODS: We reviewed portfolios to examine the number, type, and source of assessments selected by students (n = 32) to document their performance in seven competencies. The quality of assessment data selected for each competency was rated by promotion committee members (n = 14). RESULTS: Findings indicate that students cited multiple types and sources of available assessments. The promotion committee rated evidence quality highest for competencies where the program provided sufficient evidence for students to cite a broad range of assessments. When assessments were not provided by the program, students cited self-generated evidence. CONCLUSION: We found that when student-constructed portfolios are part of an overall assessment system, students generally select evidence in proportion to the number and types of assessments available.

A Systematic Approach to Provide Feedback to Presenters at Virtual and Face-to-Face Professional Meetings.

Introduction: To promote their personal and professional growth, medical educators need practical, actionable feedback on their scholarship, as well as guidance for documenting their scholarship in educator portfolios. We offer a framework and resources to provide formative and summative feedback to faculty, administrators, and/or learners delivering an oral presentation at a face-to-face or virtual health professions education meeting. Methods: In 2014, the leadership of the Central Group on Educational Affairs (CGEA) meeting planning committee developed and piloted a process to provide individuals with formative and summative feedback on their oral CGEA research presentations at face-to-face meetings and create a transparent process for determining the Best Presentation Award. The feedback process was implemented for 7 years until revised in 2021 for the CGEA's first virtual meeting. Past and present meeting organizers conducted four focus groups in 2021 with presenters and peer reviewers via Zoom. Transcripts were analyzed for major themes using conventional content analysis. Results: To date, 102 presentation assessments have been conducted, including formative and summative assessments. Sixty-two volunteer assessors have participated, 19 (31%) of whom served for more than 2 years. Focus groups identified best practices and suggestions to improve the feedback process. Discussion: This resource offers a feasible, systematic process to provide individuals with formative feedback on presentations at professional conferences, promote a community of practice for personal and professional development, and create a transparent process for determining a Best Presentation Award. Participants valued providing and receiving feedback and recommended implementation at other professional meetings.

Applying Conceptual and Theoretical Frameworks to Health Professions Education Research: An Introductory Workshop.

Introduction: Literature suggests that the quality and rigor of health professions education (HPE) research can be elevated if the research is anchored in existing theories and frameworks. This critical skill is difficult for novice researchers to master. We created a workshop to introduce the practical application of theories and frameworks to HPE research. Methods: We conducted two 60- to 75-minute workshops, one in 2019 at an in-person national conference and another in 2021 during an online national education conference. After a brief role-play introduction, participants applied a relevant theory to a case scenario in small groups, led by facilitators with expertise in HPE research. The workshop concluded with a presentation on applying the lessons learned when preparing a scholarly manuscript. We conducted a postworkshop survey to measure self-reported achievement of objectives. Results: Fifty-five individuals participated in the in-person workshop, and approximately 150 people completed the online workshop. Sixty participants (30%) completed the postworkshop survey across both workshops. As a result of participating in the workshop, 80% of participants (32) indicated they could distinguish between frameworks and theories, and 86% (32) could apply a conceptual or theoretical framework to a research question. Strengths of the workshop included the small-group activity, access to expert facilitators, and the materials provided. Discussion: The workshop has been well received by participants and fills a gap in the existing resources available to HPE researchers and mentors. It can be replicated in multiple settings to model the application of conceptual and theoretical frameworks to HPE research.

Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship.

Introduction: Education scholarship requires peer reviewers. For novice scholars, reviewing is an important developmental activity that cultivates deeper participation in the scholarship community. Yet getting started with reviewing is challenging for those not involved with the educational scholarship community. Beyond scientific expertise, reviewers must have a mentoring mindset, skills in providing constructive feedback, and knowledge of common ethical challenges associated with producing and evaluating scholarship. Methods: Our workshop introduced novice health professions educators to peer reviewing. It included four stimulus presentations about the peer reviewer's mindset and skills, followed by reinforcement activities. The workshop could be adapted to variously sized groups. An 8:1 ratio of participants to facilitators was ideal for activities. Topics covered included considerations before accepting an invitation, the review process, the good citizen approach to reviewing, and reviewer ethics. The session concluded with suggestions for continued development of peer reviewer competencies. The workshop was evaluated using a custom survey. Results: Throughout 2019 and early 2020, 58 health professions educators and trainees participated in the workshop across varied venues. Evaluations were obtained from 33 participants (57%). Nearly all rated the workshop as high quality and valuable to peer reviewer preparation. Most (26 of 33; 75%) gained confidence about their qualifications to serve as reviewers. Eighty percent (28 of 33) believed they could recognize ethical dilemmas. Discussion: This workshop provided a springboard for peer reviewing health professions education scholarship. Participants generally praised the experience for introducing them to the world of peer review and preparing them for it.

What Is an Innovation Article? A Systematic Overview of Innovation in Health Professions Education Journals.

PURPOSE: Innovation articles have their own submission category and guidelines in health professions education (HPE) journals, which suggests innovation might be a unique genre of scholarship. Yet, the requirements for innovation submissions vary among journals, suggesting ambiguity about the core content of this type of scholarship. To reduce this ambiguity, the researchers conducted a systematic overview to identify key features of innovation articles and evaluate their consistency in use across journals. Findings from this review may have implications for further development of innovation scholarship within HPE. METHOD: In this systematic overview, conducted in 2020, the researchers identified 13 HPE journals with innovation-type articles and used content analysis to identify key features from author guidelines and publications describing what editors look for in innovation articles. The researchers then audited a sample of 39 innovation articles (3/journal) published in 2019 to determine presence and consistency of 12 innovation features within and across HPE journals. Audit findings informed the researchers' evaluation of innovation as a genre in HPE. RESULTS: Findings show variability of innovation feature presence within and across journals. On average, articles included 7.8 of the 12 innovation features (SD 2.1, range 3-11). The most common features were description of: how the innovation was implemented (92%), a problem (90%), what was new or novel (79%), and data or outcomes (77%). On average, 5.5 (SD 1.5) out of 12 innovation features were consistently used in articles within each journal. CONCLUSIONS: The authors identified common features of innovation article types based on journal guidelines, but there was variability in presence and consistency of these features, suggesting HPE innovations are in an emerging state of genre development. The authors discuss potential reasons for variability within this article type and highlight the need for further discussion among authors, editors, and reviewers to improve clarity.

Effectiveness of a Leadership Development Course for Chief Residents: A Longitudinal Evaluation.

BACKGROUND: Nonclinical skills (eg, self-regulation, team leadership, conflict resolution) are essential for success as a chief resident (CR). The literature on programs teaching these skills reports few if any effectiveness outcomes. OBJECTIVE: We reported the outcomes of a leadership course for CRs using participants' self-reported outcomes and assessments from their program directors (PDs). METHODS: A 2-day curriculum focused on emotional intelligence competencies, including self-awareness, self-management, social awareness, and relationship management. We used a logic model to align 2017-2018 curriculum with targeted outcomes. Questionnaires before and after the course assessed short-term and intermediate outcomes for the participants and PD interviews evaluated observed changes in CRs' performance attributable to the course. RESULTS: A total of 74 residents participated in the course, and 65% and 59% responded to the post-course and follow-up questionnaires, respectively. Over 95% of respondents indicated developing leadership knowledge and skills and connecting with new CRs in the post-course questionnaire. During follow-up, CRs reported applying concepts learned during chief residency, using tools to address conflict, engaging in quality and patient safety projects, and continuing to interact with other participants. The relationships between reported outcomes and participants' gender/prior leadership training were not significant (P > .05), with small to medium effect sizes (0.01-0.32). All 14 PDs offered positive appraisal of the CRs, but we could not specifically attribute this growth to the course. CONCLUSIONS: Participation in this CR leadership development course was associated with enhancement and application of leadership competencies in immediate and intermediate time frames.

Best Practices for Building and Supporting Effective ACGME-Mandated Program Evaluation Committees.

Introduction: The Accreditation Council for Graduate Medical Education (ACGME) mandates that residency training programs form program evaluation committees (PECs) to monitor program delivery and outcomes, generate annual program evaluations (APEs), facilitate strategic planning, and implement continuous quality improvement projects. Though PECs provide essential documentation to position programs for successful accreditation decisions, few resources exist in the literature for PEC members. Methods: Employing Kern's model for curriculum development, we conducted a needs assessment in 2016 that resulted in adding a 2-hour workshop on building and supporting effective PECs to a certificate program for residency program directors. The workshop used a flipped classroom model with prework readings and guiding questions to familiarize participants with ACGME requirements for PECs and APEs. Several activities helped participants identify best practices for PECs and discuss authentic examples of mission statements, APEs, and action plans. Results: From 2017 to 2019, we offered this workshop on three different occasions to a total of 42 participants (34 residency program directors/associate program directors and eight program coordinators). In 2019, 14 participants completed a web-based evaluation after the session. All agreed or strongly agreed that the workshop met the learning objectives, utilized interactive teaching methods, included useful APE examples, and provided valuable resources. Discussion: This workshop addresses a gap in the literature by helping program directors identify best practices for PECs. The APE template and workshop examples can be adjusted to fit the needs of individual institutions.

Evaluation of essay questions used to assess medical students' application and integration of basic and clinical science knowledge.

BACKGROUND: Educators need approaches to assess medical students' abilities to apply and integrate concepts essential to medical practice. DESCRIPTION: We used a multimethod approach to examine the quality of essay questions intended to elicit medical students' ability to apply and integrate their understanding of medical concepts. EVALUATION: Three educators assigned essay questions (n = 120) to one of four levels of cognition. Kappa was computed before and after discussion. Faculty (n = 46) critiqued essay quality using a checklist (97% response), and students completed a questionnaire about the learning environment (91% response). CONCLUSIONS: We identified effective approaches to evaluate the quality of essay questions and to train faculty to write essay questions of sufficient complexity. This systematic review of essay questions also encouraged review of the curriculum to determine if core concepts were being taught. It is feasible to have faculty write and critique essay questions targeted at higher levels of cognition.

Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals.

Background: Despite recommendations from survey scientists, surveys appear to be utilized in medical education without the critical step of pretesting prior to survey launch. Pretesting helps ensure respondents understand questions as survey developers intended and that items and response options are relevant to respondents and adequately address constructs, topics, issues or problems. While psychometric testing is important in assessing aspects of question quality and item performance, it cannot discern how respondents, based upon their lived experiences, interpret the questions we pose.Aim: This audit study explored whether authors of medical education journal articles within audited journals reported pretesting survey instruments during survey development, as recommended by survey scientists and established guidelines/standards for survey instrument development.Methods: Five national and international medical education journals publishing survey articles from Jan. 2014 - Dec. 2015 were audited to determine whether authors reported pretesting during survey development. All abstracts within all issues of these journals were initially reviewed. Two hundred fifty-one articles met inclusion criteria using a protocol piloted and revised prior to use.Results: The number of survey articles published per journal ranged from 11 to 106. Of 251 audited articles, 181 (72.11%) described using a new instrument without pretesting, while 17 (6.77%) described using a new instrument where items were pretested. Fifty-three (21.12%) articles described using pre-existing instruments; of these, no articles (0%) reported pretesting existing survey instruments prior to use.Conclusions: Findings from this audit study indicate that reported survey pretesting appears to be lower than that reported in healthcare journals. This is concerning, as results of survey studies and evaluation projects are used to inform educational practices, guide future research, and influence policy and program development. Findings apply to both survey developers and faculty across a range of fields, including evaluation and medical education research.

Methods to assess students' acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum.

BACKGROUND: The Cleveland Clinic Lerner College of Medicine was designed to encourage medical students to pursue careers as physician investigators. Our faculty decided that assessment should enhance learning and adopted only formative assessments to document student performance in relation to nine broad-based competencies. No grades are used to judge student performance throughout the 5-year program. Instead, assessments are competency-based, relate directly to performance standards, and are stored in e-Portfolios to track progress and document student achievement. The class size is limited to 32 students a year. AIMS: Schools with competency-based curricula must provide students with formative feedback to identify performance gaps and monitor progress. We describe a systematic approach to assess medical knowledge using essay-type questions (CAPPs) and multiple choice questions (SAQs) to provide medical students with weekly, formative feedback about their abilities to acquire, apply and integrate basic and clinical science concepts. METHOD: Processes for developing performance standards, creating assessment items, training faculty, reporting student performance and monitoring outcomes are described. A case study of a Year 1 course is presented with specific examples of CAPPs and SAQs to illustrate how formative assessment data are interpreted and reported in students' e-Portfolios. RESULTS: Preliminary evidence suggests that CAPPs and SAQs have a positive impact on students' education, a justifiable cost in light of obtained benefits and growing acceptance among stakeholders. Two student cohorts performed significantly above the population mean on USMLE Step 1, which suggests that these assessment methods have not disadvantaged students. More evidence is needed to assess the reliability and validity of these tools for formative purposes. CONCLUSIONS: Using assessment data for formative purposes may encourage application and integration of knowledge, help students identify performance gaps, foster student development of learning plans and promote student responsibility for learning. Discussion provides applications for institutions with larger classes to consider.