Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine.

PURPOSE: The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM). METHOD: Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings. RESULTS: During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS). CONCLUSIONS: The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

The Evaluation of a Digital Health Intervention to Improve Human Papillomavirus Vaccine Recommendation Practices of Medical Students.

We investigated what is being taught about HPV in US medical schools and evaluated a digital health intervention for medical students to increase their intention to provide a high-quality HPV vaccine recommendation. An online survey was emailed to Academic Deans at the 124 accredited US Schools of Medicine and Osteopathic Medicine between February and April 2018. A digital educational module was emailed to medical students in June 2020. A single-subject longitudinal study design was employed. Pre- post-survey administration measured change in knowledge, attitudes, subjective norms, perceived behavioral control, and intention to recommend HPV vaccine. Gaps in medical school curricula were found. Very few schools (~ 9%) reported teaching content on HPV prevention strategies or patient education. For the digital intervention, independent and dependent variables increased positively. Perceived behavioral control improved at significant levels as participants felt more confident addressing parents' concerns about the vaccine (p < 0.001) and more confident recommending the vaccine for patients (p < 0.05) post module. On the pre-survey, only 6% of the sample knew the most effective HPV vaccine communication style and on the post-survey 81% correctly identified it (p < 0.001). The intervention increased HPV and HPV vaccine knowledge, self-efficacy, and intention to provide a high-quality HPV vaccine recommendation to patients.

Impact of Faculty Training in Health Systems Science on Scholarly Presentation of Resident Physician and Fellow Quality Improvement Projects.

The objective was to evaluate whether faculty participation in a Health Systems Science training program was associated with increased presentation and publication of quality improvement (QI) projects involving resident physicians and fellows at 1 institution. The authors evaluated annual, department-level counts of QI projects with resident physician or fellow involvement, presented locally or published, according to residency or fellowship program director and faculty participation in Teachers of Quality Academy. Ten clinical departments had 82 presentations and 2 publications. Each additional faculty member's participation in Teachers of Quality Academy increased the annual count of published or presented QI projects by 9% (P < 0.001). At this institution, participation in a Health Systems Science training program among clinical faculty improved engagement of resident physicians and fellows in local presentation of QI projects.

HPV vaccine recommendation practices of current and future physicians in North Carolina: an exploratory study.

We assessed human papillomavirus (HPV) prevention practices and HPV literacy of pediatricians, family physicians and medical students in North Carolina. An online survey was distributed to physicians in 2018, and paper surveys were collected among medical students in 2019. Surveys measured HPV literacy, HPV prevention practices and HPV prevention self-efficacy. In terms of comfort, 27% of medical students and 24% of physicians anticipated having an uncomfortable conversion when recommending the vaccine to patients. Most physicians (76%, n = 230) followed the HPV vaccine age recommendation guidelines; however, those with higher HPV vaccine knowledge were more compliant with the guidelines (P < 0.01). Female physicians were more likely to start routinely recommending the HPV vaccine to women (84%, n = 134 versus 72%, n = 92) and men (81%, n = 127 versus 71%, n = 84) between the ages of 9 and 12 years (P < 0.05). Only 27%, n = 73 of physicians and 18%, n = 19 of medical students followed/knew the 'provider-driven' HPV-recommended style. Female physicians were more likely to use this communication style (32%, n = 48 versus 20%, n = 23, P = 0.03). HPV prevention curriculum should be incorporated into medical programs. The gender-related practice patterns found indicate a need for training of male providers specifically. Quality improvement efforts are needed for all physicians to strengthen vaccine communication, recommendation practices and guideline adherence.

An Innovative Use of Twitter to Disseminate and Promote Medical Student Scholarship During the COVID-19 Pandemic: Usability Study.

BACKGROUND: Due to the emergence of the COVID-19 pandemic in March 2020, the cancellation of in-person learning activities forced every aspect of medical education and student engagement to pivot to a web-based format, including activities supporting the performance and dissemination of scholarly work. At that time, social media had been used to augment in-person conference learning, but it had not been used as the sole platform for scholarly abstract presentations. OBJECTIVE: Our aim was to assess the feasibility of using Twitter to provide a completely web-based forum for real-time dissemination of and engagement with student scholarly work as an alternative to a traditional in-person poster presentation session. METHODS: The Brody School of Medicine at East Carolina University launched an online Medical Student Scholarship Forum, using Twitter as a platform for students to present scholarly work and prepare for future web-based presentations. A single student forum participant created posts using a standardized template that incorporated student research descriptions, uniform promotional hashtags, and individual poster presentations. Tweets were released over 5 days and analytic data were collected from the Twitter platform. Outcome measures included impressions, engagements, retweets, likes, media engagements, and average daily engagement rate. RESULTS: During the conference, the student leader published 63 tweets promoting the work of 58 students (55 medical and 3 dental students) over 5 days. During the forum and the following week, tweets from the @BrodyDistinctly Twitter account received 63,142 impressions and 7487 engagements, including 187 retweets, 1427 likes, and 2082 media engagements. During the 5 days of the forum, the average daily engagement rate was 12.72%. CONCLUSIONS: Using Twitter as a means of scholarly dissemination resulted in a larger viewing community compared to a traditional in-person event. Early evidence suggests that social media platforms may be an alternative to traditional scholarly presentations. Presenting via Twitter allowed students to receive instantaneous feedback and effectively network with wider academic communities. Additional research is needed to evaluate the effectiveness of knowledge uptake, feedback, and networking.

Impact of a Hybrid-Virtual Teaching Model on the Physical Examination Skills of Fourth-Year Medical Students.

ABSTRACT: A required fourth-year advanced core neurology-physical medicine and rehabilitation clerkship was adapted to hybrid format (2-wk remote; 2-wk in-person) during the COVID-19 pandemic. With teaching of the neurological physical examination being shifted to the remote component, we sought to determine whether this negatively affected student performance on an Objective Structured Clinical Examination, particularly the physical examination component. Mean pandemic-era total Objective Structured Clinical Examination scores ( n = 79, 85.1 ± 7.3) were similar to prepandemic era ( n = 137, 83.5 ± 6.0, P = 0.082). Pandemic-era physical examination scores were slightly higher than prepandemic (86.9 ± 6.5 vs. 84.9 ± 6.6). Despite conversion of the clerkship to a hybrid curriculum, the performance of the students on the Objective Structured Clinical Examination and the physical examination were unchanged. Reasons for this lack of change may include the constructiveness and integration of the case-based virtual demonstrations combined with in-person learning or the flexibility of the virtual course to allow students more time to prepare for the Objective Structured Clinical Examination and the physical examination. Our findings demonstrate that a hybrid-virtual model can be used to teach foundational skills such as the basics of the physical examination, while allowing faculty to address higher-order skills such as integration of clinical data with medical knowledge.

Beyond the CLAIM: A comprehensive needs assessment strategy for creating an Advanced Medical Education Research Training Program (ARMED-MedEd).

BACKGROUND: The health professions education (HPE) landscape has shifted substantively with increasing professionalization of research and scholarship. Clinician educators seeking to become competitive in this domain often pursue fellowships and master's degrees in HPE, but there are few resources for the continuing professional development (CPD) of those who seek to continue developing their scholarly practice within HPE. Acknowledging the multiple players in this landscape, the authors sought to design a new "beyond beginners" HPE research program using a novel needs assessment planning process. METHODS: The authors developed and conducted a new three-phase, five-step process that sets forth a programmatic approach to conducting a needs assessment for a CPD course in HPE research. The five steps of the CLAIM method are: Competitive analysis, Literature review with thematic analysis, Ask stakeholders, Internal review by experts, and Mapping of a curriculum. These steps are organized into three phases (Discovery, Convergence, and Synthesis). RESULTS: Over a 12-month period, the authors completed a comprehensive needs assessment. The CLAIM process revealed that longitudinal digital connection, diverse and in depth exposure to HPE research methods, skills around scholarly publishing, and leadership and management of research would be beneficial to our design. CONCLUSIONS: The CLAIM method provided scaffolding to help the authors create a robust curriculum that adopts a scholarly approach for developing a HPE research course. This needs assessment methodology may be useful in other CPD contexts.

Social network analysis of publication collaboration of accelerating change in MedEd consortium.

INTRODUCTION: The American Medical Association formed the Accelerating Change in Medical Education Consortium through grants to effect change in medical education. The dissemination of educational innovations through scholarship was a priority. The objective of this study was to explore the patterns of collaboration of educational innovation through the consortium's publications. METHOD: Publications were identified from grantee schools' semi-annual reports. Each publication was coded for the number of citations, Altmetric score, domain of scholarship, and collaboration with other institutions. Social network analysis explored relationships at the midpoint and end of the grant. RESULTS: Over five years, the 32 Consortium institutions produced 168 publications, ranging from 38 papers from one institution to no manuscripts from another. The two most common domains focused on health system science (92 papers) and competency-based medical education (30 papers). Articles were published in 54 different journals. Forty percent of publications involved more than one institution. Social network analysis demonstrated rich publishing relationships within the Consortium members as well as beyond the Consortium schools. In addition, there was growth of the network connections and density over time. CONCLUSION: The Consortium fostered a scholarship network disseminating a broad range of educational innovations through publications of individual school projects and collaborations.

Preparing Faculty to Incorporate Health Systems Science into the Clinical Learning Environment: Factors Associated with Sustained Outcomes.

This study assesses participants' perceptions of long-term impacts of the Teachers of Quality Academy, a medical school faculty development program designed to prepare faculty to both practice and teach health system science. A previously published 1-year evaluation of the first cohort of 27 participants showed improved perceived skills, with positive career and health system impacts. In this 5-year evaluation, a mixed-methods design included a questionnaire followed by semistructured interviews to assess perceived long-term impacts on participants. Quantitative and qualitative analyses were completed. Questionnaire response rate was 88% (N = 22), and 14 interviews were analyzed. Results demonstrated that participants had incorporated quality improvement concepts into their clinical work and teaching, better understood interprofessionalism, and observed continued improvements in care delivery. They felt the longitudinal training, delivered in a shared setting, created a learning community with lasting positive effects in institutional culture, supported long-term professional development, and had broader institutional impact. Advancements in clinical care, medical education, and professional and academic advancements were noted.

Health systems science education: The new post-Flexner professionalism for the 21st century.

The foundations of medical education have drawn from the Flexner Report to prepare students for practice for over a century. These recommendations relied, however, upon a limited set of competencies and a relatively narrow view of the physician's role. There have been increasing calls and recommendations to expand those competencies and the professional identity of the physician to better meet the current and future needs of patients, health systems, and society. We propose a framework for the twenty-first century physician that includes an expectation of new competency in health systems science (HSS), creating 'system citizens' who are effective stewards of the health care system. Experiential educational strategies, in addition to knowledge-centered learning, are critically important for students to develop their professional identity as system citizens working alongside interprofessional colleagues. Challenges to HSS adoption range from competing priorities for learners, to the need for faculty development, to the necessity for buy-in by medical schools and their associated health care systems. Ultimately, success will depend on our ability to articulate, encourage, support, and evaluate system citizenship and its impact on health care and health care systems.

Assessing the Transition of Training in Health Systems Science From Undergraduate to Graduate Medical Education.

BACKGROUND: The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation. OBJECTIVE: This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME). METHODS: ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models. RESULTS: At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups. CONCLUSIONS: Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

Dissemination of Quality Improvement Project Results After Local Presentation.

Among quality improvement (QI) projects submitted for local presentation, the authors sought to understand how often project results were eventually disseminated through national/international presentation or peer-reviewed journal publication. Projects submitted for local presentation from 2016 to 2019 were linked to resulting publications or national/international conference presentations. Submitting authors were surveyed about their intentions, experience, and satisfaction with the process of disseminating their project results. Of 83 projects, 5 were published and another 10 were presented nationally/internationally. External dissemination was more likely with fewer project cycles and cost-focused outcomes. Survey responses indicated that most project leaders wanted to see their results published but held mixed opinions about resources and encouragement available to reach this goal. Few QI projects submitted for local presentation resulted in wider dissemination of project results. Sharing results and lessons learned beyond the local institution requires long-term planning, education, and support beginning early in the QI process.

Multi-institutional Implementation of the National Clinical Assessment Tool in Emergency Medicine: Data From the First Year of Use.

OBJECTIVES: Uniformly training physicians to provide safe, high-quality care requires reliable assessment tools to ensure learner competency. The consensus-derived National Clinical Assessment Tool in Emergency Medicine (NCAT-EM) has been adopted by clerkships across the country. Analysis of large-scale deidentified data from a consortium of users is reported. METHODS: Thirteen sites entered data into a Web-based platform resulting in over 6,400 discrete NCAT-EM assessments from 748 students and 704 assessors. Reliability, internal consistency analysis, and factorial analysis of variance for hypothesis generation were performed. RESULTS: All categories on the NCAT-EM rating scales and professionalism subdomains were used. Clinical rating scale and global assessment scores were positively skewed, similar to other assessments commonly used in emergency medicine (EM). Professionalism lapses were noted in <1% of assessments. Cronbach's alpha was >0.8 for each site; however, interinstitutional variability was significant. M4 students scored higher than M3 students, and EM-bound students scored higher than non-EM-bound students. There were site-specific differences based on number of prior EM rotations, but no overall association. There were differences in scores based on assessor faculty rank and resident training year, but not by years in practice. There were site-specific differences based on student sex, but overall no difference. CONCLUSIONS: To our knowledge, this is the first large-scale multi-institutional implementation of a single clinical assessment tool. This study demonstrates the feasibility of a unified approach to clinical assessment across multiple diverse sites. Challenges remain in determining appropriate score distributions and improving consistency in scoring between sites.

A Required, Combined Neurology-Physical Medicine and Rehabilitation Clerkship Addresses Clinical and Health Systems Knowledge Gaps for Fourth-Year Medical Students.

ABSTRACT: This study evaluated the impact of a 4-wk mandatory neurology-physical medicine and rehabilitation advanced-core clerkship for fourth-year medical students. The combined clerkship encouraged an interdisciplinary and function-based approach to the management of common neurologic, musculoskeletal, and pain complaints. Seventy-three fourth-year medical students participated in the rotation over 1 yr. A survey assessing knowledge and skill set topics was conducted before and after the clerkship. Qualitative feedback regarding the rotation was provided by the students and analyzed. Significant gaps in knowledge and skill sets were identified before the clerkship and successfully addressed by combined teaching modalities. These data demonstrate that an integrated neurology-physical medicine and rehabilitation clerkship can improve students' confidence in multiple domains. Integrating physical medicine and rehabilitation into core clerkships at other medical schools may provide an avenue to address curriculum gaps.

Promoting Value Through Patient-Centered Communication: A Multisite Validity Study of Third-Year Medical Students.

PURPOSE: To examine validity evidence for a standardized patient scenario assessing medical students' ability to promote value using patient-centered communication (in response to a patient requesting an unnecessary test) and to explore the potential effect of various implementation and curricular factors on student scores. METHOD: Third-year medical students (N = 516) from 5 U.S. MD-granting medical schools completed the communication scenario between 2014 and 2017 as part of a larger objective structured clinical examination (OSCE). Centralized raters assessed performance using an 11-item checklist. The authors collected multiple sources of validity evidence. RESULTS: The mean checklist score was 0.85 (standard deviation 0.09). Interrater reliability for checklist scores was excellent (0.87, 95% confidence interval = 0.78-0.93). Generalizability and Phi-coefficients were, respectively, 0.65 and 0.57. Scores decreased as the number of OSCE stations increased (r = -0.15, P = .001) and increased when they were used for summative purposes (r = 0.26, P < .001). Scores were not associated with curricular time devoted to high-value care (r = 0.02, P = .67) and decreased when more clerkships were completed before the assessment (r = -0.12, P = .006). CONCLUSIONS: This multisite study provides validity evidence supporting the use of scenario scores to assess the ability of medical students to promote value in clinical encounters using patient-centered communication. Findings illuminate the potential effect of OSCE structure and purpose on student performance and suggest clerkship learning experiences may not reinforce what students are taught in the formal curriculum regarding high-value care. Devoting more time to the topic appears insufficient to counteract this erosion.

Validity of the Health Systems Science Examination: Relationship Between Examinee Performance and Time of Training.

The health systems science (HSS) framework articulates systems-relevant topics that medical trainees must learn to be prepared for physician practice. As new HSS-related curricula are developed, measures demonstrating appropriate levels of reliability and validity are needed. The authors describe a collaborative effort between a consortium of medical schools and the National Board of Medical Examiners to create a multiple-choice HSS examination in the areas of evidence-based medicine/population health, patient safety, quality improvement, and teamwork. Fifteen schools administered the 100-question examination through 2 academic years a total of 1887 times to 1837 first-time takers. Total test score mean was 67% (SD 11%). Total test reliability as measured by coefficient α was .83. This examination differentiated between medical students who completed the examination before, during, and after relevant training/instruction. This new HSS examination can support and inform the efforts of institutions as they integrate HSS-related content into their curricula.

What Do Program Directors Look for in an Applicant?

Program directors (PDs) are faced with an increasing number of applicants to emergency medicine (EM) and a limited number of positions. This article will provide candidates with insight to what PDs look for in an applicant. We will elaborate on the performance in the emergency medicine clerkship, interview, clinical rotations (apart from EM), board scores, Alpha Omega Alpha membership, letters of recommendation, Medical Student Performance Evaluation or dean's letter, extracurricular activities, Gold Humanism Society membership, medical school attended, research and scholarly projects, personal statement, and commitment to EM. We stress the National Resident Matching Program process and how, ultimately, selection of a residency is equally dependent on an applicant's selection process.

The Teachers of Quality Academy: Evaluation of the Effectiveness and Impact of a Health Systems Science Training Program.

This project aimed to evaluate the effectiveness of a faculty development program in health systems science (HSS)-the Teachers of Quality Academy (TQA). Participants in TQA and a comparison group were evaluated before, during, and 1 year after the program using self-perception questionnaires, tests of HSS knowledge, and tracking of academic productivity and career advancement. Among program completers (n = 27), the mean self-assessed ratings of knowledge and skills of HSS topics immediately after the program, as compared to baseline, increased significantly compared to controls (n = 30). Participants demonstrated progressive improvement of self-perceived skills and attitudes, and retention of HSS knowledge, from baseline to completion of the program. Participants also demonstrated substantially higher HSS scholarly productivity, leadership, and career advancement compared to the comparison group. The TQA effectively created a faculty cadre able to role model, teach, and create a curriculum in HSS competencies for medical students, resident physicians, and other health professionals.

Developing Tomorrow's Leaders: A Medical Student Distinction Track in Health System Transformation and Leadership.

PROBLEM: Calls for medical education reform focus on preparing physicians to meet the challenges of today's complex health care system. Despite implementing curricula focused on health systems science (HSS), including quality improvement (QI), patient safety, team-based care, and population health, a significant gap remains in training students to meet the system's evolving needs. APPROACH: Brody School of Medicine redesigned its curriculum to prepare leaders to effect health system change. This included development of a distinction track in health system transformation and leadership, known as the Leaders in INnovative Care (LINC) Scholars Program. Selected LINC scholars spend eight weeks in a summer immersion experience designed to provide foundational knowledge and practical application. OUTCOMES: Two cohorts (15 LINC scholars) completed the summer immersion in 2015 and 2016. Participants demonstrated significant improvement in knowledge and confidence and continue to be engaged in ongoing QI projects throughout the health system. All scholars have presented their work at local, regional, or national meetings. Students rated patient navigation experiences, health system leader interviews, QI project application, and interprofessional experiences as most valuable and recommended adoption in the curriculum for all students. NEXT STEPS: A distinction track with an immersion component can be an effective method to pilot innovative HSS components for the entire curriculum while preparing a cadre of learners with advanced expertise. To longitudinally measure HSS knowledge change, behavioral impact, and organization-level outcomes, next steps must focus on development of workplace-based assessments, establishment of learner portfolios, and longitudinal tracking of student outcomes, including career trajectory.