Integrating a self-directed ultrasound curriculum for the internal medicine clerkship.

BACKGROUND: Incorporating ultrasound into the clinical curriculum of undergraduate medical education has been limited by a need for faculty support. Without integration into the clinical learning environment, ultrasound skills become a stand-alone skill and may decline by the time of matriculation into residency. A less time intensive ultrasound curriculum is needed to preserve skills acquired in preclinical years. We aimed to create a self-directed ultrasound curriculum to support and assess students' ability to acquire ultrasound images and to utilize ultrasound to inform clinical decision-making. METHODS: Third year students completed the self-directed ultrasound curriculum during their required internal medicine clerkship. Students used Butterfly iQ+ portable ultrasound probes. The curriculum included online modules that focused on clinical application of ultrasound as well as image acquisition technique. Students were graded on image acquisition quality and setting, a patient write-up focused on clinical decision-making, and a multiple-choice quiz. Student feedback was gathered with an end-of-course survey. Faculty time was tracked. RESULTS: One hundred and ten students participated. Students averaged 1.79 (scale 0-2; SD = 0.21) on image acquisition, 78% (SD = 15%) on the quiz, and all students passed the patient write-up. Most reported the curriculum improved their clinical reasoning (72%), learning of pathophysiology (69%), and patient care (55%). Faculty time to create the curriculum was approximately 45 h. Faculty time to grade student assignments was 38.5 h per year. CONCLUSIONS: Students were able to demonstrate adequate image acquisition, use of ultrasound to aid in clinical decision-making, and interpretation of ultrasound pathology with no in-person faculty instruction. Additionally, students reported improved learning of pathophysiology, clinical reasoning, and rapport with patients. The self-directed curriculum required less faculty time than prior descriptions of ultrasound curricula in the clinical years and could be considered at institutions that have limited faculty support.

Capturing growth curves of medical students' clinical skills performance.

INTRODUCTION: A benefit of a milestone or Entrustable Professional Activity (EPA) assessment framework is the ability to capture longitudinal performance with growth curves using multi-level modelling (MLM). Growth curves can inform curriculum design and individualised learning. Residency programmes have found growth curves to vary by resident and by milestone. Only one study has analysed medical students' growth curves for EPAs. Analysis of EPA growth curves is critical because no change in performance raises concerns for EPAs as an assessment framework. METHODS: Spencer Fox Eccles School of Medicine-University of Utah students' workplace-based assessment ratings for 7 EPAs were captured at 3 time-points in years 3-4 of AY2017-2018 to AY2020-2021. MLM was used to capture EPA growth curves and determine if variation in growth curves was explained by internal medicine (IM) clerkship order. FINDINGS: A curvilinear slope significantly captured 256 students' average ratings overtime for EPA1a-history-taking, EPA2-clinical reasoning, EPA3-diagnostics, EPA5-documentation and EPA6-presentation, and a linear slope significantly captured EPA9-teamwork ratings, p ≤ 0.001. Growth curves were steepest for EPA2-clinical reasoning and EPA3-diagnostics. Growth curves varied by students, p < 0.05 for all EPA ratings, but IM clerkship rotation order did not significantly explain the variance, p > 0.05. DISCUSSION: The increase in ratings from Year 3 to Year 4 provides validity evidence for use of EPAs in an assessment framework. Students may benefit from more curriculum/skills practice for EPA2-clinical reasoning and EPA3-diagnostics prior to year 3. Variation in student's growth curves is important for coaching and skill development; a one size fits all approach may not suffice.

Comparison of internal medicine applicant and resident characteristics with performance on ACGME milestones.

Internal medicine (IM) residency programs select applicants based on several metrics. Factors predicting success during residency are unclear across studies. To identify whether specific applicant or resident factors are associated with IM resident performance using ACGME milestones. We tested for associations between applicant factors available prior to the start of IM residency and resident factors measured during IM residency training, and resident performance on ACGME milestones across three consecutive years of IM training between 2015-2020. Univariable and multivariable linear regression modeling was used to test associations. Eighty-nine categorical IM residents that completed 3 consecutive years of training were included. Median age was 28 years (IQR 27-29) and 59.6% were male. Mean ACGME milestone scores increased with each post-graduate year (PGY) from 3.36 (SD 0.19) for PGY-1, to 3.80 (SD 0.15) for PGY-2, to 4.14 (SD 0.15) for PGY-3. Univariable modeling suggested referral to the clinical competency committee (CCC) for professionalism concerns was negatively associated with resident performance during each PGY. No applicant or resident factors included in the final multivariable regression models (age at starting residency, USMLE Step scores, interview score, rank list position, ITE scores) were associated with ACGME milestone scores for PGY-1 and PGY-2. Referral to the CCC for professionalism was negatively associated with resident performance during PGY-3. Residency selection factors did not predict resident milestone evaluation scores. Referral to the CCC was associated with significantly worse resident evaluation scores, suggesting professionalism may correlate with clinical performance.

The Impact of Internal Medicine Clerkship Characteristics and NBME Subject Exams on USMLE Step 2 Clinical Knowledge Exam Performance.

BACKGROUND: Residency program directors will likely emphasize the United States Medical Licensing Exam (USMLE) Step 2 clinical knowledge (CK) exam more during residency application given the recent USMLE Step 1 transition to pass/fail scoring. We examined how internal medicine clerkship characteristics and NBME subject exam scores affect USMLE Step 2 CK performance. DESIGN: The authors used univariable and multivariable generalized estimating equations to determine associations between Step 2 CK performance and internal medicine clerkship characteristics and NBME subject exams. The sample had 21,280 examinees' first Step 2 CK scores for analysis. RESULTS: On multivariable analysis, Step 1 performance (standardized ß = 0.45, p < .001) and NBME medicine subject exam performance (standardized ß = 0.40, p < .001) accounted for approximately 60% of the variance in Step 2 CK performance. Students who completed the internal medicine clerkship last in the academic year scored lower on Step 2 CK (Mdiff = -3.17 p < .001). Students who had a criterion score for passing the NBME medicine subject exam scored higher on Step 2 CK (Mdiff = 1.10, p = .03). There was no association between Step 2 CK performance and other internal medicine clerkship characteristics (all p > 0.05) nor with the total NBME subject exams completed (ß=0.05, p = .78). CONCLUSION: Despite similarities between NBME subject exams and Step 2 CK, the authors did not identify improved Step 2 CK performance for students who had more NBME subject exams. The lack of association of Step 2 CK performance with many internal medicine clerkship characteristics and more NBME subject exams has implications for future clerkship structure and summative assessment. The improved Step 2 CK performance in students that completed their internal medicine clerkship earlier warrants further study given the anticipated increase in emphasis on Step 2 CK.

Making the Personal Statement "Truly Personal": Recommendations From a Qualitative Case Study of Internal Medicine Program and Associate Program Directors.

Background: The personal statement is an integral part of a residency application but little guidance exists for medical students about what content to include. Objective: We use the framework of isomorphism, the process by which institutions model themselves after one another, to understand what internal medicine program directors (PDs) and associate program directors (APDs) recommend be included in the personal statement and how programs use personal statements in their selection of applicants to interview and rank. Methods: Semi-structured phone interviews were conducted between August and October 2020 with 13 academic PDs and APDs, who were selected for participation based on program size and geographic location. Interviews were recorded, transcribed, and coded using content analysis. Results: Effective personal statements should be well-written, present unique information, and demonstrate fit with a residency program. PDs and APDs recommended against expressing lack of interest in a program or highlighting negative personal characteristics. PDs and APDs used personal statements to distinguish between applicants and noted that personal statements help programs form an impression of the applicant. Consensus among PDs and APDs about what personal statements should include and how they are used indicates that isomorphism influences the match process. Conclusions: Our study found that the personal statement is a valued part of the residency application when it includes unique attributes and reveals personal values that align with that of the program. Additionally, PDs and APDs noted that when applicants highlight their unique characteristics, it can help distinguish themselves from others.

A comparison of natural language processing to ICD-10 codes for identification and characterization of pulmonary embolism.

INTRODUCTION: The 10th revision of the International Classification of Diseases (ICD-10) codes is frequently used to identify pulmonary embolism (PE) events, although the validity of ICD-10 has been questioned. Natural language processing (NLP) is a novel tool that may be useful for pulmonary embolism identification. METHODS: We performed a retrospective comparative accuracy study of 1000 randomly selected healthcare encounters with a CT pulmonary angiogram ordered between January 1, 2019 and January 1, 2020 at a single academic medical center. Two independent observers reviewed each radiology report and abstracted key findings related to PE presence/absence, chronicity, and anatomic location. NLP interpretations of radiology reports and ICD-10 codes were queried electronically and compared to the reference standard, manual chart review. RESULTS: A total of 970 encounters were included for analysis. The prevalence of PE was 13% by manual review. For PE identification, sensitivity was similar between NLP (96.0%) and ICD-10 (92.9%; p = 0.405), and specificity was significantly higher with NLP (97.7%) compared to ICD-10 (91.0%; p < 0.001). NLP demonstrated higher sensitivity (70.0% vs 16.5%, p < 0.001) and specificity (99.9% vs 99.4%, p = 0.014) for saddle/main PE recognition, and significantly higher sensitivity (86.7% vs 8.3%, p < 0.001) and specificity (99.8% vs 96.5%, p < 0.001) for subsegmental PE compared to ICD-10. CONCLUSIONS: NLP is highly sensitive for PE identification and more specific than ICD-10 coding. NLP outperformed ICD-10 coding for recognition of subsegmental, saddle, and chronic PE. Our results suggest NLP is an efficient and more reliable method than ICD-10 for PE identification and characterization.

Comparison of Resident, Advanced Practice Clinician, and Hospitalist Teams in an Academic Medical Center: Association With Clinical Outcomes and Resource Utilization.

BACKGROUND: Academic medical centers have expanded their inpatient medicine services with advanced practice clinicians (APCs) or nonteaching hospitalists in response to patient volumes, residency work hour restrictions, and recently, COVID-19. Reports of clinical outcomes, cost, and resource utilization differ among inpatient team structures. OBJECTIVE: Directly compare outcomes among resident, APC, and solo hospitalist inpatient general medicine teams. DESIGN: Retrospective cohort study using multivariable analysis adjusted for time of admission, interhospital transfer, and comorbidities that compares clinical outcomes, cost, and resource utilization. SUBJECTS: Patients 18 years or older discharged from an inpatient medicine service between July 2015 and July 2018 (N = 12,716). MAIN MEASURES: Length of stay (LOS), 30-day readmission, inpatient mortality, normalized total direct cost, discharge time, and consultation utilization. KEY RESULTS: Resident teams admitted fewer patients at night (32.0%; P < .001) than did APC (49.5%) and hospitalist (48.6%) teams. APCs received nearly 4% more outside transfer patients (P = .015). Hospitalists discharged patients 26 minutes earlier than did residents (mean hours after midnight [95% CI], 14.58 [14.44-14.72] vs 15.02 [14.97-15.08]). Adjusted consult utilization was 15% higher for APCs (adjusted mean consults per admission [95% CI], 1.00 [0.96-1.03]) and 8% higher for residents (0.93 [0.90-0.95]) than it was for hospitalists (0.85 [0.80-0.90]). No differences in LOS, readmission, mortality, or cost were observed between the teams. CONCLUSION: We observed similar costs, LOS, 30-day readmission, and mortality among hospitalist, APC, and resident teams. Our results suggest clinical outcomes are not significantly affected by team structure. The addition of APC or hospitalist teams represent safe and effective alternatives to traditional inpatient resident teams.

Impact of Hospitalist Team Structure on Patient-Reported Satisfaction with Physician Performance.

BACKGROUND: Patient experience is valuable because it reflects how patients perceive the care they receive within the healthcare system and is associated with clinical outcomes. Also, as part of the Hospital Value-Based Purchasing (HVBP) program, the Center for Medicare and Medicaid Services (CMS) rewards hospitals with financial incentives for patient experience as measured by the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey. It is unclear how the addition of residents and advanced practice clinicians (APCs) to hospitalist-led inpatient teams affects patient satisfaction as measured by the HCAHPS and Press Ganey survey. OBJECTIVE: To compare patient satisfaction with hospitalists on resident, APC, and solo hospitalist teams measured by HCAHPS and Press Ganey physician performance domain survey results. DESIGN: Retrospective observational cohort study. PARTICIPANTS: All patients discharged from the Internal Medicine inpatient service between July 1, 2015, and July 1, 2018, who met HCAHPS survey eligibility criteria and completed a patient experience survey. MAIN MEASURES: HCAHPS and Press Ganey physician performance domain survey results. KEY RESULTS: No differences were observed in the selection of "top box" scores on the HCAHPS physician performance domain between resident, APC, and solo hospitalist teams. Adjusted Press Ganey physician performance domain survey results demonstrated significant differences between solo hospitalist and resident teams, with solo hospitalists having higher scores in three areas: time physician spent with you (4.58 vs. 4.38, p = 0.050); physician kept you informed (4.63 vs. 4.43, p = 0.047); and physician skill (4.80 vs. 4.63, p = 0.027). Solo hospitalists were perceived to have higher physician skill in comparison with hospitalist-APC teams (4.80 vs. 4.69, p = 0.042). CONCLUSION: While Press Ganey survey results suggest that patients have greater satisfaction with physicians on solo hospitalist teams, these differences were not observed on the HCAHPS physician performance survey domain, suggesting physician team structure does not impact HVBP incentive payments by CMS.

Reducing Diabetic Ketoacidosis Intensive Care Unit Admissions Through an Electronic Health Record-Driven, Standardized Care Pathway.

Diabetic ketoacidosis (DKA) is a common condition, with wide variation in admission location and clinical practice. We aimed to decrease intensive care unit (ICU) admission for DKA by implementing a standardized, electronic health record-driven clinical care pathway that used subcutaneous insulin, rather than a continuous insulin infusion, for patients with nonsevere DKA. This is a retrospective, observational preintervention to postintervention study of 214 hospital admissions for DKA that evaluated the effect of our intervention on clinical, safety, and cost outcomes. The primary outcome was ICU admission, which decreased from 67.0% to 41.7% (p < .001). Diabetes nurse educator consultation increased from 45.3% to 63.9% (p = .006), and 30-day Emergency Department (ED) return visit decreased from 12.3% to 2.8% (p = .008). Time to initiation of basal insulin increased from 18.19 ± 1.25 hours to 22.47 ± 1.76 hours (p = .05) and reopening of the anion gap increased from 4.7% to 13.9% (p = .02). No changes in ED length of stay (LOS), hospital LOS, hypoglycemia, treatment-induced hypokalemia, 30-day hospital readmission, or inpatient mortality were observed. The implementation of a standardized DKA care pathway using subcutaneous insulin for nonsevere DKA resulted in decreased ICU use and increased diabetes education, without affecting patient safety.

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

BACKGROUND: Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback. OBJECTIVE: To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback. DESIGN: Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback. PARTICIPANTS: Seventy internal medicine residents and their faculty evaluators at the University of Utah. APPROACH: Faculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation. KEY RESULTS: Qualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre- and post-surveys. CONCLUSIONS: The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.

To What Degree Are the 13 Entrustable Professional Activities Already Incorporated Into Physicians' Performance Schemas for Medical Students?

Phenomenon: There is an abundance of literature on Entrustable Professional Activities (EPAs) in theory, but there are few studies on the EPAs in practice for undergraduate clinical education. In addition, little is known about the degree to which the EPAs are or are not aligned with physician assessors' performance schemas of the clerkship student. Investigating the degree to which physician assessors' performance schemas are already aligned with the activities described by the EPAs is critical for effective workplace assessment design. Approach: We sampled 1,032 areas of strength (strength) and areas for improvement (improvement) written evaluation comments by 423 physician assessors for clerkship students' performance in academic years 2014-15 and 2015-16 at the University of Utah School of Medicine. Two researchers independently categorized each comment by EPA and/or coded by non-EPA topic. The proportion of comment types was compared between strength comments and improvement comments with the Wilcoxon Signed-Rank Test. Findings: The most frequently mentioned EPAs in comments were about history gathering/physical exam, differential diagnosis, documentation, presentation, and interprofessional collaboration; few mentioned diagnostic tests, patient handovers, recognition of urgent patient care, and patient safety, and none mentioned orders/prescriptions and informed consent. The most frequent non-EPA topics were about medical knowledge, need to read more, learning attitude, work ethic, professionalism/maturity, and receptiveness to feedback. The proportion of comments aligned with an EPA only, a non-EPA topic only, or both an EPA and non-EPA topic was significantly different for clerkship students' strength compared to improvement. Insights: Physician assessors' performance schemas for clerkship students were aligned with EPAs to varying degrees depending on the specific EPA and whether describing strength or improvement. Of interest, the frequently mentioned non-EPA comments represented some of the competencies that contribute to effectively performing particular EPAs and are Accreditation Council for Graduate Medical Education (ACGME) core competencies (e.g., medical knowledge, professionalism), used in residency programs. Because physician assessors for undergraduate medical education often also participate in graduate medical education, the frequency of non-EPA topics aligned to ACGME competencies may suggest influence of graduate medical education evaluative frameworks on performance schemas for clerkship students; this could be important when considering implementation of EPAs in undergraduate medical education.