Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments.

BACKGROUND: Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding. OBJECTIVE: In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks. DESIGN: Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel. PARTICIPANTS: A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor. MAIN MEASURES: Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors. KEY RESULTS: IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty. CONCLUSIONS: To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool.

INTRODUCTION: Clinical reasoning encompasses the process of data collection, synthesis, and interpretation to generate a working diagnosis and make management decisions. Situated cognition theory suggests that knowledge is relative to contextual factors, and clinical reasoning in urgent situations is framed by pressure of consequential, time-sensitive decision-making for diagnosis and management. These unique aspects of urgent clinical care may limit the effectiveness of traditional tools to assess, teach, and remediate clinical reasoning. METHODS: Using two validated frameworks, a multidisciplinary group of clinicians trained to remediate clinical reasoning and with experience in urgent clinical care encounters designed the novel Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT). REACT is a behaviorally anchored assessment tool scoring five domains used to provide formative feedback to learners evaluating patients during urgent clinical situations. A pilot study was performed to assess fourth-year medical students during simulated urgent clinical scenarios. Learners were scored using REACT by a separate, multidisciplinary group of clinician educators with no additional training in the clinical reasoning process. REACT scores were analyzed for internal consistency across raters and observations. RESULTS: Overall internal consistency for the 41 patient simulations as measured by Cronbach's alpha was 0.86. A weighted kappa statistic was used to assess the overall score inter-rater reliability. Moderate reliability was observed at 0.56. DISCUSSION: To our knowledge, REACT is the first tool designed specifically for formative assessment of a learner's clinical reasoning performance during simulated urgent clinical situations. With evidence of reliability and content validity, this tool guides feedback to learners during high-risk urgent clinical scenarios, with the goal of reducing diagnostic and management errors to limit patient harm.

Characterization of Clinical Skills Remediation: A National Survey of Medical Schools.

OBJECTIVES: Clinical skills instruction is a standard part of medical school curricula, but how institutions address learners who struggle in this area is less clear. Although recommendations for the remediation of clinical skills at an institutional level have been published, how these recommendations are being implemented on a national scale is unknown. In this descriptive study, we characterize current clinical skills remediation practices at US medical schools and US-accredited Caribbean medical schools. METHODS: We conducted a cross-sectional survey of medical educators who work with struggling students. From March 24, 2020 to April 9, 2020, the Directors of Clinical Skills Remediation Working Group conducted an e-mail survey incorporating four aspects of remediation program design and function: identification, assessment, active remediation, and ongoing evaluation. RESULTS: In total, 92 individuals representing 45 institutions provided descriptive information about their respective remediation programs. The majority of respondents have a formal process of identifying (75%) and assessing (86%) students who are identified as struggling with clinical skills, but lack a standardized method of categorizing deficits. Fewer institutions have a standardized approach to active remediation and ongoing evaluation of struggling learners. Fifty-two percent of institutions provide training to faculty involved in the remediation process. CONCLUSIONS: Although most institutions are able to identify struggling students, they lack a standardized approach to intervene. Remediation effectiveness is limited by a lack of student buy-in and institutional time, expertise, and resources. These findings highlight the need for more formalized structure and standardization in remediation program design and implementation.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned.

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

Relationships between preclinical course grades and standardized exam performance.

Success in residency matching is largely contingent upon standardized exam scores. Identifying predictors of standardized exam performance could promote primary intervention and lead to design insights for preclinical courses. We hypothesized that clinically relevant courses with an emphasis on higher-order cognitive understanding are most strongly associated with performance on United States Medical Licensing Examination Step exams and National Board of Medical Examiners clinical subject exams. Academic data from students between 2007 and 2012 were collected. Preclinical course scores and standardized exam scores were used for statistical modeling with multiple linear regression. Preclinical courses were categorized as having either a basic science or a clinical knowledge focus. Medical College Admissions Test scores were included as an additional predictive variable. The study sample comprised 795 graduating medical students. Median score on Step 1 was 234 (interquartile range 219-245.5), and 10.2 % (81/795) scored lower than one standard deviation below the national average (205). Pathology course score was the strongest predictor of performance on all clinical subject exams and Step exams, outperforming the Medical College Admissions Test in strength of association. Using Pathology score <75 as a screening metric for Step 1 score <205 results in sensitivity and specificity of 37 and 97 %, respectively, and a likelihood ratio of 11.9. Performance in Pathology, a clinically relevant course with case-based learning, is significantly related to subsequent performance on standardized exams. Multiple linear regression is useful for identifying courses that have potential as risk stratifiers.

Needs assessment of essential anatomy: The perspective of adult primary care resident physicians.

Curricular development and modification involve first identifying a problem and then performing a needs assessment, which can guide the design of curricular components. Pedagogical changes, coupled with reductions in curricular time for gross anatomy, pose challenges and impose restrictions within medical school curricula. In order to make anatomy education effective and efficient, it is important to determine the anatomy considered essential for medical education through a targeted needs assessment. In this study, 50 adult primary care resident physicians in family medicine (FM) and internal medicine (IM) were surveyed to assess the importance of 907 anatomical structures, or groups of structures, across all anatomical regions from a curated list based on the boldface terms in four primary anatomy texts. There were no statistically significant differences in the ratings of structures between the two groups for any anatomical region. In total, 17.0% of structures, or groups of structures, were classified as essential, 58.0% as more important, 24.4% as less important, and 0.7% as not important. FM residents rated tissues classified as skeleton, nerves, fasciae, anatomical spaces, blood vessels, lymphatics, and surface anatomy (p < 0.0001) significantly higher than IM residents, but there were no differences in the rating of muscles or organs (p > 0.0056). It was notable that 100.0% of cranial nerves were classified as essential, and 94.5% of surface anatomy structures were classified as essential or more important. It is proposed that results of this study can serve to inform curricular development and revision.

Essential anatomy for core clerkships: A clinical perspective.

Clerkships are defining experiences for medical students in which students integrate basic science knowledge with clinical information as they gain experience in diagnosing and treating patients in a variety of clinical settings. Among the basic sciences, there is broad agreement that anatomy is foundational for medical practice. Unfortunately, there are longstanding concerns that student knowledge of anatomy is below the expectations of clerkship directors and clinical faculty. Most allopathic medical schools require eight "core" clerkships: internal medicine (IM), pediatrics (PD), general surgery (GS), obstetrics and gynecology (OB), psychiatry (PS), family medicine (FM), neurology (NU), and emergency medicine (EM). A targeted needs assessment was conducted to determine the anatomy considered important for each core clerkship based on the perspective of clinicians teaching in those clerkships. A total of 525 clinical faculty were surveyed at 24 United States allopathic medical schools. Participants rated 97 anatomical structure groups across all body regions on a 1-4 Likert-type scale (1 = not important, 4 = essential). Non-parametric ANOVAs determined if differences existed between clerkships. Combining all responses, 91% of anatomical structure groups were classified as essential or more important. Clinicians in FM, EM, and GS rated anatomical structures in most body regions significantly higher than at least one other clerkship (p = 0.006). This study provides an evidence-base of anatomy content that should be considered important for each core clerkship and may assist in the development and/or revision of preclinical curricula to support the clinical training of medical students.

Feedback Methods in an Interprofessional Mock Paging Program.

BACKGROUND: Many medical schools offer a culminating internship readiness experience. Curricula focus on particular knowledge and skills critical to internship, such as answering urgent nursing pages. Studies have shown student performance improvement with mock paging education programs, but the role of feedback versus self-regulated practice has not been studied. DESIGN AND METHODS: The interprofessional mock paging program included 156 medical students enrolled in a 4th-year internship readiness course and 44 master's level direct entry nursing students. Medical students were randomized to receive verbal feedback immediately after each of the three phone calls (intervention group) or delayed written feedback (control group) after the third phone call only. Specialty-specific case scenarios were developed and a single checklist for all scenarios was developed using the communication tool ISBAR. Medical students and nursing students had separate training sessions before the pages commenced. The nursing students administered the phone calls and evaluated the medical students by ISBAR checklist. An interrater reliability measure was obtained with physician observation of a selection of phone calls. RESULTS: After adjusting for the case effects (different case scenarios for different specialties), students showed no statistically significant differences on checklist scores for case 1 (first case, F = 1.491, df = 1, p = .224), but did show statistically significant differences on checklist scores for case 3 (final case, F = 12.238, df = 1, p = .001). Strong interrater reliability was found between the faculty physician and observed nursing students (ICC = .89). CONCLUSIONS: Immediate feedback significantly improves student checklist scores with a mock paging program. This finding suggests that coaching with feedback may have advantages above self-regulated learning.

Case-Based Teaching: Does the Addition of High-Fidelity Simulation Make a Difference in Medical Students' Clinical Reasoning Skills?

CONTEXT: Situativity theory posits that learning and the development of clinical reasoning skills are grounded in context. In case-based teaching, this context comes from recreating the clinical environment, through emulation, as with manikins, or description. In this study, we sought to understand the difference in student clinical reasoning abilities after facilitated patient case scenarios with or without a manikin. METHODS: Fourth-year medical students in an internship readiness course were randomized into patient case scenarios without manikin (control group) and with manikin (intervention group) for a chest pain session. The control and intervention groups had identical student-led case progression and faculty debriefing objectives. Clinical reasoning skills were assessed after the session using a 64-question script concordance test (SCT). The test was developed and piloted prior to administration. Hospitalist and emergency medicine faculty responses on the test items served as the expert standard for scoring. RESULTS: Ninety-six students were randomized to case-based sessions with (n = 48) or without (n = 48) manikin. Ninety students completed the SCT (with manikin n = 45, without manikin n = 45). A statistically significant mean difference on test performance between the two groups was found (t = 3.059, df = 88, p = .003), with the manikin group achieving higher SCT scores. CONCLUSION: Use of a manikin in simulated patient case discussion significantly improves students' clinical reasoning skills, as measured by SCT. These results suggest that using a manikin to simulate a patient scenario situates learning, thereby enhancing skill development.

Residents' perspective on professionalism in the learning environment.

All accrediting organizations for medical education in the US require teaching hospitals to ensure the learning environment fosters professionalism behaviors of trainees and faculty. This study analyzes the learning environment of professionalism from the perspective of residents. An on-line anonymous survey that explored the learning climate of professionalism was sent to all residents at the University of Virginia in 2013-14. Residents rated their personal commitment, their residency program's, and the institution's commitment to demonstrating professionalism behaviors, described professionalism education, reasons for not participating in curricular offerings, the quality of role modeling, and barriers to professionalism. Nearly half the residents completed the survey (47%, N = 365/771). Residents rated their personal commitment and commitment of their residency program significantly greater than the institution's commitment to demonstrating professionalism.(p < 0.001) They noted only 25% of faculty modeled these behaviors all the time; and more than half stated poor role modeling impacted their attitudes about the importance of professionalism. Other areas in need of improvement include communicating with patients with cultural differences, and inter-professional teamwork. Despite accreditation requirements for learning environments, residency curricula, and faculty development programs to promote professionalism, residents perceive their commitment to professionalism greater than the institution where they work.

Development and evaluation of an instrument for assessing brief behavioral change interventions.

OBJECTIVE: To develop an observational coding instrument for evaluating the fidelity and quality of brief behavioral change interventions based on the behavioral theories of the 5 A's, Stages of Change and Motivational Interviewing. METHODS: Content and face validity were assessed prior to an intervention where psychometric properties were evaluated with a prospective cohort of 116 medical students. Properties assessed included the inter-rater reliability of the instrument, internal consistency of the full scale and sub-scales and descriptive statistics of the instrument. Construct validity was assessed based on student's scores. RESULTS: Inter-rater reliability for the instrument was 0.82 (intraclass correlation). Internal consistency for the full scale was 0.70 (KR20). Internal consistencies for the sub-scales were as follows: MI intervention component (KR20=.7); stage-appropriate MI-based intervention (KR20=.55); MI spirit (KR20=.5); appropriate assessment (KR20=.45) and appropriate assisting (KR20=.56). CONCLUSIONS: The instrument demonstrated good inter-rater reliability and moderate overall internal consistency when used to assess performing brief behavioral change interventions by medical students. PRACTICE IMPLICATIONS: This practical instrument can be used with minimal training and demonstrates promising psychometric properties when evaluated with medical students counseling standardized patients. Further testing is required to evaluate its usefulness in clinical settings.

A PDA-based counseling tool for improving medical student smoking cessation counseling.

BACKGROUND AND OBJECTIVES: There is little research on training medical students in smoking cessation counseling (SCC). This study aimed to determine if a personal digital assistant (PDA)-based SCC tool can improve medical student SCC. METHODS: We conducted a randomized, controlled trial with third-year medical students. SCC behaviors, comfort, and knowledge were assessed using a validated survey before students attended a workshop on SCC. Student groups were then randomized to receive a paper-based reminder tool or the reminder plus a PDA-based SCC tool. The validated survey was repeated upon clerkship completion, and a videotaped standardized patient interview was assessed by trained reviewers using a 24-item SCC checklist. Focus groups assessed satisfaction with the PDA tool, usability, and barriers to use. RESULTS: SCC behaviors, knowledge, and comfort increased among all participants, with no statistical differences between groups. The PDA tool group performed 62% of key SCC activities during the videotaped interview, while the control group performed 69%. Students reported discomfort using the PDA with patients, lack of time, and lack of training as barriers to use of the tool. CONCLUSIONS: We demonstrated improvement of SCC skills by third-year medical students using a workshop combined with a supplemental reference tool. However, a PDA-based tool did not increase key SCC behaviors compared with a paper-based reminder. For a PDA intervention to be effective in this setting, the tool must be simplified and additional training provided.

A change to pass/fail grading in the first two years at one medical school results in improved psychological well-being.

PURPOSE: To measure the impact of a change in grading system in the first two years of medical school, from graded (A, B, C, D, F) to pass/fail, on medical students' academic performance, attendance, residency match, satisfaction, and psychological well-being. METHOD: For both the graded and pass/fail classes, objective data were collected on academic performance in the first- and second-year courses, the clerkships, United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK), and residency placement. Self-report data were collected using a Web survey (which included the Dupuy General Well-Being Schedule) administered each of the first four semesters of medical school. The study was conducted from 2002 to 2007 at the University of Virginia School of Medicine. RESULTS: The pass/fail class exhibited a significant increase in well-being during each of the first three semesters of medical school relative to the graded class, greater satisfaction with the quality of their medical education during the first four semesters of medical school, and greater satisfaction with their personal lives during the first three semesters of medical school. The graded and pass/fail classes showed no significant differences in performance in first- and second-year courses, grades in clerkships, scores on USMLE Step 1 and Step 2CK, success in residency placement, and attendance at academic activities. CONCLUSIONS: A change in grading from letter grades to pass/fail in the first two years of medical school conferred distinct advantages to medical students, in terms of improved psychological well-being and satisfaction, without any reduction in performance in courses or clerkships, USMLE test scores, success in residency placement, or level of attendance.

A handheld computer smoking intervention tool and its effects on physician smoking cessation counseling.

OBJECTIVE: The objective of this study was to evaluate a handheld computer smoking cessation intervention tool designed to assist physicians in their smoking cessation counseling with patients. METHODS: This study used a pre/post survey design, with a 4-month trial period for the software. Study participants included 22 faculty and resident physicians from the University of Virginia. Paired samples t tests were used to assess mean differences in the 4 main subscales (physician behavior, attitudes, comfort related to counseling patients about smoking cessation, and knowledge). RESULTS: No statistically significant mean differences were found for physician behavior (mean increase = 0.44, P = .55) or physician attitude (mean increase = 0.44, P = .16). A statistically significant mean increase of 2.29 was observed for the physician comfort subscale (t = 3.87, df = 16, P = .001). Physicians indicated improved comfort in counseling patients about smoking cessation (P = .007) and improved comfort in using the Public Health Service Clinical Practice Guidelines (P = .012). CONCLUSION: Physician comfort level in counseling patients about smoking cessation can be improved through handheld computer software. When used in conjunction with other practice modifications, this tool has the potential to improve physician smoking cessation intervention practices.