How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Evaluating the Association of a Core EPA-Oriented Patient Handover Curriculum on Medical Students' Self-reported Frequency of Observation and Skill Acquisition.

Introduction: "Giving or receiving a patient handover to transition patient care responsibility" is one of the thirteen Core Entrustable Professional Activities (Core EPAs) for Entering Residency. However, implementing a patient handover curriculum in undergraduate medical education (UME) remains challenging. Educational leaders in the multi-institutional Core EPA8 pilot workgroup developed a longitudinal patient handover UME curriculum that was implemented at two pilot institutions. Materials and Methods: We utilized multi-school graduation questionnaire data to assess the association of our patient handover curriculum on self-reported frequency of observation/feedback and skill acquisition by comparing data from the shared curriculum schools to data from other Core EPA pilot schools (three schools with school-specific curriculum; five without a dedicated handover curriculum). Questionnaire data from 1,278 graduating medical students of the class of 2020 from all ten Core EPA pilot schools were analyzed. Results: Graduates from the two medical schools that implemented the shared patient handover curriculum reported significantly greater frequency of handover observation/feedback compared to graduates at the other schools (school-specific curriculum (p < .05) and those without a handover curriculum (p < .05)). Graduates from the two shared approach schools also more strongly agreed that they possessed the skill to perform handovers compared to graduates from the other eight pilot schools that did not implement this curriculum. Conclusion: The findings of this study suggest that the implementation of a multi-institutional Core EPA-based curricular model for teaching and assessing patient handovers was successful and could be implemented at other UME institutions.

Soliciting feedback on the wards: a peer-to-peer workshop.

BACKGROUND: For medical students, soliciting feedback is a critical but difficult skill that merits proper training. This skill may be taught effectively by peers who have experienced challenges with feedback on the wards. METHODS: Two medical students developed and taught a workshop on feedback skills. The workshop was presented to 248 third-year students. The workshop trained students in soliciting, receiving and responding to feedback through interactive discussions of case scenarios. Students were given pre- and post-surveys to assess changes in their confidence in and attitudes towards the feedback process. RESULTS: There were statistically significant increases in students' likeliness to solicit feedback and confidence in their ability to solicit feedback. Students' view on the importance of feedback did not change. The most commonly cited barriers to feedback were time constraints, fear of negative feedback, emotions and skills when asking for feedback, and student-mentor relationship. The content the students valued the most was management of internal triggers to negative feedback. Students noted that the peer-to-peer format, case scenarios, and interactive questions were strengths of the workshop. DISCUSSION: Although medical students recognise the importance of feedback, they often lack the confidence and skills to obtain quality feedback. A peer-to-peer workshop on soliciting, receiving, and responding to feedback can be an effective method to improve students' confidence and skills in this area. More research needs to be done to conclude if this workshop increases the instances of students soliciting high-quality feedback on the wards and improves clinical performance.

Faculty Development Revisited: A Systems-Based View of Stakeholder Development to Meet the Demands of Entrustable Professional Activity Implementation.

In 2015, the Association of American Medical Colleges implemented an interinstitutional pilot of 13 core entrustable professional activities (EPAs) for entering residency, activities that entering residents should be expected to perform with indirect supervision. The pilot included a concept group on faculty development; this group previously offered a shared mental model focused on the development of faculty who devote their efforts to clinical teaching and assessment for learning and entrustment decision making. In this article, the authors draw from the literature of competency-based education to propose what is needed in overall approaches to faculty development to prepare institutions for undergraduate EPA implementation.Taking a systems-based view that defines the necessary tasks of EPA implementation, the authors move beyond the variably used term "faculty" and enumerate a comprehensive list of institutional stakeholders who can meaningfully support and/or engage in the relationships and organizational processes required for EPA learning and assessment. They consider each group's responsibilities and development needs according to five domains delineated by Steinert: teaching improvement, leadership and management, research-building capacity, academic career building, and organizational change.The authors argue that the EPA framework addresses barriers posed with the use of a competency-based framework. By facilitating the communication required for organizational change, enabling valid assessment with comprehensive yet feasible levels of faculty development, and incorporating all relevant data on student professional behavior into summative assessment decisions, EPAs may offer a clearer path toward the goal of competency-based education.

Relieving the Bottleneck: An Investigation of Barriers to Expansion of Supervision Networks at Genetic Counseling Training Programs.

Rapid growth in the demand for genetic counselors has led to a workforce shortage. There is a prevailing assumption that the number of training slots for genetic counseling students is linked to the availability of clinical supervisors. This study aimed to determine and compare barriers to expansion of supervision networks at genetic counseling training programs as perceived by supervisors, non-supervisors, and Program Directors. Genetic counselors were recruited via National Society of Genetic Counselors e-blast; Program Directors received personal emails. Online surveys were completed by 216 supervisors, 98 non-supervisors, and 23 Program Directors. Respondents rated impact of 35 barriers; comparisons were made using Kruskal-Wallis and Wilcoxon ranked sum tests. Half of supervisors (51%) indicated willingness to increase supervision. All non-supervisors were willing to supervise. However, all agreed that being too busy impacted ability to supervise, highlighted by supervisors' most impactful barriers: lack of time, other responsibilities, intensive nature of supervision, desire for breaks, and unfilled positions. Non-supervisors noted unique barriers: distance, institutional barriers, and non-clinical roles. Program Directors' perceptions were congruent with those of genetic counselors with three exceptions they rated as impactful: lack of money, prefer not to supervise, and never been asked. In order to expand supervision networks and provide comprehensive student experiences, the profession must examine service delivery models to increase workplace efficiency, reconsider the supervision paradigm, and redefine what constitutes a countable case or place value on non-direct patient care experiences.

Systems-based practice in graduate medical education: systems thinking as the missing foundational construct.

BACKGROUND: Since 2001, residencies have struggled with teaching and assessing systems-based practice (SBP). One major obstacle may be that the competency alone is not sufficient to support assessment. We believe the foundational construct underlying SBP is systems thinking, absent from the current Accreditation Council for Graduate Medical Education competency language. SUMMARY: Systems thinking is defined as the ability to analyze systems as a whole. The purpose of this article is to describe psychometric issues that constrain assessment of SBP and elucidate the role of systems thinking in teaching and assessing SBP. CONCLUSION: Residency programs should incorporate systems thinking models into their curricula. Trainees should be taught to understand systems at an abstract level, in order to analyze their own healthcare systems, and participate in quality and patient safety activities. We suggest that a developmental trajectory for systems thinking be developed, similar to the model described by Dreyfus and Dreyfus.

Decoding the learning environment of medical education: a hidden curriculum perspective for faculty development.

Medical student literature has broadly established the importance of differentiating between formal-explicit and hidden-tacit dimensions of the physician education process. The hidden curriculum refers to cultural mores that are transmitted, but not openly acknowledged, through formal and informal educational endeavors. The authors extend the concept of the hidden curriculum from students to faculty, and in so doing, they frame the acquisition by faculty of knowledge, skills, and values as a more global process of identity formation. This process includes a subset of formal, formative activities labeled "faculty development programs" that target specific faculty skills such as teaching effectiveness or leadership; however, it also includes informal, tacit messages that faculty absorb. As faculty members are socialized into faculty life, they often encounter conflicting messages about their role. In this article, the authors examine how faculty development programs have functioned as a source of conflict, and they ask how these programs might be retooled to assist faculty in understanding the tacit institutional culture shaping effective socialization and in managing the inconsistencies that so often dominate faculty life.

Enhancing competency in professionalism: targeting resident advance directive education.

BACKGROUND: Education about advance directives typically is incorporated into medical school curricula and is not commonly offered in residency. Residents' experiences with advance directives are generally random, nonstandardized, and difficult to assess. In 2008, an advance directive curriculum was developed by the Scott & White/Texas A&M University System Health Science Center College of Medicine (S&W/Texas A&M) internal medicine residency program and the hospital's legal department. A pilot study examining residents' attitudes and experiences regarding advance directives was carried out at 2 medical schools. METHODS: In 2009, 59 internal medicine and family medicine residents (postgraduate year 2-3 [PGY-2, 3]) completed questionnaires at S&W/Texas A&M (n  =  32) and The University of Texas Medical School at Houston (n  =  27) during a validation study of knowledge about advance directives. The questionnaire contained Likert-response items assessing attitudes and practices surrounding advance directives. Our analysis included descriptive statistics and analysis of variance (ANOVA) to compare responses across categories. RESULTS: While 53% of residents agreed/strongly agreed they had "sufficient knowledge of advance directives, given my years of training," 47% disagreed/strongly disagreed with that statement. Most (93%) agreed/strongly agreed that "didactic sessions on advance directives should be offered by my hospital, residency program, or medical school." A test of responses across residency years with ANOVA showed a significant difference between ratings by PGY-2 and PGY-3 residents on 3 items: "Advance directives should only be discussed with patients over 60," "I have sufficient knowledge of advance directives, given my years of training," and "I believe my experience with advance directives is adequate for the situations I routinely encounter." CONCLUSION: Our study highlighted the continuing need for advance directive resident curricula. Medical school curricula alone do not appear to be sufficient for residents' needs in this area.

The effect of monthly resident reflection on achieving rotation goals.

BACKGROUND: Reflection is a learning tool increasingly being used in medical education. It has not been well studied as a method of communicating resident rotation goals and objectives. PURPOSE: The purpose was to study the effect of monthly resident reflection on achieving curriculum goals. METHODS: During the first half of the 2005-06 academic year, obstetrics/gynecology residents at a community hospital received curriculum goals in a traditional written manner. During the second 6 months, the same trainees underwent 1-hr monthly structured reflection sessions on the rotation goals and objectives. RESULTS: Sixteen residents were studied. Compared to control, residents reported the rotation goals as better defined during reflection months. More trainees rated reflection as more valuable than the didactic time it replaced. All 16 residents documented more cumulative procedures in the Accreditation Council for Graduate Medical Education operative log Web site during reflection months for each of the three major categories; there was a mean increased number of procedures logged in Obstetrics (23.1 +/- 14.0, p < .001), Gynecology (18.3 +/- 10.9, p < .001), and Primary Care (21.1 +/- 23.5, p = .003). Overall, there was a mean of 62.6 +/- 32.1 (p < .001) more procedures entered during the reflection period, which translated to an 18% increase. CONCLUSIONS: Structured monthly reflection may lead to a higher success of achieving rotation goals.

A review of portfolio use in residency programs and considerations before implementation.

BACKGROUND: Portfolios, often described as collections of evidence, are discussed as a means of teaching or assessing the Accreditation Council for Graduate Medical Education competencies. Yet, it is unclear how many residency programs utilize portfolios. The purpose of this article is to (a) review the literature on portfolio use in graduate medical education; (b) examine efficacy of portfolio use, based upon studies in the field; and (c) offer a discussion of considerations for implementing portfolios. SUMMARY: Two searches of PubMed, OVID, JSTOR, SCOPUS, and FirstSearch Wilson Select were conducted between October 2006 and April 2007 to identify studies and articles related to portfolio usage. Thirty-nine articles met criteria and were reviewed. CONCLUSIONS: There is wide variation in how portfolios are utilized within U.S. residency programs. The challenge for graduate medical education is to create consensus on the definition and purpose of portfolios, such that best practices in portfolio implementation and assessment can be achieved.