Is It All About the Form? Norm- vs Criterion-Referenced Ratings and Faculty Inter-Rater Reliability.

Background: Little research to date has examined the quality of data obtained from resident performance evaluations. This study sought to address this need and compared inter-rater reliability obtained from norm-referenced and criterion-referenced evaluation scaling approaches for faculty completing resident performance evaluations. Methods: Resident performance evaluation data were examined from 2 institutions (3 programs, 2 internal medicine and 1 surgery; 426 residents in total), with 4 evaluation forms: 2 criterion-referenced (1 with an additional norm-referenced item) and 2 norm-referenced. Faculty inter-rater reliability was calculated with intraclass correlation coefficients (ICCs) (1,10) for each competency area within the form. ICCs were transformed to z-scores, and 95% CIs were computed. Reliabilities for each evaluation form and competency, averages within competency, and averages within scaling type were examined. Results: Inter-rater reliability averages were higher for all competencies that used criterion-referenced scaling relative to those that used norm-referenced scaling. Aggregate scores of all independent categories (competencies and the items assessing overall competence) for criterion-referenced scaling demonstrated higher reliability (z=1.37, CI 1.26-1.48) than norm-referenced scaling (z=0.88, CI 0.77-0.99). Moreover, examination of the distributions of composite scores (average of all competencies and raters for each individual being rated) suggested that the criterion-referenced evaluations better represented the performance continuum. Conclusion: Criterion-referenced evaluation approaches appear to provide superior inter-rater reliability relative to norm-referenced evaluation scaling approaches. Although more research is needed to identify resident evaluation best practices, using criterion-referenced scaling may provide more valid data than norm-referenced scaling.

Team FIRST framework: Identifying core teamwork competencies critical to interprofessional healthcare curricula.

Interprofessional healthcare team function is critical to the effective delivery of patient care. Team members must possess teamwork competencies, as team function impacts patient, staff, team, and healthcare organizational outcomes. There is evidence that team training is beneficial; however, consensus on the optimal training content, methods, and evaluation is lacking. This manuscript will focus on training content. Team science and training research indicates that an effective team training program must be founded upon teamwork competencies. The Team FIRST framework asserts there are 10 teamwork competencies essential for healthcare providers: recognizing criticality of teamwork, creating a psychologically safe environment, structured communication, closed-loop communication, asking clarifying questions, sharing unique information, optimizing team mental models, mutual trust, mutual performance monitoring, and reflection/debriefing. The Team FIRST framework was conceptualized to instill these evidence-based teamwork competencies in healthcare professionals to improve interprofessional collaboration. This framework is founded in validated team science research and serves future efforts to develop and pilot educational strategies that educate healthcare workers on these competencies.

Resident Engagement in a Wellness Program in a Large Academic Residency: A Follow-Up After Two Years of Wellness.

INTRODUCTION: The rate of burnout among residents has reached an alarming level and negatively impacts learning, attrition, and patient care. We implemented a comprehensive Surgery Resident Wellness Program in a large academic program and aimed to assess the degree this initiative improved resident burnout based on validated burnout and well-being models utilizing voluntary surveys. MATERIALS AND METHODS: A voluntary survey was sent to surgical residents and included the Maslach Burnout Inventory and The Psychological General Well-Being Index. These were graded according to the respective validated scale for each test. The survey was administered prior to establishing a wellness program, and at 6 month- and 24 month-intervals following its establishment. These data were analyzed using univariate analysis based on survey data from each of the time points. RESULTS: The survey had a 51% response rate. The baseline, 6-month, and 24-month mean scores for MBI personal accomplishment were 32.80, 32.91, and 35.57, emotional exhaustion was 23.48, 23.0, and 24.42, and depersonalization were 10.94, 9.54, and 11.11. Resident burnout was present in 16 of 31 (51.61%), 14 of 33 (42.42%), and 12 of 27 (44.44%) residents at each time point. The rate of change in resident burnout was -17.8% at 6 months and -13.9% at 24 months. The averaged PGWBI global scores were 66, 73, and 83 among the participants who took the survey at each time interval. CONCLUSIONS: Conclusion Implementation of a structured wellness program was associated with a decrease in resident burnout and an increase in overall resident wellness.

The Fuel Gauge: A Simple Tool for Assessing General Surgery Resident Well-Being.

OBJECTIVE: Resident well-being is an increasingly relevant issue in medical education; however, there is no consensus on how to best measure well-being. The "fuel gauge," is a simple, easy-to-use tool developed to measure resident well-being and previously applied in an Internal Medicine Residency Program at our institution. The current study sought to evaluate its acceptability and usefulness in a surgery program. DESIGN: Weekly fuel gauge data was retrospectively collected from August 2017 through December 2018 along with resident Postgraduate Year designations. SETTING: This study was conducted at a single, large general surgery residency program that rotates through a variety of hospitals, including a University hospital, a large county hospital, a Veterans Affairs hospital, and a freestanding Children's hospital. PARTICIPANTS: Categorical general surgery residents at every level of training as well as preliminary interns and off service intern rotators from urology, oral and maxillofacial surgery, and otolaryngology were eligible for the study. Fuel gauge submissions which did not denote a score were excluded from analysis. RESULTS: Out of 130 residents, 103 (79.2%) completed at least 1 fuel gauge assessment with a weekly mean response rate of 41.5%. Low scores were submitted by 39.8% of resident participants. Narrative feedback was provided in 6.2% of submissions with increased length associated with decreased fuel gauge score. CONCLUSIONS: The fuel gauge was well accepted by a large general surgery program with no decline in participation rates over the study period. The tool provided residents with a direct line of communication with their program's administration, and a feasible way for the program director's office to monitor and identify residents who were struggling with regard to their well-being.

Checking in on check-out: Survey of Learning Priorities in Primary Care Residency Teaching Clinics.

Background: Despite focus on increasing the quality of ambulatory education training, few studies have examined residents' perceptions of learning during case discussions with their preceptors (i.e., "check-out"). The objective of this study was to assess the difference between residents' and preceptors' perceptions of behaviors that should occur during check-out discussions. Methods: We conducted a cross-sectional survey of categorical internal medicine and family medicine residents and preceptors. The survey was distributed electronically and assessed 20 components of the check-out discussion. Results: Of 38 preceptors, 22 (61%) completed the survey. Of 172 residents, 82 (48%) completed the survey. For residents, we identified discrepancies in desired and perceived check-out behaviors. Specifically, utilizing a dependent sample t-test, residents felt that all 20 areas needed additional teaching during check-out (P < 0.05). Preceptors believed that demonstrating physical examination skills in the patient room during check-out was significantly more important than did residents (P = 0.01). Increasing years of preceptor experience did not statistically relate to their valuation of components important to residents. Discussion: Our research highlighted a major deficiency in training in the check-out process, with residents desiring more patient management education in all components. Moreover, faculty and residents do not necessarily agree with what is an important focus in the "teachable moment." Our results serve as a training needs assessment for future faculty development seminars and highlight the need to consider resident learning needs in general.

Trauma Surgeon and Palliative Care Physician Attitudes Regarding Goals-of-Care Delineation for Injured Geriatric Patients.

BACKGROUND: The value of defining goals of care (GoC) for geriatric patients is well known to the palliative care community but is a newer concept for many trauma surgeons. Palliative care specialists and trauma surgeons were surveyed to elicit the specialties' attitudes regarding (1) importance of GoC conversations for injured seniors; (2) confidence in their own specialty's ability to conduct these conversations; and (3) confidence in the ability of the other specialty to do so. METHODS: A 13-item survey was developed by the steering committee of a multicenter, palliative care-focused consortium and beta-tested by trauma surgeons and palliative care specialists unaffiliated with the consortium. The finalized instrument was electronically circulated to active physician members of the American Association for the Surgery of Trauma and American Academy for Hospice and Palliative Medicine. RESULTS: Respondents included 118 trauma surgeons (8.8%) and 244 palliative care specialists (5.7%). Palliative physicians rated being more familiar with GoC, were more likely to report high-quality training in performing conversations, believed more palliative specialists were needed in intensive care units, and had more interest in conducting conversations relative to trauma surgeons. Both groups believed themselves to perform GoC discussions better than the other specialty perceived them to do so and favored their own specialty leading team discussions. CONCLUSIONS: Both groups believe themselves to conduct GoC discussions for injured seniors better than the other specialty perceived them to do so, which led to disparate views on the optimal leadership of these discussions.

Burn Surgeon and Palliative Care Physician Attitudes Regarding Goals of Care Delineation for Burned Geriatric Patients.

Palliative care specialists (PCS) and burn surgeons (BS) were surveyed regarding: 1) importance of goals of care (GoC) conversations for burned seniors; 2) confidence in their own specialty's ability to conduct these conversations; and 3) confidence in the ability of the other specialty to do so. A 13-item survey was developed by the steering committee of a multicenter consortium dedicated to palliative care in the injured geriatric patient and beta-tested by BS and PCS unaffiliated with the consortium. The finalized instrument was electronically circulated to active physician members of the American Burn Association and American Academy for Hospice and Palliative Medicine. Forty-five BS (7.3%) and 244 PCS (5.7%) responded. Palliative physicians rated being more familiar with GoC, were more comfortable having a discussion with laypeople, were more likely to have reported high-quality training in performing conversations, believed more palliative specialists were needed in intensive care units, and had more interest in conducting conversations relative to BS. Both groups believed themselves to perform GoC discussions better than the other specialty perceived them to do so. BS favored leading team discussions, whereas palliative specialists preferred jointly led discussions. Both groups agreed that discussions should occur within 72 hours of admission. Both groups believe themselves to conduct GoC discussions for burned seniors better than the other specialty perceived them to do so, which led to disparate views on perceptions for the optimal leadership of these discussions.

Resident Fuel Levels: Reframing, Assessing, and Addressing Well-Being.

BACKGROUND: To optimize resident learning, programs need to readily assess resident well-being. There is a lack of easy-to-use, acceptable instruments for this task. OBJECTIVE: We created a well-being "fuel gauge," and assessed the acceptability and feasibility of this weekly electronic communication pipeline for residents to report and discuss their well-being. METHODS: A well-being fuel gauge assessment was administered weekly over the course of 1 academic year (July 2016 to June 2017) in a large internal medicine residency program. The well-being gauge asked residents to report their fuel levels using a 1 to 5 Likert-type scale (1, empty; 3, half tank; and 5, full tank). Residents who provided low scores (1 or 2) were contacted by program leadership, and the program director sent weekly e-mail updates that addressed residents' comments on their well-being fuel gauge. RESULTS: Of 163 residents, 149 (91%) provided data on their well-being fuel gauge, with a 53% average weekly response rate. Fifty-four percent of residents (80 of 149) reported a low score over the course of the year, and 4 residents only used the assessment to report a low score. Comments on average consisted of 280 characters (SD = 357) and were lengthier and more prevalent with lower fuel gauge scores. We analyzed the relationship between scores and comments. CONCLUSIONS: The well-being fuel gauge was well accepted by most residents and was easy to administer and to oversee by program directors. It facilitated ongoing monitoring of well-being and follow-up to address factors contributing to low well-being.