Is It All About the Form? Norm- vs Criterion-Referenced Ratings and Faculty Inter-Rater Reliability.

Background: Little research to date has examined the quality of data obtained from resident performance evaluations. This study sought to address this need and compared inter-rater reliability obtained from norm-referenced and criterion-referenced evaluation scaling approaches for faculty completing resident performance evaluations. Methods: Resident performance evaluation data were examined from 2 institutions (3 programs, 2 internal medicine and 1 surgery; 426 residents in total), with 4 evaluation forms: 2 criterion-referenced (1 with an additional norm-referenced item) and 2 norm-referenced. Faculty inter-rater reliability was calculated with intraclass correlation coefficients (ICCs) (1,10) for each competency area within the form. ICCs were transformed to z-scores, and 95% CIs were computed. Reliabilities for each evaluation form and competency, averages within competency, and averages within scaling type were examined. Results: Inter-rater reliability averages were higher for all competencies that used criterion-referenced scaling relative to those that used norm-referenced scaling. Aggregate scores of all independent categories (competencies and the items assessing overall competence) for criterion-referenced scaling demonstrated higher reliability (z=1.37, CI 1.26-1.48) than norm-referenced scaling (z=0.88, CI 0.77-0.99). Moreover, examination of the distributions of composite scores (average of all competencies and raters for each individual being rated) suggested that the criterion-referenced evaluations better represented the performance continuum. Conclusion: Criterion-referenced evaluation approaches appear to provide superior inter-rater reliability relative to norm-referenced evaluation scaling approaches. Although more research is needed to identify resident evaluation best practices, using criterion-referenced scaling may provide more valid data than norm-referenced scaling.

The Fuel Gauge: A Simple Tool for Assessing General Surgery Resident Well-Being.

OBJECTIVE: Resident well-being is an increasingly relevant issue in medical education; however, there is no consensus on how to best measure well-being. The "fuel gauge," is a simple, easy-to-use tool developed to measure resident well-being and previously applied in an Internal Medicine Residency Program at our institution. The current study sought to evaluate its acceptability and usefulness in a surgery program. DESIGN: Weekly fuel gauge data was retrospectively collected from August 2017 through December 2018 along with resident Postgraduate Year designations. SETTING: This study was conducted at a single, large general surgery residency program that rotates through a variety of hospitals, including a University hospital, a large county hospital, a Veterans Affairs hospital, and a freestanding Children's hospital. PARTICIPANTS: Categorical general surgery residents at every level of training as well as preliminary interns and off service intern rotators from urology, oral and maxillofacial surgery, and otolaryngology were eligible for the study. Fuel gauge submissions which did not denote a score were excluded from analysis. RESULTS: Out of 130 residents, 103 (79.2%) completed at least 1 fuel gauge assessment with a weekly mean response rate of 41.5%. Low scores were submitted by 39.8% of resident participants. Narrative feedback was provided in 6.2% of submissions with increased length associated with decreased fuel gauge score. CONCLUSIONS: The fuel gauge was well accepted by a large general surgery program with no decline in participation rates over the study period. The tool provided residents with a direct line of communication with their program's administration, and a feasible way for the program director's office to monitor and identify residents who were struggling with regard to their well-being.

Trauma Surgeon and Palliative Care Physician Attitudes Regarding Goals-of-Care Delineation for Injured Geriatric Patients.

BACKGROUND: The value of defining goals of care (GoC) for geriatric patients is well known to the palliative care community but is a newer concept for many trauma surgeons. Palliative care specialists and trauma surgeons were surveyed to elicit the specialties' attitudes regarding (1) importance of GoC conversations for injured seniors; (2) confidence in their own specialty's ability to conduct these conversations; and (3) confidence in the ability of the other specialty to do so. METHODS: A 13-item survey was developed by the steering committee of a multicenter, palliative care-focused consortium and beta-tested by trauma surgeons and palliative care specialists unaffiliated with the consortium. The finalized instrument was electronically circulated to active physician members of the American Association for the Surgery of Trauma and American Academy for Hospice and Palliative Medicine. RESULTS: Respondents included 118 trauma surgeons (8.8%) and 244 palliative care specialists (5.7%). Palliative physicians rated being more familiar with GoC, were more likely to report high-quality training in performing conversations, believed more palliative specialists were needed in intensive care units, and had more interest in conducting conversations relative to trauma surgeons. Both groups believed themselves to perform GoC discussions better than the other specialty perceived them to do so and favored their own specialty leading team discussions. CONCLUSIONS: Both groups believe themselves to conduct GoC discussions for injured seniors better than the other specialty perceived them to do so, which led to disparate views on the optimal leadership of these discussions.

Burn Surgeon and Palliative Care Physician Attitudes Regarding Goals of Care Delineation for Burned Geriatric Patients.

Palliative care specialists (PCS) and burn surgeons (BS) were surveyed regarding: 1) importance of goals of care (GoC) conversations for burned seniors; 2) confidence in their own specialty's ability to conduct these conversations; and 3) confidence in the ability of the other specialty to do so. A 13-item survey was developed by the steering committee of a multicenter consortium dedicated to palliative care in the injured geriatric patient and beta-tested by BS and PCS unaffiliated with the consortium. The finalized instrument was electronically circulated to active physician members of the American Burn Association and American Academy for Hospice and Palliative Medicine. Forty-five BS (7.3%) and 244 PCS (5.7%) responded. Palliative physicians rated being more familiar with GoC, were more comfortable having a discussion with laypeople, were more likely to have reported high-quality training in performing conversations, believed more palliative specialists were needed in intensive care units, and had more interest in conducting conversations relative to BS. Both groups believed themselves to perform GoC discussions better than the other specialty perceived them to do so. BS favored leading team discussions, whereas palliative specialists preferred jointly led discussions. Both groups agreed that discussions should occur within 72 hours of admission. Both groups believe themselves to conduct GoC discussions for burned seniors better than the other specialty perceived them to do so, which led to disparate views on perceptions for the optimal leadership of these discussions.