Group Assessment of Resident Performance: Valuable for Program Director Judgment?

BACKGROUND: Group discussion of resident performance is an emerging assessment approach in postgraduate medical education. However, groups do not necessarily make better decisions than individuals. OBJECTIVE: This study examined how group meetings concerning the assessment of residents take place, what information is shared during the meetings, and how this influences program directors' judgment of resident performance. METHODS: In 2017, the researchers observed 10 faculty group meetings where resident performance was discussed and interviewed the program directors within a month after the meetings. We used a thematic framework analysis to identify themes from the transcribed meetings and interviews. RESULTS: The information shared by group members during the meetings had 2 aims: (1) forming a judgment about the residents, and (2) faculty development. Most group members shared information without written notes, most discussions were not structured by the program director, the major focus of discussions was on residents with performance concerns, and there was a lack of a shared mental model of resident performance. The program directors who benefited most from the meetings were those who thought group members were engaged and summarized the information after every discussion. CONCLUSIONS: Unstructured discussions and a lack of a shared mental model among group members impede effective information sharing about resident performance with a developmental approach. Structured discussions with an equal amount of discussion time for every resident and creating a shared mental model about the purpose of the discussions and the assessment approach could enhance use of a developmental approach to assessing resident performance.

Using conversation analysis to explore feedback on resident performance.

Feedback on clinical performance of residents is seen as a fundamental element in postgraduate medical education. Although literature on feedback in medical education is abundant, many supervisors struggle with providing this feedback and residents experience feedback as insufficiently constructive. With a detailed analysis of real-world feedback conversations, this study aims to contribute to the current literature by deepening the understanding of how feedback on residents' performance is provided, and to formulate recommendations for improvement of feedback practice. Eight evaluation meetings between program directors and residents were recorded in 2015-2016. These meetings were analyzed using conversation analysis. This is an ethno-methodological approach that uses a data-driven, iterative procedure to uncover interactional patterns that structure naturally occurring, spoken interaction. Feedback in our data took two forms: feedback as a unidirectional activity and feedback as a dialogic activity. The unidirectional feedback activities prevailed over the dialogic activities. The two different formats elicit different types of resident responses and have different implications for the progress of the interaction. Both feedback formats concerned positive as well as negative feedback and both were often mitigated by the participants. Unidirectional feedback and mitigating or downplaying feedback is at odds with the aim of feedback in medical education. Dialogic feedback avoids the pitfall of a program director-dominated conversation and gives residents the opportunity to take ownership of their strengths and weaknesses, which increases chances to change resident behavior. On the basis of linguistic analysis of our real-life data we suggest implications for feedback conversations.

Design and evaluation of a clinical competency committee.

INTRODUCTION: In postgraduate medical education, group decision-making has emerged as an essential tool to evaluate the clinical progress of residents. Clinical competency committees (CCCs) have been set up to ensure informed decision-making and provide feedback regarding performance of residents. Despite this important task, it remains unclear how CCCs actually function in practice and how their performance should be evaluated. METHODS: In the prototyping phase of a design-based approach, a CCC meeting was developed, using three theoretical design principles: (1) data from multiple assessment tools and multiple perspectives, (2) a shared mental model and (3) structured discussions. The meetings were held in a university children's hospital and evaluated using observations, interviews with CCC members and an open-ended questionnaire among residents. RESULTS: The structured discussions during the meetings provided a broad outline of resident performance, including identification of problematic and excellent residents. A shared mental model about the assessment criteria had developed over time. Residents were not always satisfied with the feedback they received after the meeting. Feedback that had been provided to a resident after the first CCC meeting was not addressed in the second meeting. DISCUSSION: The principles that were used to design the CCC meeting were feasible in practice. Structured discussions, based on data from multiple assessment tools and multiple perspectives, provided a broad outline of resident performance. Residency programs that wish to implement CCCs can build on our design principles and adjust the prototype to their particular context. When running a CCC, it is important to consider feedback that has been provided to a resident after the previous meeting and to evaluate whether it has improved the resident's performance.

Judging residents' performance: a qualitative study using grounded theory.

BACKGROUND: Although program directors judge residents' performance for summative decisions, little is known about how they do this. This study examined what information program directors use and how they value this information in making a judgment of residents' performance and what residents think of this process. METHODS: Sixteen semi-structured interviews were held with residents and program directors from different hospitals in the Netherlands in 2015-2016. Participants were recruited from internal medicine, surgery and radiology. Transcripts were analysed using grounded theory methodology. Concepts and themes were identified by iterative constant comparison. RESULTS: When approaching semi-annual meetings with residents, program directors report primarily gathering information from the following: assessment tools, faculty members and from their own experience with residents. They put more value on faculty's comments during meetings and in the corridors than on feedback provided in the assessment tools. They are influenced by their own beliefs about learning and education in valuing feedback. Residents are aware that faculty members discuss their performance in meetings, but they believe the assessment tools provide the most important proof to demonstrate their clinical competency. CONCLUSIONS: Residents think that feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents' performance. They rely heavily on remarks of faculty in meetings instead. Therefore, residents' performance may be better judged in group meetings that are organised to enhance optimal information sharing and decision making about residents' performance.