A multispecialty ethnographic study of clinical competency committees (CCCs).

PURPOSE: Clinical competency committees (CCCs) assess residents' performance on their specialty specific milestones, however there is no 'one-size fits all' blueprint for accomplishing this. Thus, CCCs have had to develop their own procedures. The goal of this study was to examine these efforts to assist new programs embarking on this venture and established programs looking to improve their CCC practices and processes. METHODS: We purposefully sampled CCCs across multiple specialties and institutions. Data from three sources were triangulated: (1) online demographic survey, (2) ethnographic observations of CCC meetings and (3) post-observation semi-structured interviews with the program director and/or CCC chairperson. Template analysis was used to build the coding structure. RESULTS: Sixteen observations were completed with 15 different CCCs at 9 institutions. Three main thematic categories that impact the operations of CCCs emerged: (1) Membership structure and members roles, (2) Roles of the CCC in residency and 3) CCC processes, including trainee presentation to the committee and decision-making. While effective practices were observed, substantial variation existed in all three thematic areas. CONCLUSIONS: While CCCs used some known effective practices, substantial variation in structure and processes was notable across CCCs. Future work should explore the impact of this variation on educational outcomes.

Stages of Milestones Implementation: A Template Analysis of 16 Programs Across 4 Specialties.

BACKGROUND: Since 2013, US residency programs have used the competency-based framework of the Milestones to report resident progress and to provide feedback to residents. The implementation of Milestones-based assessments, clinical competency committee (CCC) meetings, and processes for providing feedback varies among programs and warrants systematic examination across specialties. OBJECTIVE: We sought to determine how varying assessment, CCC, and feedback implementation strategies result in different outcomes in resource expenditure and stakeholder engagement, and to explore the contextual forces that moderate these outcomes. METHODS: From 2017 to 2018, interviews were conducted of program directors, CCC chairs, and residents in emergency medicine (EM), internal medicine (IM), pediatrics, and family medicine (FM), querying their experiences with Milestone processes in their respective programs. Interview transcripts were coded using template analysis, with the initial template derived from previous research. The research team conducted iterative consensus meetings to ensure that the evolving template accurately represented phenomena described by interviewees. RESULTS: Forty-four individuals were interviewed across 16 programs (5 EM, 4 IM, 5 pediatrics, 3 FM). We identified 3 stages of Milestone-process implementation, including a resource-intensive early stage, an increasingly efficient transition stage, and a final stage for fine-tuning. CONCLUSIONS: Residency program leaders can use these findings to place their programs along an implementation continuum and gain an understanding of the strategies that have enabled their peers to progress to improved efficiency and increased resident and faculty engagement.

What resources do clinical competency committees (CCCs) require to do their work? A pilot study comparing CCCs across specialties.

PURPOSE: Although a growing literature describes how clinical competency committees (CCCs) make decisions about trainees' clinical performance, little is known about the resources these committees need to perform their work. In this pilot study, we examined key characteristics of CCC processes across generalist and surgical specialties. This study intended to clarify topic areas for further investigation. METHODS: A cross-sectional web-based survey of CCC chairpersons at two institutions was conducted in 2017. Survey items were designed to describe not only CCC work, including types of assessment data used and time spent discussing learners, but also resource needs such as faculty development, institutional support, and protected time for members. RESULTS: The response rate was 59% (16/27). Only 44% offered faculty development to members. There was strong support for the institution to assist with faculty development for CCC members (81.25%), workshops for program coordinators (87.5%) and optimizing residency management software to organize assessment data (81.25%). Most respondents did not receive protected time for their committee work (93.75%). CONCLUSIONS: Further studies are needed to elucidate whether CCC work varies across specialties and the associated committee resource needs. There may be opportunities for institutions to assist CCCs with resources across specialties.

How do clinical competency committees use different sources of data to assess residents' performance on the internal medicine milestones?A mixed methods pilot study.

PURPOSE: This study examines how Clinical Competency Committees (CCCs) synthesize assessment data to make judgments about residents' clinical performances. METHODS: Between 2014 and 2015, after four six-month reporting periods to the Accreditation Council for Graduate Medical Education (ACGME), 7 of 16 CCC faculty at Rush University Medical Center completed questionnaires focused on their perspectives about rating residents on their achievement of the milestones and participated in a focus group. Qualitative data were analyzed using grounded theory. Milestones ratings for two six-month ACGME reporting cycles (n = 100 categorical residents) were also analyzed. RESULTS: CCC members weighted resident rotation ratings highest (weight = 37%), followed by faculty rotation comments (weight = 27%) and personal experience with residents (weight = 14%) for making judgments about learner's milestone levels. Three assessment issues were identified from qualitative analyses: (1) "design issues" (e.g. problems with available data or lack thereof); (2) "synthesis issues" (e.g. factors influencing ratings and decision-making processes) and (3) "impact issues" (e.g. how CCC generated milestones ratings are used). CONCLUSIONS: Identifying factors that affect assessment at all stages of the CCC process can contribute to improving assessment systems, including support for faculty development for CCCs. Recognizing challenges in synthesizing first and second-hand assessment data is an important step in understanding the CCC decision-making process.