Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.

Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment.

Background: Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique. Objective: To explore faculty's perceptions of the benefits and challenges associated with FORT. Methods: Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis. Results: All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness. Conclusions: Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial.

PURPOSE: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD: This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

Qualitative Study of Orthopedic Surgery Milestones 1.0: Burdens and Benefits.

OBJECTIVE: The purpose of this study was to explore the context and mechanisms by which the first set of Milestones impacted the processes of the Clinical Competency Committee, how programs have incorporated the Milestones into their program, and to understand more about the resident perspective in order to improve Orthopedic Surgery Milestones. DESIGN, SETTING, AND PARTICIPANTS: In 2020 all 201 accredited orthopedic surgery residency programs were invited to participate in qualitative telephone interviews to assess their experience with the Milestones and complete a supplemental online survey about their Clinical Competency Committee. Participants were comprised of a self-selected sample and complemented by a purposeful sample to ensure a breadth of perspectives. Interview data were analyzed using template analysis. Survey data were analyzed with descriptive statistics. RESULTS: Interviews were completed with 101 individuals from 47 programs (23% of all programs). The two overarching themes were implementation and impact of Milestones. Subthemes within implementation were substantial variability in approaches to Milestone use in curriculum and assessment, faculty development, and methods to introduce residents to the Milestones assessment framework. The large number of subcompetencies created a significant burden for almost all programs. The structure of the Milestones was also viewed as poorly aligned with the variable design of rotation schedules across programs. Milestones have the potential to offer valuable feedback for trainees and programs overall. CONCLUSIONS/APPLICATION: While some benefits were noted and most programs appreciated the intent, the structure and design of the initial set of orthopedic surgery Milestones created substantial challenges for these programs. The results of this study helped guide a revision of the Orthopedic Surgery Milestones. These results can also be used by program leadership to encourage reflection around past, current, and future utilization of the Milestones framework. Further research will be needed to determine the impact of the revision on programs.

The Effect and Use of Milestones in the Assessment of Neurological Surgery Residents and Residency Programs.

OBJECTIVES: The purpose of this study was to determine the effect of the Accreditation Council for Graduate Medical Education Milestones on the assessment of neurological surgery residents. The authors sought to determine the feasibility, acceptability, and utility of this new framework in making judgments of progressive competence, its implementation within programs, and the influence on curricula. Residents were also surveyed to elicit the effect of Milestones on their educational experience and professional development. DESIGN, SETTING, AND PARTICIPANTS: In 2015, program leadership and residents from 21 neurological surgery residency programs participated in an online survey and telephone interview in which they reflected on their experiences with the Milestones. Survey data were analyzed using descriptive statistics. Interview transcripts were analyzed using grounded theory. RESULTS: Response themes were categorized into 2 groups: outcomes of the Milestones implementation process, and facilitators and barriers. Because of Milestones implementation, participants reported changes to the quality of the assessment process, including the ability to identify struggling residents earlier and design individualized improvement plans. Some programs revised their curricula based on training gaps identified using the Milestones. Barriers to implementation included limitations to the adoption of a developmental progression model in the context of rotation block schedules and misalignment between progression targets and clinical experience. The shift from time-based to competency-based evaluation presented an ongoing adjustment for many programs. Organized preparation before clinical competency committee meetings and diverse clinical competency committee composition led to more productive meetings and perceived improvement in promotion decisions. CONCLUSIONS: The results of this study can be used by program leadership to help guide further implementation of the Milestones and program improvement. These results also help to guide the evolution of Milestones language and their implementation across specialties.

Competency Crosswalk: A Multispecialty Review of the Accreditation Council for Graduate Medical Education Milestones Across Four Competency Domains.

PURPOSE: To identify common and overlapping themes among the interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP) milestones of the transitional year and 26 specialties. METHOD: In May 2017, milestones were accessed from the Accreditation Council for Graduate Medical Education specialties website. A thematic analysis of the ICS, PBLI, PROF, and SBP milestones was performed to determine unique and common themes across these competencies and across specialties. Keywords from the common program requirements were initially applied as codes to the milestones. Codes were then grouped into common themes. RESULTS: Twenty-two themes were identified: 15 (68%) were unique to a given competency (3 related to ICS, 4 related to PBLI, 5 related to PROF, and 3 related to SBP), and 7 (32%) appeared in the milestones of more than one core competency. Eleven themes (50%) were used by 20 or more specialties, and 6 themes (27%) by 10 or fewer specialties. No theme was present across all specialties. CONCLUSIONS: The ICS, PBLI, PROF, and SBP milestones contain multiple themes with areas of overlap among these four competencies and substantial variability across specialties. This variability may create differential expectations of residents across specialties, complicate faculty development, and make sharing assessment tools difficult. The thematic analysis provides important insights into how individual specialties interpret and operationalize the ICS, PBLI, PROF, and SBP competency domains and can inform future revisions of milestones to enable harmonization and shared understanding of these competencies across specialties where appropriate.

Commitment to Change and Challenges to Implementing Changes After Workplace-Based Assessment Rater Training.

PURPOSE: Faculty development for clinical faculty who assess trainees is necessary to improve assessment quality and impor tant for competency-based education. Little is known about what faculty plan to do differently after training. This study explored the changes faculty intended to make after workplace-based assessment rater training, their ability to implement change, predictors of change, and barriers encountered. METHOD: In 2012, 45 outpatient internal medicine faculty preceptors (who supervised residents) from 26 institutions participated in rater training. They completed a commitment to change form listing up to five commitments and ranked (on a 1-5 scale) their motivation for and anticipated difficulty implementing each change. Three months later, participants were interviewed about their ability to implement change and barriers encountered. The authors used logistic regression to examine predictors of change. RESULTS: Of 191 total commitments, the most common commitments focused on what faculty would change about their own teaching (57%) and increasing direct observation (31%). Of the 183 commitments for which follow-up data were available, 39% were fully implemented, 40% were partially implemented, and 20% were not implemented. Lack of time/competing priorities was the most commonly cited barrier. Higher initial motivation (odds ratio [OR] 2.02; 95% confidence interval [CI] 1.14, 3.57) predicted change. As anticipated difficulty increased, implementation became less likely (OR 0.67; 95% CI 0.49, 0.93). CONCLUSIONS: While higher baseline motivation predicted change, multiple system-level barriers undermined ability to implement change. Rater-training faculty development programs should address how faculty motivation and organizational barriers interact and influence ability to change.

Do Faculty Benefit From Participating in a Standardized Patient Assessment as Part of Rater Training? A Qualitative Study.

PURPOSE: To explore faculty's experience participating in a standardized patient (SP) assessment where they were observed and assessed and then received feedback about their own clinical skills as part of a rater training faculty development program on direct observation. METHOD: In 2012, 45 general internist teaching faculty from 30 residency programs participated in an eight-station SP assessment with cases covering common clinical scenarios. Twenty-one participants (47%) received verbal feedback from SPs and a performance-based score report. All participants reflected on the experience through an independent written exercise, one-on-one interviews, and a focus group discussion. Grounded theory was used to analyze all three reflections. RESULTS: Eleven participants (24%) previously completed an SP assessment post training. Most found the SP assessment valuable and experienced emotions that increased their empathy for learners' experiences being observed, being assessed, and receiving nonspecific feedback. Participants receiving verbal feedback from SPs described different themes around personal improvement plans compared with the nonfeedback group. CONCLUSIONS: Faculty experience many of the same emotions as trainees during SP encounters and view SP assessment as a valuable mechanism to improve their own clinical skills and assessments of trainees. SP assessments may be one approach to provide faculty feedback about core clinical skills needed in their own patient care as well as what they are expected to teach trainees. Although actual changes in participants' clinical or assessor skills were not measured (more research is merited), findings hint at a "dual benefit" from incorporating SP assessment into a faculty development workshop about assessment.

How faculty members experience workplace-based assessment rater training: a qualitative study.

CONTEXT: Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training. OBJECTIVES: This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed. METHODS: In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data. RESULTS: Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in. CONCLUSIONS: Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

Assessment and evaluation in interprofessional education: exploring the field.

BACKGROUND: The practice of interprofessional education (IPE) is expanding rapidly in the United States and globally. The publication of competencies from the Interprofessional Education Collaborative (IPEC) was a significant step forward to recognize the importance of health professions collaboration and to guide institutions for educational program development. However, there remains substantial difficulty in implementation, as well as considerable variability in assessment of learners' interprofessional collaborative knowledge and skills and evaluation of IPE programs. METHODS: We conducted a multi-methods project which included 20 key informant interviews, a literature review, and a meeting of an expert panel. Our goals were 1) explore the current field of IPE, 2) identify and disseminate best practices to institutions wishing to implement/augment IPE assessment and evaluation processes, 3) uncover gaps in current IPE assessment and evaluation practices, and 4) recommend next steps for the field. RESULTS: A small and growing literature indicates evidence of the effectiveness of IPE. A diverse collection of methods and tools are used to assess and evaluate IPE learners and programs; these are often used without an explicit program-evaluation framework. CONCLUSIONS: For the field to advance and to align with the demands of changing clinical care systems, robust assessment and evaluation methods, standardized use of common tools, and longitudinal assessment from diverse data streams are needed for IPE.

Reconceptualizing variable rater assessments as both an educational and clinical care problem.

The public is calling for the U.S. health care and medical education system to be accountable for ensuring high-quality, safe, effective, patient-centered care. As medical education shifts to a competency-based training paradigm, clinician educators' assessment of and feedback to trainees about their developing clinical skills becomes paramount. However, there is substantial variability in the accuracy, reliability, and validity of the assessments faculty make when they directly observe trainees with patients. These difficulties have been treated primarily as a rater cognition problem focusing on the inability of the assessor to make reliable and valid assessments of the trainee.The authors' purpose is to reconceptualize the rater cognition problem as both an educational and clinical care problem. The variable quality of faculty assessments is not just a psychometric predicament but also an issue that has implications for decisions regarding trainee supervision and the delivery of quality patient care. The authors suggest that the frame of reference for rating performance during workplace-based assessments be the ability to provide safe, effective, patient-centered care. The authors developed the Accountable Assessment for Quality Care and Supervision equation to remind faculty that supervision is a dynamic, complex process essential for patients to receive high-quality care. This fundamental shift in how assessment is conceptualized requires new models of faculty development and emphasizes the essential and irreplaceable importance of the clinician educator in trainee assessment.

Development of an instrument to evaluate residents' confidence in quality improvement.

BACKGROUND: Practice-based learning and improvement is a core competency that all medical residents must demonstrate. Because confidence is important in translating competence into action, effective quality improvement (QI) curricula should evaluate trainees' knowledge and confidence to perform QI. Past efforts to assess educational outcomes in QI have not adequately evaluated trainees' confidence from a multidimensional perspective. METHODS: Participants--732 internal medicine and family medicine residents from 42 training programs in the United States--completed the 31-item Quality Improvement Confidence Instrument (QICI), which was developed to measure confidence in six QI skill domains based on the Institute for Healthcare Improvement model ofQI. Confirmatory factor analysis was performed to support construct validity. Multivariate analysis of covariance was used to examine associations between residents' QI experience and other characteristics with confidence scores. RESULTS: Confirmatory factor analysis supported the QICI's multidimensional structure. Individual items yielded adequate variability, and reliability estimates for all six domains were high (> 0.86). On average, residents rated their confidence lowest for skills pertaining to choosing a target for improvement (specifically, using methods to evaluate interventions and to identify sources of process errors) and for testing a change in practice using specific tools for data collection and analysis. After controlling for program year and other characteristics, residents with previous QI experience reported significantly greater QI confidence. CONCLUSION: The QICI offers a psychometrically rigorous approach to evaluating residents' confidence levels. It can be used to gauge the appropriateness of a trainee's confidence against actual QI performance.

Early feedback on the use of the internal medicine reporting milestones in assessment of resident performance.

BACKGROUND: The educational milestones were designed as a criterion-based framework for assessing resident progression on the 6 Accreditation Council for Graduate Medical Education competencies. OBJECTIVE: We obtained feedback on, and assessed the construct validity and perceived feasibility and utility of, draft Internal Medicine Milestones for Patient Care and Systems-Based Practice. METHODS: All participants in our mixed-methods study were members of competency committees in internal medicine residency programs. An initial survey assessed participant and program demographics; focus groups obtained feedback on the draft milestones and explored their perceived utility in resident assessment, and an exit survey elicited input on the value of the draft milestones in resident assessment. Surveys were tabulated using descriptive statistics. Conventional content analysis method was used to assess the focus group data. RESULTS: Thirty-four participants from 17 programs completed surveys and participated in 1 of 6 focus groups. Overall, the milestones were perceived as useful in formative and summative assessment of residents. Participants raised concerns about the length and complexity of some draft milestones and suggested specific changes. The focus groups also identified a need for faculty development. In the exit survey, most participants agreed that the Patient Care and Systems-Based Practice Milestones would help competency committees assess trainee progress toward independent practice. CONCLUSIONS: Draft reporting milestones for 2 competencies demonstrated significant construct validity in both the content and response process and the perceived utility for the assessment of resident performance. To ensure success, additional feedback from the internal medicine community and faculty development will be necessary.

Comparative trial of a web-based tool to improve the quality of care provided to older adults in residency clinics: modest success and a tough road ahead.

PURPOSE: To determine whether residency programs can use a multicomponent, Web-based quality improvement tool to improve the care of older adults. METHOD: The authors conducted an exploratory, cluster-randomized, comparative before-after trial of the Care of the Vulnerable Elderly Practice Improvement Module in the ambulatory clinics of 46 internal medicine and family medicine residency programs, 2006-2008. The main outcomes were the deltas between pre- and post-performance on the Assessing Care of the Vulnerable Elderly (ACOVE) quality measures. RESULTS: Of the 46 programs initially selected for the study, 37 (80%) provided both baseline and follow-up data. Performance on all 10 ACOVE measures was poor at baseline (range 8.6%-33.6%). Intervention clinics most frequently chose for improvement fall-risk screening and documentation of end-of-life preferences. The change in the percentage of patients screened for fall risk for the intervention clinics that targeted this measure was significantly greater than the change observed by the control clinics (+23.3% versus +9.7%, P = .003, odds ratio [OR] = 2.0; 95% confidence interval [CI]: 1.25-3.75), as was the difference observed for documentation of preference for life-sustaining care (+16.4% versus +2.8%, P = .002, OR = 6.3; 95% CI: 2.0-19.6) and surrogate decision maker (+14.3% versus +2.8%, P = .003, OR = 6.8; 95% CI: 1.9-24.4). CONCLUSIONS: A multicomponent, Web-based, quality improvement tool can help residency programs improve care for older adults, but much work remains for improving the state of care for this population in training settings.

Faculty staff perceptions of feedback to residents after direct observation of clinical skills.

CONTEXT: Performance-based workplace assessments are increasingly important in clinical training. Given the inaccuracy of self-assessment, the provision of external feedback to residents about their clinical skills is necessary for the development of expertise. However, little is known about the processes used by faculty members in giving feedback to residents after observing them with patients. This study explores the factors that underpin faculty members' decisions regarding the feedback they give to residents after directly observing them with patients and the factors that influence how feedback is delivered. METHODS: In 2009, 44 general internal medicine faculty staff responsible for out-patient resident teaching from 16 internal medicine residency programmes watched four videotaped scenarios and two live scenarios of standardised residents (SRs) with standardised patients and rated the SRs using the mini-clinical evaluation exercise (mini-CEX) format. Faculty staff also provided feedback to the SRs after the live encounters. After each encounter, faculty staff were individually interviewed using a semi-structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods. RESULTS: Two broad themes were identified in faculty members' descriptions of the feedback process: variability in feedback techniques, and the factors that influence how faculty staff think and feel about delivering feedback. Multiple approaches to feedback delivery were observed. Faculty members' tensions in balancing positive and negative feedback, their own perceived self-efficacy, their perceptions of the resident's insight, receptivity, skill and potential, the faculty member-resident relationship and contextual factors impacted the feedback process. CONCLUSIONS: The provision of feedback by faculty staff to residents after observing resident-patient interactions is a complex and dynamic process and is influenced by many factors. Understanding these cognitive and affective factors may provide insight into potential new approaches to faculty development to improve faculty staff's feedback skills and the effectiveness of their feedback.

Variability in obtaining institutional review board approval for quality improvement activities in residency programs.

INTRODUCTION: Quality improvement (QI) activities are an important part of residency training. National studies are needed to inform best practices in QI training and experience for residents. The impact of the Institutional Review Board (IRB) process on such studies is not well described. METHODS: This observational study looked at time, length, comfort level, and overall quality of experience for 42 residency training programs in obtaining approval or exemption for a nationally based educational QI study. RESULTS: For the 42 programs in the study, the time period to IRB approval/exemption was highly variable, ranging from less than 1 week to 56.5 weeks; mean and median time was approximately 18 weeks (SD, 10.8). Greater reported comfort with the IRB process was associated with less time to obtain approval (r  =  -.50; P < .01; 95% CI, -0.70 to -0.23). A more positive overall quality of experience with the IRB process was also associated with less time to obtain IRB approval (r  =  -.60; P < .01; 95% CI, -0.74 to -0.36). DISCUSSION: The IRB process for residency programs initiating QI studies shows considerable variance that is not explained by attributes of the projects. New strategies are needed to assist and expedite IRB processes for QI research in educational settings and reduce interinstitutional variability and increase comfort level among educators with the IRB process.

Opening the black box of clinical skills assessment via observation: a conceptual model.

OBJECTIVES: This study was intended to develop a conceptual framework of the factors impacting on faculty members' judgements and ratings of resident doctors (residents) after direct observation with patients. METHODS: In 2009, 44 general internal medicine faculty members responsible for out-patient resident teaching in 16 internal medicine residency programmes in a large urban area in the eastern USA watched four videotaped scenarios and two live scenarios of standardised residents engaged in clinical encounters with standardised patients. After each, faculty members rated the resident using a mini-clinical evaluation exercise and were individually interviewed using a semi-structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods. RESULTS: Four primary themes that provide insights into the variability of faculty assessments of residents' performance were identified: (i) the frames of reference used by faculty members when translating observations into judgements and ratings are variable; (ii) high levels of inference are used during the direct observation process; (iii) the methods by which judgements are synthesised into numerical ratings are variable, and (iv) factors external to resident performance influence ratings. From these themes, a conceptual model was developed to describe the process of observation, interpretation, synthesis and rating. CONCLUSIONS: It is likely that multiple factors account for the variability in faculty ratings of residents. Understanding these factors informs potential new approaches to faculty development to improve the accuracy, reliability and utility of clinical skills assessment.

Listening to older adults: elderly patients' experience of care in residency and practicing physician outpatient clinics.

The population of people aged 65 and older is rapidly growing. Research has demonstrated significant quality gaps in the clinical care of older patients in the United States, especially in training programs. Little is known about how older patients' experience with care delivered in residency clinics compares with that delivered by practicing physicians. Using patient surveys from the American Board of Internal Medicine Care of the Vulnerable Elderly Practice Improvement Module, the quality of care provided to adults aged 65 and older by 52 internal medicine and family medicine residency clinics and by a group of 144 practicing physicians was studied. The residency clinics received 2,213 patient surveys, and the practicing physicians received 4,204. Controlling for age and overall health status, patients from the residency clinic sample were less likely to report receiving guidance and interventions for important aspects of care for older adults than patients from the practicing physician sample. The largest difference was observed in providing ways to help patients prevent falls or treat problems with balance or walking (42.1% vs 61.8%, P<.001). Patients from the residency clinic sample were less likely to rate their overall care as high (77.5% vs 88.8%, P<.001). Patient surveys reveal important deficiencies in processes of care that are more pronounced for patients cared for in residency clinics. Quality of patient experience and communication are vital aspects of overall quality of care, especially for older adults. Physician education at all levels, faculty development, and practice system redesign are needed to ensure that the care needs of older adults are met.