Multisource feedback in the ambulatory setting.

BACKGROUND: The Accreditation Council for Graduate Medical Education has mandated multisource feedback (MSF) in the ambulatory setting for internal medicine residents. Few published reports demonstrate actual MSF results for a residency class, and fewer still include clinical quality measures and knowledge-based testing performance in the data set. METHODS: Residents participating in a year-long group practice experience called the "long-block" received MSF that included self, peer, staff, attending physician, and patient evaluations, as well as concomitant clinical quality data and knowledge-based testing scores. Residents were given a rank for each data point compared with peers in the class, and these data were reviewed with the chief resident and program director over the course of the long-block. RESULTS: Multisource feedback identified residents who performed well on most measures compared with their peers (10%), residents who performed poorly on most measures compared with their peers (10%), and residents who performed well on some measures and poorly on others (80%). Each high-, intermediate-, and low-performing resident had a least one aspect of the MSF that was significantly lower than the other, and this served as the basis of formative feedback during the long-block. CONCLUSION: Use of multi-source feedback in the ambulatory setting can identify high-, intermediate-, and low-performing residents and suggest specific formative feedback for each. More research needs to be done on the effect of such feedback, as well as the relationships between each of the components in the MSF data set.

The ambulatory long-block: an accreditation council for graduate medical education (ACGME) educational innovations project (EIP).

INTRODUCTION: Historical bias toward service-oriented inpatient graduate medical education experiences has hindered both resident education and care of patients in the ambulatory setting. AIM: Describe and evaluate a residency redesign intended to improve the ambulatory experience for residents and patients. SETTING: Categorical Internal Medicine resident ambulatory practice at the University of Cincinnati Academic Health Center. PROGRAM DESCRIPTION: We created a year-long continuous ambulatory group-practice experience separated from traditional inpatient responsibilities called the long block as an Accreditation Council for Graduate Medical Education Educational Innovations Project. The practice adopted the Chronic Care Model and residents received extensive instruction in quality improvement and interprofessional teams. PROGRAM EVALUATION: The long block was associated with significant increases in resident and patient satisfaction as well as improvement in multiple quality process and outcome measures. Continuity and no-show rates also improved. DISCUSSION: An ambulatory long block can be associated with improvements in resident and patient satisfaction, quality measures, and no-show rates. Future research should be done to determine effects of the long block on education and patient care in the long term, and elucidate which aspects of the long block most contribute to improvement.

Systematic review of the literature on assessment, feedback and physicians' clinical performance: BEME Guide No. 7.

BACKGROUND AND CONTEXT: There is a basis for the assumption that feedback can be used to enhance physicians' performance. Nevertheless, the findings of empirical studies of the impact of feedback on clinical performance have been equivocal. OBJECTIVES: To summarize evidence related to the impact of assessment and feedback on physicians' clinical performance. SEARCH STRATEGY: The authors searched the literature from 1966 to 2003 using MEDLINE, HealthSTAR, the Science Citation Index and eight other electronic databases. A total of 3702 citations were identified. INCLUSION AND EXCLUSION CRITERIA: Empirical studies were selected involving the baseline measurement of physicians' performance and follow-up measurement after they received summaries of their performance. DATA EXTRACTION: Data were extracted on research design, sample, dependent and independent variables using a written protocol. DATA SYNTHESIS: A group of 220 studies involving primary data collection was identified. However, only 41 met all selection criteria and evaluated the independent effect of feedback on physician performance. Of these, 32 (74%) demonstrated a positive impact. Feedback was more likely to be effective when provided by an authoritative source over an extended period of time. Another subset of 132 studies examined the effect of feedback combined with other interventions such as educational programmes, practice guidelines and reminders. Of these, 106 studies (77%) demonstrated a positive impact. Two additional subsets of 29 feedback studies involving resident physicians in training and 18 studies examining proxy measures of physician performance across clinical sites or groups of patients were reviewed. The majority of these two subsets also reported that feedback had positive effects on performance. HEADLINE RESULTS: Feedback can change physicians' clinical performance when provided systematically over multiple years by an authoritative, credible source. CONCLUSIONS: The effects of formal assessment and feedback on physician performance are influenced by the source and duration of feedback. Other factors, such as physicians' active involvement in the process, the amount of information reported, the timing and amount of feedback, and other concurrent interventions, such as education, guidelines, reminder systems and incentives, also appear to be important. However, the independent contributions of these interventions have not been well documented in controlled studies. It is recommended that the designers of future theoretical as well as practical studies of feedback separate the effects of feedback from other concurrent interventions.

Academic health centers and public health departments: partnership matters.

BACKGROUND: The Institute of Medicine has called for increased population-based training for healthcare professions students, and particularly medical students. For this to be effective, students should receive such training in the locations where population-based approaches to care take place, such as public health departments (HDs). However, little is known about currently existing relationships between academic health centers (AHCs) and HDs. METHODS: During the spring and summer of 2003, e-mail surveys on this topic were sent to the 104 members of the Association of Academic Health Centers, and 500 members of the National Association of City and County Health Officers and the Association of State and Territorial Health Officers. Results were received from 50 AHCs and 201 HDs. RESULTS: Survey results suggest that large numbers of healthcare professions students--residents, medical students, and others--are currently being trained in many HDs nationwide. The web of relationships between AHCs and HDs extends beyond education into research, service, and other purposes. CONCLUSIONS: These preliminary results require verification. Nevertheless, they raise questions about the types and quality of the education being offered in health departments, the impact of AHC/HD relationships on both parties, and how existing relationships may be enhanced to meet current and future national needs.

Measuring professionalism: a review of studies with instruments reported in the literature between 1982 and 2002.

PURPOSE: To describe the measurement properties of instruments reported in the literature that faculty might use to measure professionalism in medical students and residents. METHOD: The authors reviewed studies published between 1982 and 2002 that had been located using Medline and four other databases. A national panel of 12 experts in measurement and research in medical education extracted data from research reports using a structured critique form. RESULTS: A total of 134 empirical studies related to the concept of professionalism were identified. The content of 114 involved specific elements of professionalism, such as ethics, humanism, and multiculturalism, or associated phenomena in the educational environment such as abuse and cheating. Few studies addressed professionalism as a comprehensive construct (11 studies) or as a distinct facet of clinical competence (nine studies). The purpose of 109 studies was research or program evaluation, rather than summative or formative assessment. Sixty five used self-administered instruments with no independent observation of the participants' professional behavior. Evidence of reliability was reported in 62 studies. Although content validity was reported in 86 studies, only 34 provided strong evidence. Evidence of concurrent or predictive validity was provided in 43 and 16 studies, respectively. CONCLUSIONS: There are few well-documented studies of instruments that can be used to measure professionalism in formative or summative evaluation. When evaluating the tools described in published research it is essential for faculty to look critically for evidence related to the three fundamental measurement properties of content validity, reliability, and practicality.

Understanding residents' work: moving beyond counting hours to assessing educational value.

PURPOSE: To begin to understand how residents' work affects their own educations and the hospitals in which most of their training takes place, the authors undertook a systematic review of the literature analyzing residents' activities. This review sought to analyze resident physicians' activities to assess the educational value of residents' work. METHOD: The published literature was searched in 2001 using the Medline and Science Citation Index databases, and the unpublished literature was searched using bibliographies and key informants. One hundred six studies were rated for methodological rigor using the Cochrane Collaboration protocol, as modified by Bland et al. for nonclinical trials. Only those studies undertaken following the Bell Commission's report in 1987 and whose methodological rigor score fell at or above the median for all studies rated were included in the data synthesis. Results data from 16 studies that included over 1,000 residents in six different specialties, were combined under the definitions of types of residents' activities: marginal, patient care, teaching and learning, and other. RESULTS: This preliminary analysis found that residents devoted approximately 36% of their effort to direct patient care necessary to achieve specialty-specific learning objectives, 15% to the residency program's organized teaching activities, and potentially as much as 35% to delivering patient care of marginal or no educational value. An additional 16% of residents' waking time on duty was spent in other, unspecified activities. CONCLUSION: It is possible and potentially valuable to consider not only the number of hours worked by residents, but the educational content of their work when considering residency work and hour reforms