Quality evaluation reports: Can a faculty development program make a difference?

BACKGROUND: The quality of medical student and resident clinical evaluation reports submitted by rotation supervisors is a concern. The effectiveness of faculty development (FD) interventions in changing report quality is uncertain. AIMS: This study assessed whether faculty could be trained to complete higher quality reports. METHOD: A 3-h interactive program designed to improve evaluation report quality, previously developed and tested locally, was offered at three different Canadian medical schools. To assess for a change in report quality, three reports completed by each supervisor prior to the workshop and all reports completed for 6 months following the workshop were evaluated by three blinded, independent raters using the Completed Clinical Evaluation Report Rating (CCERR): a validated scale that assesses report quality. RESULTS: A total of 22 supervisors from multiple specialties participated. The mean CCERR score for reports completed after the workshop was significantly higher (21.74 ± 4.91 versus 18.90 ± 5.00, p = 0.02). CONCLUSIONS: This study demonstrates that this FD workshop had a positive impact upon the quality of the participants' evaluation reports suggesting that faculty have the potential to be trained with regards to trainee assessment. This adds to the literature which suggests that FD is an important component in improving assessment quality.

Assessing the quality of supervisors' completed clinical evaluation reports.

CONTEXT: Although concern has been raised about the value of clinical evaluation reports for discriminating among trainees, there have been few efforts to formalise the dimensions and qualities that distinguish effective versus less useful styles of form completion. METHODS: Using brainstorming and a modified Delphi technique, a focus group determined the key features of high-quality completed evaluation reports. These features were used to create a rating scale to evaluate the quality of completed reports. The scale was pilot-tested locally; the results were psychometrically analysed and used to modify the scale. The scale was then tested on a national level. Psychometric analysis and final modification of the scale were completed. RESULTS: Sixteen features of high-quality reports were identified and used to develop a rating scale: the Completed Clinical Evaluation Report Rating (CCERR). The reliability of the scale after a national field test with 55 raters assessing 18 in-training evaluation reports (ITERs) was 0.82. Further revisions were made; the final version of the CCERR contains nine items rated on a 5-point scale. With this version, the mean ratings of three groups of 'gold-standard' ITERs (previously judged to be of high, average and poor quality) differed significantly (P < 0.05). DISCUSSION: The CCERR is a validated scale that can be used to help train supervisors to complete and assess the quality of evaluation reports.

Assessing change in clinical teaching skills: are we up for the challenge?

BACKGROUND: The faculty development community has been challenged to more rigorously assess program impact and move beyond traditional outcomes of knowledge tests and self ratings. PURPOSE: The purpose was to (a) assess our ability to measure supervisors' feedback skills as demonstrated in a clinical setting and (b) compare the results with traditional outcome measures of faculty development interventions. METHODS: A pre-post study design was used. Resident and expert ratings of supervisors' demonstrated feedback skills were compared with traditional outcomes, including a knowledge test and participant self-evaluation. RESULTS: Pre-post knowledge increased significantly (pre = 61%, post = 85%; p < .001) as did participant's self-evaluation scores (pre = 4.13, post = 4.79; p < .001). Participants' self-evaluations were moderately to poorly correlated with resident (pre r = .20, post r = .08) and expert ratings (pre r = .43, post r = -.52). Residents and experts would need to evaluate 110 and 200 participants, respectively, to reach significance. CONCLUSIONS: It is possible to measure feedback skills in a clinical setting. Although traditional outcome measures show a significant effect, demonstrating change in teaching behaviors used in practice will require larger scale studies than typically undertaken currently.

Failure to fail: the perspectives of clinical supervisors.

BACKGROUND: Clinical supervisors often do not fail students and residents even though they have judged their performance to be unsatisfactory. This study explored the factors identified by supervisors that affect their willingness to report poor clinical performance when completing In-Training Evaluation Reports (ITERs). METHOD: Semistructured interviews with 21 clinical supervisors at the University of Ottawa were conducted and qualitatively analyzed. RESULTS: Participants identified four major areas of the evaluation process that act as barriers to reporting a trainee who has performed poorly: (1) lack of documentation, (2) lack of knowledge of what to specifically document, (3) anticipating an appeal process and (4) lack of remediation options. CONCLUSIONS: The study provides insight as to why supervisors fail to fail the poorly performing student and resident. It also offers suggestions of how to support supervisors, increasing the likelihood that they will provide a valid ITER when faced with an underachieving trainee.

Quality in-training evaluation reports--does feedback drive faculty performance?

PURPOSE: Clinical faculty often complete in-training evaluation reports (ITERs) poorly. Faculty development (FD) strategies should address this problem. An FD workshop was shown to improve ITER quality, but few physicians attend traditional FD workshops. To reach more faculty, the authors developed an "at-home" FD program offering participants various types of feedback on their ITER quality based on the workshop content. Program impact is evaluated here. METHOD: Ninety-eight participants from four medical schools, all clinical supervisors, were recruited in 2009-2010; 37 participants completed the study. These were randomized into five groups: a control group and four other groups with different feedback conditions. ITER quality was assessed by two raters using a validated tool: the completed clinical evaluation report rating (CCERR). Participants were given feedback on their ITER quality based on group assignment. Six months later, participants submitted new ITERs. These ITERs were assessed using the CCERR, and feedback was sent to participants on the basis of their group assignment. This process was repeated two more times, ending in 2012. RESULTS: CCERR scores from the participants in all feedback groups were collapsed (n=27) and compared with scores from the control group (n=10). Mean CCERR scores significantly increased over time for the feedback group but not the control group. CONCLUSIONS: The results suggest that faculty are able to improve ITER quality following a minimal "at-home" FD intervention. This also adds to the growing literature that has found success with improving the quality of trainee assessments following rater training.

Ambulation monitoring of transtibial amputation subjects with patient activity monitor versus pedometer.

Our study aimed to compare the accuracy of step count and ambulation distance determined with the Yamax Digi-Walker SW-700 pedometer (DW) and the Ossur patient activity monitor (PAM) in 20 transtibial amputation subjects who were functioning at the K3 Medicare Functional Classification Level. Subjects completed four simulated household tasks in an apartment setup and a gymnasium walking course designed to simulate outdoor walking without the presence of environmental barriers or varied terrain. The mean step count accuracy of the DW and the PAM was equivalent for both the household activity (75.3% vs 70.6%) and the walking course (93.8% vs 94.0%). The mean distance measurement accuracy was better with the DW than with the PAM (household activity: 72.8% vs 0%, walking course: 92.5% vs 86.3%; p < 0.05). With acceptable step count accuracy, both devices are appropriate for assessing relatively continuous ambulation. The DW may be preferred for its more accurate distance measurements. Neither device is ideal for monitoring in-home ambulation.

Skin problems in an amputee clinic.

OBJECTIVE: To document the type and frequency of individual residual limb skin problems among patients using a lower extremity prosthesis, including the suggested etiology and management of each type of skin problem. DESIGN: This is a 6-yr retrospective chart review of skin lesions diagnosed in patients examined in an outpatient amputee clinic at a regional, referral rehabilitation hospital in Ottawa, Canada. Skin lesions were included if they were on a lower extremity residual limb for a patient who functionally used a prosthesis. Descriptive statistics were used to analyze data. RESULTS: A total of 528 skin problems were documented in 337 lower extremity residual limbs. Ulcers, irritations, inclusion cysts, calluses, and verrucous hyperplasia were the five most common skin problems representing 79.5% of all documented skin disorders. CONCLUSIONS: This study demonstrated that a wide variety of dermatologic conditions occurred frequently in the lower extremity amputee who functionally used a prosthesis. Five types of skin problems accounted for nearly 80% of the skin lesions identified. Future studies are required to evaluate prevention and management of the most frequent skin problems.

Dermatologic conditions associated with use of a lower-extremity prosthesis.

OBJECTIVES: To document the frequency of skin problems among lower-limb prosthesis users and to assess for factors associated with skin problems among patients using a prosthesis. DESIGN: Six-year retrospective chart review. SETTING: An outpatient amputee clinic at a regional, referral rehabilitation hospital in Canada. PARTICIPANTS: Seven hundred forty-five subjects with a total of 828 lower-extremity amputations participated. Subjects were included if they had a lower-extremity amputation and used a prosthesis for ambulation or transfers. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: The presence or absence of a skin problem. Descriptive and nonparametric statistics were used to analyze data. RESULTS: Three hundred thirty-seven residual limbs (40.7%) had at least 1 skin problem. Adjusted odds ratios showed that amputation level, being employed, type of walking aid, and absence of peripheral vascular disease (as a comorbidity) were independently associated with the presence of at least 1 skin problem ( P <.05). CONCLUSIONS; Dermatologic conditions are a frequent complication for the lower-extremity amputee who uses a prosthesis. The results suggest that more active amputees have an increased risk for developing skin problems. Further study in this area is warranted.

Bone overgrowth in the adult traumatic amputee.

Bone overgrowth of the residual limb after an amputation is a well documented complication in the pediatric amputee population. Bone overgrowth can cause pain, problems with skin breakdown, and poor prosthetic fit. There have been few reports of bone overgrowth in the adult amputee. Two cases of traumatic transfemoral amputations after extensive tissue damage are presented. Both patients successfully completed an in-patient amputee rehabilitation program and achieved functional ambulation with their prostheses. However, each developed distal residual limb pain within a year after their amputations that significantly limited the amount of time they could wear their prostheses and the distance they could walk. Radiographs demonstrated additional bone growth from the residual femur into adjacent soft tissues in both patients. These case examples demonstrate that bone overgrowth should be considered in the differential diagnosis of residual limb pain in the adult amputee.