Optimizing Clinical Reasoning Assessments With Analytic and Holistic Ratings: Examining the Validity, Reliability, and Cost of a Simplified Patient Note Scoring Procedure.

PURPOSE: Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback. METHOD: Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time). RESULTS: Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores. CONCLUSIONS: Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.

Evaluation of tensiometric assessment as a measure of skill degradation.

This pilot study explored the use of tensiometry as a measure of retention of knot tying skills. Medical students learned a one-handed square knot tying technique. Their final performances were video recorded and these videos were uploaded on to a website. Students were divided into two groups: an observational learning group that had access to videos before a retention test, and a control group that did not. After a two-week retention period, all students came back and performed one more trial to test the amount of retention of the skill. Tensiometry was used to measure strengths of the knots before and after the retention period. The scores showed no significant difference between the groups (p>0.308) or tests (p>0.737). We interpret the results to suggest that tensiometry is not sensitive enough to detect degradation in the performance of fundamental clinical skills as they are forgotten after being taught.

The creation of an objective assessment tool for ultrasound-guided regional anesthesia using the Delphi method.

BACKGROUND AND OBJECTIVES: The assessment of technical skills in ultrasound-guided regional anesthesia is currently subjective and relies largely on observations of the trainer. The objective of this study was to develop a checklist to assess training progress and to detect training gaps in ultrasound-guided regional anesthesia using the Delphi method. METHODS: A 30-item checklist was developed and then e-mailed to 18 reviewers for feedback. The checklist was modified on the basis of their feedback. This process of iteration was repeated until no further feedback was received, and a consensus was reached on the final composition of the checklist. A global rating scale (GRS) was introduced as a result of the feedback. RESULTS: Three rounds of feedback were required to reach consensus on the composition of the checklist and the GRS. The final checklist contains 22 items, and the GRS contains 9 categories. CONCLUSIONS: Using the Delphi method, a checklist and GRS were developed. These tools can serve as an objective means of assessing progress in ultrasound technical skills acquisition.