Can Residents Assess Other Providers' Infant Lumbar Puncture Skills?: Validity Evidence for a Global Rating Scale and Subcomponent Skills Checklist.

OBJECTIVES: The aims of this study were to provide validity evidence for infant lumbar puncture (ILP) checklist and global rating scale (GRS) instruments when used by residents to assess simulated ILP performances and to compare these metrics to previously obtained attending rater data. METHODS: In 2009, the International Network for Simulation-based Pediatric Innovation, Research, and Education (INSPIRE) developed checklist and GRS scoring instruments, which were previously validated among attending raters when used to assess simulated ILP performances. Video recordings of 60 subjects performing an LP on an infant simulator were collected; 20 performed by subjects in 3 categories (beginner, intermediate, and expert). Six blinded pediatric residents independently scored each performance (3 via the GRS, 3 via the checklist). Four of the 5 domains of validity evidence were collected: content, response process, internal structure (reliability and discriminant validity), and relations to other variables. RESULTS: Evidence for content and response process validity is presented. When used by residents, the checklist performed similarly to what was found for attending raters demonstrating good internal consistency (Cronbach α = 0.77) and moderate interrater agreement (intraclass correlation coefficient = 0.47). Residents successfully discerned beginners (P < 0.01, effect size = 2.1) but failed to discriminate between expert and intermediate subjects (P = 0.68, effect size = 0.34). Residents, however, gave significantly higher GRS scores than attending raters across all subject groups (P < 0.001). Moderate correlation was found between GRS and total checklist scores (P = 0.49, P < 0.01). CONCLUSIONS: This study provides validity evidence for the checklist instrument when used by pediatric residents to assess ILP performances. Compared with attending raters, residents appeared to over-score subjects on the GRS instrument.

Validation of global rating scale and checklist instruments for the infant lumbar puncture procedure.

INTRODUCTION: The Patient Outcomes in Simulation Education network has developed tools for the assessment of competency to perform the infant lumbar puncture (ILP) procedure. The objective of this study was to evaluate the validity and reliability of these tools in a simulated setting. METHODS: We developed a 4-point anchored global rating scale (GRS) and 15-item dichotomous checklist instrument to assess ILP performance in a simulated environment. Video recordings of 60 subjects performing an unsupervised lumbar puncture on an infant bench top simulator were collected prospectively; 20 performed by subjects in each of 3 categories (beginner, intermediate experienced, or expert). Three blinded, expert raters independently scored each subject's video recording using the GRS and checklist instruments. RESULTS: The final version of the scoring instruments is presented. Across all subject groups, higher GRS scores were found with advancing level of experience (P < 0.01). Total checklist scores were similar between the expert and intermediate experienced groups (P = 0.54). Both groups scored higher than the beginner group on the checklist instrument (P < 0.01). For each rater, a significant positive correlation was found between GRS scores and total checklist scores (median ρ = 0.75, P < 0.01). Cronbach α coefficient for the checklist was 0.77. The intraclass correlation coefficients between raters for the GRS and total checklist scores were 0.71 and 0.52, respectively. CONCLUSIONS: This study provides some initial evidence to support the validity and reliability of the ILP-anchored GRS. Acceptable internal consistency was found for the checklist instrument. The GRS instrument outperformed the checklist in its discriminant ability and interrater agreement.