Measuring learning gain during a one-day introductory bronchoscopy course.

BACKGROUND: Rigorous assessment of medical knowledge and technical skill inspires learning, reinforces confidence, and reassures the public. Identifying curricular effectiveness using objective measures of learning is therefore crucial for competency-oriented program development in a learner-centric educational environment. The aim of this study was to determine whether various measures of learning, including class-average normalized gain, can be used to assess the effectiveness of a one-day introductory bronchoscopy course curriculum. METHODS: We conducted a quasi-experimental one-group pre-test/post-test study at the University of California, Irvine. The group comprised 24 first-year pulmonary and critical care trainees from eight training institutions in southern California. Class-average normalized gain, single-student normalized gain, absolute gain, and relative gain were used as objective measures of cognitive knowledge and bronchoscopy technical skill learning. A class-average normalized gain of 30% was used to determine curricular effectiveness. Perceived educational value using Likert-scale surveys and post-course questionnaires was determined during and 3 months after course participation. RESULTS: Mean test scores of cognitive knowledge improved significantly from 48 to 66% (p = 0.043). Absolute gain for the class was 18%, relative gain was 37%, class average normalized gain was 34%, and the average of the single-student normalized gains g(ave) was 29%. Mean test scores of technical skill improved significantly from 43 to 77% (p = 0.017). Absolute gain was 34%, relative gain was 78%, class average normalized gain was 60%, and the average of the single-student normalized gains g(ave) was 59%. Statistically significant improvements in absolute gain were noted in all five elements of technical skill (p < 0.05). Likert-scale surveys, questionnaires, and surveys demonstrated strong perceived educational value. CONCLUSION: The effectiveness of a one-day introductory bronchoscopy curriculum was demonstrated using a pre-test/post-test model with calculation of normalized gain and related metrics.

Use of competency-based metrics to determine effectiveness of a postgraduate thoracoscopy course.

BACKGROUND: Despite the paradigm shift from process to competency-based education, no study has explored how competency-based metrics might be used to assess short-term effectiveness of thoracoscopy-related postgraduate medical education. OBJECTIVES: To assess the use of a single-group, pre-/post-test model comprised of multiple-choice questions (MCQ) and psychomotor skill measures to ascertain the effectiveness of a postgraduate thoracoscopy program. METHODS: A 37-item MCQ test of cognitive knowledge was administered to 17 chest physicians before and after a 2-day continued medical education-approved program. Pre- and post-course technical skills were assessed using rigid videothoracoscopy simulation stations. Competency-based metrics (mean relative gain, mean absolute gain, and class-average normalized gain ) were calculated. A >30% was used to determine curricular effectiveness. RESULTS: Mean cognitive knowledge score improved significantly from 20.9 to 28.7 (7.8 ± 1.3 points, p < 0.001), representing a relative gain of 37% and an absolute gain of 21%. Mean technical skill score improved significantly from 5.20 to 7.82 (2.62 ± 0.33 points, p < 0.001), representing a relative gain of 50% and an absolute gain of 33%. Non-parametric testing confirmed t test results (p < 0.001). Class-average normalized gains were 48 and 92%, respectively. CONCLUSION: Competency-based metrics, including class-average normalized gain, can be used to assess course effectiveness and to determine if a program meets predesignated objectives of knowledge acquisition and psychomotor technical skill.

Comparative effectiveness of low- and high-fidelity bronchoscopy simulation for training in conventional transbronchial needle aspiration and user preferences.

BACKGROUND: Conventional transbronchial needle aspiration (TBNA) can be learned using high-fidelity virtual-reality platforms and low-fidelity models comprised of molded silicone or excised animal airways. OBJECTIVES: The purpose of this study was to determine perceptions and preferences of learners and instructors regarding the comparative effectiveness of low-fidelity and high-fidelity bronchoscopy simulation for training in TBNA. METHODS: During the 2008 annual CHEST conference, a prospective randomized crossover design was used to train study participants in three methods of conventional TBNA using low- and high-fidelity models. Likert style questions were administered to learners and instructors in order to elicit preferences and opinions regarding educational effectiveness of the models. Results were tabulated and depicted in graphic format, with medians calculated. RESULTS: Learners felt that the models were equally enjoyable (13-13) and enthusiasm generating (low 17-high 15). There was preference for low-fidelity in terms of realism (23-17), ease of learning (20-6), and learning all three TBNA methods (31-7 for hub-against-wall, 31-6 for jabbing, 29-6 for piggyback). Low-fidelity was preferred as an ideal model overall (19-11). Instructors thought that low-fidelity was more useful in teaching TBNA (9-0 for all three methods). Instructors perceived the low-fidelity model overall as an ideal tool for learning TBNA (8-0) and a more effective teaching instrument (8-0). CONCLUSION: Based on learner and instructor perceptions, a low-fidelity model is superior to a high-fidelity platform for training in three methods of conventional TBNA.