Variable Learning Curve of Basic Rigid Bronchoscopy in Trainees.

BACKGROUND: Despite increased use of rigid bronchoscopy (RB) for therapeutic indications and recommendations from professional societies to use performance-based competency, an assessment tool has not been utilized to measure the competency of trainees to perform RB in clinical settings. OBJECTIVES: The aim of the study was to evaluate a previously developed assessment tool - Rigid Bronchoscopy Tool for Assessment of Skills and Competence (RIGID-TASC) - for determining the RB learning curve of interventional pulmonary (IP) trainees in the clinical setting and explore the variability of learning curve of trainees. METHODS: IP fellows at 4 institutions were enrolled. After preclinical simulation training, all RBs performed in patients were scored by faculty using RIGID-TASC until competency threshold was achieved. Competency threshold was defined as unassisted RB intubation and navigation through the central airways on 3 consecutive patients at the first attempt with a minimum score of 89. A regression-based model was devised to construct and compare the learning curves. RESULTS: Twelve IP fellows performed 178 RBs. Trainees reached the competency threshold between 5 and 24 RBs, with a median of 15 RBs (95% CI, 6-21). There were differences among trainees in learning curve parameters including starting point, slope, and inflection point, as demonstrated by the curve-fitting model. Subtasks that required the highest number of procedures (median = 10) to gain competency included ability to intubate at the first attempt and intubation time of <60 s. CONCLUSIONS: Trainees acquire RB skills at a variable pace, and RIGID-TASC can be used to assess learning curve of IP trainees in clinical settings.

Development of a Tool to Assess Basic Competency in the Performance of Rigid Bronchoscopy.

RATIONALE: Rigid bronchoscopy is increasingly used by pulmonologists for the management of central airway disorders. However, an assessment tool to evaluate the competency of operators in the performance of this technique has not been developed. We created the Rigid Bronchoscopy Tool for Assessment of Skills and Competence (RIGID-TASC) to serve as an objective, competency-oriented assessment tool of basic rigid bronchoscopic skills, including rigid bronchoscopic intubation and central airway navigation. OBJECTIVES: To assess whether RIGID-TASC scores accurately distinguish the basic rigid bronchoscopy skills of novice, intermediate, and expert operators, and to determine whether RIGID-TASC has adequate interrater reliability when used by different independent testers. METHODS: At two academic medical centers in the United States, 30 physician volunteers were selected in three categories: 10 novices at rigid bronchoscopy (performed at least 50 flexible, but no rigid, bronchoscopies), 10 operators with intermediate experience (performed 5-20 rigid bronchoscopies), and 10 experts (performed ≥100 rigid bronchoscopies). Participants included pulmonary and critical care fellows, interventional pulmonology fellows, and faculty interventional pulmonologists. Each subject then performed rigid bronchoscopic intubation and navigation on a manikin, while being scored independently by two testers, using RIGID-TASC. MEASUREMENTS AND MAIN RESULTS: Mean scores for three categories (novice, intermediate, and expert) were 58.10 (±4.6 [SE]), 78.15 (±3.8), and 94.40 (±1.1), respectively. There was significant difference between novice and intermediate (20.05, 95% confidence interval [CI] = 7.77-32.33, P = 0.001), and intermediate and expert (16.25, 95% CI = 3.97-28.53, P = 0.008) operators. The interrater reliability (intraclass correlation coefficient) between the two testers was high (r = 0.95, 95% CI = 0.90-0.98). CONCLUSIONS: RIGID-TASC showed evidence of construct validity and interrater reliability in this setting and group of subjects. It can be used to reliably and objectively score and classify operators from novice to expert in basic rigid bronchoscopic intubation and navigation.

Endobronchial ultrasound skills and tasks assessment tool: assessing the validity evidence for a test of endobronchial ultrasound-guided transbronchial needle aspiration operator skill.

RATIONALE: Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is becoming standard of care for the sampling of mediastinal adenopathy. The need for a safe, effective, accurate procedure makes EBUS-TBNA ideal for mastery training and testing. OBJECTIVES: The Endobronchial Ultrasound Skills and Tasks Assessment Tool (EBUS-STAT) was created as an objective competency-oriented assessment tool of EBUS-TBNA skills and knowledge. This study demonstrates the reliability and validity evidence of this tool. METHODS: The EBUS-STAT objectively scores the EBUS-TBNA operator's skills, including atraumatic airway introduction and navigation, ultrasound image acquisition and optimization, identification of mediastinal nodal and vascular structures, EBUS-TBNA sampling, and recognition of EBUS/computed tomography images of mediastinal structures. It can be administered at the bedside or using combination of low- and high-fidelity simulation platforms. Two independent testers administered the EBUS-STAT to 24 operators at three levels of EBUS-TBNA experience (8 beginners, 8 intermediates, and 8 experienced) at three institutions; operators were also asked to self-assess their skills. Scores were analyzed for intertester reliability, correlation with prior EBUS-TBNA experience, and association with self-assessments. MEASUREMENTS AND MAIN RESULTS: Intertester reliability between testers was very high (r = 0.9991, P < 0.00005). Mean EBUS-STAT scores for beginner, intermediate, and experienced groups, respectively, were 31.1, 74.9, and 93.6 out of 100 (F(2,21) = 118.6, P < 0.0001). Groups were nonoverlapping: post hoc tests showed each group differed significantly from the others (P < 0.001). Self-assessments corresponded closely to actual EBUS-STAT scores (r(2) = 0.81, P < 0.001). CONCLUSIONS: The EBUS-STAT can be used to reliably and objectively score and classify EBUS-TBNA operators from novice to expert. Its use to assess and document the acquisition of knowledge and skill is a step toward the goal of mastery training in EBUS-TBNA.

Scientific evidence and principles for the use of endobronchial ultrasound and transbronchial needle aspiration.

Endobronchial ultrasound (EBUS), using the radial EBUS probe and convex-probe EBUS-guided transbronchial needle aspiration, are increasingly advocated for a wide array of minimally invasive thoracic procedures. The effectiveness of EBUS-guided procedures has been demonstrated to a degree that, in many institutions, EBUS is becoming standard of practice for the diagnosis, staging and restaging of mediastinal lymphadenopathy in lung cancer, the diagnosis of sarcoidosis, and for bronchoscopic biopsy of peripheral lung lesions. Its role in other bronchoscopic procedures requires further study despite an already strong body of literature: diagnosis of lymphoma and benign infectious disease, diagnosis of early lung cancer and airway wall disorders, imaging of thoracic vascular disease such as pulmonary embolism, and therapeutic procedures such as placement of fiducial markers. In this article, we illustrate some of the principles of EBUS, describe major technical aspects pertaining to the procedure itself and provide a narrative review of original research addressing proposed roles of EBUS in a variety of indications. In closing, we describe future perspectives including new educational processes and philosophies that could favorably impact the rapid and safe dissemination of this evolving technology into clinical practice.

Measuring learning gain during a one-day introductory bronchoscopy course.

BACKGROUND: Rigorous assessment of medical knowledge and technical skill inspires learning, reinforces confidence, and reassures the public. Identifying curricular effectiveness using objective measures of learning is therefore crucial for competency-oriented program development in a learner-centric educational environment. The aim of this study was to determine whether various measures of learning, including class-average normalized gain, can be used to assess the effectiveness of a one-day introductory bronchoscopy course curriculum. METHODS: We conducted a quasi-experimental one-group pre-test/post-test study at the University of California, Irvine. The group comprised 24 first-year pulmonary and critical care trainees from eight training institutions in southern California. Class-average normalized gain, single-student normalized gain, absolute gain, and relative gain were used as objective measures of cognitive knowledge and bronchoscopy technical skill learning. A class-average normalized gain of 30% was used to determine curricular effectiveness. Perceived educational value using Likert-scale surveys and post-course questionnaires was determined during and 3 months after course participation. RESULTS: Mean test scores of cognitive knowledge improved significantly from 48 to 66% (p = 0.043). Absolute gain for the class was 18%, relative gain was 37%, class average normalized gain was 34%, and the average of the single-student normalized gains g(ave) was 29%. Mean test scores of technical skill improved significantly from 43 to 77% (p = 0.017). Absolute gain was 34%, relative gain was 78%, class average normalized gain was 60%, and the average of the single-student normalized gains g(ave) was 59%. Statistically significant improvements in absolute gain were noted in all five elements of technical skill (p < 0.05). Likert-scale surveys, questionnaires, and surveys demonstrated strong perceived educational value. CONCLUSION: The effectiveness of a one-day introductory bronchoscopy curriculum was demonstrated using a pre-test/post-test model with calculation of normalized gain and related metrics.

Use of competency-based metrics to determine effectiveness of a postgraduate thoracoscopy course.

BACKGROUND: Despite the paradigm shift from process to competency-based education, no study has explored how competency-based metrics might be used to assess short-term effectiveness of thoracoscopy-related postgraduate medical education. OBJECTIVES: To assess the use of a single-group, pre-/post-test model comprised of multiple-choice questions (MCQ) and psychomotor skill measures to ascertain the effectiveness of a postgraduate thoracoscopy program. METHODS: A 37-item MCQ test of cognitive knowledge was administered to 17 chest physicians before and after a 2-day continued medical education-approved program. Pre- and post-course technical skills were assessed using rigid videothoracoscopy simulation stations. Competency-based metrics (mean relative gain, mean absolute gain, and class-average normalized gain ) were calculated. A >30% was used to determine curricular effectiveness. RESULTS: Mean cognitive knowledge score improved significantly from 20.9 to 28.7 (7.8 ± 1.3 points, p < 0.001), representing a relative gain of 37% and an absolute gain of 21%. Mean technical skill score improved significantly from 5.20 to 7.82 (2.62 ± 0.33 points, p < 0.001), representing a relative gain of 50% and an absolute gain of 33%. Non-parametric testing confirmed t test results (p < 0.001). Class-average normalized gains were 48 and 92%, respectively. CONCLUSION: Competency-based metrics, including class-average normalized gain, can be used to assess course effectiveness and to determine if a program meets predesignated objectives of knowledge acquisition and psychomotor technical skill.

Comparative effectiveness of low- and high-fidelity bronchoscopy simulation for training in conventional transbronchial needle aspiration and user preferences.

BACKGROUND: Conventional transbronchial needle aspiration (TBNA) can be learned using high-fidelity virtual-reality platforms and low-fidelity models comprised of molded silicone or excised animal airways. OBJECTIVES: The purpose of this study was to determine perceptions and preferences of learners and instructors regarding the comparative effectiveness of low-fidelity and high-fidelity bronchoscopy simulation for training in TBNA. METHODS: During the 2008 annual CHEST conference, a prospective randomized crossover design was used to train study participants in three methods of conventional TBNA using low- and high-fidelity models. Likert style questions were administered to learners and instructors in order to elicit preferences and opinions regarding educational effectiveness of the models. Results were tabulated and depicted in graphic format, with medians calculated. RESULTS: Learners felt that the models were equally enjoyable (13-13) and enthusiasm generating (low 17-high 15). There was preference for low-fidelity in terms of realism (23-17), ease of learning (20-6), and learning all three TBNA methods (31-7 for hub-against-wall, 31-6 for jabbing, 29-6 for piggyback). Low-fidelity was preferred as an ideal model overall (19-11). Instructors thought that low-fidelity was more useful in teaching TBNA (9-0 for all three methods). Instructors perceived the low-fidelity model overall as an ideal tool for learning TBNA (8-0) and a more effective teaching instrument (8-0). CONCLUSION: Based on learner and instructor perceptions, a low-fidelity model is superior to a high-fidelity platform for training in three methods of conventional TBNA.

Realistic and affordable lo-fidelity model for learning bronchoscopic transbronchial needle aspiration.

BACKGROUND: Transbronchial needle aspiration (TBNA) is used to sample mediastinal abnormalities and lymph node stations for diagnostic purposes and lung cancer staging. The procedure is underused, operator dependent, and reputed to have a steep learning curve. Other difficulties arise from a bronchoscopist's failure to insert the needle satisfactorily into the target node. The purpose of this study was to evaluate the realism and helpfulness of a lo-fidelity, easily constructed hybrid model used for learning and practicing TBNA. METHODS: The model is constructed by attaching a porcine tracheobronchial tree to a Laerdal Airway Model mounted on polyvinyl chloride (PVC) piping. Twelve individuals with various levels of bronchoscopy training and experience were given a 15-min introductory PowerPoint presentation on TBNA strategy and planning, execution, and response to complications. Participants then practiced TBNA alone and with guidance, aided by an assistant, as many times as individually necessary to feel comfortable with the procedure. A five-point Likert scale 8-item questionnaire was then completed. RESULTS: Participants were unanimously positive about their experience (mean scores 4.25-4.91). The model was realistic, provided increased comfort with TBNA techniques, and allowed practice of communication skills. CONCLUSION: This realistic, affordable, and easily constructed hybrid lo-fidelity airway model allows beginner and experienced bronchoscopists opportunities to learn and practice basic TBNA techniques and team communication skills without placing patients at risk.

Reliability of a 25-item low-stakes multiple-choice assessment of bronchoscopic knowledge.

BACKGROUND: A need for improved patient safety, quality of care, and accountability has prompted the development of competency-based educational processes. Assessment tools related to bronchoscopy training, however, have not yet been developed or validated. PURPOSES: To determine whether 25 multiple-choice questions (MCQs) extracted from the free, Web-based Essential Bronchoscopist (EB) learning guide qualify in their original form as a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge. MATERIALS AND METHODS: Twenty-five randomly selected MCQs from among the top 70 question-answer sets of the EB were administered to 40 self-declared novice bronchoscopists (n = 13), experienced bronchoscopists (n = 21), and expert bronchoscopists (n = 6). A difficulty index and a discrimination index (DI) were calculated for each item. Internal consistency reliability was calculated using item-total correlation and Cronbach alpha. Content validity was determined by five independent experts. Ideal test items based on a difficulty index and item-total correlation were administered to a different group of 24 bronchoscopists to prospectively reassess internal consistency reliability. RESULTS: The mean (+/- SD) score for the 40 participants was 16.47 +/- 3.72 (median score, 17; score range, 7 to 22). The mean difficulty index was 0.65 +/- 0.22, and the mean DI was 0.52 +/- 0.28. Item total-correlations ranged from - 0.01 to + 0.71. Test content was unanimously validated. The Cronbach alpha was 0.69. There was no significant correlation between scores and the number of bronchoscopies performed or self-declared expertise. Eleven ideal test MCQs were identified. The internal consistency of these items remained satisfactory (Cronbach alpha = 0.75) when assessed prospectively in a different cohort. CONCLUSION: Reliable and valid MCQs were identified to initiate a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge.

Bronchoscopy simulation: a brief review.

More than 500,000 flexible bronchoscopies are performed annually by chest physicians in the United States (Ernst et al., Chest 123:1693-1717, 2003). Indications include diagnosis of lung cancer and airway tumors, benign strictures, pulmonary infections, and treatment of central airway obstruction, emphysema, and intraepithelial lesions such as carcinoma in-situ. Anesthesiologists, cardiothoracic and trauma surgeons, otolaryngologists, and critical are physicians also perform this procedure as part of difficult airway management, intubation or airway inspection and bronchial toilet. Compared to the expanding body of simulation-related literature that is available in other procedural fields, however, the volume of published work relating to bronchoscopy is scant. The purpose of this paper is to provide an overview of the available literature pertaining to the use of simulation in bronchoscopy education, and to demonstrate how this limited yet valuable body of work lays a foundation for the future use of simulator-based bronchoscopy training.

A competency-based test of bronchoscopic knowledge using the Essential Bronchoscopist: an initial concept study.

BACKGROUND AND OBJECTIVE: Competency-based training and assessment are increasingly replacing the traditional structure- and process-based model of medical education. The web-based Essential Bronchoscopist (EB) is an open access, laddered, competency-based curriculum of question-answer sets pertaining to basic bronchoscopic knowledge, accessible in five languages. The purpose of this study was to use consensus to evaluate whether question-answer sets (items) from the EB could provide material from which to devise competency-based tests of bronchoscopic knowledge that could be used in countries with different health-care environments. METHODS: Ten junior and five senior bronchoscopists in Argentina and the USA were asked to identify the material in each of the 186 items of the EB as either 'not necessary', 'necessary' or 'absolutely necessary' for inclusion in a competency-based test of bronchoscopic knowledge. The kappa statistic was used to analyse inter-rater agreement. RESULTS: More than 80% of the 30 participants rated 150 of 186 items as necessary or absolutely necessary. Seventy items were rated as absolutely necessary by more than 50% of participants and as necessary or absolutely necessary by more than 90%. There was no statistically significant difference between items selected, based on level of training or environment of practice. CONCLUSION: Consensus among junior and senior bronchoscopists from two culturally different health-care environments demonstrated that question-answer sets from the EB can be used to provide material for competency-based tests of bronchoscopic knowledge.

Validation of two instruments to assess technical bronchoscopic skill using virtual reality simulation.

BACKGROUND: In an era of increased emphasis on patient safety and competency-based education, demonstration of significant variability in the teaching of flexible bronchoscopy has led to initiatives for new standardized curricula and assessment tools. OBJECTIVES: To evaluate the efficacy of such curricula and to assess bronchoscopic skill, 2 measuring instruments have been developed: the Bronchoscopy Skills and Tasks Assessment Tool (BSTAT) and the Bronchoscopy Step-by-Step Evaluation Tool (BSET). We studied the validity and reliability of these 2 instruments. METHODS: Two independent testers simultaneously scored 22 volunteer participants at 3 levels [novice (n = 7), fellow (n = 8) and attending (n = 7)] on a virtual reality bronchoscopy simulator using the 2 instruments; each participant was tested twice, in 2 separate sessions. Intertester and test-retest reliability were analyzed with intraclass correlations (ICC); ANOVA was used to assess concurrent validity based on the subjects' expected skill level. RESULTS: The ICCs between the testers were 0.98 for both the BSTAT and BSET. Comparison of the scores between the sessions showed high test-retest reliability by ICC (0.86 and 0.85 for BSTAT and BSET respectively), with a small yet statistically significant learning effect. The novice group's scores were lower than the fellows' and attendings' (p < 0.001) for both the BSTAT and BSET; the fellows' scores were consistently lower than the attendings' on both tests, yet the differences were not statistically significant. CONCLUSION: This validation study of 2 objective tests of bronchoscopic skill demonstrated high reliability and concurrent validity. These instruments can now be used to evaluate the effectiveness of new competency-based bronchoscopy curricula.