Analysis of implementation science strategies to train laparoscopic surgical skills among Liberian surgeons using ALL-SAFE.

BACKGROUND: Significant disparities exist in laparoscopic training opportunities for surgeons in low- and middle-income countries (LMICs). ALL-SAFE is an innovative, low-cost training system for LMIC surgeons' laparoscopic development. However, strategies to implement and scale ALL-SAFE are unstudied. We aimed to assess the impact of implementation science strategies on ALL-SAFE uptake in Liberia, a novel and low-resource context. METHODS: This study used implementation science approaches to evaluate ALL-SAFE uptake in three Liberian hospitals: John F. Kennedy Medical Center, Phebe, and ELWA. Five validated implementation strategies of program orientation, local champions, network weaving, feedback mechanisms, and practical supervision were piloted. All five strategies were implemented at two hospitals, while four strategies, excluding clinician supervision, were implemented at the final site as a pseudo-control, to evaluate the impact of resource-intensive supervision. Participants included surgical consultants, residents, and medical students. Engagement was assessed with mixed methods including participant number, practice hours, module completion rates, and strategy feasibility through end-user interviews. RESULTS: Across three hospitals, 33 participants used ALL-SAFE for 87.8 total training hours (5,268 min). Participant numbers varied across sites (NJFK = 20, 80%; NPhebe = 7, 88%; NELWA = 6, 86%), as did practice times (TotalJFK = 3,060 min, Median time/ParticipantJFK = 103 min; TotalPhebe = 1,434 min, Median time/ParticipantPhebe = 126 min; TotalELWA = 774 min, Median time/ParticipantELWA = 100 min). Sites with practical supervision demonstrated higher engagement than those without (P = 0.042). Interviews (n = 8) revealed positive perceptions toward program orientation, practical supervision, and network weaving. Participants recommended thirteen additional strategies for sustainability including translating ALL-SAFE skills to patient care. CONCLUSION: Our study is one of the first to evaluate implementation science strategies for laparoscopic training in sub-Saharan Africa. The implementation bundles of five validated strategies were considered feasible by Liberian surgeons in both urban and rural hospitals. Mixed methods suggested a positive association between engagement and the resource-intensive practical supervision strategy. Future studies should focus on quantifying individual strategy contributions with rigorous implementation designs and assessing sustainability strategies.

Assessment of laparoscopic surgery practice, knowledge, and training goals in Liberia: A mixed-methods study among Liberian surgeons.

BACKGROUND: Laparoscopy training remains inaccessible in many low- and middle-income countries, including Liberia. We assessed the availability of laparoscopy and feasibility of implementing a laparoscopic program among Liberian surgeons and trainees. METHODS: This mixed-methods study utilized a 32-item survey and semi-structured interviews on laparoscopic experience, knowledge, desires, barriers, patient perceptions, and training opportunities among surgeons and trainees at Liberia's two main teaching hospitals in March 2023. Data analysis utilized summed knowledge and desirability scores, descriptive statistics, and qualitative content analysis. RESULTS: 31 interns, residents, and consultants participated, comprising 60% of Liberia's surgeons. Laparoscopic training (32%) and experience (16%) was low, with exposure limited to those training outside Liberia (p = 0.001). While laparoscopy knowledge varied (29% low, 55% medium, 16% high), participants expressed high interest in training (100%) and willingness to pay (52%). Interviews revealed four themes: desires for training, patient acceptability, feasibility of technology-based training, and barriers including limited equipment and expert trainers. At the time of survey, the only minimally invasive surgeries ever performed in Liberia were two diagnostic laparoscopies. CONCLUSIONS: This is the first mixed-methods study assessing laparoscopy in Liberia. Our sample, though small, comprised approximately 60% of Liberian surgeons in both rural and urban hospitals. Findings demonstrated limited experience, variable knowledge, and high desires for training, showing feasibility for laparoscopy implementation in Liberia.

Infant bronchial tree simulator: Success of a built-from-scratch model for single lung isolation.

BACKGROUND: One-lung ventilation in infants is a high-risk procedure. Complications include endotracheal tube occlusion, with grave consequences. Although there are commercially available bronchoscopy simulators, there are no realistic models of infant patients. This limits access to training opportunities that would ensure safe and efficient lung isolation. To bridge this gap, we developesd a realistic infant bronchial tree model for single lung intubation and evaluated preliminary validity evidence of its features and clinicians' ability to perform critical skills associated with pediatric one-lung ventilation. METHODS: Using computed tomography imaging, a stereolithography file of an infant airway was generated to 3D print a model. This model was inserted into a commercially available airway trainer to allow lung isolation using standard bronchoscopy techniques. Ten experienced pediatric anesthesiologists independently evaluated the simulator's physical attributes, realism, value, and relevance using a 29-item paper survey and rated using 4-point rating scales (4 = highest). Participants' ability to complete 5 critical tasks was self-reported using 5-point rating scales (5 = too easy). Item and domain mean ratings were calculated, and comments reviewed. RESULTS: Overall, reviews were positive, with mean scores indicating adequate realism and high value. Specific challenges were associated with right mainstem bronchus and upper lobe takeoff. Performance scores indicated that most tasks were "somewhat easy to perform," suggesting that the model's anatomy did not hinder physicians' ability to perform one-lung ventilation. CONCLUSION: Preliminary findings indicate that the novel simulator holds promise for training in lung isolation techniques after refinement. Future research will target refinement, expanding evaluation, and developing a comprehensive curriculum and competency assessment program.

Multi-institutional beta testing of a novel cervical esophagogastric anastomosis simulator.

Objective: A novel simulator developed to offer hands-on practice for the stapled side-to-side cervical esophagogastric anastomosis was tested previously in a single-center study that supported its value in surgical education. This multi-institutional trial was undertaken to evaluate validity evidence from 6 independent thoracic surgery residency programs. Methods: After a virtual session for simulation leaders, learners viewed a narrated video of the procedure and then alternated as surgeon or first assistant. Using an online survey, perceived value was measured across fidelity domains: physical attributes, realism of materials, realism of experience, value, and relevance. Objective assessment included time, number of sutures tearing, bubble test, and direct inspection. Comparison across programs was performed using the Kruskal-Wallis test. Results: Surveys were completed by 63 participants as surgeons (17 junior and 20 senior residents, 18 fellows, and 8 faculty). For 3 of 5 tasks, mean ratings of 4.35 to 4.44 correlated with "somewhat easy" to "very easy" to perform. The interrupted outer layer of the anastomosis rated lowest, suggesting this task was the most difficult. The simulator was rated as a highly valuable training tool. For the objective measurements of performance, "direct inspection" rated highest followed by "time." A total of 90.5% of participants rated the simulator as ready for use with only minor improvements. Conclusions: Results from this multi-institutional study suggest the cervical esophagogastric anastomosis simulator is a useful adjunct for training and assessment. Further research is needed to determine its value in assessing competence for independent operating and associations between improved measured performance and clinical outcomes.

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System.

OBJECTIVE: Laparoscopic surgical skill assessment and machine learning are often inaccessible to low-and-middle-income countries (LMIC). Our team developed a low-cost laparoscopic training system to teach and assess psychomotor skills required in laparoscopic salpingostomy in LMICs. We performed video review using AI to assess global surgical techniques. The objective of this study was to assess the validity of artificial intelligence (AI) generated scoring measures of laparoscopic simulation videos by comparing the accuracy of AI results to human-generated scores. DESIGN: Seventy-four surgical simulation videos were collected and graded by human participants using a modified OSATS (Objective Structured Assessment of Technical Skills). The videos were then analyzed via AI using 3 different time and distance-based calculations of the laparoscopic instruments including path length, dimensionless jerk, and standard deviation of tool position. Predicted scores were generated using 5-fold cross validation and K-Nearest-Neighbors to train classifiers. SETTING: Surgical novices and experts from a variety of hospitals in Ethiopia, Cameroon, Kenya, and the United States contributed 74 laparoscopic salpingostomy simulation videos. RESULTS: Complete accuracy of AI compared to human assessment ranged from 65-77%. There were no statistical differences in rank mean scores for 3 domains, Flow of Operation, Respect for Tissue, and Economy of Motion, while there were significant differences in ratings for Instrument Handling, Overall Performance, and the total summed score of all 5 domains (Summed). Estimated effect sizes were all less than 0.11, indicating very small practical effect. Estimated intraclass correlation coefficient (ICC) of Summed was 0.72 indicating moderate correlation between AI and Human scores. CONCLUSIONS: Video review using AI technology of global characteristics was similar to that of human review in our laparoscopic training system. Machine learning may help fill an educational gap in LMICs where direct apprenticeship may not be feasible.

Evidence supporting performance measures of laparoscopic appendectomy through a novel surgical proficiency assessment tool and low-cost laparoscopic training system.

BACKGROUND: Laparoscopic training remains inaccessible for surgeons in low- and middle-income countries, limiting its widespread adoption. We developed a novel tool for assessment of laparoscopic appendectomy skills through ALL-SAFE, a low-cost laparoscopy training system. METHODS: This pilot study in Ethiopia, Cameroon, and the USA assessed appendectomy skills using the ALL-SAFE training system. Performance measures were captured using the ALL-SAFE verification of proficiency tool (APPY-VOP), consisting of a checklist, modified Objective Structured Assessment of Technical Skills (m-OSATS), and final rating. Twenty participants, including novice (n = 11), intermediate (n = 8), and expert (n = 1), completed an online module covering appendicitis management and psychomotor skills in laparoscopic appendectomy. After viewing an expert skills demonstration video, participants recorded their performance within ALL-SAFE. Using the APPY-VOP, participants rated their own and three peer videos. We used the Kruskal-Wallis test and a Many-Facet Rasch Model to evaluate (i) capacity of APPY-VOP to differentiate performance levels, (ii) correlation among three APPY-VOP components, and (iii) rating differences across groups. RESULTS: Checklist scores increased from novice (M = 21.02) to intermediate (M = 23.64) and expert (M = 28.25), with differentiation between experts and novices, P = 0.005. All five m-OSATS domains and global summed, total summed, and final rating discriminated across all performance levels (P < 0.001). APPY-VOP final ratings adequately discriminated Competent (M = 2.0), Borderline (N = 1.8), and Not Competent (M = 1.4) performances, Χ2 (2,85) = 32.3, P = 0.001. There was a positive correlation between ALL-SAFE checklist and m-OSATS summed scores, r(83) = 0.63, P < 0.001. Comparison of ratings suggested no differences across expertise levels (P = 0.69) or location (P = 0.66). CONCLUSION: APPY-VOP effectively discriminated between novice and expert performance in laparoscopic appendectomy skills in a simulated setting. Scoring alignment across raters suggests consistent evaluation, independent of expertise. These results support the use of APPY-VOP among all skill levels inside a peer rating system. Future studies will focus on correlating proficiency to clinical practice and scaling ALL-SAFE to other settings.

Impact of the COVID-19 pandemic on American College of Surgeons-Accredited Education Institutes & American Society of Anesthesiologists-Simulation Education Network: Opportunities for interdisciplinary collaboration.

BACKGROUND: The COVID-19 pandemic presented challenges for simulation programs including American College of Surgeons Accredited Education Institutes and American Society of Anesthesiologists Simulation Education Network. American College of Surgeons Accredited Education Institutes and American Society of Anesthesiologists Simulation Education Network leadership were surveyed to identify opportunities to enhance patient safety through simulation. METHODS: Between January and June 2021, surveys consisting of 3 targeted domains: (I) Changing practice; (II) Contributions and recognition; and (III) Moving ahead were distributed to 100 American College of Surgeons Accredited Education Institutes and 54 American Society of Anesthesiologists Simulation Education Network centers. Responses were combined and percent frequencies reported. RESULTS: Ninety-six respondents, representing 51 (51%) American College of Surgeons Accredited Education Institutes, 17 (31.5%) American Society of Anesthesiologists Simulation Education Network, and 28 dually accredited centers, completed the survey. Change of practice. Although 20.3% of centers stayed fully operational at the COVID-19 onset, 82% of all centers closed: 32% were closed less than 3 months, 28% were closed 3 to 6 months, 8% were closed 7 to 9 months, and 32% remained closed as of June 6, 2021. Most impacted activities were large-group instruction and team training. Sixty-nine percent of programs converted in-person to virtual programs. Contributions. The top reported innovative contributions included policies (80%), curricula (80%), and scholarly work (74%), Moving ahead. The respondents' top concerns were returning to high-quality training to best address learners' deficiencies and re-engagement of re-directed training programs. When asked "How the American College of Surgeons/American Society of Anesthesiologists Programs could best assist your simulation center goals?" the top responses were "facilitate collaboration" and "publish best practices from this work." CONCLUSION: The Pandemic presented multiple challenges and opportunities for simulation centers. Opportunities included collaboration between American College of Surgeons Accredited Education Institutes and the American Society of Anesthesiologists Simulation Education Network to identify best practices and resources needed to enhance patient safety through simulation.

The Fundamentals of Vaginal Surgery pilot study: developing, validating, and setting proficiency scores for a vaginal surgical skills simulation system.

BACKGROUND: Surgical training in the simulation lab can develop basic skills that translate to the operating room. Standardized, basic skills programs that are supported by validated assessment measures exist for open, laparoscopic, and endoscopic surgery; however, there is yet to be a nationally recognized and widely implemented basic skills program specifically for vaginal surgery. OBJECTIVE: Develop a vaginal surgical simulation system; evaluate robust validity evidence for the simulation system and its related performance measures; and establish a proficiency score that discriminates between novice and experienced vaginal surgeon performance. STUDY DESIGN: In this 3-phased study, we developed the Fundamentals of Vaginal Surgery simulation system consisting of (1) the Fundamentals of Vaginal Surgery Trainer, a task trainer; (2) a validated regimen of tasks to be performed on the trainer; and (3) performance measures to determine proficiency. In Phase I, we developed the task trainer and selected surgical tasks by performing a needs assessment and hierarchical task analyses, with review and consensus from an expert panel. In Phase II, we conducted a national survey of vaginal surgeons to collect validity evidence regarding test content, response process, and internal structure relevant to the simulation system. In Phase III, we compared performance of novice (first and second year residents) and experienced (third and fourth year residents, fellows, and faculty) surgeons on the simulation system to evaluate relevant relationships to other variables and consequences. Performance measures were analyzed to set a proficiency score that would discriminate between novice and expert (faculty) vaginal surgical performance. RESULTS: A novel task trainer and 6 basic vaginal surgical skills were developed in Phase I. In Phase II, the survey responses of 48 participants (27 faculty surgeons, 6 fellows, and 14 residents) were evaluated on the dimensions of test content, response process, and internal structure. To support evidence of test content, the participants deemed the task trainer and surgical tasks representative of intended surgical field and supportive of typical surgical actions (mean scores, 3.8-4.4/5). For response process, rater-data analysis revealed high rating variability regarding prototype color. This early evidence confirmed the value of a white prototype. For internal structure, there was high agreement among rater groups (obstetricians and gynecologists generalists vs Female Pelvic Medicine and Reconstructive Surgery specialists: interclass correlation coefficient range, 0.59-0.91; learners vs faculty interclass correlation coefficient range, 0.64-1.0). There were no differences in ratings across institution type, surgeon volume, expertise (P>.14). In Phase III, we analyzed performance from 23 participants (15 [65%] obstetricians and gynecologists residents, 3 [13%] fellows, and 5 [22%] Female Pelvic Medicine and Reconstructive Surgery faculty). Experienced surgeons scored significantly higher than novice surgeons (median, 467.5; interquartile range, [402.5-542.5] vs median, 261.5; interquartile range, [211.5-351.0]; P<.001). Based on these data, setting a proficiency score threshold at 400 results in 0% (0/6) novices attaining the score, with 100% (5/5) experts exceeding it. CONCLUSION: We present validity evidence relevant to all 5 sources which supports the use of this novel simulation system for basic vaginal surgical skills. To complement the system, a proficiency score of 400 was established to discriminate between novices and experts.

Development of a novel mandibular distraction osteogenesis simulator using Computer Aided Design and 3D printing.

INTRODUCTION: Micrognathia, a component of Robin Sequence, can cause glossoptosis, failure of palatal fusion, and critical obstruction of the airway. Mandibular distraction osteogenesis (MDO) is at times offered to anteriorly translate the mandible and tongue, relieving airway obstruction. MDO is an intricate reconstructive procedure that may be ideal for teaching using a high-fidelity educational simulator, allowing early hands-on experience in a zero-risk environment. OBJECTIVES: To design a novel, low-cost, high-fidelity neonatal MDO simulator that can be used for trainee education and refinement of surgical technique. METHODS: A novel MDO simulator was developed using additive manufacturing techniques. Three experts in MDO surgery completed a 20-item survey, rating the simulator's physical attributes, the realism of experience, the simulator's value, its relevance to practice and the surgeon's ability to perform tasks on a 4-point Likert scale. RESULTS: Computer Aided Design (CAD) and 3D printing allowed for the production of a realistic surgical simulator that emulates important aspects of MDO surgery. This preliminary evaluation indicated adequate means across the five domains relevant to the simulator's fidelity and usability (M = 3.33 to 3.75) out of a maximum of 4 points. Lowest rated items were consistent with expert comments allowing future refinement on subsequent iterations. Consumable material costs per model were $9.39 USD. CONCLUSIONS: The MDO model demonstrated adequate fidelity and holds promise as a skill-development tool for surgeons in training. Further studies are planned to determine its utility as a training and assessment tool.

A novel cervical esophagogastric anastomosis simulator.

OBJECTIVES: At least partially technically related, a cervical esophagogastric anastomosis has a 12% to 14% leak rate, which is theoretically reducible with simulator practice. Preliminary development and testing of a cervical esophagogastric anastomosis simulator are described. METHODS: A portable, low-cost, scale reproduction of the cervical esophagogastric anastomosis operative site was engineered around a 19 × 11 × 6-cm plastic box. Silicone "esophageal" and "gastric tip" castings permitted construction of a stapled side-to-side cervical esophagogastric anastomosis guided by an illustrated curriculum. In a 2-phase pilot study, the simulator and curriculum were evaluated. Phase 1: Seven faculty evaluated fidelity using a 5-point, 24-item survey of (1) physical attributes, (2) realism of materials, (3) realism of experience, (4) value, and (5) relevance, and (6) ability to perform tasks. Overall impression of the simulator was also measured. Phase 2: Eight thoracic surgical trainees similarly evaluated the simulator and the quality of the curriculum. Faculty and trainee ratings were compared using a Rasch model, and inter-rater agreement was estimated. RESULTS: There were no overall fidelity differences across faculty and resident ratings. Combined observed averages ranged from 4.52 (Realism of Materials) to 5.00 (Relevance). Lifelike feel of esophagus had the lowest ratings (observed average = 4.40). Residents rated interrupted outer layer of anterior closure to be more difficult (observed average = 4.13) than faculty (observed average = 4.86; P = .016, d = 1.99). Global ratings (observed average = 3.33/4.00) indicated participants believed the simulator could be used for cervical esophagogastric anastomosis training now, but could be improved slightly. CONCLUSIONS: Preliminary evidence suggests the novel cervical esophagogastric anastomosis simulator is valuable as a surgical training tool.

Evaluating a Solely Mechanical Articulating Laparoscopic Device: A Prospective Randomized Crossover Study.

BACKGROUND: The FlexDex® (FD) is a solely mechanical articulating device that combines the functionality of robotic surgery with the relative low cost and simplicity of laparoscopy. We sought to evaluate the performance of first-time FD users while performing a simple suture task at locations of varying degrees of difficulty. STUDY DESIGN: A prospective, randomized crossover study was performed comparing the FD to standard laparoscopy (SL). Two specific groups were evaluated; Group 1 consisted of complete novices, and Group 2 consisted of surgical trainees. Participants performed a simple suture with both FD and SL locations of varying degrees of difficulty (Easy, Moderate, and Hard). The following outcomes were evaluated: Instrument Function and Ergonomics (Comfort/Ergonomics survey), Task Difficulty (National Aeronautics and Space Administration Task Load Index [NASA-TLX]), Task Performance Quality (Objective Structured Assessment of Technical Skills [OSATS]), and Time (seconds). RESULTS: Twenty-two participants were enrolled with 12 participants in Group 1 and 10 participants in Group 2. Group 1-FD participants experienced overall less shoulder strain (1.2 ± 0.40 versus 1.9 ± 0.90, P = .01), and Group 2-FD participants experienced less shoulder (2.5 ± 0.66 versus 4.0 ± 0.50, P = .01), back (1.1 ± 0.32 versus 1.9 ± 0.74, P = .01), and forearm strain (1.9 ± 0.88 versus 2.5 ± 1.1, P = .04). Group 1 participants using the FD experienced higher mental demand (73 ± 17 versus 48 ± 27, P < .01) and perceived effort (70 ± 20 versus 54 ± 23, P < .001). Both Group 1 and Group 2 FD participants performed tasks at the Hard location more effectively. Both Group 1 (70 versus 87, P = .21) and Group 2 (53 versus 60, P = .55) performed tasks at the Hard location in similar times, while Group 1 (80 versus 177, P = .03) and Group 2 (33 versus 70, P = .001) performed tasks at the Easy location in shorter times using SL. CONCLUSIONS: This study demonstrates the first assessment of the FD, a mechanically articulating laparoscopic tool. First-time FD users demonstrated improved ergonomics and effectiveness suturing at difficult locations. Future studies will focus on comparison to robotic surgery and translation into clinical applications.

An approach to value-based simulator selection: The creation and evaluation of the simulator value index tool.

BACKGROUND: Currently there is no reliable, standardized mechanism to support health care professionals during the evaluation of and procurement processes for simulators. A tool founded on best practices could facilitate simulator purchase processes. METHODS: In a 3-phase process, we identified top factors considered during the simulator purchase process through expert consensus (n = 127), created the Simulator Value Index (SVI) tool, evaluated targeted validity evidence, and evaluated the practical value of this SVI. A web-based survey was sent to simulation professionals. Participants (n = 79) used the SVI and provided feedback. We evaluated the practical value of 4 tool variations by calculating their sensitivity to predict a preferred simulator. RESULTS: Seventeen top factors were identified and ranked. The top 2 were technical stability/reliability of the simulator and customer service, with no practical differences in rank across institution or stakeholder role. Full SVI variations predicted successfully the preferred simulator with good (87%) sensitivity, whereas the sensitivity of variations in cost and customer service and cost and technical stability decreased (≤54%). The majority (73%) of participants agreed that the SVI was helpful at guiding simulator purchase decisions, and 88% agreed the SVI tool would help facilitate discussion with peers and leadership. CONCLUSION: Our findings indicate the SVI supports the process of simulator purchase using a standardized framework. Sensitivity of the tool improved when factors extend beyond traditionally targeted factors. We propose the tool will facilitate discussion amongst simulation professionals dealing with simulation, provide essential information for finance and procurement professionals, and improve the long-term value of simulation solutions. Limitations and application of the tool are discussed.

Simulation center best practices: A review of ACS-accredited educational institutes' best practices, 2011 to present.

BACKGROUND: Since the inception of American College of Surgeons' Accredited Educational Institute (ACS-AEI) Consortium, accreditation reviews have identified best practices in simulation-based education and center operations. A review of best practices would support the communication of these best practices, offer recognition of exemplar institutes, and facilitate discussion and sharing of resources amongst AEIs. METHODS: We examined 5 years of ACS AEI accreditation best practices identified across all standards and criteria. The goal was to identify resources that could be shared among AEIs and recognize AEI champions that have promoted best practices in surgical simulation. RESULTS: From 149 site reviews (July 2011-June 2016), reviewers identified 197 best practices across 83 AEIs (52.9% of all sites reviewed received a best practice). A total of 52.5% of best practices were associated with curriculum development, delivery of effective education and assessment, and 25 available resources were identified that could be shared among AEIs. The majority of best practices (n = 117, 59.3%) were identified at 24 AEIs, with the highest number of best practices identified at Banner Simulation Center and New Orleans Learning Center (10 each over 2 reviews). Twenty-two other institutes presented 97 best practices, with between 3 to 8 per institute (mean = 4.4, standard deviation = 1.6). Specific best practices, criteria, and AEI champions are highlighted. CONCLUSION: Review of AEI accreditation best practices identified common themes for surgical simulation programs and identified tools that could be shared to advance all programs and champion AEIs that have promoted best practices in surgical simulation.

Preliminary Evaluation of a Novel Rigid Bronchoscopy Simulator.

PURPOSE: Emergent retrieval of airway foreign bodies (AFBs) in children remains a priority skill set for pediatric surgeons. In the setting of low procedural volume, simulation-based education with deliberate practice is essential to ensure trainees reach expected surgical competency. The purposes of this work were to (1) create a realistic rigid bronchoscopy for AFB retrieval simulation model and (2) to evaluate preliminary validity evidence of a novel simulator for the use of training and assessing pediatric surgical trainees' rigid bronchoscopy skills. METHODS: After institutional review board exemption determination, 18 participants performed AFB retrieval of two different objects on a novel simulator that represented an 18-month-old pediatric tracheobronchial airway. Participants reported their experience and comfort level, and rated the simulator across two domains-Authenticity and their Ability to perform tasks. Authenticity was measured by 23 items across five subdomains (Visual Attributes, Materials' Response, Realism of Experience, Value and Relevance, and Global Value). Participants who had previously performed ≥10 rigid bronchoscopies were categorized as "experienced," while those reporting <10 were considered "novice." Validity evidence relevant to test content and internal structure was evaluated using a many-facet Rasch model. RESULTS: Novice surgeons (n = 12) had previously performed a mean of 2.7 (±2.0) rigid bronchoscopies, compared to 15.4 (±7.7) by experienced surgeons (n = 6). For both models, the Value and Relevance subdomain received the highest ratings (observed average [OA] = 3.9, while Materials' Response received the lowest (OA <3.0). Participants' Global Value rating for this model was consistent with "requires minor improvements before it can be considered for use in rigid bronchoscopy training." CONCLUSIONS: We successfully designed, assembled, and evaluated a novel pediatric rigid bronchoscopy model for AFB retrieval. The model was considered as relevant to educational needs and valuable as a testing and training tool. With recommended improvements, the model could be used for implementation with a Mastery Learning curriculum.

Preliminary Evaluation of a Laparoscopic Common Bile Duct Simulator for Pediatric Surgical Education.

PURPOSE: Laparoscopic common bile duct exploration (LCBDE) decreases overall costs and length of stay in patients with choledocolithiasis. However, utilization of LCBDE remains low. We sought to evaluate a previously developed general surgery LCBDE simulator among a cohort of pediatric surgical trainees. The study purpose was to evaluate the content validity of an LCBDE simulator to support or refute its use in pediatric surgery education. MATERIALS AND METHODS: After IRB exempt determination, 30 participants performed a transcystic LCBDE using a previously developed simulator and evaluated the simulator using a self-reported 28-item instrument. The instrument consisted of two primary domains (Quality and Ability to Perform) that were rated using twenty-five 4-point rating scales and one 4-point global rating scale. Validity evidence relevant to test content was evaluated using a many-facet Rasch model. Interitem consistency was estimated using Cronbach's alpha. P < .05 was considered statistically significant. RESULTS: The highest combined observed averages were for the Value subdomain (OA = 3.79), whereas the lowest ratings were for the Physical/visual attributes subdomain (OA = 3.19). The averaged global rating was 3.14, consistent with this simulator can be considered for use in pediatric LCBDE training, but could be improved slightly. Rasch indices were favorable and supported evidence relevant to test content. Interitem consistency estimates were also favorable, with α values of 0.94 and 0.56 for Qualities and Ability, respectively. CONCLUSIONS: Overall, participants rated the LCBDE simulator highly valuable for pediatric surgical education and felt that it could be used as an educational tool with minor modifications.

Simulator and 2 tools: Validation of performance measures from a novel neurosurgery simulation model using the current Standards framework.

BACKGROUND: Ventriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation model could also allow for assessment of trainees' proficiencies with measures that align with Accreditation Council for Graduate Medical Education milestones. This study extends previous work to evaluate validity evidence from the simulator, its newly developed performance assessment, the Ventricolostomy Procedural Assessment Tool, and the Objective Structured Assessment for Technical Skills. METHODS: After Institutional Review Board exemption, performance data were collected from 11 novice and 3 expert neurosurgeons (n = 14). Participants self-reported their ability to perform tasks on the simulator using the Ventricolostomy Procedural Assessment Tool, an 11-item, step-wise instrument with 5-point rating scales ranging from 1 (unable to perform) to 5 (performs easily and smoothly). De-identified operative performances were videotaped and independently rated by 3 neurosurgeons, using the Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills. We evaluated multiple sources of validity evidence (2014 Standards) to examine psychometric quality of the measures and to test our assumption that the tools could discriminate between novice and expert performances adequately. We used a multifacet Rasch model and traditional indices, such as Cronbach alpha, intraclass correlation, and Wilcoxon signed-rank test estimates. RESULTS: Validity evidence relevant to test content and response processes was supported adequately. Evidence of internal structure was supported by high interitem consistency (n = 0.95) and inter-rater agreement for most Ventricolostomy Procedural Assessment Tool items (Intraclass correlation coefficient = [0.00, 0.91]) and all Objective Structured Assessment for Technical Skills items (Intraclass correlation coefficient = [0.80, 0.93]). Overall, novices performed at a lower level than experts on both scales (P < .05), supporting evidence relevant to relationships to other variables. Deeper analysis of novice/expert ratings indicated novices attained lower performances ratings for all Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills items, but statistical significance was only achieved for the Objective Structured Assessment for Technical Skills items (P < .01). Rater bias estimates were favorable, supporting evidence relevant to consequences of testing. CONCLUSION: Despite a small sample, favorable evidence using current Standards supports the use of the novel simulator and both tools combined for skills training and performance assessment, but challenges (potential threats to validity) should be considered prior to implementation.

A physical simulator for endoscopic endonasal drilling techniques: technical note.

In this paper, the authors present a physical model developed to teach surgeons the requisite drilling techniques when using an endoscopic endonasal approach (EEA) to the skull base. EEA is increasingly used for treating pathologies of the ventral and ventrolateral cranial base. Endonasal drilling is a unique skill in terms of the instruments used, the long reach required, and the restricted angulation, and gaining competency requires much practice. Based on the successful experience in creating custom simulators, the authors used 3D printing to build an EEA training model from post-processed thin-cut head CT scans, formulating the materials to provide realistic haptic feedback and endoscope handling. They performed a preliminary assessment at 2 institutions to evaluate content validity of the simulator as the first step of the validation process. Overall results were positive, particularly in terms of bony landmarks and haptic response, though minor refinements were suggested prior to use as a training device.

Ongoing evaluation of video-based assessment of proctors' scoring of the fundamentals of laparoscopic surgery manual skills examination.

BACKGROUND: In continued efforts to maintain standards required for high-stakes examination administration, Society of American Gastrointestinal and Endoscopic Surgeons' Fundamentals of Laparoscopic Surgery (FLS) program requires all prospective test proctors to complete the New FLS Proctor Workshop. As the FLS proctor pool and training evolves, the influence of experience on performance is not clear. In the previous work, we proposed that asynchronous web-based performances might be used to train proctors. In this study, we assessed the accuracy of proctors' ratings from asynchronous web-based performances by comparing the sensitivity and the specificity of ratings from proctors having varied experience levels. METHODS: A sample of 42 (26% of all registered proctors) FLS proctors (recently trained novice, n = 15; intermediate, n = 20; and expert, n = 7) viewed 2 videotaped, laparoscopic-view FLS performances via the web. The first performance (error) contained 1 predetermined critical performance error in each of the 5 tasks (5 total errors), whereas the second performance (no error) contained no critical errors. For both the performances, participants noted time to complete each of the 5 tasks and any critical errors they observed (dichotomously scored). Participants also completed a demographic section that captured years of experience as a proctor. Using the Kruskal-Wallis test, we compared new trainee, intermediate, and expert proctors' recorded task times for both the performances. The sensitivity and the specificity values were also independently calculated using ratings from the error and the no error performances and then compared using the same test. RESULTS: There were no differences in recorded times across proctor groups for any of the tasks; p = 0.21 and 0.94. Rating sensitivity was 79% (novice), 75% (intermediate), and 83% (expert), with no significant differences across experience levels; p > 0.46. Following removal of the peg transfer task that had particularly low sensitivity, the sensitivity for the remaining 4 tasks increased [88% and 92%]. The specificity was 93% (novice), 86% (intermediate), and 100% (expert); p = 0.046. Comments from less-experienced proctors indicated that their focus on noncritical performance issues, such as "loop not on black line" and "air knot," were consistent with decreased specificity. CONCLUSIONS: Favorable results of this pilot study suggest that web-based FLS performances may be used for assessing FLS proctors' rating quality with adequate sensitivity and specificity. Decreased rating specificity from intermediate proctors indicates that these proctors would benefit from increased testing frequencies, updated training on current FLS proctoring standards, and ongoing performance review.

Evaluation of Three Sources of Validity Evidence for a Synthetic Thoracoscopic Esophageal Atresia/Tracheoesophageal Fistula Repair Simulator.

PURPOSE: Thoracoscopic esophageal atresia (EA)/tracheoesophageal fistula (TEF) repair is technically challenging. We have previously reported our experiences with a high-fidelity hybrid model for simulation-based educational instruction in thoracoscopic EA/TEF, including the high cost of the tissue for these models. The purposes of this study were (1) to create a low-cost synthetic tissue EA/TEF repair simulation model and (2) to evaluate the content validity of the synthetic tissue simulator. MATERIALS AND METHODS: Review of the literature and computed tomography images were used to create computer-aided drawings (CAD) for a synthetic, size-appropriate EA/TEF tissue insert. The inverse of the CAD image was then printed in six different sections to create a mold that could be filled with platinum-cured silicone. The silicone EA/TEF insert was then placed in a previously described neonatal thorax and covered with synthetic skin. Following institutional review board-exempt determination, 47 participants performed some or all of a simulated thoracoscopic EA/TEF during two separate international meetings (International Pediatric Endosurgery Group [IPEG] and World Federation of Associations of Pediatric Surgeons [WOFAPS]). Participants were identified as "experts," having 6-50 self-reported thoracoscopic EA/TEF repairs, and "novice," having 0-5 self-reported thoracoscopic EA/TEF repairs. Participants completed a self-report, six-domain, 24-item instrument consisting of 23 5-point rating scales and one 4-point Global Rating Scale. Validity evidence relevant to test content and response processes was evaluated using the many-facet Rasch model, and evidence of internal structure (interitem consistency) was estimated using Cronbach's alpha. RESULTS: A review of the participants' ratings indicates there were no overall differences across sites (IPEG versus WOFAPS, P=.84) or experience (expert versus novice, P=.17). The highest observed averages were 4.4 (Value of Simulator as a Training Tool), 4.3 (Physical Attributes-chest circumference, chest depth, and intercostal space), and 4.3 (Realism of Experience-fistula location). The lowest observed averages were 3.5 (Ability to Perform-closure of fistula), 3.7 (Ability to Perform-acquisition target trocar sites), 3.8 (Physical Attributes-landmark visualization), 3.8 (Ability to Perform-anastomosis and dissection of upper pouch), and 3.9 (Realism of Materials-skin). The Global Rating Scale was 2.9, coinciding with a response of "this simulator can be considered for use in neonatal TEF repair training, but could be improved slightly." Material costs for the synthetic EA/TEF inserts were less than $2 U.S. per insert. CONCLUSIONS: We have successfully created a low-cost synthetic EA/TEF tissue insert for use in a neonatal thoracoscopic EA/TEF repair simulator. Analysis of the participants' ratings of the synthetic EA/TEF simulation model indicates that it has value and can be used to train pediatric surgeons, especially those early in their learning curve, to begin to perform a thoracoscopic EA/TEF repair. Areas for model improvement were identified, and these areas will be the focus for future modifications to the synthetic EA/TEF repair simulator.