Evidence supporting performance measures of laparoscopic appendectomy through a novel surgical proficiency assessment tool and low-cost laparoscopic training system.

BACKGROUND: Laparoscopic training remains inaccessible for surgeons in low- and middle-income countries, limiting its widespread adoption. We developed a novel tool for assessment of laparoscopic appendectomy skills through ALL-SAFE, a low-cost laparoscopy training system. METHODS: This pilot study in Ethiopia, Cameroon, and the USA assessed appendectomy skills using the ALL-SAFE training system. Performance measures were captured using the ALL-SAFE verification of proficiency tool (APPY-VOP), consisting of a checklist, modified Objective Structured Assessment of Technical Skills (m-OSATS), and final rating. Twenty participants, including novice (n = 11), intermediate (n = 8), and expert (n = 1), completed an online module covering appendicitis management and psychomotor skills in laparoscopic appendectomy. After viewing an expert skills demonstration video, participants recorded their performance within ALL-SAFE. Using the APPY-VOP, participants rated their own and three peer videos. We used the Kruskal-Wallis test and a Many-Facet Rasch Model to evaluate (i) capacity of APPY-VOP to differentiate performance levels, (ii) correlation among three APPY-VOP components, and (iii) rating differences across groups. RESULTS: Checklist scores increased from novice (M = 21.02) to intermediate (M = 23.64) and expert (M = 28.25), with differentiation between experts and novices, P = 0.005. All five m-OSATS domains and global summed, total summed, and final rating discriminated across all performance levels (P < 0.001). APPY-VOP final ratings adequately discriminated Competent (M = 2.0), Borderline (N = 1.8), and Not Competent (M = 1.4) performances, Χ2 (2,85) = 32.3, P = 0.001. There was a positive correlation between ALL-SAFE checklist and m-OSATS summed scores, r(83) = 0.63, P < 0.001. Comparison of ratings suggested no differences across expertise levels (P = 0.69) or location (P = 0.66). CONCLUSION: APPY-VOP effectively discriminated between novice and expert performance in laparoscopic appendectomy skills in a simulated setting. Scoring alignment across raters suggests consistent evaluation, independent of expertise. These results support the use of APPY-VOP among all skill levels inside a peer rating system. Future studies will focus on correlating proficiency to clinical practice and scaling ALL-SAFE to other settings.

A multidisciplinary international collaborative implementing low cost, high fidelity 3D printed airway models to enhance Ethiopian anesthesia resident emergency cricothyroidotomy skills.

BACKGROUND: Similar to other sub-Saharan countries, Ethiopia suffers from a severe shortage of adequately trained health professionals. Academic partnerships can support sustainable training programs and build capacity for low-resource settings. 3D modeling and simulation-based training provide necessary tools, especially for rarely-encountered clinical situations, such as needle cricothyroidotomy. METHODS: Departments of Anesthesiology, Otolaryngology, and Learning Health Sciences collaborated to develop a low-cost, high-fidelity simulator and Cricothryoidotomy Skills Maintenance Program (CSMP). Twelve anesthesia residents at St. Paul's Hospital Medical Millennium College in Addis Ababa, Ethiopia participated in CSMP. The program consisted of a didactic session with presentation and demonstration and an immersive CICO scenario. Program evaluation was performed using pre/post-training knowledge and 2 procedural performance assessments-the CSMP Global Rating Scale and the Checklist. With consent, performances were videotaped and rated independently by 3 University of Michigan faculty. RESULTS: Improvements were identified in all areas, including residents' knowledge, measured by mean summed test scores (Mpre = 3.31,Mpost = 4.46,p = 0.003), time to perform cricothyroidotomy (Mpre = 96.64,Mpost = 72.82,p = 0.12), residents' performance quality, measured by overall mean Global ratings, (Mpre = 0.20; Mpost = 0.70) with improvements identified at the item-level, p = 0.001 with moderate-large effect sizes, and residents' ability to complete tasks, measured by mean Checklist ratings (Mpre = 0.51,Mpost = 0.90, with item-level improvements observed, p ≤ 0.01, with small-large effect sizes. Residents' self-reported confidence also improved (Mpre = 1.69, Mpost = 3.08,p = 0.001). CONCLUSION: Our work shows that cricothyroidotomy skills taught to anesthesia residents at SPHMMC with a 3D printed laryngotracheal model improves knowledge, skills, and confidence. The creation of a low-cost, high-fidelity simulator and a CSMP has the potential to impact patient care and safety world-wide.

Development and Validation of an Objective Scoring Tool for Robot-Assisted Radical Prostatectomy: Prostatectomy Assessment and Competency Evaluation.

PURPOSE: Comprehensive training and skill acquisition by urological surgeons are vital to optimize surgical outcomes and patient safety. We sought to develop and validate PACE (Prostatectomy Assessment and Competence Evaluation), an objective and procedure specific tool to assess the quality of robot-assisted radical prostatectomy. MATERIALS AND METHODS: Development and content validation of PACE was performed by deconstructing robot-assisted radical prostatectomy into 7 key domains utilizing the Delphi methodology. Reliability and construct validation were then assessed using de-identified videos performed by practicing surgeons and fellows. Consensus for each domain was defined as achieving a content validity index of 0.75 or greater. Reliability was assessed by the intraclass correlation and construct validation using a mixed linear model accounting for multiple ratings on the same video. RESULTS: After 3 rounds consensus was reached on wording, relevance of the skills assessed and concordance between the score assigned and the skill assessed. An intraclass correlation of 0.4 or greater was achieved for all domains. The expert group outperformed trainees in all domains but reached statistical significance in bladder drop (4.5 vs 3.4, p = 0.002), preparation of the prostate (4.4 vs 3.2, p <0.0001), seminal vesicle and posterior plane dissection (8.3 vs 6.8, p = 0.03), and neurovascular bundle preservation (4.1 vs 2.4, p <0.0001). Limitations included the lack of assessment of other key skills such as communication and decision making. CONCLUSIONS: PACE is a structured, procedure specific and reliable tool that objectively measures surgical performance during robot-assisted radical prostatectomy. It can differentiate different levels of expertise and provide structured feedback to customize training and surgical quality improvement.

Ongoing evaluation of video-based assessment of proctors' scoring of the fundamentals of laparoscopic surgery manual skills examination.

BACKGROUND: In continued efforts to maintain standards required for high-stakes examination administration, Society of American Gastrointestinal and Endoscopic Surgeons' Fundamentals of Laparoscopic Surgery (FLS) program requires all prospective test proctors to complete the New FLS Proctor Workshop. As the FLS proctor pool and training evolves, the influence of experience on performance is not clear. In the previous work, we proposed that asynchronous web-based performances might be used to train proctors. In this study, we assessed the accuracy of proctors' ratings from asynchronous web-based performances by comparing the sensitivity and the specificity of ratings from proctors having varied experience levels. METHODS: A sample of 42 (26% of all registered proctors) FLS proctors (recently trained novice, n = 15; intermediate, n = 20; and expert, n = 7) viewed 2 videotaped, laparoscopic-view FLS performances via the web. The first performance (error) contained 1 predetermined critical performance error in each of the 5 tasks (5 total errors), whereas the second performance (no error) contained no critical errors. For both the performances, participants noted time to complete each of the 5 tasks and any critical errors they observed (dichotomously scored). Participants also completed a demographic section that captured years of experience as a proctor. Using the Kruskal-Wallis test, we compared new trainee, intermediate, and expert proctors' recorded task times for both the performances. The sensitivity and the specificity values were also independently calculated using ratings from the error and the no error performances and then compared using the same test. RESULTS: There were no differences in recorded times across proctor groups for any of the tasks; p = 0.21 and 0.94. Rating sensitivity was 79% (novice), 75% (intermediate), and 83% (expert), with no significant differences across experience levels; p > 0.46. Following removal of the peg transfer task that had particularly low sensitivity, the sensitivity for the remaining 4 tasks increased [88% and 92%]. The specificity was 93% (novice), 86% (intermediate), and 100% (expert); p = 0.046. Comments from less-experienced proctors indicated that their focus on noncritical performance issues, such as "loop not on black line" and "air knot," were consistent with decreased specificity. CONCLUSIONS: Favorable results of this pilot study suggest that web-based FLS performances may be used for assessing FLS proctors' rating quality with adequate sensitivity and specificity. Decreased rating specificity from intermediate proctors indicates that these proctors would benefit from increased testing frequencies, updated training on current FLS proctoring standards, and ongoing performance review.

A validated low-cost training model for suprapubic catheter insertion.

OBJECTIVE: To describe an anatomically correct simulator for use in suprapubic catheter (SPC) insertion training. METHODS: A scale reproduction of an adult male pelvis with bony landmarks and a subcutaneous fluid-filled reservoir was created using platinum cured silicone rubber. This model was evaluated by 6 expert urologists for content validity with a 16-item 5-point rating scale used to evaluate domains relevant to the simulator­physical attributes, realism of experience, realism of materials, and global rating. The simulator was used by 25 general surgeons from rural practices participating in a 2-day comprehensive specialized surgical skills course. RESULTS: The domains were scored between 1 and 5; 1 being "not at all realistic" and 5 being "highly realistic, no changes needed." The average expert ratings of the domains were 3.9 (physical attributes), 4.3 (realism of experience), and 3.9 (realism of materials). The simulator was rated more valuable as a training tool (4.5) compared with a testing tool (3.8) with an average global rating of 4.1. CONCLUSION: Experts and trainees reported high satisfaction with their experience using this simulator. Preliminary evidence suggests this simulator is a useful tool that can be integrated into training programs to facilitate learning this necessary urologic skill.

Fundamentals of Laparoscopic Surgery manual test: is videotaped performance assessment an option?

BACKGROUND: In efforts to maintain standards required to evaluate the high-stakes assessment, Society of American Gastrointestinal and Endoscopic Surgeons Fundamentals of Laparoscopic Surgery (FLS) requires all new proctors to complete the train-the-proctor workshop. As the pool of FLS proctors expands, new methods to streamline training and quality assurance programs should be considered. We propose that videotaped performances of the FLS manual tasks may be an alternative proxy to live assessment for training of new proctors, but evaluation of proctors' measures from videotaped FLS performances is required before implementation. METHODS: A 2-phased research consisted of capturing newly trained proctors' (n = 20) ratings of 3 similar FLS performances across 3 stations-live (Live), videotaped-laparoscopic only (Lap Only) view, and videotaped-dual (Dual) views, during the 2012 Society of American Gastrointestinal and Endoscopic Surgeons FLS train-the-proctor workshop. A month later, a sample of proctors (n = 9) viewed videotaped versions of live FLS performances originally observed during the workshop. Captured metrics include recognition of a predefined critical error for each task (dichotomously scored and summed) and time to complete each of the 5 tasks. Analysis of variance compared the proctors' summed ratings for similar performances across Live, Lap Only, and Dual views, whereas paired t test compared recorded times of Lap Only vs Dual views, Live vs web ratings, and proctors' recorded times across the Lap Only and Dual views. RESULTS: There were neither differences in ratings across Live, Lap Only, and Dual views (p = 0.49) nor in recorded times for performances viewed across Lap Only and Dual viewing options (p = 0.29 and 0.76, respectively). Mean summed performance ratings observed live (4.6) were higher than those observed via the web (4.0), although not significant (p = 0.051). There were no differences in recorded times for identical performances across Live and web observations (p = 0.18 and 0.69, respectively), although findings were limited by sample size for some tasks. CONCLUSIONS: In spite of limitations, favorable results of this preliminary study supports use of videotaped FLS performances for streamlining training mechanisms for FLS proctors, and developing best practices in standard setting and long-term evaluation of the of FLS proctors.

Evaluation of Three Sources of Validity Evidence for a Synthetic Thoracoscopic Esophageal Atresia/Tracheoesophageal Fistula Repair Simulator.

PURPOSE: Thoracoscopic esophageal atresia (EA)/tracheoesophageal fistula (TEF) repair is technically challenging. We have previously reported our experiences with a high-fidelity hybrid model for simulation-based educational instruction in thoracoscopic EA/TEF, including the high cost of the tissue for these models. The purposes of this study were (1) to create a low-cost synthetic tissue EA/TEF repair simulation model and (2) to evaluate the content validity of the synthetic tissue simulator. MATERIALS AND METHODS: Review of the literature and computed tomography images were used to create computer-aided drawings (CAD) for a synthetic, size-appropriate EA/TEF tissue insert. The inverse of the CAD image was then printed in six different sections to create a mold that could be filled with platinum-cured silicone. The silicone EA/TEF insert was then placed in a previously described neonatal thorax and covered with synthetic skin. Following institutional review board-exempt determination, 47 participants performed some or all of a simulated thoracoscopic EA/TEF during two separate international meetings (International Pediatric Endosurgery Group [IPEG] and World Federation of Associations of Pediatric Surgeons [WOFAPS]). Participants were identified as "experts," having 6-50 self-reported thoracoscopic EA/TEF repairs, and "novice," having 0-5 self-reported thoracoscopic EA/TEF repairs. Participants completed a self-report, six-domain, 24-item instrument consisting of 23 5-point rating scales and one 4-point Global Rating Scale. Validity evidence relevant to test content and response processes was evaluated using the many-facet Rasch model, and evidence of internal structure (interitem consistency) was estimated using Cronbach's alpha. RESULTS: A review of the participants' ratings indicates there were no overall differences across sites (IPEG versus WOFAPS, P=.84) or experience (expert versus novice, P=.17). The highest observed averages were 4.4 (Value of Simulator as a Training Tool), 4.3 (Physical Attributes-chest circumference, chest depth, and intercostal space), and 4.3 (Realism of Experience-fistula location). The lowest observed averages were 3.5 (Ability to Perform-closure of fistula), 3.7 (Ability to Perform-acquisition target trocar sites), 3.8 (Physical Attributes-landmark visualization), 3.8 (Ability to Perform-anastomosis and dissection of upper pouch), and 3.9 (Realism of Materials-skin). The Global Rating Scale was 2.9, coinciding with a response of "this simulator can be considered for use in neonatal TEF repair training, but could be improved slightly." Material costs for the synthetic EA/TEF inserts were less than $2 U.S. per insert. CONCLUSIONS: We have successfully created a low-cost synthetic EA/TEF tissue insert for use in a neonatal thoracoscopic EA/TEF repair simulator. Analysis of the participants' ratings of the synthetic EA/TEF simulation model indicates that it has value and can be used to train pediatric surgeons, especially those early in their learning curve, to begin to perform a thoracoscopic EA/TEF repair. Areas for model improvement were identified, and these areas will be the focus for future modifications to the synthetic EA/TEF repair simulator.

Filling a void: developing a standard subjective assessment tool for surgical simulation through focused review of current practices.

BACKGROUND: Simulation-based education (SBE) has advanced greatly in surgery the past decade, partly through development of validated performance measurement. Standard measures are commonly used to evaluate performance (eg, Objective Structured Assessment of Technical Skills, Fundamentals of Laparoscopic Surgery, Global Operative Assessment of Laparoscopic Skills [GOALS]). However, subjective measures are necessary for ensuring content validity during evaluation of novel simulations or simulators. Although commonly assessed, there are no standardized instruments for such surveys of participants. This lack of standardization limits the ability to compare simulations and/or simulators. We performed a focused literature review to assess current uses of subjective measures, and develop a template for a standardized assessment tool. METHODS: A representative sample of recently collected subjective measures was generated through a focused literature review of the journals Surgery, Journal of Surgical Education, and Journal of the American College of Surgeons (January 2008-November 2012) using the key words "surgical" and "simulation." RESULTS: Of the 137 articles relevant to development and/or evaluation of surgical skills curricula or simulators, 19 (12%) reported subjective measures from participants. Ten domains were identified, including Self-efficacy/Confidence/Comfort (11, 57%), Model quality/characteristics (7, 37%), Educational/Program value (6, 32%), Previous experience with procedure/simulation (5, 26%) Relevance to practice (3, 16%), Quality- trainer feedback (3, 16%), Quality-experience (2, 11%), Attitude toward specific aspects (2, 11%), Satisfaction/Enjoyability (2, 11%), Ability to perform relevant task(s) (2, 11%). Response options varied and included 4-, 5-, and 10-point rating scales, Visual Analog Scales, and open written responses. CONCLUSION: These results suggest that simulation experiences are not currently assessed in a systematic manner. However, analysis of the literature suggests that the full range of subjective measures commonly used for subjective assessment could be addressed by a unified assessment instrument. To this end, the Michigan Standard Simulation Experience Scale (MiSSES) template has been developed and is available on-line. Such a tool would provide practitioners a freely available resource used to measure performance and preferences in SBE.

Fundamentals of laparoscopic surgery (FLS) manual skills assessment: surgeon vs nonsurgeon raters.

OBJECTIVE: The American Board of Surgery has recently started requiring completion of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) fundamentals of laparoscopic surgery (FLS) program for board certification in general surgery. Although most SAGES Testing Centers utilize nonsurgeons as FLS proctors, the effectiveness of using nonsurgeons as FLS proctors has not been evaluated. METHODS: Surgeons and nonsurgeons attending FLS proctor training workshops were studied. Participants reviewed training materials before course attendance. Subjects watched a videotaped FLS performance containing 9 "critical" errors, which participants were asked to identify. This assessment was repeated after hands-on training. RESULTS: Thirteen surgeon and 17 nonsurgeon subjects participated. At baseline, surgeons detected 66% of errors, vs 65% for nonsurgeons, with no statistical difference between groups. Analysis of individual tasks also showed no difference between groups, except for intracorporeal knot-tying (p = 0.049). Both groups improved after training (p < 0.01), with surgeons detecting 81% of errors vs 83% for nonsurgeons (no difference in overall or task-specific ratings). CONCLUSIONS: This study suggests that trained nonsurgeons may be as effective as surgeon proctors in detecting errors associated with the FLS manual test. This finding supports the utility of using trained nonsurgeons as FLS proctors as surgical training programs face increasing economic constraints.

Administrative considerations when implementing ACS/APDS Skills Curriculum.

BACKGROUND: With time and cost constraints, implementing an effective, yet efficient, skills curriculum poses significant challenges. Our purpose is to describe a successful curriculum administrative structure that promoted faculty buy-in and accountability, learner responsibility, and acceptable resource usage. METHODS: A total of 14 American College of Surgery (ACS) modules were included in the postgraduate year 1 curriculum. Before arrival, 2 modules were sent to newly matched residents. Remaining modules were administered over a 4-month period, with integrated, independent practice opportunities, as well as 4 mentored and 1 peer practice sessions. A total of 2 verifications of proficiency (VOP) progress exams and 1 final comprehensive VOP were administered. To promote faculty ownership, 1 faculty member was asked to lead each module. Module leaders attended an orientation and development session, and created an instructional management plan. Each module was taught by the leader and 2 additional faculty coinstructors, and evaluated by residents. Equipment, resource costs, and man-hours were tracked. RESULTS: Faculty buy-in was demonstrated by enthusiastic participation, with only 2 absences. Residents gave high ratings to all the modules (range, 4.22-4.89/5). Curriculum costs were approximately $21,500, reduced from potential costs of $187,000 if all simulators would have been purchased new. The estimated budget for year 2 is $17,000. CONCLUSION: It is critical for new curricula to have resident and faculty buy-in, accountability for quality teaching and learning, and reasonable resource use. We provide suggestions for structuring a curriculum to ensure accomplishment of these important drivers.