Is Technical Expertise Necessary to Assess Technical Expertise? Let's Ask the Kids.

Background: Simulation has an increasing role in surgical training; however, using validated tools such as the Objective Structured Assessment of Technical Skills (OSATS) is time-consuming, which may be a potential barrier to simulation-based training. This study tests the hypothesis that assessors with technical expertise are necessary to objectively score a technical task. Methods: Three tasks, ring transfer, needle pass, and atresia cut, were performed using a synthetic thoracoscopic simulator for esophageal atresia/tracheoesophageal fistula. Three pediatric surgeons, 6 novice adults, and 3 children aged 9-13 years scored each attempt using the overall global OSATS rating from 1 to 5 (1 "repeated or awkward movements" to 5 "fluid movement. No awkwardness"). Results: For the ring transfer, all assessors scored novice participants significantly less than expert and intermediate scores (surgeon P = .0004, nonsurgeon adults P = .0009 and children P = .0003). For the needle pass, all assessors gave significantly different scores between novices and experts (surgeon P = .0007, nonsurgeon adult P = .0008, and children P = .0040). For the atresia cut, surgeon assessors gave significantly higher scores for experts and intermediate and novice (P = .0004). Nonsurgeon assessors gave experts and intermediates significantly higher scores than novices (P = .0001). Surgeon assessors achieved good reliability for ring transfer (0.8252) and needle pass (0.7769) compared with nonsurgeon assessors who showed poor reliability for the ring transfer (0.3959) and moderate for the needle pass (0.6551). Conclusions: Expertise in performing these procedures is not a prerequisite for an assessor to evaluate the technical skill, hence assessors of skill acquisition can be nonexpert, a nonsurgeon, or even a child. The variability in all groups suggests that reliability overall is increased with multiple assessors. Although nonsurgeon assessors may be appropriate for formative assessments, they lack the reliability to provide assessment of competence for high stakes complex tasks. Summative assessment will likely require at least 1 surgeon/expert assessor to provide reliability.

Construct validation of a 3D printed neonatal thoracoscopic simulator: Can it measure expertise?

BACKGROUND: acquiring technical expertise for neonatal thoracoscopy is challenging. To address this, we designed a fully synthetic thoracoscopic simulator for which we established its construct validity. METHODS: three thoracoscopic tasks were assessed: ring transfer, needle pass and incision of a blind upper esophageal pouch (EA cut). Participants watched instructional videos with accompanying written instructions for each task before having their attempt video recorded. All tasks were marked by three blinded pediatric surgeons using a modified Objective Structured Assessment of Technical Skills (OSATS). Scores were assessed using appropriate statistical analysis and inter-rater reliability was analyzed by interclass correlation coefficient (ICC). RESULTS: 23 participants completed the ring transfer and needle pass and 21 the EA cut: 5 experts (consultant surgeons), 5 intermediate (registrars on a training program) and 13 novices (medical students, house surgeons or non-training registrars). All three tasks distinguished between novice and intermediate/expert (ring transfer p = 0.00001, needle pass p = 0.0004 and EA cut p = 0.0014, respectively). Interrater reliability was good for ring transfer and needle pass but poor for EA cut. CONCLUSION: the tasks distinguished between novice and intermediate/expert but not between expert and intermediate. In needle pass and EA cut, there was a trend for the experts to score higher than intermediate participants. Ring transfer and needle pass tasks achieved construct validity, had good interrater reliability and were found to be useful in assessing a novice surgeon's progression towards the intermediate level. Distinguishing between intermediate and expert may require assessment of more complex tasks such as intracorporeal suturing and tying. LEVEL OF EVIDENCE: II.

Master and Apprentice or a Slave to Technology? A Randomized Controlled Trial of Minimal Access Surgery Simulation-Based Training Techniques.

Introduction: This study set out to assess the efficacy of three different approaches to simulation-based minimal access surgery (MAS) training using a three-dimensional printed neonatal thoracoscopic simulator and a virtual simulator. Materials and Methods: Randomized controlled trial of medical students (N = 32), as novices to MAS. The participants performed two construct validated tasks on a thoracoscopic simulator and were then randomly allocated into four intervention groups: (1) three consultant-led sessions on a thoracoscopic simulator; (2) three self-directed learning sessions on the same simulator; (3) self-directed "virtual training" on the "SimuSurg" application; and (4) control. Postintervention participants repeated both tasks. Videos of all task attempts were de-identified and marked by a blinded consultant pediatric surgeon. Results: There were no statistically significant differences in baseline objective structured assessment of technical skills (OSATS) scores or demographics in any group. For the "ring transfer" task, Groups 1 and 2 showed significant improvement after intervention, with no significant change in Groups 3 or 4. There was no significant difference between Groups 1 or 2 in postintervention scores. For the "needle pass" task, no group demonstrated a statistically significant improvement after intervention. Conclusion: Practice on a physical simulator either consultant-led or self-directed led to improved scores for MAS novices compared with a virtual simulator or no intervention for a simple "ring transfer" task. This suggests that time on the physical simulator was the most important factor and implies that trainees could usefully practice simple tasks at their convenience rather than require consultant supervision. This improvement is not seen in more challenging tasks such as the "needle pass."

End-User Input into the Design and Validation of a Synthetic Thoracoscopic Esophageal Atresia/Tracheo-Esophageal Fistula Simulator.

Introduction: Thoracoscopic repair of esophageal atresia and tracheo-esophageal fistula (EA/TEF) is challenging. We addressed this by designing a fully synthetic simulator of the procedure and described the design process and how its content validity was assessed. Methods: An iterative design and assessment of content validity was undertaken in three stages. Data were collected from participants who trialed the model and completed a survey of their experience (adapted from Barsness et al.). Results: The model was trialed by participants of varying experience. Each design refinement improved the model's fidelity and validity. For the last iteration of the simulator, the observed averages (out of a maximum of 5) were: value as a training tool 4.8, relevance 4.6, physical attributes 4.5, realism of material 4.25, realism experience 4.17, and ability to perform tasks 3.77. Conclusion: An iterative design process based on end-user feedback has led to a synthetic simulator that has achieved a high level of content validity. This model has advantages over other EA/TEF simulators in that it is relatively inexpensive and does not use animal tissue, thus removing ethical and procurement issues. It was rated highly for its value and relevance to training.

Development of an instrumented thoracoscopic surgical trainer for objective evaluation of esophageal atresia/tracheoesophageal fistula repair.

Operative repair of complex conditions such as esophageal atresia and tracheoesophageal fistula (EA/TEF) is technically demanding, but few training opportunities exist outside the operating theater for surgeons to attain these skills. Learning them during surgery on actual neonates where the stakes are high, margins for error narrow, and where outcomes are influenced by technical expertise, is problematic. There is an increasing demand for high-fidelity simulation that can objectively measure performance. We developed such a simulator to measure force and motion reliably, allowing quantitative feedback of technical skill. A 3D-printed simulator for thoracoscopic repair of EA/TEF was instrumented with motion and force tracking components. A 3D mouse, inertial measurement unit (IMU), and optical sensor that captured force and motion data in four degrees of freedom (DOF) were calibrated and verified for accuracy. The 3D mouse had low average relative errors of 2.81%, 3.15%, and 6.15% for 0 mm, 10 mm offset in Y, and 10 mm offset in X, respectively. This increased to - 23.5% at an offset of 42 mm. The optical sensors and IMU displayed high precision and accuracy with low SDs and average relative errors, respectively. These parameters can be a useful measurement of performance for thoracoscopic EA/TEF simulation prior to surgery. Graphical abstract Inclusion of sensors into a high-fidelity simulator design can produce quantitative feedback which can be used to objectively asses performance of a technically difficult procedure. As a result, more surgical training can be done prior to operating on actual patients in the operating theater.

Critical design and validation considerations for the development of neonatal minimally invasive surgery simulators.

BACKGROUND/PURPOSE: Pediatric surgical trainees have limited exposure to advanced minimally invasive surgery (MIS) during their clinical training, particularly for cases such as esophageal atresia/tracheoesophageal fistula (EA/TEF). Simulation on validated neonatal models offers an alternative means of training that may augment traditional forms of training; but to be useful, they must fulfill certain criteria. METHODOLOGY: Review of the currently available MIS, thoracoscopic and laparoscopic, simulators for pediatric surgery, and identification of those factors that contribute to their fidelity and validity as a training tool that must be incorporated in the design of future simulation models. RESULTS: There are few neonatal laparoscopic and thoracoscopic models currently available, or in the research stage. To our knowledge, there is no commercially available, synthetic, high fidelity and low cost thoracoscopic model in existence. Use of animal tissue has disadvantages of ethical dilemmas, cost, and logistic and procurement issues. Newer synthetic models need to be validated for fidelity, to replicate those components of the operation that pose the greatest technical challenge, and incorporate means of measuring acquisition of technical expertise. CONCLUSION: This review describes the principles that need to be considered to develop low cost, validated high-fidelity MIS simulator that can be used for training, and that is capable of measuring the acquisition of the technical skills that can be applied to the repair of complex procedures such as esophageal atresia. Level of evidence III.