A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy.

Importance: Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure. Objective: To develop and validate a procedure-specific competency assessment tool (CAT) for MIE. Design, Setting, and Participants: In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023. Exposure: Performance assessment of transthoracic MIE with an intrathoracic anastomosis. Main Outcomes and Measures: Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT. Results: Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases. Conclusions and Relevance: The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Validity evidence for a new portable, lower-cost platform for the fundamentals of endoscopic surgery skills test.

BACKGROUND: The fundamentals of endoscopic surgery (FES) examination measures the knowledge and skills required to perform safe flexible endoscopy. A potential limitation of the FES skills test is the size and cost of the simulator on which it was developed (GI Mentor II virtual reality endoscopy simulator; Simbionix LTD, Israel). A more compact and lower-cost alternative (GI Mentor Express) was developed to address this issue. The purpose of this study was to obtain evidence for the validity of scores obtained on the Express platform, so that it can be used for testing. STUDY DESIGN: General surgery residents at various levels of training and practicing endoscopists at five institutions participated. Each completed the five FES tasks on both simulator platforms in random order, with 3-14 days between tests. Scores were calculated using the same standardized computer-generated algorithm and compared using Pearson's correlation coefficient. RESULTS: There were 58 participants (mean age 32; 76% male) with a broad range of endoscopic experience. The mean (95% confidence interval) FES scores were 72 (67:77) on the GI Mentor II and 66 (60:71) on the Express. The correlation between scores on the two platforms was 0.86 (0.77:0.91; p < 0.0001). CONCLUSION: There is a high correlation between FES manual skills scores measured on the original platform and the new Express, providing evidence to support the use of the GI Mentor Express for FES testing.