Technical assessment in minimally invasive complete mesocolic excision: Is the complete mesocolic excision competency assessment tool valid and reliable?

AIM: The complete mesocolic excision competency assessment tool (CMECAT) is a novel tool designed to assess technical skills in minimally invasive complete mesocolic excision (CME) surgery. The aim of this study was to assess construct validity and reliability of CMECAT in a clinical context. METHOD: Colorectal surgeons were asked to submit video recorded laparoscopic CME resections for independent assessment of their technical abilities. The videos were grouped by surgeons' training level, and four established CME experts were recruited as CMECAT assessors. Extended reliability analysis (G-theory) was applied to describe assessor agreement. RESULTS: A total of 19 videos and 72 assessments were included in the analysis. Overall, technical skills assessed by CMECAT improved with increased training level: the experts scored significantly better than the untrained surgeons (3.3 vs. 2.5 points; p < 0.01). On right-sided resections, significantly higher scores were reported with increased training level for all categories and sections, while for left-sided resections, the variance across groups was smaller and significantly higher scores were only reported for oncological safety describing items. Overall, assessor agreement was high (G-coefficient: 0.81). CONCLUSION: This study confirms that CMECAT can be applied to video recorded CME cases for technical skill assessment. Further, it can reliably assess technical performance in right sided CME surgery, where construct validity has now been established. More videos are required to evaluate its validity on left colonic CME. In the future, we hope CMECAT can improve feedback during CME training, serve as a tool in certification processes and contribute to distinguishing CME from conventional surgery in future research.

Development of a procedure-specific tool for skill assessment in left- and right-sided laparoscopic complete mesocolic excision.

AIM: To (1) develop an assessment tool for laparoscopic complete mesocolic excision (LCME) and (2) report evidence of its content validity. METHOD: Assessment statements were revealed through (1) semi-structured expert interviews and (2) consensus by the Delphi method, both involving an expert panel of five LCME surgeons. All experts were interviewed and then asked to rate LCME describing statements from 1 (strongly disagree) to 5 (strongly agree). Responses were returned anonymously to the panel until consensus was reached. Statements were directly included as content in the assessment tool if ≥60% of the experts responded "agree" or "strongly agree" (ratings 4 and 5), with the remaining responses being "neither agree nor disagree" (rating 3). Interclass correlation coefficient (ICC) was calculated for expert agreement evaluation. All included statements were subsequently reformulated as tool items and approved by the experts. RESULTS: Four Delphi rounds were performed to reach consensus. Disagreement was reported for statements describing instrument handling around pancreas; visualisation of landmarks before inferior mesenteric artery ligation; lymphadenectomy around the inferior mesenteric artery, and division of the terminal ileum and transverse colon. ICC in the last Delphi-round was 0.84. The final tool content included 73 statements, converted to 48 right- and 40 left-sided items for LCME assessment. CONCLUSION: A procedure-specific, video-based tool, named complete mesocolic excision competency assessment tool (CMECAT), has been developed for LCME skill assessment. In the future, we hope it can facilitate assessment of LCME surgeons, resulting in improved patient outcome after colon cancer surgery.

Implementation and validation of a competency assessment tool for laparoscopic cholecystectomy.

BACKGROUND: Achieving proficiency in a surgical procedure is a milestone in the career of a trainee. We introduced a competency assessment tool for laparoscopic cholecystectomy in our residency program. Our aim was to assess the inter-rater reliability of this tool. METHODS: We included all laparoscopic cholecystectomies performed by residents under the supervision of board certified surgeons. All residents were assessed at the end of the procedure by the supervising surgeon (live reviewer) using our competency assessment tool. Video records of the same procedure were analyzed by two independent reviewers (reviewer A and B), who were blinded to the performing trainee's. The assessment had three parts: a laparoscopic cholecystectomy-specific assessment tool (LCAT), the objective structured assessment of technical skills (OSATS) and a 5-item visual analogue scale (VAS) to address the surgeon's autonomy in each part of the cholecystectomy. We compared the assessment scores of the live supervising surgeon and the video reviewers. RESULTS: We included 15 junior residents who performed 42 laparoscopic cholecystectomies. Scoring results from live and video reviewer were comparable except for the OSATS and VAS part. The score for OSATS by the live reviewer and reviewer B were 3.68 vs. 4.26 respectively (p = 0.04) and for VAS (5.17 vs. 4.63 respectively (p = 0.03). The same difference was found between reviewers A and B with OSATS score (3.75 vs. 4.26 respectively (p = 0.001)) and VAS (5.56 vs. 4.63 respectively; p = 0.004)). CONCLUSION: Our competency assessment tool for the evaluation of surgical skills specific to laparoscopic cholecystectomy has been shown to be objective and comparable in-between raters during live procedure or on video material.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery.

Importance: Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear. Objective: To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes. Design, Setting, and Participants: This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases. Interventions: Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons. Main Outcomes and Measures: Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores. Results: The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03). Conclusions and Relevance: Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Surgical Quality Assurance in COLOR III: Standardization and Competency Assessment in a Randomized Controlled Trial.

OBJECTIVE: The aim of this study was to develop an objective and reliable surgical quality assurance system (SQA) for COLOR III, an international multicenter randomized controlled trial (RCT) comparing transanal total mesorectal excision (TaTME) with laparoscopic approach for rectal cancer. BACKGROUND OF SUMMARY DATA: SQA influences outcome measures in RCTs such as lymph nodes harvest, in-hospital mortality, and locoregional cancer recurrence. However, levels of SQA are variable. METHOD: Hierarchical task analysis of TaTME was performed. A 4-round Delphi methodology was applied for standardization of TaTME steps. Semistructured interviews were conducted in round 1 to identify key steps and tasks, which were rated as mandatory, optional, or prohibited in rounds 2 to 4 using questionnaires. Competency assessment tool (CAT) was developed and its content validity was examined by expert surgeons. Twenty unedited videos were assessed to test reliability using generalizability theory. RESULTS: Eighty-three of 101 surgical tasks identified reached 70% agreement (26 mandatory, 56 optional, and 1 prohibited). An operative guide of standardized TaTME was created. CAT is matrix of 9 steps and 4 performance qualities: exposure, execution, adverse event, and end-product. The overall G-coefficient was 0.883. Inter-rater and interitem reliability were 0.883 and 0.986. To enter COLOR III, 2 unedited TaTME and 1 laparoscopic TME videos were submitted and assessed by 2 independent assessors using CAT. CONCLUSION: We described an iterative approach to develop an objective SQA within multicenter RCT. This approach provided standardization, the development of reliable and valid CAT, and the criteria for trial entry and monitoring surgical performance during the trial.

Development and implementation of the Structured Training Trainer Assessment Report (STTAR) in the English National Training Programme for laparoscopic colorectal surgery.

BACKGROUND: There is a lack of educational tools available for surgical teaching critique, particularly for advanced laparoscopic surgery. The aim was to develop and implement a tool that assesses training quality and structures feedback for trainers in the English National Training Programme for laparoscopic colorectal surgery. METHODS: Semi-structured interviews were performed and analysed, and items were extracted. Through the Delphi process, essential items pertaining to desirable trainer characteristics, training structure and feedback were determined. An assessment tool (Structured Training Trainer Assessment Report-STTAR) was developed and tested for feasibility, acceptability and educational impact. RESULTS: Interview transcripts (29 surgical trainers, 10 trainees, four educationalists) were analysed, and item lists created and distributed for consensus opinion (11 trainers and seven trainees). The STTAR consisted of 64 factors, and its web-based version, the mini-STTAR, included 21 factors that were categorised into four groups (training structure, training behaviour, trainer attributes and role modelling) and structured around a training session timeline (beginning, middle and end). The STTAR (six trainers, 48 different assessments) demonstrated good internal consistency (α = 0.88) and inter-rater reliability (ICC = 0.75). The mini-STTAR demonstrated good inter-item reliability (α = 0.79) and intra-observer reliability on comparison of 85 different trainer/trainee combinations (r = 0.701, p = <0.001). Both were found to be feasible and acceptable. The educational report for trainers was found to be useful (4.4 out of 5). CONCLUSIONS: An assessment tool that evaluates training quality was developed and shown to be reliable, acceptable and of educational value. It has been successfully implemented into the English National Training Programme for laparoscopic colorectal surgery.

Re-validating the Observational Teamwork Assessment for Surgery tool (OTAS-D): cultural adaptation, refinement, and psychometric evaluation.

BACKGROUND: The nontechnical and team skills of surgical teams are critical for safety and efficiency in the operating room. Assessment of nontechnical and team skills can facilitate improvement by encouraging both self-reflection and team reflection, identifying training needs, and informing operating room (OR) team training approaches. The observational teamwork assessment for surgery (OTAS) tool is a well-validated and robust tool for capturing teamwork in the operating room. The aims of the present study were to systematically adapt and refine the OTAS for German-speaking OR staff and to test the adapted assessment tool (OTAS-D) for psychometric properties and metric equivalence. METHODS: The study was carried out in three stages: at stage 1, OTAS was translated into German. At stage 2, experienced German OR experts (surgeons, OR nurses, anesthetists) were interviewed. At stage 3, two blinded assessors observed 11 general surgical operations (general surgical and vascular procedures) and interrater reliability was tested for refined OTAS-D behavioral exemplars and scorings. RESULTS: The German OR experts confirmed the applicability and content validity of the vast majority of translated behavioral exemplars. After their evaluation, 32 items were changed slightly, six were changed substantially, and one item was added. During observations, perfect and substantial interobserver agreement was found for 77 behavioral exemplars (67.1 % of the items, kappa coefficient >0.60). Rating at all OTAS behaviors showed acceptable levels of reliability (intraclass correlation coefficients >0.72). CONCLUSIONS: The OTAS-D is a tool for valid and reliable assessment of nontechnical skills that contribute to safe and effective surgical performance in ORs staffed by German-speaking professionals. Furthermore, our study serves as an example for systematically adapting and customizing well-established observational tools across different healthcare environments.

Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England.

OBJECTIVES: To develop, validate, and implement a competency assessment tool (CAT) for technical surgical performance in the context of a summative assessment process for the National Training Programme in Laparoscopic Colorectal Surgery (NTP). BACKGROUND: The NTP is an educational initiative by the National Cancer Action Team in England to safely increase the uptake of laparoscopic colorectal surgery. It is the first competency-based national educational initiative for specialist surgeons (consultants), and performance assessment is an integral part of the program. METHODS: Content validity was sought using expert opinion by semistructured interviews and the Delphi method. For validity and reliability studies, NTP apprentices and experts were asked to submit video-recorded cases. Construct validity was established between delegates who passed the assessment and those who failed. Concurrent validity was tested by comparing scores with error counts as identified by observational clinical human reliability analysis. A fully crossed design, using generalizability theory methods and D-studies, was used for reliability. FINDINGS: Interviews and the Delphi method revealed a list of characteristics for assessment. A hybrid structure combining task-specific and generic items was used to include important characteristics into the assessment format. Fifty-four cases were submitted. Overall reliability reached G(ACI) = 0.803 when using 2 cases and 2 assessors. Experts scored significantly better than apprentices (3.19 vs 2.60; P = 0.004), and apprentices who passed had better scores than those who failed (2.95 vs 2.28; P < 0.001). There was an inverse correlation between CAT scores and observational clinical human reliability analysis error counts (ρ = -0.520, P < 0.001). The combination of both methods reached overall sensitivity of 100%, specificity of 83.3%, a positive predictive value of 93.8%, and a negative predictive value of 100%. CONCLUSIONS: The CAT can reliably assess technical performance in laparoscopic colorectal surgery. The use of CATs to judge specialist technical performance before embarking on independent practice of new procedures is achievable on a national scale and can be adapted by other specialties.

Root cause analysis of internal hernia and Roux limb compression after laparoscopic Roux-en-Y gastric bypass using observational clinical human reliability assessment.

BACKGROUND: Internal hernia (IH) and Roux limb compression (RC) are recognized complications after retrocolic laparoscopic Roux-en-Y gastric bypass for obesity. The aim of the present study was to systematically identify the surgical technical errors leading to these complications. METHODS: An observational clinical human reliability assessment approach was used to analyze the operating videos of 3 groups: an IH group (n = 12), a Roux compression group (n = 13), and a control group (no complications, n = 21). Two investigators, unaware of the outcomes, reviewed all videos, using special rating software. All errors were categorized using the external error mode system and further described if a direct consequential error (e.g., bleeding) was found. RESULTS: An analysis of data showed that, on average, more errors occurred in the complication groups than in the control group (IH 5.85, Roux compression 3.54, control .90, P < .001). The strongest differences were found for missing intermesenteric stitches on both sides of the Roux limb. Logistic regression analysis showed that a missed stitch between the mesentery of the Roux limb and the transverse mesocolon was an independent predictor for IH (B = 1.727, P = .025). No technical or consequential errors could be identified as responsible for RC. CONCLUSION: The observational clinical human reliability analysis is a useful method to identify operative failure. For retrocolic, retrogastric laparoscopic Roux-en-Y gastric bypass, a systematic approach for the closure of the transverse mesenteric window might prevent IH complications.

Observational clinical human reliability analysis (OCHRA) for competency assessment in laparoscopic colorectal surgery at the specialist level.

BACKGROUND: There are no valid and reliable tools to assess competency in advanced laparoscopic surgery at a specialist level. The observational clinical human reliability analysis (OCHRA) may have the required characteristics of such a tool. The aim of this study was to evaluate construct and concurrent validity of OCHRA for competency assessment at a specialist level. METHODS: Thirty-two video-recorded laparoscopic colorectal resections, performed by experts and delegates of the National Training Program in England, were evaluated. Each video was analysed using OCHRA by identifying errors enacted during surgery. The number of tissue-handling, instrument-misuse, and consequential errors was recorded using video-rating software. Times spent on dissecting (D) and on exposing (E) tissues were also measured (D/E ratio). In addition, two independent expert surgeons globally assessed each video regarding competency (pass vs. fail). Logistic regression was used to predict outcomes. RESULTS: A total of 399 errors were identified. There was a significant difference when comparing the expert, pass, and fail groups for total errors (median counts for experts = 4, pass = 10, fail = 17; P < 0.001). When comparing the pass and fail groups excluding experts, differences could be found for tissue-handling errors (7 vs. 12; P = 0.005), but not for consequential errors (4 vs. 7; P = 0.059) and instrument-handling errors (4 vs. 5; P = 0.320). The D/E ratio was significantly lower for delegates than for experts (0.6 vs. 1.0; P = 0.001). When all four independent variables were used to predict delegates who passed or failed, the area under the receiver operating characteristic curve was 0.867, sensitivity was 71.4%, and specificity was 90.9%. CONCLUSION: OCHRA is a valid tool for assessing competency at a specialist level in advanced laparoscopic surgery. It has the potential to be used for recertification and revalidation of specialists.

Self vs expert assessment of technical and non-technical skills in high fidelity simulation.

BACKGROUND: Accurate assessment is imperative for learning, feedback and progression. The aim of this study was to examine whether surgeons can accurately self-assess their technical and nontechnical skills compared with expert faculty members' assessments. METHODS: Twenty-five surgeons performed a laparoscopic cholecystectomy (LC) in a simulated operating room. Technical and nontechnical performance was assessed by participants and faculty members using the validated Objective Structured Assessment of Technical Skills (OSATS) and the Non-Technical Skills for Surgeons scale (NOTSS). RESULTS: Assessment of technical performance correlated between self and faculty members' ratings for experienced (median score, 30.0 vs 31.0; ρ = .831; P = .001) and inexperienced (median score, 22.0 vs 28.0; ρ = .761; P = .003) surgeons. Assessment of nontechnical skills between self and faculty members did not correlate for experienced surgeons (median score, 8.0 vs 10.5; ρ = -.375; P = .229) or their more inexperienced counterparts (median score, 9.0 vs 7.0; ρ = -.018; P = .953). CONCLUSIONS: Surgeons can accurately self-assess their technical skills in virtual reality LC. Conversely, formal assessment with faculty members' input is required for nontechnical skills, for which surgeons lack insight into their behaviours.

Observational tools for assessment of procedural skills: a systematic review.

BACKGROUND: Assessment by direct observation of procedural skills is an important source of constructive feedback. The aim of this study was to identify observational tools for technical skill assessment, to assess characteristics of these tools, and to assess their usefulness for assessment. METHODS: Included studies reported tools for observational assessment of technical skills. A total of 106 articles were included. RESULTS: Three main categories included global assessment scales evaluating generic skills (n = 29), task-specific methods assessing procedure-specific skills (n = 30), and combinations of tools evaluating both generic and task-specific skills (n = 47). In most studies, content validity was not evaluated using an accepted scientific method. All tools were assessed for inter-rater reliability and construct validity. Data on feasibility, acceptability, and educational impact were sparse. CONCLUSIONS: There is evidence of validity and reliability for observational assessment tools at the trainee level. In most studies a comprehensive analysis of the tools was not achieved. Evaluation of technical skill using current observational assessment tools is not reliable and valid at the specialist level. Future research needs to focus on further systematic tool development and analysis, especially at the specialist level.