Performance in Irish Selection and Future Performance in Surgical Training.

Importance: Selection processes for surgical training should aim to identify candidates who will become competent independent practitioners and should aspire to high standards of reliability and validity. Objective: To determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training, examined via rate of progression to Higher Specialist Training (HST), attrition rates, and performance as assessed through a multimodal framework of workplace-based and simulation-based assessments. Design, Setting, and Participants: This retrospective observational cohort study included data from all successful applicants to the Royal College of Surgeons in Ireland (RCSI) national Core Surgical Training (CST) program. Participants included all trainees recruited to dedicated postgraduate surgical training from 2016 to 2020. These data were analyzed from July 11, 2016, through July 10, 2022. Exposures: Selection decisions were based on a composite score that was derived from technical aptitude assessments, undergraduate academic performance, and a 4-station multiple mini-interview. Main outcomes and measures: Assessment data, attrition rates, and rates of progression to HST were recorded for each trainee. CST performance was assessed using workplace-based and simulation-based technical and nontechnical skill assessments. Potential associations between selection and assessment measures were explored using Pearson correlation, logistic regression, and multiple linear-regression analyses. Results: Data were available for 303 trainees. Composite scores were positively associated with progression to HST (odds ratio [OR], 1.09; 95% CI, 1.05-1.13). There was a weak positive correlation, ranging from 0.23 to 0.34, between scores and performance across all CST assessments. Multivariable linear regression analysis showed technical aptitude scores at application were associated with future operative performance assessment scores, both in the workplace (ß = 0.31; 95% CI, 0.14-0.48) and simulated environments (ß = 0.57; 95% CI, 0.33-0.81). There was evidence that the interpersonal skills interview station was associated with future performance in simulated communication skill assessments (ß = 0.55; 95% CI, 0.22-0.87). Conclusions and Relevance: In this study, performance at the time of Irish national selection, measured across technical and nontechnical domains in a multimodal fashion, was associated with future performance in the workplace and in simulated environments. Future studies will be required to explore the consequential validity of selection, including potential unintended effects of selection and ranking on candidate performance.

Assessing operative competence in core surgical training: A reliability analysis.

BACKGROUND: This study quantifies the number of observations required to reliably assess the operative competence of Core Surgical Trainees (CSTs) in Ireland, using the Supervised Structured Assessment of Operative Performance (SSAOP) tool. METHODS: SSAOPs (April 2016-February 2021) were analysed across a mix of undifferentiated procedures, as well as for three commonly performed general surgery procedures in CST: appendicectomy, abdominal wall hernia repair, and skin/subcutaneous lesion excision. Generalizability and Decision studies determined the number of observations required to achieve dependability indices ≥0.8, appropriate for use in high-stakes assessment. RESULTS: A total of 2,294 SSAOPs were analysed. Four assessors, each observing 10 cases, can generate scores sufficiently reliable for use in high-stakes assessments. Focusing on a selection of core procedures yields more favourable reliability indices. CONCLUSION: Trainers should conduct repeated assessments across a smaller number of procedures to improve reliability. Programs should increase the assessor mix to yield sufficient dependability indices for high-stakes assessment.

Baseline Assessments of Psychomotor Aptitude Are Associated With the Future Operative Performance of Surgical Trainees: A Prospective Observational Cohort Study.

OBJECTIVE: This study sought to investigate the association between validated psychomotor ability tests and future in-theater and simulated operative performance. BACKGROUND: Assessments of visuospatial ability, perceptual ability, and manual dexterity correlate with simulated operative performance. Data showing the predictive value of such assessments in relation to future performance in the workplace is lacking. METHODS: Core surgical residents in Ireland recruited from 2016 to 2019 participated in assessments of baseline perceptual, visuospatial, and psychomotor ability; Pictorial Surface Orientation (PicSOr) testing, digital visuospatial ability testing, and manual dexterity testing. Operative performance was prospectively assessed using the in-theater Supervised Structured Assessment of Operative Performance (SSAOP) tool, and simulation-based Operative Surgical Skill (OSS) assessments performed over a 2-year core training period. SSAOP assessments were scored using a 15-point checklist and a global 5-point operative performance score. OSS assessments were scored using procedure-specific checklists. Univariate correlations and multiple linear regression analyses were used to explore the association between fundamental ability measures and operative performance. RESULTS: A total of 242 residents completed baseline psychomotor ability assessments. Aggregated fundamental ability scores were associated with performance in submitted workplace-based SSAOP assessments using the Total Checklist score ( P =0.002) and Overall Performance scores ( P =0.002), independent of operative experience, and undergraduate centile scores. Aggregated ability scores were also positively associated with simulation-based OSS assessment scores on multivariable analysis ( P =0.03). CONCLUSION: This study indicates that visuospatial, psychomotor, and perceptual ability testing scores are associated with the future operative performance of surgical residents.

How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?

OBJECTIVE: The aim of this study is to assess the quality of feedback provided to surgical trainees in the operating theatre, and to further investigate how trainees and trainers use workplace-based assessment in practice with regards to frequency and timing of assessments. DESIGN: A retrospective study of all submitted Supervised Structured Assessments of Operative Performance (SSAOPs) from April 25, 2016 to February 2, 2021 was conducted. SETTING: Surgical trainees in the Republic of Ireland across all national surgical training sites submitted SSAOPs through an online platform. PARTICIPANTS: Assessments of operative competence (SSAOPs) from all Core Surgical Trainees (in their first two years of dedicated post-graduate surgical training) were included for analysis, regardless of surgical subspecialty. A total of 2294 assessments were submitted from April 25, 2016 to February 2, 2021 by 330 core surgical trainees and 379 surgeon assessors. Five hundred of these assessments were randomly selected and scored for quality of feedback using a modified "Task, Gap, Action (TGA)" framework. RESULTS: Of all 2294 submitted assessments, 1905 (83.04%) were submitted in the latter 3 months of each rotation, and 803 (35%) were submitted in the last month. Only 51 of 270 (18.89%) of trainees in their first year and 33 of 236 trainees in their second year (13.98%) submitted more than the minimum required number of assessments (6 per year). Of 500 randomly selected assessments, 362 (72.4%) had documented written feedback. The mean 'Gap' and 'Action' scores were low, at 0.44/3 and 0.53/3 respectively. CONCLUSIONS: Trainees do not submit more than the required number of operative workplace-based assessments. Assessments are submitted at the end of the trainee's rotation, limiting their formative value. The quality of written feedback is poor and could be improved significantly by encouraging a "Task," "Gap" and "Action" approach.

The correlation between fundamental characteristics and first-time performance in laparoscopic tasks.

BACKGROUND: The ability of characteristics to predict first time performance in laparoscopic tasks is not well described. Videogame experience predicts positive performance in laparoscopic experiences but its mechanism and confounding-association with aptitude remains to be elucidated. This study sought to evaluate for innate predictors of laparoscopic performance in surgically naive individuals with minimal videogame exposure. METHODS: Participants with no prior laparoscopic exposure and minimal videogaming experience were recruited consecutively from preclinical years at a medical university. Participants completed four visuospatial, one psychomotor aptitude test and an electronic survey, followed by four laparoscopic tasks on a validated Virtual Reality simulator (LAP Mentor™). RESULTS: Twenty eligible individuals participated with a mean age of 20.8 (±3.8) years. Significant intra-aptitude performance correlations were present amongst 75% of the visuospatial tests. These visuospatial aptitudes correlated significantly with multiple laparoscopic task metrics: number of movements of a dominant instrument (rs ≥ -0.46), accuracy rate of clip placement (rs ≥ 0.50) and time taken (rs ≥ -0.47) (p < 0.05). Musical Instrument experience predicted higher average speed of instruments (rs ≥ 0.47) (p < 0.05). Participant's revised competitive index level predicted lower proficiency in laparoscopic metrics including: pathlength, economy and number of movements of dominant instrument (rs ≥ 0.46) (p < 0.05). CONCLUSION: Multiple visuospatial aptitudes and innate competitive level influenced baseline laparoscopic performances across several tasks in surgically naïve individuals.

Development and evaluation of a trauma decision-making simulator in Oculus virtual reality.

BACKGROUND: Consumer-available virtual-reality technology was launched in 2016 with strong foundations in the entertainment-industry. We developed an innovative medical-training simulator on the Oculus™ Gear-VR platform. This novel application was developed utilising internationally recognised Advanced Trauma Life Support (ATLS) principles, requiring decision-making skills for critically-injured virtual-patients. METHODS: Participants were recruited in June, 2016 at a single-centre trauma-course (ATLS, Leinster, Ireland) and trialled the platform. Simulator performances were correlated with individual expertise and course-performance measures. A post-intervention questionnaire relating to validity-aspects was completed. RESULTS: Eighteen(81.8%) eligible-candidates and eleven(84.6%) course-instructors voluntarily participated. The survey-responders mean-age was 38.9(±11.0) years with 80.8% male predominance. The instructor-group caused significantly less fatal-errors (p < 0.050) and proportions of incorrect-decisions (p < 0.050). The VR-hardware and trauma-application's mean ratings were 5.09 and 5.04 out of 7 respectively. Participants reported it was an enjoyable method of learning (median-6.0), the learning platform of choice (median-5.0) and a cost-effective training tool (median-5.0). CONCLUSION: Our research has demonstrated evidence of validity-criteria for a concept application on virtual-reality headsets. We believe that virtual-reality technology is a viable platform for medical-simulation into the future.

Objective scoring of an electronic surgical logbook: Analysis of impact and observations within a surgical training body.

BACKGROUND: Historically, evaluating operative-volumes has proven difficult due to mass-variability in operative-complexities and participation. This study aimed to introduce a national scoring interface for residents' operative-logs while forming meaningful observations on specialities, training-institutes and technical competency. METHODS: A weighted-scoring algorithm was applied prospectively to residents' operative volumes since July 8th, 2013 with daily web-based quantitative feedback. Pre and post intervention analyses were performed with historical volumes. Operative volumes were correlated with work-based and university technical-skills' assessments. RESULTS: Ninety-five residents completed two-year preliminary training since 2013 recording 79,490 operations. These residents recorded significant (p < 0.050) increases in mean-score (case-load), total, performed and assisted operations of >16,528 (50%), 234 (45%), 115 (66%) and 113 (33%) respectively. The number of resident-performed operations was a significant predictor of performance in work-based and university technical-skills assessments (p < 0.050). There were no associations between these measures and the volume of assisted-operations. CONCLUSIONS: Open-benchmarking of surgical-volumes stimulates residents to actively pursue operative-opportunities and record those experiences. It provides objective performance data on residents and training-institutes while providing evidence that level of operative participation is significant in technical skills development.

Basic Surgical Skill Retention: Can Patriot Motion Tracking System Provide an Objective Measurement for it?

BACKGROUND: Knot tying is a fundamental skill that surgical trainees have to learn early on in their training. The aim of this study was to establish the predictive and concurrent validity of the Patriot as an assessment tool and determine the skill retention in first-year surgical trainees after 5 months of training. METHODS: First-year surgical trainees were recruited in their first month of the training program. Experts were invited to set the proficiency level. The subjects performed hand knot tying on a bench model. The skill was assessed at baseline in the first month of training and at 5 months. The assessment tools were the Patriot electromagnetic tracking system and Objective Structured Assessment of Technical Skills (OSATS). The trainees' scores were compared to the proficiency score. The data were analyzed using paired t-test and Pearson correlation analysis. RESULTS: A total of 14 first-year trainees participated in this study. The time taken to complete the task and the path length (PL) were significantly shorter (p = 0.007 and p = 0.0085, respectively) at 5 months. OSATS scoring showed a significant improvement (p = 0.0004). There was a significant correlation between PL and OSATS at baseline (r = -0.873) and at Month 5 (r = -0.774). In all, 50% of trainees reached the proficiency PL at baseline and at Month 5. Among them, 3 trainees improved their PL to reach proficiency and the other 3 trainees failed to reach proficiency. CONCLUSION: The parameters from the Patriot motion tracker demonstrated a significant correlation with the classical observational assessment tool and were capable of highlighting the skill retention in surgical trainees. Therefore, the automated scoring system has a significant role in the surgical training curriculum as an adjunct to the available assessment tool.

The impact of a surgical boot camp on early acquisition of technical and nontechnical skills by novice surgical trainees.

BACKGROUND: Acquisition of skills early in surgical training represents a significant challenge at present because of training time constraints. The aim of this study was to investigate if an intensive surgical boot camp was effective in transferring skills at the beginning of a surgical training program. METHODS: New core surgical trainees (n = 58) took part in a 5-day boot camp. There were pretest and posttest assessments of knowledge, technical skills, and confidence levels. The boot camp used simulation and senior surgical faculty to teach a defined range of technical and nontechnical skills. RESULTS: The scores for knowledge (53.8% vs 68.4%, P < .01), technical skills (35.9% to 60.6% vs 50.6% to 78.2%, P < .01), and confidence levels improved significantly during boot camp. Skills improvements were still present a year later. CONCLUSION: The 5-day surgical boot camp proved to be an effective way to rapidly acquire surgical knowledge and skills while increasing the confidence levels of trainees.

Zone calculation as a tool for assessing performance outcome in laparoscopic suturing.

BACKGROUND: Simulator performance is measured by metrics, which are valued as an objective way of assessing trainees. Certain procedures such as laparoscopic suturing, however, may not be suitable for assessment under traditionally formulated metrics. Our aim was to assess if our new metric is a valid method of assessing laparoscopic suturing. STUDY DESIGN: A software program was developed to order to create a new metric, which would calculate the percentage of time spent operating within pre-defined areas called "zones." Twenty-five candidates (medical students N = 10, surgical residents N = 10, and laparoscopic experts N = 5) performed the laparoscopic suturing task on the ProMIS III(®) simulator. New metrics of "in-zone" and "out-zone" scores as well as traditional metrics of time, path length, and smoothness were generated. Performance was also assessed by two blinded observers using the OSATS and FLS rating scales. This novel metric was evaluated by comparing it to both traditional metrics and subjective scores. RESULTS: There was a significant difference in the average in-zone and out-zone scores between all three experience groups (p < 0.05). The new zone metrics scores correlated significantly with the subjective-blinded observer scores of OSATS and FLS (p = 0.0001). The new zone metric scores also correlated significantly with the traditional metrics of path length, time, and smoothness (p < 0.05). CONCLUSION: The new metric is a valid tool for assessing laparoscopic suturing objectively. This could be incorporated into a competency-based curriculum to monitor resident progression in the simulated setting.

The impact of aptitude on the learning curve for laparoscopic suturing.

BACKGROUND: Within surgery, several specialties demand advanced technical skills, specifically in the minimally invasive environment. METHODS: Two groups of 10 medical students were recruited on the basis of their aptitude (visual-spatial ability, depth perception, and psychomotor ability). All subjects were tested consecutively using the ProMIS III simulator until they reached proficiency performing laparoscopic suturing. Simulator metrics, critical error scores, observed structured assessment of technical skills scores, and Fundamentals of Laparoscopic Surgery scores were recorded. RESULTS: Group A (high aptitude) achieved proficiency after a mean of 7 attempts (range, 4-10). In group B (low aptitude), 30% achieved proficiency after a mean of 14 attempts (range, 10-16). In group B, 40% demonstrated improvement but did not attain proficiency, and 30% failed to progress. CONCLUSIONS: Distinct learning curves for laparoscopic suturing can be mapped on the basis of fundamental ability. High aptitude is directly related to earlier completion of the learning curve. A proportion of subjects with low aptitude are unable to reach proficiency despite repeated attempts.