Development and preliminary validation of an Objective Structured Assessment of Technical Skills (OSATS) for a partial pulpotomy procedure.

AIM: To develop an Objective Structured Assessment of Technical Skills (OSATS) tool for partial pulpotomy and to preliminary assess the validity of this tool. Secondary objectives were to check internal consistency as well as the interrater reliability of this specific checklist and to compare it with the previously validated global rating score (GRS). METHODOLOGY: The study assessed three groups of dental participants (mix of practitioners or students) possessing differing levels of clinical experience (7 novices, 7 intermediates, 7 experts), during a partial pulpotomy procedure on a simulated tooth with deep caries, using a bespoke procedure-specific checklist (PSC) and the GRS. Two independent examiners received the study participants in groups of two, in order to be able to directly observe their actions and grade the different steps of the procedure from 1 to 5 on the PSC and GRS. The internal validity of the specific checklist was assessed using a Cronbach's alpha test. As the construct validity of such a tool can be determined when the tool differentiates performance based on the level of experience, the total score of the PSC of the three groups (e.g., novice group, intermediate group, experienced group) were compared using the Kruskal-Wallis and post-hoc Mann-Whitney pairwise U test. The total scores of the GRS were also analysed in the same manner. A correlation test (Correlation Matrices Test) was carried out for the entire sample between the specific checklist and the GRS (on total score). The intraclass correlation coefficient was used to measure the degree of agreement between the two evaluators. The statistical analysis was performed with XLSTAT® and Statistica® (significance p = .05). RESULTS: The partial pulpotomy specific checklist graded from one to five demonstrated good internal consistency (0.86) and a good interrater reliability (0.91), a correct construct validity (p generally < .05) and a good positive correlation with the validated GRS (r = .92). CONCLUSIONS: To address the need to develop education in vital pulp treatment, and more specifically the technical skills assessment of partial pulpotomy, an OSATS-specific checklist was developed and preliminary validated. The checklist will need to be validated in larger cohorts.

Objective structured assessment of medical students' technical skills in second-degree perineal laceration repair with sponge model-based training.

PURPOSE: In this cohort study, we used a sponge simulator to train students in second-degree perineal laceration repair. We examined whether the training course improved the students' skills, as measured with an objective structured assessment of technical skills (OSATS) and by a senior physician. We also examined the correlation between these ratings to assess the validity of OSATS application in this context. METHODS: Between April and July 2022, 40 medical students took part in gynecological/obstetrics training that included a lecture about perineal trauma and the viewing of a video that demonstrated second-degree perineal laceration repair using a sponge model. They then underwent initial evaluation by a senior physician and OSATS application, yielding two independent scores. After training with the sponge model, a second evaluation was performed. The OSATS assessed practical skills (8 items) and suture results (2 items). The senior physician assigned ratings on a five-point ordinal scale ranging from 1 (excellent) to 5 (poor). RESULTS: Training with the sponge simulator significantly increased students' OSATS (practical skills, p < 0.001; suture results, p < 0.05) and senior physician (p < 0.001) ratings. The OSATS and senior physician ratings correlated strongly (Spearman's r: first assessment, - 0.72; second assessment, - 0.74; p < 0.01). CONCLUSION: The sponge-based training improves students' skills for the repair of a second-degree perineal laceration. The OSATS for the sponge model might be a valid option to examine medical students in an obstetrical course.

Assessment of suturing and scaling skills of periodontology and oral medicine residents by OSATS method: a pilot study.

INTRODUCTION: Updating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method. METHODS: All six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents' satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used. RESULTS: There was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96-0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86-0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores. CONCLUSION: Considering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

Methods of assessment of the level of surgeons for minimally invasive fixation of distal radius fractures: "expertise" versus "performance".

INTRODUCTION: The results of surgical techniques vary depending on the level of the surgeon's experience. The level of expertise can be evaluated quantitatively or qualitatively. In this study, we evaluated the duration of the procedure and the size of the incision of minimally invasive fixation of distal radius fractures as a means of grading expertise and surgical performance. The null hypothesis of our study was that the level of expertise did not match the level of performance. MATERIAL AND METHODS: The study included 52 surgeons split in five levels of expertise who had performed 467 minimally invasive fixations of distal radius fractures. The performance of the surgeons in terms of duration of the procedure and size of the incision were scored according to the five levels of surgical experience using four different methodologies: clinical (setting limits compared to data found in literature), statistical (setting statistical limits of comparison in percentiles (20th, 40th, 60th, 80th) for each of the two parameters measured), arithmetical (setting limits compared to equal intervals for each of the 2 variables), and success rate (setting a threshold for each of the 2 variables). RESULTS: Our results showed a great disparity between levels of experience depending on the method used. The scores for levels 1 and 2 were 72% for expertise, 13% for performance according to the clinical method, 75% for the statistical method, 0% for the arithmetical method, and 57% according the success rate. The rate of level 3 was 23% for expertise, 41% for performance by the clinical method, 17% by the statistical method, 17% by the arithmetical method, and 15% by the success rate. The rates of levels 4 and 5 were 5% for expertise, 46% for performance clinical method, 8% by the statistical method, 83% by the arithmetical method, and 28% by the success rate. DISCUSSION: The null hypothesis of our study was confirmed by the results which show that there was no correlation between levels of expertise and performance. The clinical method of assessment appears to reflect best the true level of performance of the surgeon. CONCLUSION: Publications reporting the results of a surgical technique for a given pathology should always mention the level of performance as measured on a clinical scale.

An Innovative Technique of Microsurgical Training on Fresh "Chicken Quarter" Model: Our Experience.

Purpose Regular practice, quality clinical exposure, and academic discussion are essential in any surgical specialty training. This study discusses and validates the option of using a fresh "chicken quarter" model with a measurable scoring system, as a standard training regimen in microvascular surgery. This can be a very effective, economical, and easily accessible model for residents. Materials and Methods This study was conducted in the Department of Plastic surgery, from October 2020 to May 2021. Twenty-four fresh "chicken quarter" specimens were dissected and the ischial arteries and femoral veins' external diameter (ED) were measured. The microsurgical skills of the trainee were assessed in 6 months intervals using the Objective Structured Assessment of Technical Skills Scale (OSATS) as well as the time taken for anastomosis. All the data were analyzed using SPSS (statistical package for social sciences) version 21. Results A task-specific score value of 50% on October 2020 improved to 85.7% by May 2021. This was found to be statistically significant ( p = 0.043). The mean ED of the ischial artery and femoral vein was 2.07 and 2.26 mm, respectively. The mean width of the vein measured at the lower one-third of the tibia was 2.08 mm. A greater than 50% reduction in anastomosis time was observed after a period of 6 months. Conclusion In our minimal experience, the "chicken quarter model" with OSATS scoring system seems to be effective, economical, very affordable, and easily accessible microsurgery training model for the residents. Our study is done only as a pilot project due to limited resources and we have the plan to introduce it as a proper training method in the near future with more residents.

Surgical skill and oncological outcome of laparoscopic radical hysterectomy: JGOG1081s-A1, an ancillary analysis of the Japanese Gynecologic Oncology Group Study JGOG1081.

OBJECTIVES: We investigated whether surgical skill and procedure were related to oncological outcomes in cervical cancer patients who underwent Laparoscopic Radical Hysterectomy (LRH). METHODS: We previously assessed data of LRH from 251 patients with FIGO stage (2009) IA2, IB1and IIA1 cervical cancer collected for JGOG 1081s study. 1) The JGOG 1081s cohort study was re-examined to refine the surgical details and extend the follow-up period as chart review. 2) Unedited videos for recurrent cases and matched non-recurrent control cases were newly compared by experts for various surgical skills and surgical procedures using the modified Objective Structured Assessment of Technical Skills (OSATS) tool, without awareness of the recurrence status as video review. RESULTS: After a median follow-up of 46 months, tumors had recurred in 31 of the 251 patients. The five-year Recurrence-Free Survival rate was 86.9% (81.8-90.6) and five-year Overall Survival rate was 93.7% (87.5-96.8). Multivariate analysis from chart reviews found that an experience with LRH of less than 20 cases per institution was an independent prognostic factor for recurrence (Hazard Ratio (HR) 2.49, 95%CI 1.12-5.53, p = 0.025). For the surgical video review, we compared 23 videos of recurrent cases with 23 background-matched non-recurrent controls. Lower modified OSATS scores from the video review were consistently trended to have a higher risk of recurrence. CONCLUSIONS: Our new study has found that LRH surgical experience and skill trended to have better oncological outcomes.

Performance with robotic surgery versus 3D- and 2D-laparoscopy during pancreatic and biliary anastomoses in a biotissue model: pooled analysis of two randomized trials.

BACKGROUND: Robotic surgery may improve surgical performance during minimally invasive pancreatoduodenectomy as compared to 3D- and 2D-laparoscopy but comparative studies are lacking. This study assessed the impact of robotic surgery versus 3D- and 2D-laparoscopy on surgical performance and operative time using a standardized biotissue model for pancreatico- and hepatico-jejunostomy using pooled data from two randomized controlled crossover trials (RCTs). METHODS: Pooled analysis of data from two RCTs with 60 participants (36 surgeons, 24 residents) from 11 countries (December 2017-July 2019) was conducted. Each included participant completed two pancreatico- and two hepatico-jejunostomies in biotissue using 3D-robotic surgery, 3D-laparoscopy, or 2D-laparoscopy. Primary outcomes were the objective structured assessment of technical skills (OSATS: 12-60) rating, scored by observers blinded for 3D/2D and the operative time required to complete both anastomoses. Sensitivity analysis excluded participants with excess experience compared to others. RESULTS: A total of 220 anastomoses were completed (robotic 80, 3D-laparoscopy 70, 2D-laparoscopy 70). Participants in the robotic group had less surgical experience [median 1 (0-2) versus 6 years (4-12), p < 0.001], as compared to the laparoscopic group. Robotic surgery resulted in higher OSATS ratings (50, 43, 39 points, p = .021 and p < .001) and shorter operative time (56.5, 65.0, 81.5 min, p = .055 and p < .001), as compared to 3D- and 2D-laparoscopy, respectively, which remained in the sensitivity analysis. CONCLUSION: In a pooled analysis of two RCTs in a biotissue model, robotic surgery resulted in better surgical performance scores and shorter operative time for biotissue pancreatic and biliary anastomoses, as compared to 3D- and 2D-laparoscopy.

Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score.

INTRODUCTION: The aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS). MATERIALS AND METHODS: The Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8). RESULTS: The Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale. CONCLUSION: With the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Comparison of a practice-based versus theory-based training program for conducting vacuum-assisted deliveries: a randomized-controlled trial.

PURPOSE: Vacuum-assisted deliveries (VAD) are complex procedures that require training and experience to be performed proficiently. We aimed to evaluate if a more resource intensive practice-based training program for conducting VAD is more efficient compared to a purely theory-based training program, with respect to immediate training effects and persistence of skills 4-8 weeks after the initial training. METHODS: In this randomized-controlled study conducted in maternity staff, participants performed a simulated low-cavity non-rotational vacuum delivery before (baseline test) and immediately after the training (first post-training test) as well as 4-8 weeks thereafter (second post-training test). The study's primary endpoint was to compare training effectiveness between the two study groups using a validated objective structured assessment of technical skills (OSATS) rating scale. RESULTS: Sixty-two participants were randomized to either the theory-based group (n = 31) or the practice-based group (n = 31). Total global and specific OSATS scores, as well as distance of cup application to the flexion point improved significantly from baseline test to the first post-training test in both groups (pall < 0.007). Skill deterioration after 4-8 weeks was only found in the theory-based group, whereas skills remained stable in the practice-based group. CONCLUSION: A practice-based training program for conducting VAD results in comparable immediate improvement of skills compared to a theory-based training program, but the retention of skills 4-8 weeks after training is superior in a practice-based program. Future studies need to evaluate, whether VAD simulation training improves maternal and neonatal outcome after VAD.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice.

BACKGROUND: Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods. METHODS: The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300). RESULTS: The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6). CONCLUSION: Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

Added value of 3D-vision during robotic pancreatoduodenectomy anastomoses in biotissue (LAEBOT 3D2D): a randomized controlled cross-over trial.

BACKGROUND: We tested the added value of 3D-vision on procedure time and surgical performance during robotic pancreatoduodenectomy anastomoses in biotissue. Robotic surgery has the advantage of articulating instruments and 3D-vision. Consensus is lacking on the added value of 3D-vision during laparoscopic surgery. Given the improved dexterity with robotic surgery, the added value of 3D-vision may be even less with robotic surgery. METHODS: In this experimental randomized controlled cross-over trial, 20 surgeons and surgical residents from 5 countries performed robotic pancreaticojejunostomy and hepaticojejunostomy anastomoses in a biotissue organ model using the da Vinci® system and were randomized to start with either 3D- or 2D-vision. Primary endpoint was the time required to complete both anastomoses. Secondary endpoint was the objective structured assessment of technical skill (OSATS; range 12-60) rating; scored by two observers blinded to 3D/2D. RESULTS: Robotic 3D-vision reduced the combined operative time from 78.1 to 57.3 min (24.6% reduction, p < 0.001; 20.8 min reduction, 95% confidence intervals 12.8-28.8 min). This reduction was consistent for both anastomoses and between surgeons and residents, p < 0.001. Robotic 3D-vision improved OSATS performance by 6.1 points (20.8% improvement, p = 0.003) compared to 2D (39.4 to 45.1 points, ± 5.5). CONCLUSION: 3D-vision has a considerable added value during robotic pancreatoduodenectomy anastomoses in biotissue in both time reduction and improved surgical performance as compared to 2D-vision.

Simulator training and residents' first laparoscopic hysterectomy: a randomized controlled trial.

BACKGROUND: Hysterectomy rates are decreasing in many countries, and virtual reality simulators bring new opportunities into residents' surgical education. The objective of this study was to evaluate the effect of training in laparoscopic hysterectomy module with virtual reality simulator on surgical outcomes among residents performing their first laparoscopic hysterectomy. METHODS: This randomized study was carried out at the Department of Obstetrics and Gynecology in Helsinki University Hospital and Hyvinkää Hospital. We recruited twenty residents and randomly signed half of them to train ten times with the laparoscopic hysterectomy module on a virtual reality simulator, while the rest represented the control group. Their first laparoscopic hysterectomy was video recorded and assessed later by using the Objective Structured Assessment of Technical Skills (OSATS) forms and Visual Analog Scale (VAS). The scores and surgical outcomes were compared between the groups. RESULTS: The mean OSATS score for the Global Rating Scale (GRS) was 17.0 (SD 3.1) in the intervention group and 11.2 (SD 2.4) in the control group (p = 0.002). The mean procedure-specific OSATS score was 20.0 (SD 3.3) and 16.0 (SD 2.8) (p = 0.012), and the mean VAS score was 55.0 (SD 14.8) and 29.9 (SD 14.9) (p = 0.001). Operative time was 144 min in the intervention group and 165 min in the control group, but the difference did not reach statistical significance (p = 0.205). There were no differences between the groups in blood loss or direct complications. CONCLUSION: Residents training with a virtual reality simulator prior to the first laparoscopic hysterectomy seem to perform better in the actual live operation. Thus, a virtual reality simulator hysterectomy module could be considered as a part of laparoscopic training curriculum.

Video-based coaching in surgical education: a systematic review and meta-analysis.

BACKGROUND: In the era of competency-based surgical education, VBC has gained increased attention and may enhance the efficacy of surgical education. The objective of this systematic review was to summarize the existing evidence of video-based coaching (VBC) and compare VBC to traditional master-apprentice-based surgical education. METHODS: We performed a systematic review and meta-analysis of randomized controlled trials (RCT) assessing VBC according to the PRISMA and Cochrane guidelines. The MEDLINE, EMBASE, and COCHRANE and Researchgate databases were searched for eligible manuscripts. Standard mean difference (SMD) of performance scoring scales was used to assess the effect of VBC versus traditional training without VBC (control). RESULTS: Of 627 studies identified, 24 RCTs were eligible and evaluated. The studies included 778 surgical trainees (n = 386 VBC vs. n = 392 control). 13 performance scoring scales were used to assess technical competence; OSATS-GRS was the most common (n = 15). VBC was provided preoperative (n = 11), intraoperative (n = 1), postoperative (n = 10), and perioperative (n = 2). The majority of studies were unstructured, where identified coaching frameworks were PRACTICE (n = 1), GROW (n = 2) and Wisconsin Coaching Framework (n = 1). There was an effect on performance scoring scales in favor of VBC coaching (SMD 0.87, p < 0.001). In subgroup analyses, the residents had a larger relative effect (SMD 1.13; 0.61-1.65, p < 0.001) of VBC compared to medical students (SMD 0.43, 0.06-0.81, p < 0.001). The greatest source of potential bias was absence of blinding of the participants and personnel (n = 20). CONCLUSION: Video-based coaching increases technical performance of medical students and surgical residents. There exist significant study and intervention heterogeneity that warrants further exploration, showing the need to structure and standardize video-based coaching tools.

Simulation as tool for evaluating and improving technical skills in laparoscopic gynecological surgery.

BACKGROUND: Simulation in laparoscopic surgery is nowadays recognized as a valid instrument for learning and training surgeons in different procedures. However, its role as evaluation test and self-assessment tool to verify basic surgical skills is still under discussion. METHODS: Thirty-three residents in obstetrics and gynecology at University of Pisa, Italy were recruited, and they received a simulation program consisting of 5 tasks. They had to perform basic laparoscopic surgery maneuvers as creating pneumoperitoneum, positioning trocars under vision, demonstrating the appropriate use of dominant and non-dominant hand and making single stitch and knot. They were evaluated with a modified OSATs scale. RESULTS: Senior trainees had better score than junior trainees (p value< 0,005) and after different sessions of simulation scores of both groups significantly improved (p < 0,001), especially for the junior group. All the trainees reported self-assessments that matched with the evaluation of external observers demonstrating the importance of simulation also as auto-evaluation test. CONCLUSIONS: In this study, we demonstrated the role of simulation as powerful tool to evaluate and to self-assess surgical technical skills and to improve own capacities, with the use of a modified OSATs scale adapted to specific exercises.

Development and validation of an objective assessment scale for chest tube insertion under 'direct' and 'indirect' rating.

BACKGROUND: There is an increasing need for objective and validated educational concepts. This holds especially true for surgical procedures like chest tube insertion (CTI). Thus, we developed an instrument for objectification of learning successes: the assessment scale based on Objective Structured Assessment of Technical Skill (OSATS) for chest tube insertion, which is evaluated in this study. Primary endpoint was the evaluation of intermethod reliability (IM). Secondary endpoints are 'indirect' interrater reliability (IR) and construct validity of the scale (CV). METHODS: Every participant (N = 59) performed a CTI on a porcine thorax. Participants received three ratings (one 'direct' on site, two 'indirect' via video rating). IM compares 'direct' with 'indirect' ratings. IR was assessed between 'indirect' ratings. CV was investigated by subgroup analysis based on prior experience in CTI for 'direct' and 'indirect' rating. RESULTS: We included 59 medical students to our study. IM showed moderate conformity ('direct' vs. 'indirect 1' ICC = 0.735, 95% CI: 0.554-0.843; 'direct' vs. 'indirect 2' ICC = 0.722, 95% CI 0.533-0.835) and good conformity between 'direct' vs. 'average indirect' rating (ICC = 0.764, 95% CI: 0.6-0.86). IR showed good conformity (ICC = 0.84, 95% CI: 0.707-0.91). CV was proven between subgroups in 'direct' (p = 0.037) and 'indirect' rating (p = 0.013). CONCLUSION: Results for IM suggest equivalence for 'direct' and 'indirect' ratings, while both IR and CV was demonstrated in both rating methods. Thus, the assessment scale seems a reliable method for rating trainees' performances 'directly' as well as 'indirectly'. It may help to objectify and facilitate the assessment of training of chest tube insertion.

Training or non-surgical factors-what determines a good surgical performance? A randomised controlled trial.

BACKGROUND: Acquiring laparoscopic skills is a necessity for every young surgeon. Whether it is a talent or a non-surgical skill that determines the surgical performance of an endoscopic operation has been discussed for years. In other disciplines aptitude testing has become the norm. Airlines, for example, have implemented assessments to test the natural aptitude of future pilots to predict their performance later on. In the medical field, especially surgery, there are no similar comparable tests implemented or even available. This study investigates the influence of potential factors that may predict the successful performance of a complex laparoscopic operation, such as the surgeon's age, gender or learning method. METHODS: This study focussed 70 surgical trainees. It was designed as a secondary analysis of data derived from a 2 × 2 factorial randomised controlled trial of practical training and/or multimedia training (four groups) in an experimental exercise. Both before and then after the training sessions, the participating trainees performed a laparoscopic cholecystectomy in a pelvitrainer. Surgical performance was then evaluated using a modified objective structured assessment of technical skills (OSATS). Participants were classified as 'Skilled' (high score in the pre-test), 'Good Learner' (increase from pre- to post-test) or 'Others' based on the OSATS results. Based on the results of the recorded performance, the training methods as well as non-surgical skills were eventually evaluated in a univariate and in a multivariate analysis. RESULTS: In the pre-training performance 11 candidates were categorised as 'Skilled' (15.7%), 35 participants as 'Good Learners' (50.0%) and 24 participants were classified as 'Others'. The univariate analysis showed that the age, a residency in visceral surgery, and participation in a multimedia training were significantly associated with this grouping. Multivariate analyses revealed that residency in visceral surgery was the most predictive factor for the 'Skilled' participants (p = 0.059), and multimedia training was most predictive for the 'Good Learner' (p = 0.006). Participants in the group of 'Others' who were neither 'Skilled' nor improved in the training phase were younger (p = 0.011) and did not receive multimedia (p < 0.001) or practical (p = 0.025) training. CONCLUSION: The type of learning method has been shown to be the most effective factor to improve laparoscopic skills, with multimedia training proving to be more effective than practical training.

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery.

BACKGROUND: There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs). METHODS: An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion. RESULTS: Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692). DISCUSSION/CONCLUSION: A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

Development and validation of an objective structured assessment of technical skill tool for the practice of breech presentation delivery.

OBJECTIVE: To develop and test the validity of an Objective Structured Assessment of Technical Skills (OSATS) tool for breech presentation delivery. MATERIALS AND METHODS: Monocentric prospective study conducted in the Department of Gynecology, Obstetrics, Fetal Medicine and Reproductive Medicine at the University Hospital of Nice. The study consisted of two parts, the development of the OSATS scoring system and its objective validation. Several experts in obstetrics from university hospital centers and private French hospitals were invited to participate in the development phase of the scoring system. For the validation phase, we formed a group of 20 novices and a group of 20 experts, who had to perform a breech presentation delivery on a simulator, according to a standardized scenario. Each participant was filmed and two experts would then evaluate their performance by viewing anonymized videos and using the OSATS score. RESULTS: The scores obtained by the expert group were significantly higher than those of the novice group, with a total score of 21.73/25 versus 6.95/25 (p < 0.0001), a task-specific score of 87.2/110 versus 44.3/110 (p < 0.0001) and an overall score of 108.93/135 versus 51.25/135 (p < 0.0001), respectively. CONCLUSION: The OSATS score developed in this study for breech presentation delivery is a reliable model to assess the competence level in procedural skills using a simulator.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.