Surgical data recording in the operating room: a systematic review of modalities and metrics.

INTRODUCTION: Operating room recording, via video, audio and sensor-based recordings, is increasingly common. Yet, surgical data science is a new field without clear guidelines. The purpose of this study is to examine existing published studies of surgical recording modalities to determine which are available for use in the operating room, as a first step towards developing unified standards for this field. METHODS: Medline, EMBASE, CENTRAL and PubMed databases were systematically searched for articles describing modalities of data collection in the operating room. Search terms included 'video-audio media', 'bio-sensing techniques', 'sound', 'movement', 'operating rooms' and others. Title, abstract and full-text screening were completed to identify relevant articles. Descriptive statistical analysis was performed for included studies. RESULTS: From 3756 citations, 91 studies met inclusion criteria. These studies described 10 unique data-collection modalities for 17 different purposes in the operating room. Data modalities included video, audio, kinematic and eye-tracking among others. Data-collection purposes described included surgical trainee assessment, surgical error, surgical team communication and operating room efficiency. CONCLUSION: Effective data collection and utilization in the operating room are imperative for the provision of superior surgical care. The future operating room landscape undoubtedly includes multiple modalities of data collection for a plethora of purposes. This review acts as a foundation for employing operating room data in a way that leads to meaningful benefit for patient care.

Implementing structured team debriefing using a Black Box in the operating room: surveying team satisfaction.

BACKGROUND: Surgical safety may be improved using a medical data recorder (MDR) for the purpose of postoperative team debriefing. It provides the team in the operating room (OR) with the opportunity to look back upon their joint performance objectively to discuss and learn from suboptimal situations or possible adverse events. The aim of this study was to investigate the satisfaction of the OR team using an MDR, the OR Black Box®, in the OR as a tool providing output for structured team debriefing. METHODS: In this longitudinal survey study, 35 gastro-intestinal laparoscopic operations were recorded using the OR Black Box® and the output was subsequently debriefed with the operating team. Prior to study, a privacy impact assessment was conducted to ensure alignment with applicable legal and regulatory requirements. A structured debrief model and an OR Back Box® performance report was developed. A standardized survey was used to measure participant's satisfaction with the team debriefing, the debrief model used and the performance report. Factor analysis was performed to assess the questionnaire's quality and identified contributing satisfaction factors. Multivariable analysis was performed to identify variables associated with participants' opinions. RESULTS: In total, 81 team members of various disciplines in the OR participated, comprising 35 laparoscopic procedures. Mean satisfaction with the OR Black Box® performance report and team debriefing was high for all 3 identified independent satisfaction factors. Of all participants, 98% recommend using the OR Black Box® and the outcome report in team debriefing. CONCLUSION: The use of an MDR in the OR for the purpose of team debriefing is considered to be both beneficial and important. Team debriefing using the OR Black Box® outcome report is highly recommended by 98% of team members participating.

Relationship between intraoperative non-technical performance and technical events in bariatric surgery.

BACKGROUND: The operating theatre is a unique environment with complex team interactions, where technical and non-technical performance affect patient outcomes. The correlation between technical and non-technical performance, however, remains underinvestigated. The purpose of this study was to explore these interactions in the operating theatre. METHODS: A prospective single-centre observational study was conducted at a tertiary academic medical centre. One surgeon and three fellows participated as main operators. All patients who underwent a laparoscopic Roux-en-Y gastric bypass and had the procedures captured using the Operating Room Black Box® platform were included. Technical assessment was performed using the Objective Structured Assessment of Technical Skills and Generic Error Rating Tool instruments. For non-technical assessment, the Non-Technical Skills for Surgeons (NOTSS) and Scrub Practitioners' List of Intraoperative Non-Technical Skills (SPLINTS) tools were used. Spearman rank-order correlation and N-gram statistics were conducted. RESULTS: Fifty-six patients were included in the study and 90 procedural steps (gastrojejunostomy and jejunojejunostomy) were analysed. There was a moderate to strong correlation between technical adverse events (rs = 0·417-0·687), rectifications (rs = 0·380-0·768) and non-technical performance of the surgical and nursing teams (NOTSS and SPLINTS). N-gram statistics showed that after technical errors, events and prior rectifications, the staff surgeon and the scrub nurse exhibited the most positive non-technical behaviours, irrespective of operator (staff surgeon or fellow). CONCLUSION: This study demonstrated that technical and non-technical performances are related, on both an individual and a team level. Valuable data can be obtained around intraoperative errors, events and rectifications.

Systematic review to establish absolute standards for technical performance in surgery.

BACKGROUND: Standard setting allows educators to create benchmarks that distinguish between those who pass and those who fail an assessment. It can also be used to create standards in clinical and simulated procedural skill. The objective of this review was to perform a systematic review of the literature using absolute standard-setting methodology to create benchmarks in technical performance. METHODS: A systematic review was conducted by searching MEDLINE, Embase, PsycINFO and the Cochrane Database of Systematic Reviews. Abstracts of retrieved studies were reviewed and those meeting the inclusion criteria were selected for full-text review. The quality of evidence presented in the included studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI), where a score of 14 or more of 18 indicates high-quality evidence. RESULTS: Of 1809 studies identified, 37 used standard-setting methodology for assessment of procedural skill. Of these, 24 used participant-centred and 13 employed item-centred methods. Thirty studies took place in a simulated environment, and seven in a clinical setting. The included studies assessed residents (26 of 37), fellows (6 of 37) and staff physicians (17 of 37). Seventeen articles achieved a MERSQI score of 14 or more of 18, whereas 20 did not meet this mark. CONCLUSION: Absolute standard-setting methodologies can be used to establish cut-offs for procedural skill assessments.

Setting pass scores for assessment of technical performance by surgical trainees.

BACKGROUND: One of the major challenges of competency-based training is defining a score representing a competent performance. The objective of this study was to set pass scores for the Objective Structured Assessment of Technical Skill. METHODS: Pass scores for the examination were set using three standard setting methods applied to data collected prospectively from first-year surgical residents (trainees). General surgery residents were then assigned an overall pass-fail status for each method. Using a compensatory model, residents passed the eight station examinations if they met the overall pass score; using a conjunctive model, residents passed if they met the overall pass score and passed at least 50 per cent of the stations. The consistency of the pass-fail decision across the three methods, and between a compensatory and conjunctive model, were compared. RESULTS: Pass scores were stable across all three methods using data from 513 residents, 133 of whom were general surgeons. Consistency of the pass-fail decision across the three methods was 95.5 and 93.2 per cent using compensatory and conjunctive models respectively. Consistency of the pass-fail status between compensatory and conjunctive models for all three methods was also very high (91.7, 95.5 and 96.2 per cent). CONCLUSION: Consistency in pass-fail status between the various methods builds evidence of validity for the set scores. These methods can be applied and studied across a variety of assessment platforms, helping to increase the use of standard setting for competency-based training.

Consensus-based training and assessment model for general surgery.

BACKGROUND: Surgical education is becoming competency-based with the implementation of in-training milestones. Training guidelines should reflect these changes and determine the specific procedures for such milestone assessments. This study aimed to develop a consensus view regarding operative procedures and tasks considered appropriate for junior and senior trainees, and the procedures that can be used as technical milestone assessments for trainee progression in general surgery. METHODS: A Delphi process was followed where questionnaires were distributed to all 17 Canadian general surgery programme directors. Items were ranked on a 5-point Likert scale, with consensus defined as Cronbach's α of at least 0·70. Items rated 4 or above on the 5-point Likert scale by 80 per cent of the programme directors were included in the models. RESULTS: Two Delphi rounds were completed, with 14 programme directors taking part in round one and 11 in round two. The overall consensus was high (Cronbach's α = 0·98). The training model included 101 unique procedures and tasks, 24 specific to junior trainees, 68 specific to senior trainees, and nine appropriate to all. The assessment model included four procedures. CONCLUSION: A system of operative procedures and tasks for junior- and senior-level trainees has been developed along with an assessment model for trainee progression. These can be used as milestones in competency-based assessments.

Randomized clinical trial to evaluate mental practice in enhancing advanced laparoscopic surgical performance.

BACKGROUND: Mental practice, the cognitive rehearsal of a task without physical movement, is known to enhance performance in sports and music. Investigation of this technique in surgery has been limited to basic operations. The purpose of this study was to develop mental practice scripts, and to assess their effect on advanced laparoscopic skills and surgeon stress levels in a crisis scenario. METHODS: Twenty senior surgical trainees were randomized to either conventional training or mental practice groups, the latter being trained by an expert performance psychologist. Participants' skills were assessed while performing a porcine laparoscopic jejunojejunostomy as part of a crisis scenario in a simulated operating room, using the Objective Structured Assessment of Technical Skill (OSATS) and bariatric OSATS (BOSATS) instruments. Objective and subjective stress parameters were measured, as well as non-technical skills using the Non-Technical Skills for Surgeons rating tool. RESULTS: An improvement in OSATS (P = 0.003) and BOSATS (P = 0.003) scores was seen in the mental practice group compared with the conventional training group. Seven of ten trainees improved their technical performance during the crisis scenario, whereas four of the ten conventionally trained participants deteriorated. Mental imagery ability improved significantly following mental practice training (P = 0.011), but not in the conventional group (P = 0.083). No differences in objective or subjective stress levels or non-technical skills were evident. CONCLUSION: Mental practice improves technical performance for advanced laparoscopic tasks in the simulated operating room, and allows trainees to maintain or improve their performance despite added stress.

Contemporary management of paraesophaegeal hernias: establishing a European expert consensus.

BACKGROUND: The surgical treatment of paraesophageal hernias remains a challenge due to the lack of consensus regarding principles of operative treatment. The objectives of this study were to achieve consensus on key topics through expert opinion using a Delphi methodology. METHODS: A Delphi survey combined with a face-to-face meeting was conducted. A panel of European experts in foregut surgery from high-volume centres generated items in the first survey round. In subsequent rounds, the panel rated agreement with statements on a 5-point Likert-type scale. Internal consistency (consensus) was predefined as Cronbach's α > .80. Items that >70 % of the panel either rated as irrelevant/unimportant, or relevant/important were selected as consensus items, while topics that did not reach this cut-off were termed "undecided/controversial". RESULTS: Three survey rounds were completed: 19 experts from 10 countries completed round one, 18 continued through rounds two and three. Internal consistency was high in rounds two and three (α > .90). Fifty-eight additional/revised items derived from comments and free-text entries were included in round three. In total, 118 items were rated; consensus agreement was achieved for 70 of these. Examples of consensus topics are the relevance of the disease profile for assessing surgical urgency and complexity, the role of clinical history as the mainstay of patient follow-up, indications for revision surgery, and training and credentialing recommendations. Topics with the most "undecided/controversial" items were follow-up, postoperative care and surgical technique. CONCLUSIONS: This Delphi study achieved expert consensus on key topics in the operative management of paraesophageal hernias, providing an overview of the current opinion among European foregut surgeons. Moreover, areas with substantial variability in opinions were identified reflecting the current lack of empirical evidence and opportunities for future research.

Error rating tool to identify and analyse technical errors and events in laparoscopic surgery.

BACKGROUND: Surgical error analysis is essential for investigating mechanisms of errors, events and adverse outcomes. Furthermore, it provides valuable information for formative feedback and quality control. The aim of the present study was to design and validate a technical error rating tool in laparoscopic surgery. METHODS: The framework consisted of nine task groups and four error modes. Unedited videos of laparoscopic Roux-en-Y gastric bypass procedures were rated and analysed. The Objective Structured Assessment of Technical Skill (OSATS) global rating scale was used to assess technical skills. The incidence of errors and of injuries (events) were the main outcome measures, and were used to calculate the reliability, and construct and concurrent validity of the instrument. RESULTS: Two observers analysed 25 procedures. Inter-rater reliability was high regarding total number of errors (intraclass correlation coefficient (ICC) 0·90) and events (ICC 0·85). The median (interquartile range) error rate was 35 (26-44) and the event rate 3 (2-3) per procedure. Error frequencies and OSATS scores correlated significantly in all operative steps (rs = -0·75 to -0·40, P = <0·001-0·046). Surgeons demonstrating high OSATS scores had lower median (i.q.r.) error rates than surgeons with low scores in three of four steps: measuring bowel (4 (2-7) versus 10 (9-11); P = 0·004), jejunojejunostomy formation (5 (2-6) versus 10 (9-11); P = 0·001) and pouch formation (4 (3-6) versus 9 (5-12); P = 0·004). CONCLUSION: The proposed error rating tool allows an objective and reliable assessment of operative performance in laparoscopic gastric bypass procedures.

Simulation-based training and learning curves in laparoscopic Roux-en-Y gastric bypass.

BACKGROUND: Ex vivo simulation-based technical skills training has been shown to improve operating room performance and shorten learning curves for basic laparoscopic procedures. The application of such training for laparoscopic Roux-en-Y gastric bypass (LRYGBP) has not been reviewed. METHODS: Relevant studies were identified by one author from a search of MEDLINE and Embase databases from 1 January 1994 to 30 November 2010. Studies examining the learning curves and ex vivo training methods for LRYGBP were included; all other types of bariatric operations were excluded. A manual search of the references was also performed to identify additional potentially relevant papers. RESULTS: Twelve studies (5 prospective and 7 retrospective case series) were selected for review. The learning curve for LRYGBP was reported to be 50-100 procedures. Bench-top laparoscopic jejunojejunostomy, anaesthetized animals and Thiel human cadavers made up the bulk of the reported models for ex vivo training. Most studies were of relatively poor quality. An evidence-based ex vivo training curriculum for LRYGBP is currently lacking. CONCLUSION: Better quality studies are needed to define the learning curve for LRYGBP. Future studies should focus on the design and validation of training models, and a comprehensive curriculum for training and assessment of cognitive, technical and non-technical components of competency for laparoscopic bariatric surgery.

h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance.

BACKGROUND: The h-index is a tool that is increasingly used to measure individual research productivity. It is unknown whether its use as an evaluation of individual research impact is reliable and valid within the context of anaesthesia. METHODS: We calculated the h-indices of 268 faculty members of a university department of anaesthesia using Scopus™ and Web of Science(®). Agreement between the databases was investigated with a Bland-Altman plot. The construct validity was examined by comparing the h-indices for faculty grouped by academic rank. RESULTS: The mean bias between the Scopus™ and Web of Science(®) h-indices was 0.09 but 1.96 sd limits of agreement were -5.7 to 5.9. The Web of Science(®)-derived h-indices showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 2 for assistant professors, 9 for associate professors, and 16 for full professors. The Scopus™-derived h-indices also showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 1 for assistant professors, 9 for associate professors, and 17 for full professors. Post hoc testing found statistically significant differences in all comparisons between academic ranks (all P<0.01). Ignoring self-citations did not affect construct validity of the h-index. We found no evidence that the h-index is superior to counting the total number of publications. CONCLUSIONS: Agreement between the two databases was problematic. There was evidence of construct validity; however, the overlap between academic ranks limits the discriminative power of a low h-index.

Laparoscopic gastric surgery in an enhanced recovery programme.

BACKGROUND: Laparoscopy is associated with less pain and organ dysfunction than open surgery. Improved perioperative care (enhanced recovery programmes, fast-track methodology) has also led to reduced morbidity and a shorter hospital stay. The effects of a combination of laparoscopic resection and accelerated recovery have not been examined previously in the context of gastric surgery. METHODS: This was a prospective study of 32 consecutive patients undergoing laparoscopic gastric resection combined with an enhanced recovery protocol (early oral intake, no drains or nasogastric tubes, no epidural analgesia, use of a urinary catheter for less than 24 h and planned discharge 72 h after surgery). Outcomes included length of hospital stay, intraoperative and postoperative complications, readmission rate and 30-day mortality. RESULTS: Operative procedures were elective distal or subtotal gastrectomy (22 patients) and total gastrectomy (10). Median length of hospital stay was 4 (range 2-30) days. There were two major complications: postoperative bleeding requiring reoperation and pulmonary embolism. Two patients required readmission, one for a wound abscess and one for treatment of a urinary tract infection. There were no deaths within 30 days. CONCLUSION: Minimally invasive gastrectomy with enhanced postoperative recovery results in a short hospital stay and low morbidity rate.

Transatlantic comparison of the competence of surgeons at the start of their professional career.

BACKGROUND: Although the objective in European Union and North American surgical residency programmes is similar-to train competent surgeons-residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula. METHODS: At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used. RESULTS: No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0.001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0.001) and OSATS global rating scale (P < 0.001) scores. CONCLUSION: Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.

Objective assessment of surgical competence in gynaecological laparoscopy: development and validation of a procedure-specific rating scale.

OBJECTIVE: The purpose of this study was to develop a global- and a procedure-specific rating scale based on a well-validated generic model (objective structured assessment of technical skills) for assessment of technical skills in laparoscopic gynaecology. Furthermore, we aimed to investigate the construct validity and the interrater agreement (IRA) of the rating scale. We investigated both the gamma coefficient (Kendall's rank correlation), which is a measure of the strength of dependence between observations, and the kappa value for each of the ten individual items included in the rating scale. DESIGN: Prospective cohort, observer-blinded study. SETTING: Departments of Obstetrics and Gynaecology in Zealand, Denmark. POPULATION: Twenty one gynaecologists or gynaecological trainees. MATERIAL AND METHODS: Twenty-one video recordings of right side laparoscopic salpingectomies were collected prospectively, eight from novices (defined as <10 procedures), seven from intermediate experienced (20-50 procedures) and six from experts (> 200 procedures). All operations were performed by the same operative principles and using a standardised technique. The recordings were analysed by two independent, blinded observers. MAIN OUTCOME MEASURES: Construct validity of the rating scale based on operative performance (median of total score) and interrater reliability. RESULTS: There were significant differences between the three groups: median score of novices 24.00 versus intermediate 29.50 versus expert 39.50, P < 0.003) The IRA was 0.83 overall. The gamma correlation coefficient was 0.91. The kappa values varied from 0.510-0.933 for each of the individual items of the rating scale. CONCLUSIONS: The procedure-specific rating scale for laparoscopic salpingectomy is a valid and reliable tool for assessment of technical skills in gynaecological laparoscopy.

Ultrasound of the acute abdomen performed by surgeons in training.

AIM: Ultrasound has a well-established role in the diagnostic assessment of acute abdominal pain where some ultrasonically easily-accessible organs account for several diagnostic possibilities. The objective of the present study was to evaluate whether surgeons without ultrasound experience could perform valid abdominal ultrasound examinations of patients referred with acute abdominal pain. METHODS: Patients referred with acute abdominal pain had an ultrasound examination by a surgeon in training as well as by an experienced consultant radiologist whose results served as the gold standard. All participating surgeons were without any pre-existing ultrasound experience and received one hour of introduction to abdominal ultrasound. RESULTS: Thirty patients underwent 40 surgeon performed and 30 radiologist performed ultrasound examinations. Regarding gallstone and cholecholecystitis the sensitivity, specificity and kappa-agreement of the surgeon performed ultrasound examination was 1.00 (0.77-1.00), 0.96 (0.79-0.99), 0.94 (0.3-1.00) and 0.40 (0.12-0.77), 0.97 (0.83-0.99), 0.44 (0.00-0.96); respectively. Visualization of the common bile duct was poor having 73% non-diagnostic surgeon-performed ultrasound examinations. CONCLUSION: Surgeons in training without pre-existing ultrasound experience and only a minimum of formal ultrasound education can perform valid and reliable ultrasound examinations of the gallbladder in patients admitted with acute abdominal pain.

The cost of intraoperative adverse events in abdominal and pelvic surgery: A systematic review.

BACKGROUND: The assessment of intra-operative adverse events (iAEs) is a vastly under researched area with the potential to provide new methods on how to improve patient outcomes and hospital costs. Our objective was to determine the relationship between iAEs and total hospital costs in abdominal and pelvic surgery. DATA SOURCES: We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework. Embase, MEDLINE and EBM Reviews online databases were searched to identify all studies that reported iAE rates and total hospital costs. We then analyzed the costing approach used in each article using the Drummond tool and evaluated articles quality using the GRADE method. CONCLUSIONS: In total, 1709 unique references were identified through our literature search. After review, 23 were included. All studies that reported iAE rates and cost as the primary outcome found that iAEs significantly increased total hospital costs. We identified a relationship between iAEs and increased hospital costs. Future studies need to be performed to further evaluate the relationship between iAEs and cost as current studies are of low quality.

Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator.

BACKGROUND: Safe realistic training and unbiased quantitative assessment of technical skills are required for laparoscopy. Virtual reality (VR) simulators may be useful tools for training and assessing basic and advanced surgical skills and procedures. This study aimed to investigate the construct validity of the LapSimGyn VR simulator, and to determine the learning curves of gynecologists with different levels of experience. METHODS: For this study, 32 gynecologic trainees and consultants (juniors or seniors) were allocated into three groups: novices (0 advanced laparoscopic procedures), intermediate level (>20 and <60 procedures), and experts (>100 procedures). All performed 10 sets of simulations consisting of three basic skill tasks and an ectopic pregnancy program. The simulations were carried out on 3 days within a maximum period of 2 weeks. Assessment of skills was based on time, economy of movement, and error parameters measured by the simulator. RESULTS: The data showed that expert gynecologists performed significantly and consistently better than intermediate and novice gynecologists. The learning curves differed significantly between the groups, showing that experts start at a higher level and more rapidly reach the plateau of their learning curve than do intermediate and novice groups of surgeons. CONCLUSION: The LapSimGyn VR simulator package demonstrates construct validity on both the basic skills module and the procedural gynecologic module for ectopic pregnancy. Learning curves can be obtained, but to reach the maximum performance for the more complex tasks, 10 repetitions do not seem sufficient at the given task level and settings. LapSimGyn also seems to be flexible and widely accepted by the users.