Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on video.

BACKGROUND: Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos. METHODS: For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool. RESULTS: Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies. CONCLUSION: Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education).

BACKGROUND: Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose. METHODS: Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted. RESULTS: The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data. CONCLUSION: This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

Voxel-based diktiometry: Combining convolutional neural networks with voxel-based analysis and its application in diffusion tensor imaging for Parkinson's disease.

Extracting population-wise information from medical images, specifically in the neurological domain, is crucial to better understanding disease processes and progression. This is frequently done in a whole-brain voxel-wise manner, in which a population of patients and healthy controls are registered to a common co-ordinate space and a statistical test is performed on the distribution of image intensities for each location. Although this method has yielded a number of scientific insights, it is further from clinical applicability as the differences are often small and altogether do not permit for a high-performing classifier. In this article, we take the opposite approach of using a high-performing classifier, specifically a traditional convolutional neural network, and then extracting insights from it which can be applied in a population-wise manner, a method we call voxel-based diktiometry. We have applied this method to diffusion tensor imaging (DTI) analysis for Parkinson's disease (PD), using the Parkinson's Progression Markers Initiative database. By using the network sensitivity information, we can decompose what elements of the DTI contribute the most to the network's performance, drawing conclusions about diffusion biomarkers for PD that are based on metrics which are not readily expressed in the voxel-wise approach.

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy.

INTRODUCTION: Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon's technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)-objective measures related to instrument movements-can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes. METHODS: A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query "robot-assisted surgery OR robotic surgery AND performance metrics" between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed. RESULTS: The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies. CONCLUSIONS: APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required.

Subthalamic nucleus local field potentials recordings reveal subtle effects of promised reward during conflict resolution in Parkinson's disease.

Cognitive action control depends on cortical-subcortical circuits, involving notably the subthalamic nucleus (STN), as evidenced by local field potentials recordings (LFPs) studies. The STN consistently shows an increase in theta oscillations power during conflict resolution. Some studies have shown that cognitive action control in Parkinson's disease (PD) could be influenced by the occurrence of monetary reward. In this study, we investigated whether incentive motivation could modulate STN activity, and notably STN theta activity, during response conflict resolution. To achieve this objective, we recorded STN LFPs during a motivated Simon task in PD patients who had undergone deep brain stimulation surgery. Behavioral results revealed that promised rewards increased the difficulty in resolving conflict situations, thus replicating previous findings. Signal analyses locked on the imperative stimulus onset revealed the typical pattern of increased theta power in a conflict situation. However, this conflict-related modulation of theta power was not influenced by the size of the reward cued. We nonetheless identified a significant effect of the reward size on local functional organization (indexed by inter-trial phase clustering) of theta oscillations, with higher organization associated with high rewards while resolving conflict. When focusing on the period following the onset of the reward cue, we unveiled a stronger beta power decrease in higher reward conditions. However, these LFPs results were not correlated to behavioral results. Our study suggests that the STN is involved in how reward information can influence computations during conflict resolution. However, considering recent studies as well as the present results, we suspect that these effects are subtle.

Relationships Between Expertise, Crew Familiarity and Surgical Workflow Disruptions: An Observational Study.

BACKGROUND: Teamwork is an essential factor in reducing workflow disruption (WD) in the operating room. Team familiarity (TF) has been recognized as an antecedent to surgical quality and safety. To date, no study has examined the link between team members' role and expertise, TF and WD in surgical setting. This study aimed to examine the relationships between expertise, surgeon-scrub nurse familiarity and WD. METHODS: We observed a convenience sample of 12 elective neurosurgical procedures carried out by 4 surgeons and 11 SN with different levels of expertise and different degrees of familiarity between surgeons and SN. We calculated the number of WD per unit of coding time to control for the duration of operation. We explored the type and frequency of WD, and the differences between the surgeons and SN. We examined the relationships between duration of WD, staff expertise and surgeon-scrub nurse familiarity. RESULTS: 9.91% of the coded surgical time concerned WD. The most frequent causes of WD were distractions (29.7%) and colleagues' interruptions (25.2%). This proportion was seen for SN, whereas teaching moments and colleagues' interruptions were the most frequent WD for surgeons. The WD was less high among expert surgeons and less frequent when surgeon was familiar with SN. CONCLUSIONS: The frequency of WD during surgical time can compromise surgical quality and patient safety. WD seems to decrease in teams with high levels of surgeon-scrub nurse familiarity and with development of surgical expertise. Favoring TF and giving feedback to the team about WD issues could be interesting ways to improve teamwork.

Data-Driven Prediction of the Therapeutic Window during Subthalamic Deep Brain Stimulation Surgery.

BACKGROUND: Moving from awake surgery under local anesthesia to asleep surgery under general anesthesia will require to precisely predict the outcome of deep brain stimulation. OBJECTIVE: To propose a data-driven prediction of both the therapeutic effect and side effects of the surgery. METHODS: The retrospective intraoperative data from 30 patients operated on in the subthalamic nucleus were used to train an artificial neural network to predict the deep brain stimulation outcome. A leave-one-out validation was undertaken to give a predictive performance that would reflect the performance of the predictive model in clinical practice. Three-dimensional coordinates and the amount of current of the electrodes were used to train the model. RESULTS: 130 electrode positions were reviewed. The areas under the curve were 0.902 and 0.89 for therapeutic and side effects, respectively. The mean sensitivity and specificity were 93.07% (SD 0.95) and 69.24% (SD 5.27) for the therapeutic effect, 73.47% (SD 10.55) and 91.82% (SD 0.12) for the side effect. CONCLUSION: Data-driven prediction could be an additional modality to predict deep brain stimulation outcome. Further validation is needed to precisely use this method for performing surgery under general anesthesia.

Distinguishing surgical behavior by sequential pattern discovery.

OBJECTIVE: Each surgical procedure is unique due to patient's and also surgeon's particularities. In this study, we propose a new approach to distinguish surgical behaviors between surgical sites, levels of expertise and individual surgeons thanks to a pattern discovery method. METHODS: The developed approach aims to distinguish surgical behaviors based on shared longest frequent sequential patterns between surgical process models. To allow clustering, we propose a new metric called SLFSP. The approach is validated by comparison with a clustering method using Dynamic Time Warping as a metric to characterize the similarity between surgical process models. RESULTS: Our method outperformed the existing approach. It was able to make a perfect distinction between surgical sites (accuracy of 100%). We reached an accuracy superior to 90% and 85% for distinguishing levels of expertise and individual surgeons. CONCLUSION: Clustering based on shared longest frequent sequential patterns outperformed the previous study based on time analysis. SIGNIFICANCE: The proposed method shows the feasibility of comparing surgical process models, not only by their duration but also by their structure of activities. Furthermore, patterns may show risky behaviors, which could be an interesting information for surgical training to prevent adverse events.

Investigation of morphometric variability of subthalamic nucleus, red nucleus, and substantia nigra in advanced Parkinson's disease patients using automatic segmentation and PCA-based analysis.

Subthalamic nucleus (STN) deep brain stimulation (DBS) is an effective surgical therapy to treat Parkinson's disease (PD). Conventional methods employ standard atlas coordinates to target the STN, which, along with the adjacent red nucleus (RN) and substantia nigra (SN), are not well visualized on conventional T1w MRIs. However, the positions and sizes of the nuclei may be more variable than the standard atlas, thus making the pre-surgical plans inaccurate. We investigated the morphometric variability of the STN, RN and SN by using label-fusion segmentation results from 3T high resolution T2w MRIs of 33 advanced PD patients. In addition to comparing the size and position measurements of the cohort to the Talairach atlas, principal component analysis (PCA) was performed to acquire more intuitive and detailed perspectives of the measured variability. Lastly, the potential correlation between the variability shown by PCA results and the clinical scores was explored.

A preliminary exploration into top-down and bottom-up deep-learning approaches to localising neuro-interventional point targets in volumetric MRI.

PURPOSE: Point localisation is a critical aspect of many interventional planning procedures, specifically representing anatomical regions of interest or landmarks as individual points. This could be seen as analogous to the problem of visual search in cognitive psychology, in which this search is performed either: bottom-up, constructing increasingly abstract and coarse-resolution features over the entire image; or top-down, using contextual cues from the entire image to refine the scope of the region being investigated. Traditional convolutional neural networks use the former, but it is not clear if this is optimal. This article is a preliminary investigation as to how this motivation affects 3D point localisation in neuro-interventional planning. METHODS: Two neuro-imaging datasets were collected: one for cortical point localisation for repetitive transcranial magnetic stimulation and the other for sub-cortical anatomy localisation for deep brain stimulation. Four different frameworks were developed using top-down versus bottom-up paradigms as well as representing points as co-ordinates or heatmaps. These networks were applied to point localisation for transcranial magnetic stimulation and subcortical anatomy localisation. These networks were evaluated using cross-validation and a varying number of training datasets to analyse their sensitivity to quantity of training data. RESULTS: Each network shows increasing performance as the amount of available training data increases, with the co-ordinate-based top-down network consistently outperforming the others. Specifically, the top-down architectures tend to outperform the bottom-up ones. An analysis of their memory consumption also encourages the top-down co-ordinate based architecture as it requires significantly less memory than either bottom-up architectures or those representing their predictions via heatmaps. CONCLUSION: This paper is a preliminary foray into a fundamental aspect of machine learning architectural design: that of the top-down/bottom-up divide from cognitive psychology. Although there are additional considerations within the particular architectures investigated that could affect these results and the number of architectures investigated is limited, our results do indicate that the less commonly used top-down paradigm could lead to more efficient and effective architectures in the future.

Generalisation capabilities of machine-learning algorithms for the detection of the subthalamic nucleus in micro-electrode recordings.

PURPOSE: Micro-electrode recordings (MERs) are a key intra-operative modality used during deep brain stimulation (DBS) electrode implantation, which allow for a trained neurophysiologist to infer the anatomy in which the electrode is placed. As DBS targets are small, such inference is necessary to confirm that the electrode is correctly positioned. Recently, machine learning techniques have been used to augment the neurophysiologist's capability. The goal of this paper is to investigate the generalisability of these methods with respect to different clinical centres and training paradigms. METHODS: Five deep learning algorithms for binary classification of MER signals have been implemented. Three databases from two different clinical centres have also been collected with differing size, acquisition hardware, and annotation protocol. Each algorithm has initially been trained on the largest database, then either directly tested or fine-tuned on the smaller databases in order to estimate their generalisability. As a reference, they have also been trained from scratch on the smaller databases as well in order to estimate the effect of the differing database sizes and annotation systems. RESULTS: Each network shows significantly reduced performance (on the order of a 6.5% to 16.0% reduction in balanced accuracy) when applied out-of-distribution. This reduction can be ameliorated through fine-tuning the network on the new database through transfer learning. Although, even for these small databases, it appears that retraining from scratch may still offer equivalent performance as fine-tuning with transfer learning. However, this is at the expense of significantly longer training times. CONCLUSION: Generalisability is an important criterion for the success of machine learning algorithms in clinic. We have demonstrated that a variety of recent machine learning algorithms for MER classification are negatively affected by domain shift, but that this can be quickly ameliorated through simple transfer learning procedures that can be readily performed for new centres.

The influence of virtual reality simulation on surgical residents' heart rate during an assessment of arthroscopic technical skills: A prospective, paired observational study.

HYPOTHESIS: To demonstrate that a virtual reality (VR) simulation training program reduces heart rate variability during an assessment of surgical trainees' technical skills in arthroscopy. STUDY DESIGN: Prospective observational matched study. MATERIALS & METHODS: Thirty-six orthopaedic surgery residents, new to arthroscopy, received standard training in arthroscopic knee surgery, supplemented by additional monthly training for 6months on a VR simulator for 16 of them. At inclusion, the 2 groups (VR and NON-VR) answered a questionnaire and performed a meniscectomy on a VR simulator. After 6months of training, two independent trainers blinded to the inclusion arms evaluated the technical skills of the two groups during meniscectomies on a model and on an anatomical subject. Heart rate variability (HRV) was measured using a wireless heart rate monitor during baseline, VR training, and assessment. RESULTS: After removing incomplete data, the analysis focused on 10 VR residents matched at inclusion with 10 NON-VR residents. The VR group had a significantly lower heart rate at the final assessment (p=0.02) and lower overall HRV (p=0.05). The low/high frequency ratio (LF/HF) was not significantly different between the groups (1.84 vs 2.05, p=0.66) but the before-after training comparison showed a greater decrease in this ratio in the VR group compared to the NON-VR group -0.76 (-41%) vs -0.08 (-4%). CONCLUSION: This study demonstrates a significant difference in heart rate variability between trained residents versus untrained residents during the final assessment of their technical skills at 6months. It appears that improving stress management should be an integral part of training programs in arthroscopic surgery. CLINICAL INTEREST: VR simulators in arthroscopy could improve non-technical skills such as heart rate variability, from the perspective of accountability. LEVEL OF EVIDENCE: III.

Analysis of convolutional neural networks for fronto-temporal dementia biomarker discovery.

PURPOSE: Frontotemporal lobe dementia (FTD) results from the degeneration of the frontal and temporal lobes. It can manifest in several different ways, leading to the definition of variants characterised by their distinctive symptomatologies. As these variants are detected based on their symptoms, it can be unclear if they represent different types of FTD or different symptomatological axes. The goal of this paper is to investigate this question with a constrained cohort of FTD patients in order to see if the heterogeneity within this cohort can be inferred from medical images rather than symptom severity measurements. METHODS: An ensemble of convolutional neural networks (CNNs) is used to classify diffusion tensor images collected from two databases consisting of 72 patients with behavioural variant FTD and 120 healthy controls. FTD biomarkers were found using voxel-based analysis on the sensitivities of these CNNs. Sparse principal components analysis (sPCA) is then applied on the sensitivities arising from the patient cohort in order to identify the axes along which the patients express these biomarkers. Finally, this is correlated with their symptom severity measurements in order to interpret the clinical presentation of each axis. RESULTS: The CNNs result in sensitivities and specificities between 83 and 92%. As expected, our analysis determines that all the robust biomarkers arise from the frontal and temporal lobes. sPCA identified four axes in terms of biomarker expression which are correlated with symptom severity measurements. CONCLUSION: Our analysis confirms that behavioural variant FTD is not a singular type or spectrum of FTD, but rather that it has multiple symptomatological axes that relate to distinct regions of the frontal and temporal lobes. This analysis suggests that medical images can be used to understand the heterogeneity of FTD patients and the underlying anatomical changes that lead to their different clinical presentations.

Probe motion during mid-trimester fetal anomaly scan in the clinical setting: A prospective observational study.

INTRODUCTION: This study aims to investigate probe motion during full mid-trimester anomaly scans. METHODS: We undertook a prospective, observational study of obstetric sonographers at a UK University Teaching Hospital. We collected prospectively full-length video recordings of routine second-trimester anomaly scans synchronized with probe trajectory tracking data during the scan. Videos were reviewed and trajectories analyzed using duration, path metrics (path length, velocity, acceleration, jerk, and volume) and angular metrics (spectral arc, angular area, angular velocity, angular acceleration, and angular jerk). These trajectories were then compared according to the participant level of expertise, fetal presentation, and patient BMI. RESULTS: A total of 17 anomaly scans were recorded. The average velocity of the probe was 12.9 ± 3.4 mm/s for the consultants versus 24.6 ± 5.7 mm/s for the fellows (p = 0.02), the average acceleration 170.4 ± 26.3 mm/s2 versus 328.9 ± 62.7 mm/s2 (p = 0.02), and the average jerk 7491.7 ± 1056.1 mm/s3 versus 14944.1 ± 3146.3 mm/s3 (p = 0.02), the working volume 9.106 ± 4.106 mm3 versus 29.106 ± 11.106 mm3 (p = 0.03), respectively. The angular metrics were not significantly different according to the participant level of expertise, the fetal presentation, or to patients BMI. CONCLUSION: Some differences in the probe path metrics (velocity, acceleration, jerk and working volume) were noticed according to operator's level.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Effects of low-dose protocols in endovascular treatment of intracranial aneurysms: development of workflow task analysis during cerebral endovascular procedures.

OBJECTIVE: Reducing radiation exposure through the use of low-dose protocols during cerebral endovascular procedures is recommended, but evaluation of the impact on the procedure itself is difficult and subjective. A workflow task analysis could provide an objective comparison of two different radiation exposure protocols. SUBJECTS AND METHODS: Twenty endovascular aneurysm treatments were analyzed using a low-dose protocol (reducing radiation exposure by 20%) in 10 cases and a normal-dose protocol in the other 10 cases. The procedure was subdivided into five phases, each comprising a sequence of tasks. Each task was defined as a triplet, associating an action, an instrument, and an anatomic structure. A workflow editor was used to record tasks and phases with a tablet PC. The total duration of the entire procedure, the duration of each task, and the number of task repetitions were isolated and used as the metric. Moreover, the tasks involving x-ray use, essential for navigation and treatment phases, were separated and analyzed. RESULTS: For the microcatheter navigation and treatment phases, no statistically significant difference was found between the two radiation exposure protocols. For guide catheter navigation in cervical vessels, the total phase duration and total and mean time of tasks specifically involving x-ray use increased with age, but there was no difference between the two radiation protocols. CONCLUSION: Workflow task analysis of endovascular aneurysm treatment shows no difference between low-dose and normal-dose protocols in the guide catheter navigation, microcatheter navigation, or treatment phases.

Multi-site study of surgical practice in neurosurgery based on surgical process models.

Surgical Process Modelling (SPM) was introduced to improve understanding the different parameters that influence the performance of a Surgical Process (SP). Data acquired from SPM methodology is enormous and complex. Several analysis methods based on comparison or classification of Surgical Process Models (SPMs) have previously been proposed. Such methods compare a set of SPMs to highlight specific parameters explaining differences between populations of patients, surgeons or systems. In this study, procedures performed at three different international University hospitals were compared using SPM methodology based on a similarity metric focusing on the sequence of activities occurring during surgery. The proposed approach is based on Dynamic Time Warping (DTW) algorithm combined with a clustering algorithm. SPMs of 41 Anterior Cervical Discectomy (ACD) surgeries were acquired at three Neurosurgical departments; in France, Germany, and Canada. The proposed approach distinguished the different surgical behaviors according to the location where surgery was performed as well as between the categorized surgical experience of individual surgeons. We also propose the use of Multidimensional Scaling to induce a new space of representation of the sequences of activities. The approach was compared to a time-based approach (e.g. duration of surgeries) and has been shown to be more precise. We also discuss the integration of other criteria in order to better understand what influences the way the surgeries are performed. This first multi-site study represents an important step towards the creation of robust analysis tools for processing SPMs. It opens new perspectives for the assessment of surgical approaches, tools or systems as well as objective assessment and comparison of surgeon's expertise.

Development of workflow task analysis during cerebral diagnostic angiographies: time-based comparison of junior and senior tasks.

OBJECTIVE: Assessing neuroradiologists' skills in the operating room (OR) is difficult and often subjective. This study used a workflow time-based task analysis approach while performing cerebral angiography. METHODS: Eight angiographies performed by a senior neuroradiologist and eight performed by a junior neuroradiologist were compared. Dedicated software with specific terminology was used to record the tasks. Procedures were subdivided into phases, each comprising multiple tasks. Each task was defined as a triplet, associating an action, an instrument and an anatomical structure. The duration of each task was the metric. Total duration of the procedure, task duration and the number of times a task was repeated were identified. The focus was on tasks using fluoroscopy and for moving the X-ray table/tube. RESULTS: The total duration of tasks to complete the entire procedure was longer for the junior operators than for the seniors (P=0.012). The mean duration per task during the navigation phase was 86s for the juniors and 43s for the seniors (P=0.002). The total and mean durations of tasks involving the use of fluoroscopy were also longer for the juniors (P=0.002 and P=0.033, respectively). For tasks involving the table/tube, the total and mean durations were again longer for the juniors (P=0.019 and P=0.082, respectively). CONCLUSION: This approach allows reliable skill assessment in the radiology OR and comparison of junior and senior competencies during cerebral diagnostic angiography. This new tool can improve the quality and safety of procedures, and facilitate the learning process for neuroradiologists.