Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Introduction: Medical Mobilities in the Modern Middle East and North Africa.

This themed section contributes to efforts to conceptualize medical mobility. It does so by observing medical histories within the Middle East while following concrete movements. This focus on what moves and how, rather than on largely static and fixed units of analysis on where to, is central to the studies in this issue. The location of the Middle East, as a crossroad for imperial mobilities-is ideal for exploring transnational medical movements. Bringing together historians of the Middle East and North Africa, the articles explore intersections among medicine, health, and the body and histories of cross-regional mobility. This section spans the period from the early twentieth century to the 1970s. The articles are based on primary sources in Greek, Turkish, English, French, Spanish, and Arabic, located in the national archives of the UK, Israel, and Cyprus; in French diplomatic and military archives; and in the Overseas Nursing Association's publications.

Longitudinal multiple sclerosis lesion segmentation: Resource and challenge.

In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Balancing new technology: Virtual reality for balance measurement case report.

RATIONALE: Falling and the inability to maintain balance are the second leading cause of unintentional injury deaths globally. There are a number of chronic and acute conditions characterized by balance difficulties, including neurological diseases, and sport injuries. Therefore, methods to monitor and quantify balance are critical for clinical decision-making regarding risk management and balance rehabilitation. New advances in virtual reality (VR) technology has identified VR as a novel therapeutic platform. VRSway is a VR application that uses sensors attached to a virtual reality headset, and handheld remote controllers for measurement and analysis of postural stability by measuring changes in spatial location relative to the center of mass and calculates various postural stability indexes. This case report evaluates balance measures in 2 healthy participants with no previous history of balance disorders using the VRSway software application and compares to output generated by the current gold standard of balance measurement, force platform technology. CASE PRESENTATION: The primary objective of this case study was to validate the VRSway stability score for evaluation of balance. Here, we present posturography measures of the VRSway in comparison with force plate readouts in 2 healthy participants. Body Sway measurements were recorded simultaneously in both the force plate and VRSway systems. Data calculated by proprietary software is highly correlative to the data generated by force plates for each of the following measurements for participant-1 and participant-2, respectively: Sway index (r 1 = 0.985, P < .001; r 2 = 0.970, P < .001), total displacement (r 1 = 0.982, P < .001; r 2 = 0.935, P < .001), center of pressure mean velocity (r 1 = 0.982, P < .001; r 2 = 0.935, P < .001), ellipse radius 1 (r 1 = 0.979, P < .001; r 2 = 0.965, P < .001), ellipse radius 2 (r 1 = 0.982, P < .001; r 2 = 0.969, P < .001), and ellipse area (r 1 = 0.983, P < .001; r 2 = 0.969, P < .001). CONCLUSIONS: Data from this case study suggest that VRSway measurements are highly correlated with output from force plate technology posing that VRSway is a novel approach to evaluate balance measures with VR. More research is required to understand possible uses of VR-based use for balance measurement in a larger and more diverse cohort.

Virtual reality in the management of stress and anxiety disorders: A retrospective analysis of 61 people treated in the metaverse.

Mental health is the second largest group of health disorders associated with prolonged disability. Treating conditions such as stress and anxiety are a global health challenge due to inadequate funding and resources. Therefore, providing virtual treatment in the metaverse may provide a novel method of treatment for these conditions. We conducted a retrospective analysis of health records of people experiencing stress and anxiety who were treated principally in the metaverse using virtual reality. The main objective was to determine if virtual mental health treatment was achievable and safe, with measurable outcomes repeated at multiple time points. Here, 61 participants health records were evaluated (50% were female, 19% male, 31% identified as other). The cohort was 45.7 ± 15.7 years of age and reported no adverse effects with outcomes measured. Specifically, anxiety (via Generalized Anxiety Disorder Scale) decreased by 34% (p = 0.002) and stress (via Perceived Stress Scale) decreased by 32% (p < 0.001) after virtual intervention. The data suggests that this method of treatment was feasible, safe, and outcomes were obtainable over a range of time points. This early data suggest that management in the metaverse for these conditions may be beneficial, however, further prospective studies are necessary to better understand these clinical findings.

Virtual reality in the management of patients with low back and neck pain: a retrospective analysis of 82 people treated solely in the metaverse.

BACKGROUND: Clinically, neck pain disorders (NPD) and non-specific low back pain (NS-LBP) are respectively the fourth and first most common conditions associated with the greatest number of years lived with disability. Remote delivery of care may benefit healthcare sustainability, reduce environmental pollution, and free up space for those requiring care non-virtual care. METHODS: A retrospective analysis was performed on 82 participants with NS-LBP and/or NPD who received exercise therapy delivered solely in the metaverse using virtually reality. The study was to determine if this was achievable, safe, had appropriate outcome measures that could be collected, and if there was any early evidence of beneficial effects. RESULTS: The study demonstrated that virtual reality treatment delivered via the metaverse appears to be safe (no adverse events or side effects). Data for more than 40 outcome measures were collected. Disability from NS-LBP was significantly reduced (Modified Oswestry Low Back Pain Disability Index) by 17.8% (p < 0.001) and from NPD (Neck Disability Index) by 23.2% (p = 0.02). CONCLUSIONS: The data suggest that this method of providing exercise therapy was feasible, and safe (no adverse events reported), that complete reports were obtained from a large selection of patients, and that software acquired outcomes were obtainable over a range of time points. Further prospective research is necessary to better understand our clinical findings.

Estimating individual treatment effect on disability progression in multiple sclerosis using deep learning.

Disability progression in multiple sclerosis remains resistant to treatment. The absence of a suitable biomarker to allow for phase 2 clinical trials presents a high barrier for drug development. We propose to enable short proof-of-concept trials by increasing statistical power using a deep-learning predictive enrichment strategy. Specifically, a multi-headed multilayer perceptron is used to estimate the conditional average treatment effect (CATE) using baseline clinical and imaging features, and patients predicted to be most responsive are preferentially randomized into a trial. Leveraging data from six randomized clinical trials (n = 3,830), we first pre-trained the model on the subset of relapsing-remitting MS patients (n = 2,520), then fine-tuned it on a subset of primary progressive MS (PPMS) patients (n = 695). In a separate held-out test set of PPMS patients randomized to anti-CD20 antibodies or placebo (n = 297), the average treatment effect was larger for the 50% (HR, 0.492; 95% CI, 0.266-0.912; p = 0.0218) and 30% (HR, 0.361; 95% CI, 0.165-0.79; p = 0.008) predicted to be most responsive, compared to 0.743 (95% CI, 0.482-1.15; p = 0.179) for the entire group. The same model could also identify responders to laquinimod in another held-out test set of PPMS patients (n = 318). Finally, we show that using this model for predictive enrichment results in important increases in power.

Propagating Uncertainty Across Cascaded Medical Imaging Tasks for Improved Deep Learning Inference.

Although deep networks have been shown to perform very well on a variety of medical imaging tasks, inference in the presence of pathology presents several challenges to common models. These challenges impede the integration of deep learning models into real clinical workflows, where the customary process of cascading deterministic outputs from a sequence of image-based inference steps (e.g. registration, segmentation) generally leads to an accumulation of errors that impacts the accuracy of downstream inference tasks. In this paper, we propose that by embedding uncertainty estimates across cascaded inference tasks, performance on the downstream inference tasks should be improved. We demonstrate the effectiveness of the proposed approach in three different clinical contexts: (i) We demonstrate that by propagating T2 weighted lesion segmentation results and their associated uncertainties, subsequent T2 lesion detection performance is improved when evaluated on a proprietary large-scale, multi-site, clinical trial dataset acquired from patients with Multiple Sclerosis. (ii) We show an improvement in brain tumour segmentation performance when the uncertainty map associated with a synthesised missing MR volume is provided as an additional input to a follow-up brain tumour segmentation network, when evaluated on the publicly available BraTS-2018 dataset. (iii) We show that by propagating uncertainties from a voxel-level hippocampus segmentation task, the subsequent regression of the Alzheimer's disease clinical score is improved.

Feature-based morphometry: discovering group-related anatomical patterns.

This paper presents feature-based morphometry (FBM), a new fully data-driven technique for discovering patterns of group-related anatomical structure in volumetric imagery. In contrast to most morphometry methods which assume one-to-one correspondence between subjects, FBM explicitly aims to identify distinctive anatomical patterns that may only be present in subsets of subjects, due to disease or anatomical variability. The image is modeled as a collage of generic, localized image features that need not be present in all subjects. Scale-space theory is applied to analyze image features at the characteristic scale of underlying anatomical structures, instead of at arbitrary scales such as global or voxel-level. A probabilistic model describes features in terms of their appearance, geometry, and relationship to subject groups, and is automatically learned from a set of subject images and group labels. Features resulting from learning correspond to group-related anatomical structures that can potentially be used as image biomarkers of disease or as a basis for computer-aided diagnosis. The relationship between features and groups is quantified by the likelihood of feature occurrence within a specific group vs. the rest of the population, and feature significance is quantified in terms of the false discovery rate. Experiments validate FBM clinically in the analysis of normal (NC) and Alzheimer's (AD) brain images using the freely available OASIS database. FBM automatically identifies known structural differences between NC and AD subjects in a fully data-driven fashion, and an equal error classification rate of 0.80 is achieved for subjects aged 60-80 years exhibiting mild AD (CDR=1).

Exploring uncertainty measures in deep networks for Multiple sclerosis lesion detection and segmentation.

Deep learning networks have recently been shown to outperform other segmentation methods on various public, medical-image challenge datasets, particularly on metrics focused on large pathologies. For diseases such as Multiple Sclerosis (MS), however, monitoring all the focal lesions visible on MRI sequences, even very small ones, is essential for disease staging, prognosis, and evaluating treatment efficacy. Small lesion segmentation presents significant challenges to popular deep learning models. This, coupled with their deterministic predictions, hinders their clinical adoption. Uncertainty estimates for these predictions would permit subsequent revision by clinicians. We present the first exploration of multiple uncertainty estimates based on Monte Carlo (MC) dropout (Gal and Ghahramani, 2016) in the context of deep networks for lesion detection and segmentation in medical images. Specifically, we develop a 3D MS lesion segmentation CNN, augmented to provide four different voxel-based uncertainty measures based on MC dropout. We train the network on a proprietary, large-scale, multi-site, multi-scanner, clinical MS dataset, and compute lesion-wise uncertainties by accumulating evidence from voxel-wise uncertainties within detected lesions. We analyze the performance of voxel-based segmentation and lesion-level detection by choosing operating points based on the uncertainty. Uncertainty filtering improves both voxel and lesion-wise TPR and FDR on remaining, certain predictions compared to sigmoid-based TPR/FDR curves. Small lesions and lesion-boundaries are the most uncertain regions, which is consistent with human-rater variability.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Evaluating White Matter Lesion Segmentations with Refined Sørensen-Dice Analysis.

The Sørensen-Dice index (SDI) is a widely used measure for evaluating medical image segmentation algorithms. It offers a standardized measure of segmentation accuracy which has proven useful. However, it offers diminishing insight when the number of objects is unknown, such as in white matter lesion segmentation of multiple sclerosis (MS) patients. We present a refinement for finer grained parsing of SDI results in situations where the number of objects is unknown. We explore these ideas with two case studies showing what can be learned from our two presented studies. Our first study explores an inter-rater comparison, showing that smaller lesions cannot be reliably identified. In our second case study, we demonstrate fusing multiple MS lesion segmentation algorithms based on the insights into the algorithms provided by our analysis to generate a segmentation that exhibits improved performance. This work demonstrates the wealth of information that can be learned from refined analysis of medical image segmentations.

Detection, localization, and sex classification of faces from arbitrary viewpoints and under occlusion.

This paper presents a novel framework for detecting, localizing, and classifying faces in terms of visual traits, e.g., sex or age, from arbitrary viewpoints and in the presence of occlusion. All three tasks are embedded in a general viewpoint-invariant model of object class appearance derived from local scale-invariant features, where features are probabilistically quantified in terms of their occurrence, appearance, geometry, and association with visual traits of interest. An appearance model is first learned for the object class, after which a Bayesian classifier is trained to identify the model features indicative of visual traits. The framework can be applied in realistic scenarios in the presence of viewpoint changes and partial occlusion, unlike other techniques assuming data that are single viewpoint, upright, prealigned, and cropped from background distraction. Experimentation establishes the first result for sex classification from arbitrary viewpoints, an equal error rate of 16.3 percent, based on the color FERET database. The method is also shown to work robustly on faces in cluttered imagery from the CMU profile database. A comparison with the geometry-free bag-of-words model shows that geometrical information provided by our framework improves classification. A comparison with support vector machines demonstrates that Bayesian classification results in superior performance.

Stroke Lesion Segmentation in FLAIR MRI Datasets Using Customized Markov Random Fields.

Robust and reliable stroke lesion segmentation is a crucial step toward employing lesion volume as an independent endpoint for randomized trials. The aim of this work was to develop and evaluate a novel method to segment sub-acute ischemic stroke lesions from fluid-attenuated inversion recovery (FLAIR) magnetic resonance imaging (MRI) datasets. After preprocessing of the datasets, a Bayesian technique based on Gabor textures extracted from the FLAIR signal intensities is utilized to generate a first estimate of the lesion segmentation. Using this initial segmentation, a customized voxel-level Markov random field model based on intensity as well as Gabor texture features is employed to refine the stroke lesion segmentation. The proposed method was developed and evaluated based on 151 multi-center datasets from three different databases using a leave-one-patient-out validation approach. The comparison of the automatically segmented stroke lesions with manual ground truth segmentation revealed an average Dice coefficient of 0.582, which is in the upper range of previously presented lesion segmentation methods using multi-modal MRI datasets. Furthermore, the results obtained by the proposed technique are superior compared to the results obtained by two methods based on convolutional neural networks and three phase level-sets, respectively, which performed best in the ISLES 2015 challenge using multi-modal imaging datasets. The results of the quantitative evaluation suggest that the proposed method leads to robust lesion segmentation results using FLAIR MRI datasets only as a follow-up sequence.

Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care.

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: a pilot study of eight cases.

We present our work investigating the feasibility of combining intraoperative ultrasound for brain shift correction and augmented reality (AR) visualization for intraoperative interpretation of patient-specific models in image-guided neurosurgery (IGNS) of brain tumors. We combine two imaging technologies for image-guided brain tumor neurosurgery. Throughout surgical interventions, AR was used to assess different surgical strategies using three-dimensional (3-D) patient-specific models of the patient's cortex, vasculature, and lesion. Ultrasound imaging was acquired intraoperatively, and preoperative images and models were registered to the intraoperative data. The quality and reliability of the AR views were evaluated with both qualitative and quantitative metrics. A pilot study of eight patients demonstrates the feasible combination of these two technologies and their complementary features. In each case, the AR visualizations enabled the surgeon to accurately visualize the anatomy and pathology of interest for an extended period of the intervention. Inaccuracies associated with misregistration, brain shift, and AR were improved in all cases. These results demonstrate the potential of combining ultrasound-based registration with AR to become a useful tool for neurosurgeons to improve intraoperative patient-specific planning by improving the understanding of complex 3-D medical imaging data and prolonging the reliable use of IGNS.

Why rankings of biomedical image analysis competitions should be interpreted with care.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

A statistical parts-based model of anatomical variability.

In this paper, we present a statistical parts-based model (PBM) of appearance, applied to the problem of modeling intersubject anatomical variability in magnetic resonance (MR) brain images. In contrast to global image models such as the active appearance model (AAM), the PBM consists of a collection of localized image regions, referred to as parts, whose appearance, geometry and occurrence frequency are quantified statistically. The parts-based approach explicitly addresses the case where one-to-one correspondence does not exist between all subjects in a population due to anatomical differences, as model parts are not required to appear in all subjects. The model is constructed through a fully automatic machine learning algorithm, identifying image patterns that appear with statistical regularity in a large collection of subject images. Parts are represented by generic scale-invariant features, and the model can, therefore, be applied to a wide variety of image domains. Experimentation based on 2-D MR slices shows that a PBM learned from a set of 102 subjects can be robustly fit to 50 new subjects with accuracy comparable to 3 human raters. Additionally, it is shown that unlike global models such as the AAM, PBM fitting is stable in the presence of unexpected, local perturbation.