Quantification of Celiac Disease Severity Using Video Capsule Endoscopy: A Comparison of Human Experts and Machine Learning Algorithms.

BACKGROUND: Video capsule endoscopy (VCE) is an attractive method for diagnosing and objectively monitoring disease activity in celiac disease (CeD). Its use, facilitated by artificial intelligence- based tools, may allow computer-assisted interpretation of VCE studies, transforming a subjective test into a quantitative and reproducible measurement tool. OBJECTIVE: To evaluate and compare objective CeD severity assessment as determined with VCE by expert human readers and a machine learning algorithm (MLA). METHODS: Patients ≥ 18 years with histologically proven CeD underwent VCE. Examination frames were scored by three readers from one center and the MLA, using a 4-point ordinal scale for assessing the severity of CeD enteropathy. After scoring, curves representing CeD severity across the entire small intestine (SI) and individual tertiles (proximal, mid, and distal) were fitted for each reader and the MLA. All comparisons used Krippendorff's alpha; values > 0.8 represent excellent to 'almost perfect' inter-reader agreement. RESULTS: VCEs from 63 patients were scored. Readers demonstrated strong inter-reader agreement on celiac villous damage (alpha=0.924), and mean value reader curves showed similarly excellent agreement with MLA curves (alpha=0.935). Average reader and MLA curves were comparable for mean and maximum values for the first SI tertile (alphas=0.932 and 0.867, respectively) and the mean value over the entire SI (alpha=0.945). CONCLUSION: A novel MLA demonstrated excellent agreement on whole SI imaging with three expert gastroenterologists. An ordinal scale permitted high inter-reader agreement, accurately and reliably replicated by the MLA. Interpreting VCEs using MLAs may allow automated diagnosis and disease burden assessment in CeD.

Multiparametric Quantitative Imaging Biomarker as a Multivariate Descriptor of Health: A Roadmap.

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

Multiparametric Data-driven Imaging Markers: Guidelines for Development, Application and Reporting of Model Outputs in Radiomics.

This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of "Radiomic ANalysis and DAta Modeling"), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies.

Multiparametric Quantitative Imaging in Risk Prediction: Recommendations for Data Acquisition, Technical Performance Assessment, and Model Development and Validation.

Combinations of multiple quantitative imaging biomarkers (QIBs) are often able to predict the likelihood of an event of interest such as death or disease recurrence more effectively than single imaging measurements can alone. The development of such multiparametric quantitative imaging and evaluation of its fitness of use differs from the analogous processes for individual QIBs in several key aspects. A computational procedure to combine the QIB values into a model output must be specified. The output must also be reproducible and be shown to have reasonably strong ability to predict the risk of an event of interest. Attention must be paid to statistical issues not often encountered in the single QIB scenario, including overfitting and bias in the estimates of model performance. This is the fourth in a five-part series on statistical methodology for assessing the technical performance of multiparametric quantitative imaging. Considerations for data acquisition are discussed and recommendations from the literature on methodology to construct and evaluate QIB-based models for risk prediction are summarized. The findings in the literature upon which these recommendations are based are demonstrated through simulation studies. The concepts in this manuscript are applied to a real-life example involving prediction of major adverse cardiac events using automated plaque analysis.

A Framework for Evaluating the Technical Performance of Multiparameter Quantitative Imaging Biomarkers (mp-QIBs).

Multiparameter quantitative imaging incorporates anatomical, functional, and/or behavioral biomarkers to characterize tissue, detect disease, identify phenotypes, define longitudinal change, or predict outcome. Multiple imaging parameters are sometimes considered separately but ideally are evaluated collectively. Often, they are transformed as Likert interpretations, ignoring the correlations of quantitative properties that may result in better reproducibility or outcome prediction. In this paper we present three use cases of multiparameter quantitative imaging: i) multidimensional descriptor, ii) phenotype classification, and iii) risk prediction. A fourth application based on data-driven markers from radiomics is also presented. We describe the technical performance characteristics and their metrics common to all use cases, and provide a structure for the development, estimation, and testing of multiparameter quantitative imaging. This paper serves as an overview for a series of individual articles on the four applications, providing the statistical framework for multiparameter imaging applications in medicine.

Comparative analytical performance of multiple plasma Aß42 and Aß40 assays and their ability to predict positron emission tomography amyloid positivity.

INTRODUCTION: This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aß) peptides in plasma and the extent to which they improve the prediction of amyloid positivity. METHODS: Alzheimer's Disease Neuroimaging Initiative plasma samples with corresponding amyloid positron emission tomography (PET) data were run on six plasma Aß assays. Statistical tests were performed to determine whether the plasma Aß measures significantly improved the area under the receiver operating characteristic curve for predicting amyloid PET status compared to age and apolipoprotein E (APOE) genotype. RESULTS: The age and APOE genotype model predicted amyloid status with an area under the curve (AUC) of 0.75. Three assays improved AUCs to 0.81, 0.81, and 0.84 (P < .05, uncorrected for multiple comparisons). DISCUSSION: Measurement of Aß in plasma contributes to addressing the amyloid component of the ATN (amyloid/tau/neurodegeneration) framework and could be a first step before or in place of a PET or cerebrospinal fluid screening study. HIGHLIGHTS: The Foundation of the National Institutes of Health Biomarkers Consortium evaluated six plasma amyloid beta (Aß) assays using Alzheimer's Disease Neuroimaging Initiative samples. Three assays improved prediction of amyloid status over age and apolipoprotein E (APOE) genotype. Plasma Aß42/40 predicted amyloid positron emission tomography status better than Aß42 or Aß40 alone.

Radiologists and Clinical Trials: Part 1 The Truth About Reader Disagreements.

The debate over human visual perception and how medical images should be interpreted have persisted since X-rays were the only imaging technique available. Concerns over rates of disagreement between expert image readers are associated with much of the clinical research and at times driven by the belief that any image endpoint variability is problematic. The deeper understanding of the reasons, value, and risk of disagreement are somewhat siloed, leading, at times, to costly and risky approaches, especially in clinical trials. Although artificial intelligence promises some relief from mistakes, its routine application for assessing tumors within cancer trials is still an aspiration. Our consortium of international experts in medical imaging for drug development research, the Pharma Imaging Network for Therapeutics and Diagnostics (PINTAD), tapped the collective knowledge of its members to ground expectations, summarize common reasons for reader discordance, identify what factors can be controlled and which actions are likely to be effective in reducing discordance. Reinforced by an exhaustive literature review, our work defines the forces that shape reader variability. This review article aims to produce a singular authoritative resource outlining reader performance's practical realities within cancer trials, whether they occur within a clinical or an independent central review.

Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance.

Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.

Cerebrospinal Fluid Biomarkers for Alzheimer's Disease: A View of the Regulatory Science Qualification Landscape from the Coalition Against Major Diseases CSF Biomarker Team.

Alzheimer's disease (AD) drug development is burdened with the current requirement to conduct large, lengthy, and costly trials to overcome uncertainty in patient progression and effect size on treatment outcome measures. There is an urgent need for the discovery, development, and implementation of novel, objectively measured biomarkers for AD that would aid selection of the appropriate subpopulation of patients in clinical trials, and presumably, improve the likelihood of successfully evaluating innovative treatment options. Amyloid deposition and tau in the brain, which are most commonly assessed either in cerebrospinal fluid (CSF) or by molecular imaging, are consistently and widely accepted. Nonetheless, a clear gap still exists in the accurate identification of subjects that truly have the hallmarks of AD. The Coalition Against Major Diseases (CAMD), one of 12 consortia of the Critical Path Institute (C-Path), aims to streamline drug development for AD and related dementias by advancing regulatory approved drug development tools for clinical trials through precompetitive data sharing and adoption of consensus clinical data standards. This report focuses on the regulatory process for biomarker qualification, briefly comments on how it contrasts with approval or clearance of companion diagnostics, details the qualifications currently available to the field of AD, and highlights the current challenges facing the landscape of CSF biomarkers qualified as hallmarks of AD. Finally, it recommends actions to accelerate regulatory qualification of CSF biomarkers that would, in turn, improve the efficiency of AD therapeutic development.

Spatially selective 2D RF inner field of view (iFOV) diffusion kurtosis imaging (DKI) of the pediatric spinal cord.

Magnetic resonance based diffusion imaging has been gaining more utility and clinical relevance over the past decade. Using conventional echo planar techniques, it is possible to acquire and characterize water diffusion within the central nervous system (CNS); namely in the form of Diffusion Weighted Imaging (DWI) and Diffusion Tensor Imaging (DTI). While each modality provides valuable clinical information in terms of the presence of diffusion and its directionality, both techniques are limited to assuming an ideal Gaussian distribution for water displacement with no intermolecular interactions. This assumption neglects pathological processes that are not Gaussian therefore reducing the amount of potentially clinically relevant information. Additions to the Gaussian distribution measured by the excess kurtosis, or peakedness, of the probabilistic model provide a better understanding of the underlying cellular structure. The objective of this work is to provide mathematical and experimental evidence that Diffusion Kurtosis Imaging (DKI) can offer additional information about the micromolecular environment of the pediatric spinal cord. This is accomplished by a more thorough characterization of the nature of random water displacement within the cord. A novel DKI imaging sequence based on a tilted 2D spatially selective radio frequency pulse providing reduced field of view (FOV) imaging was developed, implemented, and optimized on a 3 Tesla MRI scanner, and tested on pediatric subjects (healthy subjects: 15; patients with spinal cord injury (SCI):5). Software was developed and validated for post processing of the DKI images and estimation of the tensor parameters. The results show statistically significant differences in mean kurtosis (p < 0.01) and radial kurtosis (p < 0.01) between healthy subjects and subjects with SCI. DKI provides incremental and novel information over conventional diffusion acquisitions when coupled with higher order estimation algorithms.

Metrology Standards for Quantitative Imaging Biomarkers.

Although investigators in the imaging community have been active in developing and evaluating quantitative imaging biomarkers (QIBs), the development and implementation of QIBs have been hampered by the inconsistent or incorrect use of terminology or methods for technical performance and statistical concepts. Technical performance is an assessment of how a test performs in reference objects or subjects under controlled conditions. In this article, some of the relevant statistical concepts are reviewed, methods that can be used for evaluating and comparing QIBs are described, and some of the technical performance issues related to imaging biomarkers are discussed. More consistent and correct use of terminology and study design principles will improve clinical research, advance regulatory science, and foster better care for patients who undergo imaging studies.

Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment.

Technological developments and greater rigor in the quantitative measurement of biological features in medical images have given rise to an increased interest in using quantitative imaging biomarkers to measure changes in these features. Critical to the performance of a quantitative imaging biomarker in preclinical or clinical settings are three primary metrology areas of interest: measurement linearity and bias, repeatability, and the ability to consistently reproduce equivalent results when conditions change, as would be expected in any clinical trial. Unfortunately, performance studies to date differ greatly in designs, analysis method, and metrics used to assess a quantitative imaging biomarker for clinical use. It is therefore difficult or not possible to integrate results from different studies or to use reported results to design studies. The Radiological Society of North America and the Quantitative Imaging Biomarker Alliance with technical, radiological, and statistical experts developed a set of technical performance analysis methods, metrics, and study designs that provide terminology, metrics, and methods consistent with widely accepted metrological standards. This document provides a consistent framework for the conduct and evaluation of quantitative imaging biomarker performance studies so that results from multiple studies can be compared, contrasted, or combined.

The imaging core lab perspective on glioblastoma imaging and response assessment in clinical trials.

Multicenter clinical trials that include medical images as a key component of response assessment often involve an imaging service provider (a core laboratory or contract research organization) to collect images and often to provide independent assessments of treatment response. The brief discussion and recommendations provided here are not intended as a rigorous academic analysis but reflect the practical experience accumulated at one such institution, which has conducted the image collection and review for numerous glioblastoma trials, in every phase of drug development, encompassing over 4000 patients scanned at over 900 sites.

Coalition Against Major Diseases/European Medicines Agency biomarker qualification of hippocampal volume for enrichment of clinical trials in predementia stages of Alzheimer's disease.

BACKGROUND: Regulatory qualification of a biomarker for a defined context of use provides scientifically robust assurances to sponsors and regulators that accelerate appropriate adoption of biomarkers into drug development. METHODS: The Coalition Against Major Diseases submitted a dossier to the Scientific Advice Working Party of the European Medicines Agency requesting a qualification opinion on the use of hippocampal volume as a biomarker for enriching clinical trials in subjects with mild cognitive impairment, incorporating a scientific rationale, a literature review and a de novo analysis of Alzheimer's Disease Neuroimaging Initiative data. RESULTS: The literature review and de novo analysis were consistent with the proposed context of use, and the Committee for Medicinal Products for Human Use released an opinion in November 2011. CONCLUSIONS: We summarize the scientific rationale and the data that supported the first qualification of an imaging biomarker by the European Medicines Agency.

Robustness of automated hippocampal volumetry across magnetic resonance field strengths and repeat images.

BACKGROUND: Low HCV has recently been qualified by the European Medicines Agency as a biomarker for enrichment of clinical trials in predementia stages of Alzheimer's disease. For automated methods to meet the necessary regulatory requirements, it is essential they be standardized and their performance be well characterized. METHODS: The within-image and between-field strength reproducibility of automated hippocampal volumetry using the Learning Embeddings for Atlas Propagation (or LEAP) algorithm was assessed on 153 Alzheimer's Disease Neuroimaging Initiative subjects. RESULTS: Tests/retests at 1.5 T and 3 T, and a comparison between 1.5 T and 3 T, yielded average unsigned variabilities in HCVs of 1.51%, 1.52%, and 2.68%. A small bias between field strengths (mean signed difference, 1.17%; standard deviation, 3.07%) was observed. CONCLUSIONS: The measured reproducibility characteristics confirm the suitability of using automated magnetic resonance imaging analyses to assess HCVs quantitatively and to represent a fundamental characterization that is critical to meet the regulatory requirements for using hippocampal volumetry in clinical trials and health care.

A computational neurodegenerative disease progression score: method and results with the Alzheimer's disease Neuroimaging Initiative cohort.

While neurodegenerative diseases are characterized by steady degeneration over relatively long timelines, it is widely believed that the early stages are the most promising for therapeutic intervention, before irreversible neuronal loss occurs. Developing a therapeutic response requires a precise measure of disease progression. However, since the early stages are for the most part asymptomatic, obtaining accurate measures of disease progression is difficult. Longitudinal databases of hundreds of subjects observed during several years with tens of validated biomarkers are becoming available, allowing the use of computational methods. We propose a widely applicable statistical methodology for creating a disease progression score (DPS), using multiple biomarkers, for subjects with a neurodegenerative disease. The proposed methodology was evaluated for Alzheimer's disease (AD) using the publicly available AD Neuroimaging Initiative (ADNI) database, yielding an Alzheimer's DPS or ADPS score for each subject and each time-point in the database. In addition, a common description of biomarker changes was produced allowing for an ordering of the biomarkers. The Rey Auditory Verbal Learning Test delayed recall was found to be the earliest biomarker to become abnormal. The group of biomarkers comprising the volume of the hippocampus and the protein concentration amyloid beta and Tau were next in the timeline, and these were followed by three cognitive biomarkers. The proposed methodology thus has potential to stage individuals according to their state of disease progression relative to a population and to deduce common behaviors of biomarkers in the disease itself.

Disease progression modeling using Hidden Markov Models.

The development of novel treatments for many slowly progressing diseases, such as Alzheimer's disease (AD), is dependent on the ability to monitor and detect changes in disease progression. In some diseases the distinct clinical stages of the disease progress far too slowly to enable a quick evaluation of the efficacy of a given proposed treatment. To help improve the assessment of disease progression, we propose using Hidden Markov Models (HMM's) to model, in a more granular fashion, disease progression as compared to the clinical stages of the disease. Unlike many other applications of Hidden Markov Models, we train our HMM in an unsupervised way and then evaluate how effective the model is at uncovering underlying statistical patterns in disease progression by considering HMM states as disease stages. In this study, we focus on AD and show that our model, when evaluated on the cross validation data, can identify more granular disease stages than the three currently accepted clinical stages of "Normal", "MCI" (Mild Cognitive Impairment), and "AD".

Quantitative imaging test approval and biomarker qualification: interrelated but distinct activities.

UNLABELLED: Quantitative imaging biomarkers could speed the development of new treatments for unmet medical needs and improve routine clinical care. However, it is not clear how the various regulatory and nonregulatory (eg, reimbursement) processes (often referred to as pathways) relate, nor is it clear which data need to be collected to support these different pathways most efficiently, given the time- and cost-intensive nature of doing so. The purpose of this article is to describe current thinking regarding these pathways emerging from diverse stakeholders interested and active in the definition, validation, and qualification of quantitative imaging biomarkers and to propose processes to facilitate the development and use of quantitative imaging biomarkers. A flexible framework is described that may be adapted for each imaging application, providing mechanisms that can be used to develop, assess, and evaluate relevant biomarkers. From this framework, processes can be mapped that would be applicable to both imaging product development and to quantitative imaging biomarker development aimed at increasing the effectiveness and availability of quantitative imaging. SUPPLEMENTAL MATERIAL: http://radiology.rsna.org/lookup/suppl/doi:10.1148/radiol.10100800/-/DC1.