AI in imaging: the regulatory landscape.

Artificial intelligence (AI) methods have been applied to medical imaging for several decades, but in the last few years, the number of publications and the number of AI-enabled medical devices coming on the market have significantly increased. While some AI-enabled approaches are proving very valuable, systematic reviews of the AI imaging field identify significant weaknesses in a significant proportion of the literature. Medical device regulators have recently become more proactive in publishing guidance documents and recognizing standards that will require that the development and validation of AI-enabled medical devices need to be more rigorous than required for tradition "rule-based" software. In particular, developers are required to better identify and mitigate risks (such as bias) that arise in AI-enabled devices, and to ensure that the devices are validated in a realistic clinical setting to ensure their output is clinically meaningful. While this evolving regulatory landscape will mean that device developers will take longer to bring novel AI-based medical imaging devices to market, such additional rigour is necessary to address existing weaknesses in the field and ensure that patients and healthcare professionals can trust AI-enabled devices. There would also be benefits in the academic community taking into account this regulatory framework, to improve the quality of the literature and make it easier for academically developed AI tools to make the transition to medical devices that impact healthcare.

Identifying and characterising sources of variability in digital outcome measures in Parkinson's disease.

Smartphones and wearables are widely recognised as the foundation for novel Digital Health Technologies (DHTs) for the clinical assessment of Parkinson's disease. Yet, only limited progress has been made towards their regulatory acceptability as effective drug development tools. A key barrier in achieving this goal relates to the influence of a wide range of sources of variability (SoVs) introduced by measurement processes incorporating DHTs, on their ability to detect relevant changes to PD. This paper introduces a conceptual framework to assist clinical research teams investigating a specific Concept of Interest within a particular Context of Use, to identify, characterise, and when possible, mitigate the influence of SoVs. We illustrate how this conceptual framework can be applied in practice through specific examples, including two data-driven case studies.

A novel activated-zinc antiseptic solution effective against MRSA and Pseudomonas aeruginosa: a pig model.

OBJECTIVE: The purpose of this study was to investigate the antimicrobial efficacy of a novel activated zinc solution against meticillin-resistant Staphylococcus aureus (MRSA) and Pseudomonas aeruginosa after one hour, and to evaluate any untoward effect of the solution on local wound tissue at 24 hours after solution exposure in a pig wound model. METHOD: A pathogen-free, commercially raised, Yorkshire-cross female pig was acquired 12 days prior to the procedure. Within one week prior to the procedure, a small loopful of test bacteria, Pseudomonas aeruginosa (pig-isolate) and MRSA (ATCC-6538), were streaked and cultured on a non-selective agar. Full-thickness wounds (n=24) were created and evenly divided into three groups: control wounds (exposed to bacteria but untreated, n=8); wounds treated with Compound 1 (n=8), and wounds treated with Compound 2 (n=8). All wounds were dressed and monitored for one hour and 24 hours. RESULTS: After one hour, the wounds treated with Compound 1 and Compound 2 had a mean recoverable total bacteria of 2.8 log colony forming units (CFUs) and 3.5 logCFUs, respectively. After one hour, the wounds treated with Compound 1 and Compound 2 had a mean recoverable MRSA of 2.3 logCFUs and 1.6 logCFUs, respectively (p=0.009). After one hour, the wounds treated with Compound 1 and Compound 2 had a mean recoverable Pseudomonas aeruginosa of 0.3 logCFUs and 0.0 logCFUs, respectively (p=0.000). After 24 hours of exposure to Compound 1 and Compound 2, there was no statistically significant increased necrosis (p=0.12, p=0.31, respectively) or neutrophilic infiltrate (Compound 2, p=0.12) when compared with control wounds. CONCLUSION: The novel activated-zinc compound used in this study demonstrated a 99.5-99.9% reduction in total bacteria, a 99.9-99.98% reduction in MRSA, and 100% eradication of Pseudomonas aeruginosa one hour after exposure. This novel solution may provide another significant tool to treat and/or prevent wound infections.

A Novel Activated Zinc Solution with Improved Efficacy Against Pseudomonas and MRSA Biofilm Compared to Chlorhexidine and Povidone-Iodine.

INTRODUCTION: The search for the optimal agent for infection eradication in periprosthetic joint infection (PJI) remains challenging as there are limited efficacious and safe options. The ideal solution should have significant bactericidal and anti-biofilm activity to be able to eradicate infection with the preservation of prosthetic components. Therefore, the purpose of this study was to 1) investigate the anti-biofilm efficacy of a novel activated zinc solution against Pseudomonas aeruginosa and methicillin-resistant Staphylococcus aureus (MRSA) biofilm in vitro and 2) compare its efficacy against two leading commercially available antimicrobial irrigants (CHG and 0.35% povidone-iodine [PI]). MATERIALS AND METHODS: A modified Robbins device (MRD) was utilized to replicate Pseudomonas aeruginosa and MRSA biofilms. The primary outcome was to determine bacterial reduction after two hours of biofilm exposure to an activated zinc solution, CHG, and PI, and compare to untreated controls. RESULTS: Against Pseudomonas biofilm, activated zinc demonstrated a 4.5-log (99.996%) reduction, chlorhexidine demonstrated a 0.9-log (87.4%) reduction (p<0.001), and PI demonstrated a 0.8-log (83.1%) reduction (p<0.001). After two hours of exposure, activated zinc had undetectable MRSA with a 7.08-log (100%) reduction, chlorhexidine had a 1.9-log (98.7%) reduction (p<0.01), and PI had a 3.2-log (99.9%) reduction (p<0.01). CONCLUSIONS: Our novel activated zinc compound demonstrated a 99.996% reduction in Pseudomonas biofilm and a 100% reduction in MRSA biofilm. This novel solution may provide a significant tool in the arsenal to treat and/or prevent PJI and other wound infections. Future in vivo studies are warranted to demonstrate clinical utility, efficacy, and safety.

Metadata Framework to Support Deployment of Digital Health Technologies in Clinical Trials in Parkinson's Disease.

Sensor data from digital health technologies (DHTs) used in clinical trials provides a valuable source of information, because of the possibility to combine datasets from different studies, to combine it with other data types, and to reuse it multiple times for various purposes. To date, there exist no standards for capturing or storing DHT biosensor data applicable across modalities and disease areas, and which can also capture the clinical trial and environment-specific aspects, so-called metadata. In this perspectives paper, we propose a metadata framework that divides the DHT metadata into metadata that is independent of the therapeutic area or clinical trial design (concept of interest and context of use), and metadata that is dependent on these factors. We demonstrate how this framework can be applied to data collected with different types of DHTs deployed in the WATCH-PD clinical study of Parkinson's disease. This framework provides a means to pre-specify and therefore standardize aspects of the use of DHTs, promoting comparability of DHTs across future studies.

A standard system phantom for magnetic resonance imaging.

PURPOSE: A standard MRI system phantom has been designed and fabricated to assess scanner performance, stability, comparability and assess the accuracy of quantitative relaxation time imaging. The phantom is unique in having traceability to the International System of Units, a high level of precision, and monitoring by a national metrology institute. Here, we describe the phantom design, construction, imaging protocols, and measurement of geometric distortion, resolution, slice profile, signal-to-noise ratio (SNR), proton-spin relaxation times, image uniformity and proton density. METHODS: The system phantom, designed by the International Society of Magnetic Resonance in Medicine ad hoc committee on Standards for Quantitative MR, is a 200 mm spherical structure that contains a 57-element fiducial array; two relaxation time arrays; a proton density/SNR array; resolution and slice-profile insets. Standard imaging protocols are presented, which provide rapid assessment of geometric distortion, image uniformity, T1 and T2 mapping, image resolution, slice profile, and SNR. RESULTS: Fiducial array analysis gives assessment of intrinsic geometric distortions, which can vary considerably between scanners and correction techniques. This analysis also measures scanner/coil image uniformity, spatial calibration accuracy, and local volume distortion. An advanced resolution analysis gives both scanner and protocol contributions. SNR analysis gives both temporal and spatial contributions. CONCLUSIONS: A standard system phantom is useful for characterization of scanner performance, monitoring a scanner over time, and to compare different scanners. This type of calibration structure is useful for quality assurance, benchmarking quantitative MRI protocols, and to transition MRI from a qualitative imaging technique to a precise metrology with documented accuracy and uncertainty.

Quantitative magnetic resonance imaging phantoms: A review and the need for a system phantom.

The MRI community is using quantitative mapping techniques to complement qualitative imaging. For quantitative imaging to reach its full potential, it is necessary to analyze measurements across systems and longitudinally. Clinical use of quantitative imaging can be facilitated through adoption and use of a standard system phantom, a calibration/standard reference object, to assess the performance of an MRI machine. The International Society of Magnetic Resonance in Medicine AdHoc Committee on Standards for Quantitative Magnetic Resonance was established in February 2007 to facilitate the expansion of MRI as a mainstream modality for multi-institutional measurements, including, among other things, multicenter trials. The goal of the Standards for Quantitative Magnetic Resonance committee was to provide a framework to ensure that quantitative measures derived from MR data are comparable over time, between subjects, between sites, and between vendors. This paper, written by members of the Standards for Quantitative Magnetic Resonance committee, reviews standardization attempts and then details the need, requirements, and implementation plan for a standard system phantom for quantitative MRI. In addition, application-specific phantoms and implementation of quantitative MRI are reviewed. Magn Reson Med 79:48-61, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

Enrichment of clinical trials in MCI due to AD using markers of amyloid and neurodegeneration.

OBJECTIVE: To investigate the effect of enriching mild cognitive impairment (MCI) clinical trials using combined markers of amyloid pathology and neurodegeneration. METHODS: We evaluate an implementation of the recent National Institute for Aging-Alzheimer's Association (NIA-AA) diagnostic criteria for MCI due to Alzheimer disease (AD) as inclusion criteria in clinical trials and assess the effect of enrichment with amyloid (A+), neurodegeneration (N+), and their combination (A+N+) on the rate of clinical progression, required sample sizes, and estimates of trial time and cost. RESULTS: Enrichment based on an individual marker (A+ or N+) substantially improves all assessed trial characteristics. Combined enrichment (A+N+) further improves these results with a reduction in required sample sizes by 45% to 60%, depending on the endpoint. CONCLUSIONS: Operationalizing the NIA-AA diagnostic criteria for clinical trial screening has the potential to substantially improve the statistical power of trials in MCI due to AD by identifying a more rapidly progressing patient population.

Amyloid-ß 11C-PiB-PET imaging results from 2 randomized bapineuzumab phase 3 AD trials.

OBJECTIVE: To evaluate the effects of bapineuzumab on brain ß-amyloid (Aß) burden using (11)C-Pittsburgh compound B ((11)C-PiB)-PET. METHODS: Two phase 3 clinical trials, 1 each in apolipoprotein APOE ε4 carriers and noncarriers, were conducted in patients with mild to moderate Alzheimer disease dementia. Bapineuzumab, an anti-Aß monoclonal antibody, or placebo, was administered by IV infusion every 13 weeks for 78 weeks. PET substudies assessed change in brain fibrillar Aß over 71 weeks using an (11)C-PiB-PET standardized uptake value ratio (SUVr) global cortical average (GCA) comprising the average SUVr from 5 cortical regions of interest with cerebellar gray matter as the reference region. RESULTS: A total of 115 carriers and 39 noncarriers were analyzed. The difference (δ) in mean baseline to 71 week change in (11)C-PiB-PET GCA between bapineuzumab and placebo was significant in carriers (0.5 mg/kg vs placebo δ = -0.101; p = 0.004) and in pooled analyses of both carriers and noncarriers (0.5 mg/kg vs placebo δ = -0.068; p = 0.027; 1.0 mg/kg vs placebo δ = -0.133; p = 0.028) but not in the noncarrier trial separately. Analyses by individual region of interest and in mild disease yielded findings similar to the main trial results. CONCLUSIONS: The (11)C-PiB-PET imaging results demonstrated reduction of fibrillar Aß accumulation in patients with Alzheimer disease treated with bapineuzumab; however, as no clinical benefit was observed, the findings are consistent with the hypotheses that bapineuzumab may not have been initiated early enough in the disease course, the doses were insufficient, or the most critical Aß species were inadequately targeted.

Design, operation, and safety of single-room interventional MRI suites: practical experience from two centers.

The design and operation of a facility in which a magnetic resonance imaging (MRI) scanner is incorporated into a room used for surgical or endovascular cardiac interventions presents several challenges. MR safety must be maintained in the presence of a much wider variety of equipment than is found in a diagnostic unit, and of staff unfamiliar with the MRI environment, without compromising the safety and practicality of the interventional procedure. Both the MR-guided cardiac interventional unit at Kings College London and the intraoperative imaging suite at the National Hospital for Neurology and Neurosurgery are single-room interventional facilities incorporating 1.5 T cylindrical-bore MRI scanners. The two units employ similar strategies to maintain MR safety, both in original design and day-to-day operational workflows, and between them over a decade of incident-free practice has been accumulated. This article outlines these strategies, highlighting both similarities and differences between the units, as well as some lessons learned and resulting procedural changes made in both units since installation.

Coalition Against Major Diseases/European Medicines Agency biomarker qualification of hippocampal volume for enrichment of clinical trials in predementia stages of Alzheimer's disease.

BACKGROUND: Regulatory qualification of a biomarker for a defined context of use provides scientifically robust assurances to sponsors and regulators that accelerate appropriate adoption of biomarkers into drug development. METHODS: The Coalition Against Major Diseases submitted a dossier to the Scientific Advice Working Party of the European Medicines Agency requesting a qualification opinion on the use of hippocampal volume as a biomarker for enriching clinical trials in subjects with mild cognitive impairment, incorporating a scientific rationale, a literature review and a de novo analysis of Alzheimer's Disease Neuroimaging Initiative data. RESULTS: The literature review and de novo analysis were consistent with the proposed context of use, and the Committee for Medicinal Products for Human Use released an opinion in November 2011. CONCLUSIONS: We summarize the scientific rationale and the data that supported the first qualification of an imaging biomarker by the European Medicines Agency.

Operationalizing hippocampal volume as an enrichment biomarker for amnestic mild cognitive impairment trials: effect of algorithm, test-retest variability, and cut point on trial cost, duration, and sample size.

The objective of this study was to evaluate the effect of computational algorithm, measurement variability, and cut point on hippocampal volume (HCV)-based patient selection for clinical trials in mild cognitive impairment (MCI). We used normal control and amnestic MCI subjects from the Alzheimer's Disease Neuroimaging Initiative 1 (ADNI-1) as normative reference and screening cohorts. We evaluated the enrichment performance of 4 widely used hippocampal segmentation algorithms (FreeSurfer, Hippocampus Multi-Atlas Propagation and Segmentation (HMAPS), Learning Embeddings Atlas Propagation (LEAP), and NeuroQuant) in terms of 2-year changes in Mini-Mental State Examination (MMSE), Alzheimer's Disease Assessment Scale-Cognitive Subscale (ADAS-Cog), and Clinical Dementia Rating Sum of Boxes (CDR-SB). We modeled the implications for sample size, screen fail rates, and trial cost and duration. HCV based patient selection yielded reduced sample sizes (by ∼40%-60%) and lower trial costs (by ∼30%-40%) across a wide range of cut points. These results provide a guide to the choice of HCV cut point for amnestic MCI clinical trials, allowing an informed tradeoff between statistical and practical considerations.

Steps to standardization and validation of hippocampal volumetry as a biomarker in clinical trials and diagnostic criterion for Alzheimer's disease.

BACKGROUND: The promise of Alzheimer's disease biomarkers has led to their incorporation in new diagnostic criteria and in therapeutic trials; however, significant barriers exist to widespread use. Chief among these is the lack of internationally accepted standards for quantitative metrics. Hippocampal volumetry is the most widely studied quantitative magnetic resonance imaging measure in Alzheimer's disease and thus represents the most rational target for an initial effort at standardization. METHODS AND RESULTS: The authors of this position paper propose a path toward this goal. The steps include the following: (1) Establish and empower an oversight board to manage and assess the effort, (2) adopt the standardized definition of anatomic hippocampal boundaries on magnetic resonance imaging arising from the European Alzheimer's Disease Centers-Alzheimer's Disease Neuroimaging Initiative hippocampal harmonization effort as a reference standard, (3) establish a scientifically appropriate, publicly available reference standard data set based on manual delineation of the hippocampus in an appropriate sample of subjects (Alzheimer's Disease Neuroimaging Initiative), and (4) define minimum technical and prognostic performance metrics for validation of new measurement techniques using the reference standard data set as a benchmark. CONCLUSIONS: Although manual delineation of the hippocampus is the best available reference standard, practical application of hippocampal volumetry will require automated methods. Our intent was to establish a mechanism for credentialing automated software applications to achieve internationally recognized accuracy and prognostic performance standards that lead to the systematic evaluation and then widespread acceptance and use of hippocampal volumetry. The standardization and assay validation process outlined for hippocampal volumetry was envisioned as a template that could be applied to other imaging biomarkers.

Comparison of phantom and registration scaling corrections using the ADNI cohort.

Rates of brain atrophy derived from serial magnetic resonance (MR) studies may be used to assess therapies for Alzheimer's disease (AD). These measures may be confounded by changes in scanner voxel sizes. For this reason, the Alzheimer's Disease Neuroimaging Initiative (ADNI) included the imaging of a geometric phantom with every scan. This study compares voxel scaling correction using a phantom with correction using a 9 degrees of freedom (9DOF) registration algorithm. We took 129 pairs of baseline and 1-year repeat scans, and calculated the volume scaling correction, previously measured using the phantom. We used the registration algorithm to quantify any residual scaling errors, and found the algorithm to be unbiased, with no significant (p=0.97) difference between control (n=79) and AD subjects (n=50), but with a mean (SD) absolute volume change of 0.20 (0.20) % due to linear scalings. 9DOF registration was shown to be comparable to geometric phantom correction in terms of the effect on atrophy measurement and unbiased with respect to disease status. These results suggest that the additional expense and logistic effort of scanning a phantom with every patient scan can be avoided by registration-based scaling correction. Furthermore, based upon the atrophy rates in the AD subjects in this study, sample size requirements would be approximately 10-12% lower with (either) correction for voxel scaling than if no correction was used.

Longitudinal regional brain volume changes quantified in normal aging and Alzheimer's APP x PS1 mice using MRI.

In humans, mutations of amyloid precursor protein (APP) and presenilins (PS) 1 and 2 are associated with amyloid deposition, brain structural change and cognitive decline, like in Alzheimer's disease (AD). Mice expressing these proteins have illuminated neurodegenerative disease processes but, unlike in humans, quantitative imaging has been little used to systematically determine their effects, or those of normal aging, on brain structure in vivo. Accordingly, we investigated wildtype (WT) and TASTPM mice (expressing human APP(695(K595N, M596L)) x PS1(M146V)) longitudinally using MRI. Automated global and local image registration, allied to a standard digital atlas, provided pairwise segmentation of 13 brain regions. We found the mature mouse brain, unlike in humans, enlarges significantly from 6-14 months old (WT 3.8+/-1.7%, mean+/-SD, P<0.0001). Significant changes were also seen in other WT brain regions, providing an anatomical benchmark for comparing other mouse strains and models of brain disorder. In TASTPM, progressive amyloidosis and astrogliosis, detected immunohistochemically, reflected even larger whole brain changes (5.1+/-1.4%, P<0.0001, transgenexage interaction P=0.0311). Normalising regional volumes to whole brain measurements revealed significant, prolonged, WT-TASTPM volume differences, suggesting transgene effects establish at <6 months old of age in most regions. As in humans, gray matter-rich regions decline with age (e.g. thalamus, cerebral cortex and caudoputamen); ventricles and white matter (corpus callosum, corticospinal tract, fornix system) increase; in TASTPMs such trends often varied significantly from WT (especially hippocampus). The pervasive, age-related structural changes between WT and AD transgenic mice (and mouse and human) suggest subtle but fundamental species differences and AD transgene effects.

Publication rate of abstracts presented at the shoulder and elbow session of the American Academy of Orthopaedic Surgery.

Many shoulder and elbow abstracts presented at the American Academy of Orthopaedic Surgeons (AAOS) annual meeting are cited in the orthopaedic literature or are used to guide orthopaedic practice, but not all of these abstracts are submitted, survive peer review, or eventually are published. Presuming unpublished works have not been scientifically confirmed, one could question whether it is academically responsible to cite abstracts presented at the AAOS before they are peer-reviewed and published. To partly address this issue we determined the peer-reviewed publication rate for 558 abstracts (233 papers and 325 posters) presented at the shoulder and elbow sessions of the AAOS from 1999 to 2004. In April 2007, we searched the computerized database MEDLINE and PubMed for published articles based on these abstracts. We examined the published articles to assess publication rate, time to publication, change in contents, change in authors, and change in conclusions of abstracts. The overall publication rate in peer-reviewed journals was 58% (321 of 558), similar to other orthopaedic meetings and medical disciplines. We believe it is unacceptable to cite shoulder and elbow abstracts submitted to the AAOS because only slightly more than (1/2) (58%) of them are authenticated scientifically.

Analysis of serial magnetic resonance images of mouse brains using image registration.

The aim of this paper is to investigate techniques that can identify and quantify cross-sectional differences and longitudinal changes in vivo from magnetic resonance images of murine models of brain disease. Two different approaches have been compared. The first approach is a segmentation-based approach: Each subject at each time point is automatically segmented into a number of anatomical structures using atlas-based segmentation. This allows cross-sectional and longitudinal analyses of group differences on a structure-by-structure basis. The second approach is a deformation-based approach: Longitudinal changes are quantified by the registration of each subject's follow-up images to that subject's baseline image. In addition the baseline images can be registered to an atlas allowing voxel-wise analysis of cross-sectional differences between groups. Both approaches have been tested on two groups of mice: A transgenic model of Alzheimer's disease and a wild-type background strain, using serial imaging performed over the age range from 6-14 months. We show that both approaches are able to identify longitudinal and cross-sectional differences. However, atlas-based segmentation suffers from the inability to detect differences across populations and across time in regions which are much smaller than the anatomical regions. In contrast to this, the deformation-based approach can detect statistically significant differences in highly localized areas.

Issues with threshold masking in voxel-based morphometry of atrophied brains.

There is great interest in using automatic computational neuroanatomy tools to study ageing and neurodegenerative disease. Voxel-based morphometry (VBM) is one of the most widely used of such techniques. VBM performs voxel-wise statistical analysis of smoothed spatially normalised segmented Magnetic Resonance Images. There are several reasons why the analysis should include only voxels within a certain mask. We show that one of the most commonly used strategies for defining this mask runs a major risk of excluding from the analysis precisely those voxels where the subjects' brains were most vulnerable to atrophy. We investigate the issues related to mask construction, and recommend the use of alternative strategies which greatly decrease this danger of false negatives.

Accuracy assessment of global and local atrophy measurement techniques with realistic simulated longitudinal Alzheimer's disease images.

The evaluation of atrophy quantification methods based on magnetic resonance imaging have been usually hindered by the lack of realistic gold standard data against which to judge these methods or to help refine them. Recently [Camara, O., Schweiger, M., Scahill, R., Crum, W., Sneller, B., Schnabel, J., Ridgway, G., Cash, D., Hill, D., Fox, N., 2006. Phenomenological model of diffuse global and regional atrophy using finite-element methods. IEEE Trans. Med.l Imaging 25, 1417-1430], we presented a technique in which atrophy is realistically simulated in different tissue compartments or neuroanatomical structures with a phenomenological model. In this study, we have generated a cohort of realistic simulated Alzheimer's disease (AD) images with known amounts of atrophy, mimicking a set of 19 real controls and 27 probable AD subjects, with an improved version of our atrophy simulation methodology. This database was then used to assess the accuracy of several well-known computational anatomy methods which provide global (BSI and SIENA) or local (Jacobian integration) estimates of longitudinal atrophy in brain structures using MR images. SIENA and BSI results correlated very well with gold standard data (Pearson coefficient of 0.962 and 0.969 respectively), achieving small mean absolute differences with respect to the gold standard (percentage change from baseline volume): BSI of 0.23%+/-0.26%; SIENA of 0.22%+/-0.28%. Jacobian integration was guided by both fluid and FFD-based registration techniques and resulting deformation fields and associated Jacobians were compared, region by region, with gold standard ones. The FFD-based technique outperformed the fluid one in all evaluated structures (mean absolute differences from the gold standard in percentage change from baseline volume): whole brain, FFD=0.31%, fluid=0.58%; lateral ventricles, FFD=0.79%; fluid=1.45%; left hippocampus, FFD=0.82%; fluid=1.42%; right hippocampus, FFD=0.95%; fluid=1.62%. The largest errors for both local techniques occurred in the sulcal CSF (FFD=2.27%; fluid=3.55%) regions. For large structures such as the whole brain, these mean absolute differences, relative to the applied atrophy, represented similar percentages for the BSI, SIENA and FFD techniques (controls/patients): BSI, 51.99%/16.36%; SIENA, 62.34%/21.59%; FFD, 41.02%/24.95%. For small structures such as the hippocampi, these percentages were larger, especially for controls where errors were approximately equal to the small applied changes (controls/patients): FFD, 92.82%/43.61%. However, these apparently large relative errors have not prevented the global or hippocampal measures from finding significant group separation in our study. The evaluation framework presented here will help in quantifying whether the accuracy of future methodological developments is sufficient for analysing change in smaller or less atrophied local brain regions. Results obtained in our experiments with realistic simulated data confirm previously published estimates of accuracy for both evaluated global techniques. Regarding Jacobian Integration methods, the FFD-based one demonstrated promising results and potential for being used in clinical studies alongside (or in place of) the more common global methods. The generated gold standard data has also allowed us to identify some stages and sets of parameters in the evaluated techniques--the brain extraction step in the global techniques and the number of multi-resolution levels and the stopping criteria in the registration-based methods--that are critical for their accuracy.