RESUMO
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Assuntos
Algoritmos , Processamento de Imagem Assistida por Computador , Aprendizado de Máquina , SemânticaRESUMO
PURPOSE: To evaluate the impact of an AI-based software trained to detect cerebral aneurysms on TOF-MRA on the diagnostic performance and reading times across readers with varying experience levels. METHODS: One hundred eighty-six MRI studies were reviewed by six readers to detect cerebral aneurysms. Initially, readings were assisted by the CNN-based software mdbrain. After 6 weeks, a second reading was conducted without software assistance. The results were compared to the consensus reading of two neuroradiological specialists and sensitivity (lesion and patient level), specificity (patient level), and false positives per case were calculated for the group of all readers, for the subgroup of physicians, and for each individual reader. Also, reading times for each reader were measured. RESULTS: The dataset contained 54 aneurysms. The readers had no experience (three medical students), 2 years experience (resident in neuroradiology), 6 years experience (radiologist), and 12 years (neuroradiologist). Significant improvements of overall specificity and the overall number of false positives per case were observed in the reading with AI support. For the physicians, we found significant improvements of sensitivity on lesion and patient level and false positives per case. Four readers experienced reduced reading times with the software, while two encountered increased times. CONCLUSION: In the reading with the AI-based software, we observed significant improvements in terms of specificity and false positives per case for the group of all readers and significant improvements of sensitivity and false positives per case for the physicians. Further studies are needed to investigate the effects of the AI-based software in a prospective setting.
Assuntos
Aneurisma Intracraniano , Angiografia por Ressonância Magnética , Sensibilidade e Especificidade , Software , Humanos , Aneurisma Intracraniano/diagnóstico por imagem , Angiografia por Ressonância Magnética/métodos , Masculino , Feminino , Pessoa de Meia-Idade , Competência Clínica , Interpretação de Imagem Assistida por Computador/métodos , Inteligência Artificial , Idoso , AdultoRESUMO
A crucial step in the clinical adaptation of an AI-based tool is an external, independent validation. The aim of this study was to investigate brain atrophy in patients with confirmed, progressed Huntington's disease using a certified software for automated volumetry and to compare the results with the manual measurement methods used in clinical practice as well as volume calculations of the caudate nuclei based on manual segmentations. Twenty-two patients were included retrospectively, consisting of eleven patients with Huntington's disease and caudate nucleus atrophy and an age- and sex-matched control group. To quantify caudate head atrophy, the frontal horn width to intercaudate distance ratio and the intercaudate distance to inner table width ratio were obtained. The software mdbrain was used for automated volumetry. Manually measured ratios and automatically measured volumes of the groups were compared using two-sample t-tests. Pearson correlation analyses were performed. The relative difference between automatically and manually determined volumes of the caudate nuclei was calculated. Both ratios were significantly different between the groups. The automatically and manually determined volumes of the caudate nuclei showed a high level of agreement with a mean relative discrepancy of - 2.3 ± 5.5%. The Huntington's disease group showed significantly lower volumes in a variety of supratentorial brain structures. The highest degree of atrophy was shown for the caudate nucleus, putamen, and pallidum (all p < .0001). The caudate nucleus volume and the ratios were found to be strongly correlated in both groups. In conclusion, in patients with progressed Huntington's disease, it was shown that the automatically determined caudate nucleus volume correlates strongly with measured ratios commonly used in clinical practice. Both methods allowed clear differentiation between groups in this collective. The software additionally allows radiologists to more objectively assess the involvement of a variety of brain structures that are less accessible to standard semiquantitative methods.
Assuntos
Núcleo Caudado , Aprendizado Profundo , Doença de Huntington , Humanos , Doença de Huntington/patologia , Doença de Huntington/diagnóstico por imagem , Masculino , Feminino , Pessoa de Meia-Idade , Núcleo Caudado/diagnóstico por imagem , Núcleo Caudado/patologia , Estudos Retrospectivos , Encéfalo/patologia , Encéfalo/diagnóstico por imagem , Atrofia/patologia , Imageamento por Ressonância Magnética/métodos , Adulto , Idoso , Software , Tamanho do Órgão , Processamento de Imagem Assistida por Computador/métodosRESUMO
The functionality of photoreceptors, rods, and cones is highly dependent on their outer segments (POS), a cellular compartment containing highly organized membranous structures that generate biochemical signals from incident light. While POS formation and degeneration are qualitatively assessed on microscopy images, reliable methodology for quantitative analyses is still limited. Here, we developed methods to quantify POS (QuaPOS) maturation and quality on retinal sections using automated image analyses. POS formation was examined during the development and in adulthood of wild-type mice via light microscopy (LM) and transmission electron microscopy (TEM). To quantify the number, size, shape, and fluorescence intensity of POS, retinal cryosections were immunostained for the cone POS marker S-opsin. Fluorescence images were used to train the robust classifier QuaPOS-LM based on supervised machine learning for automated image segmentation. Characteristic features of segmentation results were extracted to quantify the maturation of cone POS. Subsequently, this quantification method was applied to characterize POS degeneration in "cone photoreceptor function loss 1" mice. TEM images were used to establish the ultrastructural quantification method QuaPOS-TEM for the alignment of POS membranes. Images were analyzed using a custom-written MATLAB code to extract the orientation of membranes from the image gradient and their alignment (coherency). This analysis was used to quantify the POS morphology of wild-type and two inherited retinal degeneration ("retinal degeneration 19" and "rhodopsin knock-out") mouse lines. Both automated analysis technologies provided robust characterization and quantification of POS based on LM or TEM images. Automated image segmentation by the classifier QuaPOS-LM and analysis of the orientation of membrane stacks by QuaPOS-TEM using fluorescent or TEM images allowed quantitative evaluation of POS formation and quality. The assessments showed an increase in POS number, volume, and membrane coherency during wild-type postnatal development, while a decrease in all three observables was detected in different retinal degeneration mouse models. All the code used for the presented analysis is open source, including example datasets to reproduce the findings. Hence, the QuaPOS quantification methods are useful for in-depth characterization of POS on retinal sections in developmental studies, for disease modeling, or after therapeutic interventions affecting photoreceptors.
RESUMO
The aim of this study was to employ artificial intelligence (AI)-based magnetic resonance imaging (MRI) brain volumetry to potentially distinguish between idiopathic normal pressure hydrocephalus (iNPH), Alzheimer's disease (AD), and age- and sex-matched healthy controls (CG) by evaluating cortical, subcortical, and ventricular volumes. Additionally, correlations between the measured brain and ventricle volumes and two established semi-quantitative radiologic markers for iNPH were examined. An IRB-approved retrospective analysis was conducted on 123 age- and sex-matched subjects (41 iNPH, 41 AD, and 41 controls), with all of the iNPH patients undergoing routine clinical brain MRI prior to ventriculoperitoneal shunt implantation. Automated AI-based determination of different cortical and subcortical brain and ventricular volumes in mL, as well as calculation of population-based normalized percentiles according to an embedded database, was performed; the CE-certified software mdbrain v4.4.1 or above was used with a standardized T1-weighted 3D magnetization-prepared rapid gradient echo (MPRAGE) sequence. Measured brain volumes and percentiles were analyzed for between-group differences and correlated with semi-quantitative measurements of the Evans' index and corpus callosal angle: iNPH patients exhibited ventricular enlargement and changes in gray and white matter compared to AD patients and controls, with the most significant differences observed in total ventricular volume (+67%) and the lateral (+68%), third (+38%), and fourth (+31%) ventricles compared to controls. Global ventriculomegaly and marked white matter reduction with concomitant preservation of gray matter compared to AD and CG were characteristic of iNPH, whereas global and frontoparietally accentuated gray matter reductions were characteristic of AD. Evans' index and corpus callosal angle differed significantly between the three groups and moderately correlated with the lateral ventricular volumes in iNPH patients [Evans' index (r > 0.83, p ≤ 0.001), corpus callosal angle (r < -0.74, p ≤ 0.001)]. AI-based MRI volumetry in iNPH patients revealed global ventricular enlargement and focal brain atrophy, which, in contrast to healthy controls and AD patients, primarily involved the supratentorial white matter and was marked temporomesially and in the midbrain, while largely preserving gray matter. Integrating AI volumetry in conjunction with traditional radiologic measures could enhance iNPH identification and differentiation, potentially improving patient management and therapy response assessment.
RESUMO
OBJECTIVES: Reducing gadolinium-based contrast agents to lower costs, the environmental impact of gadolinium-containing wastewater, and patient exposure is still an unresolved issue. Published methods have never been compared. The purpose of this study was to compare the performance of 2 reimplemented state-of-the-art deep learning methods (settings A and B) and a proposed method for contrast signal extraction (setting C) to synthesize artificial T1-weighted full-dose images from corresponding noncontrast and low-dose images. MATERIALS AND METHODS: In this prospective study, 213 participants received magnetic resonance imaging of the brain between August and October 2021 including low-dose (0.02 mmol/kg) and full-dose images (0.1 mmol/kg). Fifty participants were randomly set aside as test set before training (mean age ± SD, 52.6 ± 15.3 years; 30 men). Artificial and true full-dose images were compared using a reader-based study. Two readers noted all false-positive lesions and scored the overall interchangeability in regard to the clinical conclusion. Using a 5-point Likert scale (0 being the worst), they scored the contrast enhancement of each lesion and its conformity to the respective reference in the true image. RESULTS: The average counts of false-positives per participant were 0.33 ± 0.93, 0.07 ± 0.33, and 0.05 ± 0.22 for settings A-C, respectively. Setting C showed a significantly higher proportion of scans scored as fully or mostly interchangeable (70/100) than settings A (40/100, P < 0.001) and B (57/100, P < 0.001), and generated the smallest mean enhancement reduction of scored lesions (-0.50 ± 0.55) compared with the true images (setting A: -1.10 ± 0.98; setting B: -0.91 ± 0.67, both P < 0.001). The average scores of conformity of the lesion were 1.75 ± 1.07, 2.19 ± 1.04, and 2.48 ± 0.91 for settings A-C, respectively, with significant differences among all settings (all P < 0.001). CONCLUSIONS: The proposed method for contrast signal extraction showed significant improvements in synthesizing postcontrast images. A relevant proportion of images showing inadequate interchangeability with the reference remains at this dosage.
RESUMO
AIM: To determine the effectiveness of functional stress testing and computed tomography angiography (CTA) for diagnosis of obstructive coronary artery disease (CAD). METHODS AND RESULTS: Two-thousand nine-hundred twenty symptomatic stable chest pain patients were included in the international Collaborative Meta-Analysis of Cardiac CT consortium to compare CTA with exercise electrocardiography (exercise-ECG) and single-photon emission computed tomography (SPECT) for diagnosis of CAD defined as ≥ 50% diameter stenosis by invasive coronary angiography (ICA) as reference standard. Generalised linear mixed models were used for calculating the diagnostic accuracy of each diagnostic test including non-diagnostic results as dependent variables in a logistic regression model with random intercepts and slopes. Covariates were the reference standard ICA, the type of diagnostic method, and their interactions. CTA showed significantly better diagnostic performance (p < 0.0001) with a sensitivity of 94.6% (95% CI 92.7-96) and a specificity of 76.3% (72.2-80) compared to exercise-ECG with 54.9% (47.9-61.7) and 60.9% (53.4-66.3), SPECT with 72.9% (65-79.6) and 44.9% (36.8-53.4), respectively. The positive predictive value of CTA was ≥ 50% in patients with a clinical pretest probability of 10% or more while this was the case for ECG and SPECT at pretest probabilities of ≥ 40 and 28%. CTA reliably excluded obstructive CAD with a post-test probability of below 15% in patients with a pretest probability of up to 74%. CONCLUSION: In patients with stable chest pain, CTA is more effective than functional testing for the diagnosis as well as for reliable exclusion of obstructive CAD. CTA should become widely adopted in patients with intermediate pretest probability. SYSTEMATIC REVIEW REGISTRATION: PROSPERO Database for Systematic Reviews-CRD42012002780. CRITICAL RELEVANCE STATEMENT: In symptomatic stable chest pain patients, coronary CTA is more effective than functional testing for diagnosis and reliable exclusion of obstructive CAD in intermediate pretest probability of CAD. KEY POINTS: Coronary computed tomography angiography showed significantly better diagnostic performance (p < 0.0001) for diagnosis of coronary artery disease compared to exercise-ECG and SPECT. The positive predictive value of coronary computed tomography angiography was ≥ 50% in patients with a clinical pretest probability of at least 10%, for ECG ≥ 40%, and for SPECT 28%. Coronary computed tomography angiography reliably excluded obstructive coronary artery disease with a post-test probability of below 15% in patients with a pretest probability of up to 74%.