RESUMO
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.
Assuntos
Inteligência ArtificialRESUMO
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Assuntos
Algoritmos , Processamento de Imagem Assistida por Computador , Aprendizado de Máquina , SemânticaRESUMO
Research has identified clinical, genomic, and neurophysiological markers associated with suicide attempts (SA) among individuals with psychiatric illness. However, there is limited research among those with an alcohol use disorder (AUD), despite their disproportionately higher rates of SA. We examined lifetime SA in 4,068 individuals with DSM-IV alcohol dependence from the Collaborative Study on the Genetics of Alcoholism (23% lifetime suicide attempt; 53% female; 17% Admixed African American ancestries; mean age: 38). We 1) conducted a genome-wide association study (GWAS) of SA and performed downstream analyses to determine whether we could identify specific biological pathways of risk, and 2) explored risk in aggregate across other clinical conditions, polygenic scores (PGS) for comorbid psychiatric problems, and neurocognitive functioning between those with AD who have and have not reported a lifetime suicide attempt. The GWAS and downstream analyses did not produce any significant associations. Participants with an AUD who had attempted suicide had greater rates of trauma exposure, major depressive disorder, post-traumatic stress disorder, and other substance use disorders compared to those who had not attempted suicide. Polygenic scores for suicide attempt, depression, and PTSD were associated with reporting a suicide attempt (ORs = 1.22-1.44). Participants who reported a SA also had decreased right hemispheric frontal-parietal theta and decreased interhemispheric temporal-parietal alpha electroencephalogram resting-state coherences relative to those who did not, but differences were small. Overall, individuals with alcohol dependence who report SA appear to experience a variety of severe comorbidities and elevated polygenic risk for SA. Our results demonstrate the need to further investigate suicide attempts in the presence of substance use disorders.