Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Clinical, genomic, and neurophysiological correlates of lifetime suicide attempts among individuals with alcohol dependence.

Research has identified clinical, genomic, and neurophysiological markers associated with suicide attempts (SA) among individuals with psychiatric illness. However, there is limited research among those with an alcohol use disorder (AUD), despite their disproportionately higher rates of SA. We examined lifetime SA in 4,068 individuals with DSM-IV alcohol dependence from the Collaborative Study on the Genetics of Alcoholism (23% lifetime suicide attempt; 53% female; 17% Admixed African American ancestries; mean age: 38). We 1) conducted a genome-wide association study (GWAS) of SA and performed downstream analyses to determine whether we could identify specific biological pathways of risk, and 2) explored risk in aggregate across other clinical conditions, polygenic scores (PGS) for comorbid psychiatric problems, and neurocognitive functioning between those with AD who have and have not reported a lifetime suicide attempt. The GWAS and downstream analyses did not produce any significant associations. Participants with an AUD who had attempted suicide had greater rates of trauma exposure, major depressive disorder, post-traumatic stress disorder, and other substance use disorders compared to those who had not attempted suicide. Polygenic scores for suicide attempt, depression, and PTSD were associated with reporting a suicide attempt (ORs = 1.22-1.44). Participants who reported a SA also had decreased right hemispheric frontal-parietal theta and decreased interhemispheric temporal-parietal alpha electroencephalogram resting-state coherences relative to those who did not, but differences were small. Overall, individuals with alcohol dependence who report SA appear to experience a variety of severe comorbidities and elevated polygenic risk for SA. Our results demonstrate the need to further investigate suicide attempts in the presence of substance use disorders.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Clinical, genomic, and neurophysiological correlates of lifetime suicide attempts among individuals with alcohol dependence.

Research has identified clinical, genomic, and neurophysiological markers associated with suicide attempts (SA) among individuals with psychiatric illness. However, there is limited research among those with an alcohol use disorder, despite their disproportionately higher rates of SA. We examined lifetime SA in 4,068 individuals with DSM-IV alcohol dependence from the Collaborative Study on the Genetics of Alcoholism (23% lifetime suicide attempt; 53% female; 17% Admixed African American ancestries; mean age: 38). We 1) explored clinical risk factors associated with SA, 2) conducted a genome-wide association study of SA, 3) examined whether individuals with a SA had elevated polygenic scores for comorbid psychiatric conditions (e.g., alcohol use disorders, lifetime suicide attempt, and depression), and 4) explored differences in electroencephalogram neural functional connectivity between those with and without a SA. One gene-based finding emerged, RFX3 (Regulatory Factor X, located on 9p24.2) which had supporting evidence in prior research of SA among individuals with major depression. Only the polygenic score for suicide attempts was associated with reporting a suicide attempt (OR = 1.20, 95% CI = 1.06, 1.37). Lastly, we observed decreased right hemispheric frontal-parietal theta and decreased interhemispheric temporal-parietal alpha electroencephalogram resting-state coherences among those participants who reported a SA relative to those who did not, but differences were small. Overall, individuals with alcohol dependence who report SA appear to experience a variety of severe comorbidities and elevated polygenic risk for SA. Our results demonstrate the need to further investigate suicide attempts in the presence of substance use disorders.

Exclusion of the non-English-speaking world from the scientific literature: Recommendations for change for addiction journals and publishers.

Background: While English is only the native language of 7.3% of the world's population and less than 20% can speak the language, nearly 75% of all scientific publications are English. Aim: To describe how and why scientific contributions from the non-English-speaking world have been excluded from addiction literature, and put forward suggestions for making this literature more accessible to the non-English-speaking population. Methods: A working group of the International Society of Addiction Journal Editors (ISAJE) conducted an iterative review of issues related to scientific publishing from the non-English-speaking world. Findings: We discuss several issues stemming from the predominance of English in the scientific addiction literature, including historical drivers, why this matters, and proposed solutions, focusing on the increased availability of translation services. Conclusion: The addition of non-English-speaking authors, editorial team members, and journals will increase the value, impact, and transparency of research findings and increase the accountability and inclusivity of scientific publications.

Services Available at United States Addiction Treatment Facilities That Offer Medications versus Behavioral Treatment Only: A Cross-Sectional, Observational Analysis.

Purpose: Substance use disorders (SUDs) are widespread and cause significant morbidity and mortality, yet most people in the United States with a SUD do not receive treatment. Recommendations call for widespread use of pharmacotherapy, including medications for opioid use disorder (MOUD). However, many facilities do not offer a full array of medication treatments. This study aims to characterize programs that do and do not offer pharmacotherapy as part of addiction treatment services. We hypothesized that the availability of pharmacotherapy would predict the existence of other recommended components of treatment. Patients and Methods: We analyzed characteristics regarding treatment facilities (n = 15,782) recorded by the 2019 National Survey of Substance Abuse Treatment Services (N-SSATS) to determine how many SUD treatment facilities offer any pharmacotherapy. We compared facilities that offer any pharmacotherapy to facilities that offer none. Results: We found that 65% of SUD treatment facilities that responded to the N-SSATS survey provided at least one pharmacotherapy, while 35% of SUD treatment facilities did not. The facilities that provided at least one pharmacotherapy offered, on average, 6 additional treatment options (Cohen's d = 0.87; 95% CI: 0.84-0.91). Psychiatric medications were the most commonly available pharmacotherapy, followed by buprenorphine/naloxone and naltrexone. Conclusion: These results support that pharmacotherapy availability, such as MOUD, at SUD treatment facilities is associated with an increased number of recommended treatment components. Since MOUD has been shown elsewhere to reduce mortality for people with OUD, it should be universally available at SUD treatment facilities. Further efforts are needed to make pharmacotherapy more widely available.

Ten simple rules to host an inclusive conference.

Conferences are spaces to meet and network within and across academic and technical fields, learn about new advances, and share our work. They can help define career paths and create long-lasting collaborations and opportunities. However, these opportunities are not equal for all. This article introduces 10 simple rules to host an inclusive conference based on the authors' recent experience organizing the 2021 edition of the useR! statistical computing conference, which attracted a broad range of participants from academia, industry, government, and the nonprofit sector. Coming from different backgrounds, career stages, and even continents, we embraced the challenge of organizing a high-quality virtual conference in the context of the Coronavirus Disease 2019 (COVID-19) pandemic and making it a kind, inclusive, and accessible experience for as many people as possible. The rules result from our lessons learned before, during, and after the organization of the conference. They have been written mainly for potential organizers and selection committees of conferences and contain multiple practical tips to help a variety of events become more accessible and inclusive. We see this as a starting point for conversations and efforts towards building more inclusive conferences across the world. * Translated versions of the English abstract and the list of rules are available in 10 languages in S1 Text: Arabic, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Tamil, and Thai.

Older Adult Substance Use Treatment First-Time Admissions Between 2008 and 2018.

OBJECTIVES: To see whether the percentage of older adults entering substance use treatment for their first time continued to increase and whether there were changes in the use patterns leading to the treatment episode, particularly an increase in illicit drugs. DESIGN: Public administrative health record study. SETTING: The Treatment Episode Data Sets publicly available from the Substance Abuse Mental Health Services Administration from 2008 to 2018. PARTICIPANTS: Young adults age 30-54 years (N = 3,327,903) and older adults age 55 years and older (N = 453,598) with a first-time admission for a publicly funded substance use treatment. MEASUREMENTS: Demographic and substance use history variables at admission. RESULTS: The proportion of older adults going for substance use treatment for the first time continued to increase between 2008 and 2018 relative to younger adults, continuing the trend of increasing first-time admission between 1998 and 2008. For the first time, the primary substance at admission for older adults was an illicit substance only, surpassing alcohol only and the combination of alcohol and illicit drug use. In this period, use of opioids, particularly heroin, and methamphetamine increased among older adults entering treatment. CONCLUSIONS: As our population ages and substance use trends change, healthcare providers that take care of older adults must have skills to prevent, screen for, diagnose, and treat substance use disorders. Given recent trends in substance use and treatment among older adults, substance use treatment programs must adapt to meet the needs of an older population.

Practical foundations of machine learning for addiction research. Part I. Methods and techniques.

Machine learning assembles a broad set of methods and techniques to solve a wide range of problems, such as identifying individuals with substance use disorders (SUD), finding patterns in neuroimages, understanding SUD prognostic factors and their association, or determining addiction genetic underpinnings. However, the addiction research field underuses machine learning. This two-part narrative review focuses on machine learning tools and concepts, providing an introductory insight into their capabilities to facilitate their understanding and acquisition by addiction researchers. This first part presents supervised and unsupervised methods such as linear models, naive Bayes, support vector machines, artificial neural networks, and k-means. We illustrate each technique with examples of its use in current addiction research. We also present some open-source programming tools and methodological good practices that facilitate using these techniques. Throughout this work, we emphasize a continuum between applied statistics and machine learning, we show their commonalities, and provide sources for further reading to deepen the understanding of these methods. This two-part review is a primer for the next generation of addiction researchers incorporating machine learning in their projects. Researchers will find a bridge between applied statistics and machine learning, ways to expand their analytical toolkit, recommendations to incorporate well-established good practices in addiction data analysis (e.g., stating the rationale for using newer analytical tools, calculating sample size, improving reproducibility), and the vocabulary to enhance collaboration between researchers who do not conduct data analyses and those who do.

Practical foundations of machine learning for addiction research. Part II. Workflow and use cases.

In a continuum with applied statistics, machine learning offers a wide variety of tools to explore, analyze, and understand addiction data. These tools include algorithms that can leverage useful information from data to build models; these models can solve particular tasks to answer addiction scientific questions. In this second part of a two-part review on machine learning, we explain how to apply machine learning methods to addiction research. Like other analytical tools, machine learning methods require a careful implementation to carry out a reproducible and transparent research process with reliable results. This review describes a workflow to guide the application of machine learning in addiction research, detailing study design, data collection, data pre-processing, modeling, and results communication. How to train, validate, and test a model, detect and characterize overfitting, and determine an adequate sample size are some of the key issues when applying machine learning. We also illustrate the process and particular nuances with examples of how researchers in addiction have applied machine learning techniques with different goals, study designs, or data sources as well as explain the main limitations of machine learning approaches and how to best address them. A good use of machine learning enriches the addiction research toolkit.

Substance Use in Pregnancy: Identifying Stigma and Improving Care.

This review examines the impact of stigma on pregnant people who use substances. Stigma towards people who use drugs is pervasive and negatively impacts the care of substance-using people by characterizing addiction as a weakness and fostering beliefs that undermine the personal resources needed to access treatment and recover from addiction, including self-efficacy, help seeking and belief that they deserve care. Stigma acts on multiple levels by blaming people for having a problem and then making it difficult for them to get help, but in spite of this, most pregnant people who use substances reduce or stop using when they learn they are pregnant. Language, beliefs about gender roles, and attitudes regarding fitness for parenting are social factors that can express and perpetuate stigma while facilitating punitive rather than therapeutic approaches. Because of stigmatizing attitudes that a person who uses substances is unfit to parent, pregnant people who use substances are at heightened risk of being screened for substance use, referred to child welfare services, and having their parental rights taken away; these outcomes are even more likely for people of color. Various treatment options can successfully support recovery in substance-using pregnant populations, but treatment is underutilized in all populations including pregnant people, and more knowledge is needed on how to sustain engagement in treatment and recovery activities. To combat stigma when working with substance-using pregnant people throughout the peripartum period, caregivers should utilize a trauma-informed approach that incorporates harm reduction and motivational interviewing with a focus on building trust, enhancing self-efficacy, and strengthening the personal skills and resources needed to optimize health of the parent-baby dyad.

Learning deep features for dead and living breast cancer cell classification without staining.

Automated cell classification in cancer biology is a challenging topic in computer vision and machine learning research. Breast cancer is the most common malignancy in women that usually involves phenotypically diverse populations of breast cancer cells and an heterogeneous stroma. In recent years, automated microscopy technologies are allowing the study of live cells over extended periods of time, simplifying the task of compiling large image databases. For instance, there have been several studies oriented towards building machine learning systems capable of automatically classifying images of different cell types (i.e. motor neurons, stem cells). In this work we were interested in classifying breast cancer cells as live or dead, based on a set of automatically retrieved morphological characteristics using image processing techniques. Our hypothesis is that live-dead classification can be performed without any staining and using only bright-field images as input. We tackled this problem using the JIMT-1 breast cancer cell line that grows as an adherent monolayer. First, a vast image set composed by JIMT-1 human breast cancer cells that had been exposed to a chemotherapeutic drug treatment (doxorubicin and paclitaxel) or vehicle control was compiled. Next, several classifiers were trained based on well-known convolutional neural networks (CNN) backbones to perform supervised classification using labels obtained from fluorescence microscopy images associated with each bright-field image. Model performances were evaluated and compared on a large number of bright-field images. The best model reached an AUC = 0.941 for classifying breast cancer cells without treatment. Furthermore, it reached AUC = 0.978 when classifying breast cancer cells under drug treatment. Our results highlight the potential of machine learning and computational image analysis to build new diagnosis tools that benefit the biomedical field by reducing cost, time, and stimulating work reproducibility. More importantly, we analyzed the way our classifiers clusterize bright-field images in the learned high-dimensional embedding and linked these groups to salient visual characteristics in live-dead cell biology observed by trained experts.

Three-month outcomes from a patient-centered program to treat opioid use disorder in Iowa, USA.

BACKGROUND: Opioid use disorder (OUD), a chronic disease, is a major public health problem. Despite availability of effective treatment, too few people receive it and treatment retention is low. Understanding barriers and facilitators of treatment access and retention is needed to improve outcomes for people with OUD. OBJECTIVES: To assess 3-month outcomes pilot data from a patient-centered OUD treatment program in Iowa, USA, that utilized flexible treatment requirements and prioritized engagement over compliance. METHODS: Forty patients (62.5% female: mean age was 35.7 years, SD 9.5) receiving medication, either buprenorphine or naltrexone, to treat OUD were enrolled in an observational study. Patients could select or decline case management, counseling, and peer recovery groups. Substance use, risk and protective factors, and recovery capital were measured at intake and 3 months. RESULTS: Most participants reported increased recovery capital. The median Assessment of Recovery Capital (ARC) score went from 37 at enrollment to 43 (p < 0.01). Illegal drug use decreased, with the median days using illegal drugs in the past month dropping from 10 to 0 (p < 0.001). Cravings improved: 29.2% reported no cravings at intake and 58.3% reported no cravings at 3 months (p < 0.001). Retention rate was 92.5% at 3 months. Retention rate for participants who were not on probation/parole was higher (96.9%) than for those on probation/parole (62.5%, p = 0.021). CONCLUSION: This study shows preliminary evidence that a care model based on easy and flexible access and strategies to improve treatment retention improves recovery capital, reduces illegal drug use and cravings, and retains people in treatment.

Associations between Suicidal Thoughts and Behaviors and Genetic Liability for Cognitive Performance, Depression, and Risk-Taking in a High-Risk Sample.

Background: Suicidal thoughts and behaviors (STBs) and nonsuicidal self-injury (NSSI) behaviors are moderately heritable and may reflect an underlying predisposition to depression, impulsivity, and cognitive vulnerabilities to varying degrees. Objectives: We aimed to estimate the degrees of association between genetic liability to depression, impulsivity, and cognitive performance and STBs and NSSI in a high-risk sample. Methods: We used data on 7,482 individuals of European ancestry and 3,359 individuals of African ancestry from the Collaborative Study on the Genetics of Alcoholism to examine the links between polygenic scores (PGSs) for depression, impulsivity/risk-taking, and cognitive performance with 3 self-reported indices of STBs (suicidal ideation, persistent suicidal ideation defined as ideation occurring on at least 7 consecutive days, and suicide attempt) and with NSSI. Results: The PGS for depression was significantly associated with all 4 primary self-harm measures, explaining 0.6-2.5% of the variance. The PGS for risk-taking behaviors was also associated with all 4 self-harm behaviors in baseline models, but was no longer associated after controlling for a lifetime measure of DSM-IV alcohol dependence and abuse symptom counts. Polygenic predisposition for cognitive performance was negatively associated with suicide attempts (q = 3.8e-4) but was not significantly associated with suicidal ideation nor NSSI. We did not find any significant associations in the African ancestry subset, likely due to smaller sample sizes. Conclusions: Our results encourage the study of STB as transdiagnostic outcomes that show genetic overlap with a range of risk factors.

Late- and Typical-Onset Heroin Use Among Older Adults Seeking Treatment for Opioid Use Disorder.

OBJECTIVE: Analyze 10-year trends in opioid use disorder with heroin (OUD-H) among older persons and to compare those with typical-onset (age <30 years) to those with late (age 30+) onset. DESIGN: Naturalistic observation using the most recent (2008-2017) Treatment Episode Data Set-Admissions (TEDS-A). SETTING: Admission records in TEDS-A come from all public and private U.S. programs for substance use disorder treatment receiving public funding. PARTICIPANTS: U.S. adults aged 55 years and older entering treatment for the first time between 2008 and 2017 to treat OUD-H. MEASUREMENTS: Admission trends, demographics, substance use history. RESULTS: The number of older adults who entered treatment for OUD-H nearly tripled between 2007 and 2017. Compared to those with typical-onset (before age 30), those with late-onset heroin use were more likely to be white, female, more highly educated, and rural. Older adults with late-onset were more likely to be referred to treatment by an employer and less likely to be referred by the criminal justice system. Those with late-onset were more likely to use heroin more frequently but less likely to inject heroin than those with typical-onset. Those with typical onset were more likely to receive medication for addiction treatment than those with late-onset. CONCLUSION: Late-onset heroin use is increasing among older U.S. adults. Research is needed to understand the unique needs of this population better. As this population grows, geriatric psychiatrists may be increasingly called upon to provide specialized care to people with late-onset OUD-H.

Polygenic contributions to alcohol use and alcohol use disorders across population-based and clinically ascertained samples.

BACKGROUND: Studies suggest that alcohol consumption and alcohol use disorders have distinct genetic backgrounds. METHODS: We examined whether polygenic risk scores (PRS) for consumption and problem subscales of the Alcohol Use Disorders Identification Test (AUDIT-C, AUDIT-P) in the UK Biobank (UKB; N = 121 630) correlate with alcohol outcomes in four independent samples: an ascertained cohort, the Collaborative Study on the Genetics of Alcoholism (COGA; N = 6850), and population-based cohorts: Avon Longitudinal Study of Parents and Children (ALSPAC; N = 5911), Generation Scotland (GS; N = 17 461), and an independent subset of UKB (N = 245 947). Regression models and survival analyses tested whether the PRS were associated with the alcohol-related outcomes. RESULTS: In COGA, AUDIT-P PRS was associated with alcohol dependence, AUD symptom count, maximum drinks (R2 = 0.47-0.68%, p = 2.0 × 10-8-1.0 × 10-10), and increased likelihood of onset of alcohol dependence (hazard ratio = 1.15, p = 4.7 × 10-8); AUDIT-C PRS was not an independent predictor of any phenotype. In ALSPAC, the AUDIT-C PRS was associated with alcohol dependence (R2 = 0.96%, p = 4.8 × 10-6). In GS, AUDIT-C PRS was a better predictor of weekly alcohol use (R2 = 0.27%, p = 5.5 × 10-11), while AUDIT-P PRS was more associated with problem drinking (R2 = 0.40%, p = 9.0 × 10-7). Lastly, AUDIT-P PRS was associated with ICD-based alcohol-related disorders in the UKB subset (R2 = 0.18%, p < 2.0 × 10-16). CONCLUSIONS: AUDIT-P PRS was associated with a range of alcohol-related phenotypes across population-based and ascertained cohorts, while AUDIT-C PRS showed less utility in the ascertained cohort. We show that AUDIT-P is genetically correlated with both use and misuse and demonstrate the influence of ascertainment schemes on PRS analyses.