Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Ten simple rules to host an inclusive conference.

Conferences are spaces to meet and network within and across academic and technical fields, learn about new advances, and share our work. They can help define career paths and create long-lasting collaborations and opportunities. However, these opportunities are not equal for all. This article introduces 10 simple rules to host an inclusive conference based on the authors' recent experience organizing the 2021 edition of the useR! statistical computing conference, which attracted a broad range of participants from academia, industry, government, and the nonprofit sector. Coming from different backgrounds, career stages, and even continents, we embraced the challenge of organizing a high-quality virtual conference in the context of the Coronavirus Disease 2019 (COVID-19) pandemic and making it a kind, inclusive, and accessible experience for as many people as possible. The rules result from our lessons learned before, during, and after the organization of the conference. They have been written mainly for potential organizers and selection committees of conferences and contain multiple practical tips to help a variety of events become more accessible and inclusive. We see this as a starting point for conversations and efforts towards building more inclusive conferences across the world. * Translated versions of the English abstract and the list of rules are available in 10 languages in S1 Text: Arabic, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Tamil, and Thai.

Older Adult Substance Use Treatment First-Time Admissions Between 2008 and 2018.

OBJECTIVES: To see whether the percentage of older adults entering substance use treatment for their first time continued to increase and whether there were changes in the use patterns leading to the treatment episode, particularly an increase in illicit drugs. DESIGN: Public administrative health record study. SETTING: The Treatment Episode Data Sets publicly available from the Substance Abuse Mental Health Services Administration from 2008 to 2018. PARTICIPANTS: Young adults age 30-54 years (N = 3,327,903) and older adults age 55 years and older (N = 453,598) with a first-time admission for a publicly funded substance use treatment. MEASUREMENTS: Demographic and substance use history variables at admission. RESULTS: The proportion of older adults going for substance use treatment for the first time continued to increase between 2008 and 2018 relative to younger adults, continuing the trend of increasing first-time admission between 1998 and 2008. For the first time, the primary substance at admission for older adults was an illicit substance only, surpassing alcohol only and the combination of alcohol and illicit drug use. In this period, use of opioids, particularly heroin, and methamphetamine increased among older adults entering treatment. CONCLUSIONS: As our population ages and substance use trends change, healthcare providers that take care of older adults must have skills to prevent, screen for, diagnose, and treat substance use disorders. Given recent trends in substance use and treatment among older adults, substance use treatment programs must adapt to meet the needs of an older population.

Practical foundations of machine learning for addiction research. Part I. Methods and techniques.

Machine learning assembles a broad set of methods and techniques to solve a wide range of problems, such as identifying individuals with substance use disorders (SUD), finding patterns in neuroimages, understanding SUD prognostic factors and their association, or determining addiction genetic underpinnings. However, the addiction research field underuses machine learning. This two-part narrative review focuses on machine learning tools and concepts, providing an introductory insight into their capabilities to facilitate their understanding and acquisition by addiction researchers. This first part presents supervised and unsupervised methods such as linear models, naive Bayes, support vector machines, artificial neural networks, and k-means. We illustrate each technique with examples of its use in current addiction research. We also present some open-source programming tools and methodological good practices that facilitate using these techniques. Throughout this work, we emphasize a continuum between applied statistics and machine learning, we show their commonalities, and provide sources for further reading to deepen the understanding of these methods. This two-part review is a primer for the next generation of addiction researchers incorporating machine learning in their projects. Researchers will find a bridge between applied statistics and machine learning, ways to expand their analytical toolkit, recommendations to incorporate well-established good practices in addiction data analysis (e.g., stating the rationale for using newer analytical tools, calculating sample size, improving reproducibility), and the vocabulary to enhance collaboration between researchers who do not conduct data analyses and those who do.

Practical foundations of machine learning for addiction research. Part II. Workflow and use cases.

In a continuum with applied statistics, machine learning offers a wide variety of tools to explore, analyze, and understand addiction data. These tools include algorithms that can leverage useful information from data to build models; these models can solve particular tasks to answer addiction scientific questions. In this second part of a two-part review on machine learning, we explain how to apply machine learning methods to addiction research. Like other analytical tools, machine learning methods require a careful implementation to carry out a reproducible and transparent research process with reliable results. This review describes a workflow to guide the application of machine learning in addiction research, detailing study design, data collection, data pre-processing, modeling, and results communication. How to train, validate, and test a model, detect and characterize overfitting, and determine an adequate sample size are some of the key issues when applying machine learning. We also illustrate the process and particular nuances with examples of how researchers in addiction have applied machine learning techniques with different goals, study designs, or data sources as well as explain the main limitations of machine learning approaches and how to best address them. A good use of machine learning enriches the addiction research toolkit.

Polygenic contributions to alcohol use and alcohol use disorders across population-based and clinically ascertained samples.

BACKGROUND: Studies suggest that alcohol consumption and alcohol use disorders have distinct genetic backgrounds. METHODS: We examined whether polygenic risk scores (PRS) for consumption and problem subscales of the Alcohol Use Disorders Identification Test (AUDIT-C, AUDIT-P) in the UK Biobank (UKB; N = 121 630) correlate with alcohol outcomes in four independent samples: an ascertained cohort, the Collaborative Study on the Genetics of Alcoholism (COGA; N = 6850), and population-based cohorts: Avon Longitudinal Study of Parents and Children (ALSPAC; N = 5911), Generation Scotland (GS; N = 17 461), and an independent subset of UKB (N = 245 947). Regression models and survival analyses tested whether the PRS were associated with the alcohol-related outcomes. RESULTS: In COGA, AUDIT-P PRS was associated with alcohol dependence, AUD symptom count, maximum drinks (R2 = 0.47-0.68%, p = 2.0 × 10-8-1.0 × 10-10), and increased likelihood of onset of alcohol dependence (hazard ratio = 1.15, p = 4.7 × 10-8); AUDIT-C PRS was not an independent predictor of any phenotype. In ALSPAC, the AUDIT-C PRS was associated with alcohol dependence (R2 = 0.96%, p = 4.8 × 10-6). In GS, AUDIT-C PRS was a better predictor of weekly alcohol use (R2 = 0.27%, p = 5.5 × 10-11), while AUDIT-P PRS was more associated with problem drinking (R2 = 0.40%, p = 9.0 × 10-7). Lastly, AUDIT-P PRS was associated with ICD-based alcohol-related disorders in the UKB subset (R2 = 0.18%, p < 2.0 × 10-16). CONCLUSIONS: AUDIT-P PRS was associated with a range of alcohol-related phenotypes across population-based and ascertained cohorts, while AUDIT-C PRS showed less utility in the ascertained cohort. We show that AUDIT-P is genetically correlated with both use and misuse and demonstrate the influence of ascertainment schemes on PRS analyses.

Late- and Typical-Onset Heroin Use Among Older Adults Seeking Treatment for Opioid Use Disorder.

OBJECTIVE: Analyze 10-year trends in opioid use disorder with heroin (OUD-H) among older persons and to compare those with typical-onset (age <30 years) to those with late (age 30+) onset. DESIGN: Naturalistic observation using the most recent (2008-2017) Treatment Episode Data Set-Admissions (TEDS-A). SETTING: Admission records in TEDS-A come from all public and private U.S. programs for substance use disorder treatment receiving public funding. PARTICIPANTS: U.S. adults aged 55 years and older entering treatment for the first time between 2008 and 2017 to treat OUD-H. MEASUREMENTS: Admission trends, demographics, substance use history. RESULTS: The number of older adults who entered treatment for OUD-H nearly tripled between 2007 and 2017. Compared to those with typical-onset (before age 30), those with late-onset heroin use were more likely to be white, female, more highly educated, and rural. Older adults with late-onset were more likely to be referred to treatment by an employer and less likely to be referred by the criminal justice system. Those with late-onset were more likely to use heroin more frequently but less likely to inject heroin than those with typical-onset. Those with typical onset were more likely to receive medication for addiction treatment than those with late-onset. CONCLUSION: Late-onset heroin use is increasing among older U.S. adults. Research is needed to understand the unique needs of this population better. As this population grows, geriatric psychiatrists may be increasingly called upon to provide specialized care to people with late-onset OUD-H.

Density and Dichotomous Family History Measures of Alcohol Use Disorder as Predictors of Behavioral and Neural Phenotypes: A Comparative Study Across Gender and Race/Ethnicity.

BACKGROUND: Family history (FH) is an important risk factor for the development of alcohol use disorder (AUD). A variety of dichotomous and density measures of FH have been used to predict alcohol outcomes; yet, a systematic comparison of these FH measures is lacking. We compared 4 density and 4 commonly used dichotomous FH measures and examined variations by gender and race/ethnicity in their associations with age of onset of regular drinking, parietal P3 amplitude to visual target, and likelihood of developing AUD. METHODS: Data from the Collaborative Study on the Genetics of Alcoholism (COGA) were utilized to compute the density and dichotomous measures. Only subjects and their family members with DSM-5 AUD diagnostic information obtained through direct interviews using the Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA) were included in the study. Area under receiver operating characteristic curves were used to compare the diagnostic accuracy of FH measures at classifying DSM-5 AUD diagnosis. Logistic and linear regression models were used to examine associations of FH measures with alcohol outcomes. RESULTS: Density measures had greater diagnostic accuracy at classifying AUD diagnosis, whereas dichotomous measures presented diagnostic accuracy closer to random chance. Both dichotomous and density measures were significantly associated with likelihood of AUD, early onset of regular drinking, and low parietal P3 amplitude, but density measures presented consistently more robust associations. Further, variations in these associations were observed such that among males (vs. females) and Whites (vs. Blacks), associations of alcohol outcomes with density (vs. dichotomous) measures were greater in magnitude. CONCLUSIONS: Density (vs. dichotomous) measures seem to present more robust associations with alcohol outcomes. However, associations of dichotomous and density FH measures with different alcohol outcomes (behavioral vs. neural) varied across gender and race/ethnicity. These findings have great applicability for alcohol research examining FH of AUD.

Mechanisms of Alcohol Addiction: Bridging Human and Animal Studies.

AIM: The purpose of this brief narrative review is to address the complexities and benefits of extending animal alcohol addiction research to the human domain, emphasizing Allostasis and Incentive Sensitization, two models that inform many pre-clinical and clinical studies. METHODS: The work reviewed includes a range of approaches, including: a) animal and human studies that target the biology of craving and compulsive consumption; b) human investigations that utilize alcohol self-administration and alcohol challenge paradigms, in some cases across 10 years; c) questionnaires that document changes in the positive and negative reinforcing effects of alcohol with increasing severity of addiction; and d) genomic structural equation modeling based on data from animal and human studies. RESULTS: Several general themes emerge from specific study findings. First, positive reinforcement is characteristic of early stage addiction and sometimes diminishes with increasing severity, consistent with both Allostasis and Incentive Sensitization. Second, evidence is less consistent for the predominance of negative reinforcement in later stages of addiction, a key tenant of Allostasis. Finally, there are important individual differences in motivation to drink at a given point in time as well as person-specific change patterns across time. CONCLUSIONS: Key constructs of addiction, like stage and reinforcement, are by necessity operationalized differently in animal and human studies. Similarly, testing the validity of addiction models requires different strategies by the two research domains. Although such differences are challenging, they are not insurmountable, and there is much to be gained in understanding and treating addiction by combining pre-clinical and clinical approaches.

A Pilot Follow-Up Study of Older Alcohol-Dependent COGA Adults.

BACKGROUND: Alcohol consumption and problems are increasing among older adults, who are at elevated risk for alcohol-related accidents and medical problems. This paper describes a pilot follow-up of older adults with a history of alcohol dependence that was designed to determine the feasibility of conducting a more extensive investigation. METHODS: The sample consisted of previously assessed subjects in the Collaborative Studies on the Genetics of Alcoholism who: (i) were age 50+; (ii) had lifetime DSM-IV AD; and (iii) had DNA available. Individuals were located through family contacts, Internet searches, and death registries. A brief telephone interview assessed demographics, health, and alcohol involvement. RESULTS: Of the total sample (N = 2,174), 36% were contacted, 24% were deceased, and 40% were not yet located. Most (89%) contacted subjects were interviewed, and 99% of them agreed to future evaluation. Thirty percent of interviewed subjects reported abstinence for 10+ years, 56% reported drinking within the past year, and 14% last drank between >1 and 10 years ago. There were no age-related past-year differences in weekly consumption (overall sample mean: 16 drinks), number of drinking weeks (30.8), maximum number of drinks in 24 hours (8.1), or prevalence of weekly risky drinking (19%). Among those who drank within the past 5 years, the 3 most common alcohol-related problems were spending excessive time drinking or recovering (49%), drinking more/longer than intended (35%), and driving while intoxicated (35%); and about a third (32%) received some form of treatment. CONCLUSIONS: Over a 1-year period, we located 60% of individuals last seen an average of 23 years ago. The majority of contacted individuals were interviewed and willing to be evaluated again. Although the proportion of individuals currently drinking diminished with age, subjects exhibited troublesome levels of alcohol consumption and problems. Our findings suggest the importance and feasibility of a more comprehensive follow-up.

Treating Alcohol Use Disorder in U.S. Veterans: The Role of Traumatic Brain Injury.

OBJECTIVE: The authors examined the efficacy of valproate to reduce relapse to heavy drinking among veterans with alcohol use disorder (AUD) and neuropsychiatric comorbidities and whether antecedent traumatic brain injury (TBI) or posttraumatic stress disorder (PTSD) affected treatment response. METHODS: Participants were male veterans 18-60 years old with an AUD and no other substance use besides nicotine or cannabis. Sixty-two patients were randomly assigned to receive either valproate or naltrexone. Participants were evaluated at baseline and followed weekly for 24 weeks. All participants received standardized psychosocial interventions as well as treatment for coexistent psychiatric conditions. RESULTS: During the follow-up period, nine study subjects in the naltrexone group and 14 in the valproate group relapsed to heavy drinking, but the difference did not reach statistical significance. Participants with a history of moderate to severe TBI were more likely to relapse to heavy drinking compared with those with no TBI (hazard ratio=4.834, 95% CI=1.103-21.194, p=0.033). PTSD status did not significantly affect outcome. CONCLUSIONS: Intensive outpatient programs are efficacious alternatives to treat AUD in veterans, although the role of pharmacological treatment is not completely elucidated. Glutamatergic agents appear to be less effective than opiate antagonists to prevent relapse to heavy drinking and to increase cumulative abstinence. Future studies should examine novel pharmacological and nonpharmacological options.

Reliability and validity of an internalizing symptom scale based on the adolescent and adult Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA).

BACKGROUND: The Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA) is an interview that assesses psychiatric symptoms and diagnoses, including substance use disorders and anxiety and mood (i.e., internalizing) disorders. Although the SSAGA is widely used, there exists no overall internalizing characteristics scale based on items drawn from SSAGA's mood and anxiety disorder sections. OBJECTIVES: To design and assess a SSAGA-based measurement instrument capturing the overall internalizing dimension that underlies more specific internalizing conditions. METHODS: We developed, assessed, and characterized a new scale for measuring internalizing problematic characteristics derived from the SSAGA interview. All samples were drawn from the Collaborative Studies on the Genetics of Alcoholism, a prospective multi-site genetic study of families at high risk for alcohol use disorders. All participants taking part in the study between September 2005 and September 2017 were eligible (n = 904, 52.2% female). RESULTS: The scale had adequate internal consistency (ordinal α = 0.85, 95% CI = [0.81, 0.89]). Construct validity was supported by its association with other measures of internalizing characteristics (Internalizing Scale from Achenbach Self Reports; Neuroticism Scale from the Neuroticism-Extraversion-Openness Five-Factor Personality Inventory). Several indices of alcohol, marijuana, and nicotine misuse were also positively associated with Internalizing Scale scores. CONCLUSIONS: The Internalizing Scale has very good psychometric properties and can be used in studies that incorporate the SSAGA interview to study the association between internalizing characteristics and problematic alcohol and other substance use. These associations can potentially be utilized to identify individuals at risk for substance problems and to design treatments targeting such individuals.

Repeated mild traumatic brain injury produces neuroinflammation, anxiety-like behaviour and impaired spatial memory in mice.

PRIMARY OBJECTIVE: Repeated traumatic brain injuries (rmTBI) are frequently associated with debilitating neuropsychiatric conditions such as cognitive impairment, mood disorders, and post-traumatic stress disorder. We tested the hypothesis that repeated mild traumatic brain injury impairs spatial memory and enhances anxiety-like behaviour. RESEARCH DESIGN: We used a between groups design using single (smTBI) or repeated (rmTBI) controlled cranial closed skull impacts to mice, compared to a control group. METHODS AND PROCEDURES: We assessed the effects of smTBI and rmTBI using measures of motor performance (Rotarod Test [RT]), anxiety-like behaviour (Elevated Plus Maze [EPM] and Open Field [OF] tests), and spatial memory (Morris Water Maze [MWM]) within 12 days of the final injury. In separate groups of mice, astrocytosis and microglial activation were assessed 24 hours after the final injury using GFAP and IBA-1 immunohistochemistry. MAIN OUTCOMES AND RESULTS: RmTBI impaired spatial memory in the MWM and increased anxiety-like behaviour in the EPM and OFT. In addition, rmTBI elevated GFAP and IBA-1 immunohistochemistry throughout the mouse brain. RmTBI produced astrocytosis and microglial activation, and elicited impaired spatial memory and anxiety-like behaviour. CONCLUSIONS: rmTBI produces acute cognitive and anxiety-like disturbances associated with inflammatory changes in brain regions involved in spatial memory and anxiety.

Preliminary Investigation of Cerebral Blood Flow and Amyloid Burden in Veterans With and Without Combat-Related Traumatic Brain Injury.

This study aimed to examine global and regional cerebral blood flow and amyloid burden in combat veterans with and without traumatic brain injury (TBI). Cerebral blood flow (in milliliters per minute per 100 mL) was measured by quantitative [(15)O]water, and amyloid burden was measured by [(11)C]PIB imaging. Mean global cerebral blood flow was significantly lower in veterans with TBI compared with non-TBI veterans. There were essentially no differences between groups for globally normalized regional cerebral blood flow. Amyloid burden did not differ between TBI and non-TBI veterans. Veterans who have suffered a TBI have significantly lower cerebral blood flow than non-TBI controls but did not manifest increased levels of amyloid, globally or regionally.

Clinical, genomic, and neurophysiological correlates of lifetime suicide attempts among individuals with alcohol dependence.

Research has identified clinical, genomic, and neurophysiological markers associated with suicide attempts (SA) among individuals with psychiatric illness. However, there is limited research among those with an alcohol use disorder (AUD), despite their disproportionately higher rates of SA. We examined lifetime SA in 4,068 individuals with DSM-IV alcohol dependence from the Collaborative Study on the Genetics of Alcoholism (23% lifetime suicide attempt; 53% female; 17% Admixed African American ancestries; mean age: 38). We 1) conducted a genome-wide association study (GWAS) of SA and performed downstream analyses to determine whether we could identify specific biological pathways of risk, and 2) explored risk in aggregate across other clinical conditions, polygenic scores (PGS) for comorbid psychiatric problems, and neurocognitive functioning between those with AD who have and have not reported a lifetime suicide attempt. The GWAS and downstream analyses did not produce any significant associations. Participants with an AUD who had attempted suicide had greater rates of trauma exposure, major depressive disorder, post-traumatic stress disorder, and other substance use disorders compared to those who had not attempted suicide. Polygenic scores for suicide attempt, depression, and PTSD were associated with reporting a suicide attempt (ORs = 1.22-1.44). Participants who reported a SA also had decreased right hemispheric frontal-parietal theta and decreased interhemispheric temporal-parietal alpha electroencephalogram resting-state coherences relative to those who did not, but differences were small. Overall, individuals with alcohol dependence who report SA appear to experience a variety of severe comorbidities and elevated polygenic risk for SA. Our results demonstrate the need to further investigate suicide attempts in the presence of substance use disorders.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

How reliable are county and regional health rankings?

Surveillance system indicators of morbidity, mortality, or behaviors are used to provide regional information for ever-smaller areas, most recently for ranking counties. These rankings are thought to provide information about the relative standing of regions and provide information about problem areas and the success of programs. We investigate the ability of such rankings to reliably assess health. We assess the reliability of several ranked health indices used at the county level and the consistency of an index's quality across different states. Reliability is assessed using an index of reliability, simulations, and the ability of an index to consistently identify the top 10 % (worst) counties within a state. There is marked variability across measures to provide consistent ranks, across states, and, many times, across states for a particular measure. A few health measures do consistently well, e.g., Teen Birth, Chlamydia, and Years of Potential Life Lost rates. While a few health rankings appear worthy of use for policy and in the identification of local problems, in general, they are not consistent across regions.

Exclusion of the non-English-speaking world from the scientific literature: Recommendations for change for addiction journals and publishers.

Background: While English is only the native language of 7.3% of the world's population and less than 20% can speak the language, nearly 75% of all scientific publications are English. Aim: To describe how and why scientific contributions from the non-English-speaking world have been excluded from addiction literature, and put forward suggestions for making this literature more accessible to the non-English-speaking population. Methods: A working group of the International Society of Addiction Journal Editors (ISAJE) conducted an iterative review of issues related to scientific publishing from the non-English-speaking world. Findings: We discuss several issues stemming from the predominance of English in the scientific addiction literature, including historical drivers, why this matters, and proposed solutions, focusing on the increased availability of translation services. Conclusion: The addition of non-English-speaking authors, editorial team members, and journals will increase the value, impact, and transparency of research findings and increase the accountability and inclusivity of scientific publications.