Estimating classification consistency of machine learning models for screening measures.

This article illustrates novel quantitative methods to estimate classification consistency in machine learning models used for screening measures. Screening measures are used in psychology and medicine to classify individuals into diagnostic classifications. In addition to achieving high accuracy, it is ideal for the screening process to have high classification consistency, which means that respondents would be classified into the same group every time if the assessment was repeated. Although machine learning models are increasingly being used to predict a screening classification based on individual item responses, methods to describe the classification consistency of machine learning models have not yet been developed. This article addresses this gap by describing methods to estimate classification inconsistency in machine learning models arising from two different sources: sampling error during model fitting and measurement error in the item responses. These methods use data resampling techniques such as the bootstrap and Monte Carlo sampling. These methods are illustrated using three empirical examples predicting a health condition/diagnosis from item responses. R code is provided to facilitate the implementation of the methods. This article highlights the importance of considering classification consistency alongside accuracy when studying screening measures and provides the tools and guidance necessary for applied researchers to obtain classification consistency indices in their machine learning research on diagnostic assessments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Concordance between substance use self-report and hair analysis in community-based adolescents.

Background: Accurate drug use identification through subjective self-report and toxicological biosample (hair) analysis are necessary to determine substance use sequelae in youth. Yet consistency between self-reported substance use and robust, toxicological analysis in a large sample of youth is understudied.Objectives: We aim to assess concordance between self-reported substance use and hair toxicological analysis in community-based adolescents.Methods: Hair results by LC-MS/MS and GC-MS/MS and self-reported past-year substance use from an Adolescent Brain Cognitive Development (ABCD) Study subsample (N = 1,390; ages 9-13; 48% female) were compared. The participants were selected for hair selection through two methods: high scores on a substance risk algorithm selected 93%; 7% were low-risk, randomly selected participants. Kappa coefficients the examined concordance between self-report and hair results.Results: 10% of youth self-reported any past-year substance use (e.g. alcohol, cannabis, nicotine, and opiates), while a mostly non-overlapping 10% had hair results indicating recent substance use (cannabis, alcohol, non-prescription amphetamines, cocaine, nicotine, opiates, and fentanyl). In randomly selected low-risk cases, 7% were confirmed positive in hair. Combining methods, 19% of the sample self-reported substance use and/or had a positive hair sample. Kappa coefficient of concordance between self-report and hair results was low (kappa = 0.07; p = .007).Conclusions: Hair toxicology identified substance use in high-risk and low-risk ABCD cohort subsamples. Given low concordance between hair results and self-report, reliance on either method alone would incorrectly categorize 9% as non-users. Multiple methods for characterizing substance use history in youth improves accuracy. Larger representative samples are needed to assess the prevalence of substance use in youth.

Did the acute impact of the COVID-19 pandemic on drinking or nicotine use persist? Evidence from a cohort of emerging adults followed for up to nine years.

OBJECTIVE: This study examined the impact of the COVID-19 pandemic on drinking and nicotine use through June of 2021 in a community-based sample of young adults. METHOD: Data were from 348 individuals (49% female) enrolled in a long-term longitudinal study with an accelerated longitudinal design: the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA) Study. Individuals completed pre-pandemic assessments biannually from 2016 to early 2020, then completed up to three web-based, during-pandemic surveys in June 2020, December 2020, and June 2021. Assessments when individuals were 18.8-22.4 years old (N = 1,458) were used to compare drinking and nicotine use pre-pandemic vs. at each of the three during-pandemic timepoints, adjusting for the age-related increases expected over time. RESULTS: Compared to pre-pandemic, participants were less likely to report past-month drinking in June or December 2020, but there was an increase in drinking days among drinkers in June 2020. By June 2021, both the prevalence of past-month drinking and number of drinking days among drinks were similar to pre-pandemic levels. On average, there were no statistically significant differences between pre-pandemic and during-pandemic time points for binge drinking, typical drinking quantity, or nicotine use. Young adults who reported an adverse financial impact of the pandemic showed increased nicotine use while their peers showed stable or decreased nicotine use. CONCLUSION: Initial effects of the pandemic on alcohol use faded by June 2021, and on average there was little effect of the pandemic on nicotine use.

Validating a brief screening measure for early-onset substance use during adolescence in a diverse, nationwide birth cohort.

The Loeber Risk Score (LRS) was developed to predict early-onset cannabis use in adolescence from late childhood, facilitating early identification. However, the LRS was developed in non-representative historical samples, leaving uncertain its generalizability to children/adolescents across the U.S. today. We externally validated the LRS in a diverse, nationwide cohort (N = 4,898) weighted to the composition of the U.S. Census. Participants in 20 cities completed assessments when youth were approximately 5, 9, and 15 years old. Parents completed the LRS at the age ∼5 and ∼9 interviews. At the age ∼15 interview, youth reported on the onset of alcohol/drug use before age 15, monthly drinking/binge drinking at ages 14-16, and use of cannabis multiple times per month at ages 14-16. First, we validated the LRS measured at age ∼9. Area under the receiver operating curve was 0.62 for onset of cannabis use before age 15, 0.68 for onset of cigarette use before age 15, and 0.62 for use of cannabis multiple times per month at ages 14-16. For drinking outcomes, LRS performance could not be distinguished from chance prediction. The recommended screening cutoff of LRS ≥ 2 identified 24% of children, among whom early-onset cannabis/cigarette use outcomes occurred 1.4-2.2 times more frequently than the general population. The LRS' performance did not vary significantly by sex, race, or ethnicity. When the LRS was measured at age ∼5, AUROC was significantly lower for some outcomes. Together, findings support the LRS measure as a potential tool for identifying children in early or late childhood at risk of early-onset drug use in adolescence.

Smoking-Specific Risk Factors in Early Adulthood That Mediate Risk of Daily Smoking by Age 29 for Children with ADHD.

OBJECTIVE: To test whether smoking-specific risk factors in early adulthood mediate prediction to daily smoking from childhood ADHD. METHODS: Participants were 237 with and 164 without childhood ADHD. A smoking risk profile score comprising smoking-specific factors measured between ages 18 to 25 (e.g., craving severity) and age of initiation was tested as mediator of the association between childhood ADHD and age 29 daily smoking. RESULTS: Childhood ADHD predicted age 29 smoking (ß = -.15, p = .019), 35% of ADHD versus 17% of nonADHD, and the profile score (ß = -.07, p = .004), which in turn mediated prediction to age 29 daily smoking (ß = -.03; p = .007). When tested individually, three profile variables (# cigarettes/day, difficulty concentrating during abstinence, and nicotine dependence) were significant mediators (ps = 0.005-0.038), above and beyond early adult smoking, ADHD persistence, and delinquency. CONCLUSIONS: These behavioral smoking characteristics help explain later daily cigarette smoking for adults with ADHD histories and may need to be targeted in intervention.

Estimating classification consistency of screening measures and quantifying the impact of measurement bias.

Screening measures are used in psychology and medicine to identify respondents who are high or low on a construct. Based on the screening, the evaluator assigns respondents to classes corresponding to different courses of action: Make a diagnosis versus reject a diagnosis; provide services versus withhold services; or conduct further assessment versus conclude the assessment process. When measures are used to classify individuals, it is important that the decisions be consistent and equitable across groups. Ideally, if respondents completed the screening measure repeatedly in quick succession, they would be consistently assigned into the same class each time. In addition, the consistency of the classification should be unrelated to the respondents' background characteristics, such as sex, race, or ethnicity (i.e., the measure is free of measurement bias). Reporting estimates of classification consistency is a common practice in educational testing, but there has been limited application of these estimates to screening in psychology and medicine. In this article, we present two procedures based on item response theory that are used (a) to estimate the classification consistency of a screening measure and (b) to evaluate how classification consistency is impacted by measurement bias across respondent groups. We provide R functions to conduct the procedures, illustrate the procedures with real data, and use Monte Carlo simulations to guide their appropriate use. Finally, we discuss how estimates of classification consistency can help assessment specialists make more informed decisions on the use of a screening measure with protected groups (e.g., groups defined by gender, race, or ethnicity). (PsycInfo Database Record (c) 2021 APA, all rights reserved).

When Does Differential Item Functioning Matter for Screening? A Method for Empirical Evaluation.

When items in a screening measure exhibit differential item functioning (DIF) across groups (e.g., males vs. females), DIF might affect which individuals are "caught" in the screening. This phenomenon is common, but DIF detection procedures do not typically provide guidance on whether the presence of DIF will meaningfully affect screening accuracy. Millsap and Kwok proposed a method to quantify the impact of DIF on screening accuracy, but their approach had limitations that prevent its use in scenarios where items are discrete. We extend the Millsap and Kwok procedure to accommodate discrete items and provide R functions to apply the procedure to the user's own data. We illustrate our approach using published screening information and evaluate the proposed methodology with a small simulation study. Overall, we encourage researchers to use empirical methods to evaluate the extent to which the presence of DIF in a screening measure materially affects screening performance.

Reduced Activation in the Pallidal-Thalamic-Motor Pathway Is Associated With Deficits in Reward-Modulated Inhibitory Control in Adults With a History of Attention-Deficit/Hyperactivity Disorder.

BACKGROUND: Attention-deficit/hyperactivity disorder (ADHD) symptoms persist into adulthood and are associated with functional impairments. Neuroimaging studies of reward-modulated inhibitory control can identify potential objective markers of impairment and may deepen our understanding of why probands engage in costly behaviors leading to adverse outcomes. The study aimed to identify reward-modulated inhibitory control neural circuitries, their association with ADHD symptoms, and real-world implications of a decreased capacity to engage in reward-modulated inhibitory control. METHODS: A total of 106 adults (90% male) with rigorous childhood diagnoses of ADHD were scanned with functional magnetic resonance imaging during the Monetary Incentive Go/NoGo task. Adulthood symptoms of inattention and hyperactivity/impulsivity based on self- and informant report were assessed. The number of lifetime attempts taken to quit smoking were also assessed as an exemplar real-world outcome. RESULTS: Hyperactivity/impulsivity was negatively associated with activation in the pallidum and primary motor cortex when inhibiting a previously rewarded Go stimulus that yielded a small immediate reward in order to obtain a larger reward later on. Reduced recruitment of the pallidal-thalamic-motor circuit mediated the negative association between hyperactivity/impulsivity and reward-modulated inhibitory control accuracy. Reduced pallidum activation, in response to reward-modulated inhibitory control, was also associated with more attempts made to successfully quit smoking. CONCLUSIONS: Probands with persistent hyperactivity/impulsivity symptoms have alterations in brain regions that calculate the value of inhibiting an action that yields an immediate reward in order to obtain delayed larger rewards. This deficit results in poor inhibitory control on basic tasks and during real-world behaviors that rely on similar processes.

Functional Outcomes of Young Adults with Childhood ADHD: A Latent Profile Analysis.

Adults with childhood attention-deficit hyperactivity disorder (ADHD) experience impairment in core functional domains (e.g., educational attainment, occupational status, social relationships, substance abuse, and criminal behavior), but it is currently unclear which impairments co-occur and whether subgroups experience differentiable patterns, none, or all aforementioned functional domains. Latent profile analysis (LPA) was used to characterize patterns of impairment. Data from the Pittsburgh ADHD Longitudinal Study were used. The 317 participants were 25 years old and had childhood ADHD. LPA characterized the variability across substance use (alcohol consumption, cigarette smoking, marijuana use), criminal behavior, peer impairment, educational attainment, maternal relationship, financial dependence, and sexual activity among young adults with childhood ADHD. Childhood predictors of profiles were examined, and ADHD profiles were compared to a matched comparison group without ADHD also followed longitudinally (n = 217). Five profiles were found: prototypic impairment group (54%), high binge-drinking group (17%), high marijuana use group (10%), high criminal activity group (3%), and high cross-domain impairment group (17%). All profiles were impaired compared to non-ADHD young adults. Childhood variables rarely significantly predicted profiles. Young adults with childhood ADHD have differentiable impairment patterns that vary based on substance use, criminal behavior, and number of clinically impaired domains. Nearly all young adult ADHD profiles were impaired in peer, educational, and financial domains, and there was not a nonimpaired ADHD profile. Use of specific substances was elevated among subgroups of, but not all, young adults with ADHD histories. Finally, the high cross-domain impairment profile was impaired in all domains.

Can Machine Learning Improve Screening for Targeted Delinquency Prevention Programs?

The cost-effectiveness of targeted delinquency prevention programs for children depends on the accuracy of the screening process. Screening accuracy is often poor, resulting in wasted resources and missed opportunities to avert negative outcomes. This study examined whether screening approaches based on logistic regression or machine learning algorithms could improve accuracy relative to traditional sum-score approaches when identifying boys in the 5th grade (N = 1012) who would be repeatedly arrested for violent and serious crimes from ages 13 to 30. Screening algorithms were developed that incorporated facets of teacher-reported externalizing problems and other known risk factors (e.g., peer rejection). The predictive performance of these algorithms was evaluated and compared in holdout (i.e., test) data using the area under the receiver operating curve (AUROC) and Brier score. Both the logistic and machine learning methods yielded AUROC superior to traditional sum-score screening approaches when a broad set of risk factors for future delinquency was considered. However, this improvement was modest and was not present when using item-level information from a composite scale assessing externalizing problems. Contrary to expectations, machine learning algorithms performed no better than simple logistic models. There was a large apparent advantage of machine learning that disappeared after appropriate cross-validation, underscoring the importance of careful evaluation of these methods. Results suggest that screening using logistic regression could improve the cost-effectiveness of targeted delinquency prevention programs in some cases, but screening using machine learning would confer no marginal benefit under currently realistic conditions.

A Brief Measure of Language Skills at 3 Years of Age and Special Education Use in Middle Childhood.

OBJECTIVE: To test whether a language screener administered during early childhood predicts special education referrals and placement in middle childhood. STUDY DESIGN: A series of logistic regressions was conducted in a longitudinal study of 731 children. Predictor variables included scores on the early language screener (Fluharty Preschool Speech and Language Screening Test-Second Edition [Fluharty-2]) at ages 3 and 4 years, a standardized measure of academic achievement at age 5 years, and parent report of special education services at ages 7.5, 8.5, and 9.5 years. RESULTS: Results showed that higher scores on the Fluharty-2 predicted a reduced likelihood of having an individualized education program (OR 0.48), being referred for special education (OR 0.55), and being held back a grade (OR 0.37). These findings did not vary by sex, race, or ethnicity, and remained significant after controlling for male sex, behavior problems, parental education, and family income. The Fluharty-2 remained predictive of special education outcomes even after controlling for children's academic skills at age 5 years. CONCLUSIONS: Results suggest that structured, brief assessments of language in early childhood are robust predictors of children's future engagement in special education services and low academic achievement. Primary care physicians may use a multipronged developmental surveillance and monitoring protocol designed to identify children who may need comprehensive evaluation and intervention. Early intervention may reduce the need for costly special education services in the future and reduce comorbid conditions.

Cigarette smoking and ADHD: An examination of prognostically relevant smoking behaviors among adolescents and young adults.

Attention-deficit/hyperactivity disorder (ADHD) is associated with health risks in adolescence which includes the potential for smoking cigarettes, early smoking initiation, and rapid progression to daily smoking. Much less is known, however, about prognostically relevant smoking behaviors among individuals with childhood ADHD. Further research in this area is important for identifying individuals at pronounced risk for nicotine addiction, and for developing effective interventions for this population. This study examined initiation of cigarette smoking, progression to regular smoking, quantity of use, indicators of tobacco dependence, and quit rates among adolescents and young adults with (n = 364) and without (n = 240) childhood ADHD. Individuals with, versus without, ADHD histories were significantly more likely to become daily smokers independent of conduct disorder (CD). They were also more likely to initiate smoking at younger ages and to progress to regular smoking more quickly. There were no significant group differences in cigarettes smoked per day, Fagerström Test of Nicotine Dependence (FTND), or Nicotine Dependence Syndrome Scale (NDSS) scores or in smoking within 30 min of waking. However, smokers with ADHD reported more intense withdrawal and craving during periods of abstinence than non-ADHD smokers. There were no significant group differences in number of quit attempts. Lastly, there were no significant differences among symptom persisters and desisters in daily smoking and amount. Individuals with ADHD histories are at high risk for persistent smoking given their early onset, rapid course, and abstinence characteristics. Smoking cessation programs may need to be adapted or otherwise intensified for those with ADHD. (PsycINFO Database Record

The indirect effects of childhood attention deficit hyperactivity disorder on alcohol problems in adulthood through unique facets of impulsivity.

BACKGROUND AND AIMS: Research has not studied unique impulsivity dimensions as prospective links between childhood attention deficit hyperactivity disorder (ADHD) and alcohol problems. We examined the association between childhood ADHD, five facets of impulsivity and alcohol problems in adulthood and the indirect effects of childhood ADHD-to-impulsivity-to-alcohol problems. DESIGN, SETTING AND PARTICIPANTS: Participants were from a longitudinal study (n = 289, meanage  = 28.67) of individuals with (n = 170) and without (n = 119) childhood ADHD. The ADHD sample, as part of an ADHD summer treatment program, was diagnosed initially in childhood (1987-96, Pittsburgh, PA, USA) via structured parent interview and standardized parent/teacher ratings. The ADHD sample and demographically similar individuals without ADHD (living in the greater Pittsburgh area) were recruited to participate in the Pittsburgh ADHD Longitudinal Study (PALS) between 1999 and 2003. MEASUREMENTS: Self-reported impulsivity (UPPS-P) and alcohol-related problems were assessed in adulthood (2011-14). FINDINGS: Adults with, compared with those without, childhood ADHD had more alcohol problems (ß = 0.34, P < 0.05) and higher levels of negative urgency, positive urgency, lack of perseverance and lack of planning (ßs = 0.24-0.31, Ps < 0.001). Impulsivity facets (except sensation-seeking) were related to number of alcohol problems (ßs = 0.34-0.61, Ps < 0.05). Negative (ß = 0.26, P < 0.01) and positive (ß = 0.27, P < 0.01) urgency mediated the association between childhood ADHD and number of adult alcohol problems. CONCLUSIONS: Elevated levels of emotional impulsivity (negative/positive urgency) may place children with attention deficit hyperactivity disorder at increased risk for alcohol problems in adulthood.

ADHD and Marijuana-Use Expectancies in Young Adulthood.

OBJECTIVE: This study examined mean level differences in marijuana expectancies and the differential associations between expectancies and marijuana use for individuals with and without a history of Attention-Deficit/Hyperactivity Disorder (ADHD). BACKGROUND: Substance-use expectancies are a widely studied risk factor for alcohol and other drug use. The relations between marijuana-use expectancies and self-reported marijuana use have not been examined in young adults with ADHD, a population shown to be at risk for marijuana use. METHOD: Participants were 306 (190 ADHD and 116 non-ADHD) young adults (M age = 20.06, SD = 2.03) from the Pittsburgh ADHD Longitudinal Study (PALS) who provided data about marijuana use and marijuana-use expectancies. RESULTS: Individuals in the ADHD group reported lower levels of social enhancement, tension reduction, and cognitive and behavioral-impairment expectancies compared to individuals in the non-ADHD group. Positive and negative marijuana-use expectancies were associated with marijuana use frequency in the whole sample and statistically significant ADHD group by expectancy interactions were found. Sexual-enhancement expectancies were more strongly associated with marijuana use frequency among individuals with ADHD histories while cognitive behavioral-impairment expectancies were more strongly associated with marijuana use frequency among individuals without ADHD. CONCLUSIONS: Marijuana-use expectancies may be acquired, and operate differently, for individuals with and without ADHD histories. Although future research is needed to test this speculation, these differences may be associated with ADHD-related difficulties in higher order cognitive processes that affect the encoding and utilization of expectations regarding marijuana's effects.

The role of early childhood ADHD and subsequent CD in the initiation and escalation of adolescent cigarette, alcohol, and marijuana use.

Adolescents with attention deficit/hyperactivity disorder (ADHD) are at an increased risk for substance use but the pathways through which this risk emerges are insufficiently understood. Tobacco, alcohol, and marijuana outcomes were compared between adolescents diagnosed with ADHD in early childhood (N = 113) and demographically similar controls (N = 65). Participants were assessed from age 5 until age 18. A comprehensive history of adolescent substance use was compiled for each participant and growth in ADHD and conduct disorder (CD) were modeled as they related to substance use outcomes. Results indicated that when compared with controls, adolescents with ADHD were more likely to try cigarettes, initiate alcohol use at early ages, and smoke marijuana more frequently. Furthermore, adolescents with ADHD were 4 to 5 times more likely than controls to escalate to heavy cigarette and marijuana use after trying these substances once. Adolescents with ADHD who escalated to heavy use patterns were more likely to display early cigarette use and marked problems with family members, but displayed fewer peer problems. There was evidence of baseline effects (latent intercept, measured at age 5) for both ADHD and CD on substance use outcomes. Furthermore, growth in ADHD symptoms accounted for much of the growth in CD symptoms, and consequently, escalating CD symptoms in childhood (latent slope) were viewed as a mediator of the relationship between ADHD and cigarette and marijuana use. Maternal drinking in early childhood was the strongest predictor of early adolescent alcohol use. These findings are discussed with respect to the role of ADHD in the development of adolescent risk outcomes.

Substance use and delinquency among adolescents with childhood ADHD: the protective role of parenting.

Several domains of parenting have been identified as important for adolescent well-being. Whether these same domains are equally beneficial for adolescents with ADHD histories remains an empirical and clinically important question. This study examined whether parental knowledge of their teen's activities and whereabouts, consistency, support, and parent-adolescent conflict are associated with substance use and delinquency similarly for adolescents with and without a diagnosis of ADHD in childhood. A sample of 242 adolescents, 142 diagnosed with ADHD in childhood and prospectively followed into adolescence, and 100 without ADHD in childhood, were the focus of study. The relations between adolescent-reported outcomes (i.e., substance use and delinquency) and parenting behaviors were tested using latent variable modeling to determine both the effects of general (common) and specific (unique) parenting behaviors for participants with and without a history of ADHD. Adolescents' report of parental knowledge was a significant correlate of delinquency and substance use above and beyond other parenting variables and the variance in common across the parenting variables. More knowledge was associated with less delinquency and substance use for all participants, but parental knowledge was more strongly associated with alcohol use for adolescents with versus without childhood ADHD. These correlational findings suggest that, despite the increased difficulty of parenting youths with ADHD histories, actions taken by parents and youth to increase parental awareness may provide some protection against behavioral transgressions known to be elevated in this population.

Coping skills and parent support mediate the association between childhood attention-deficit/hyperactivity disorder and adolescent cigarette use.

OBJECTIVE: To examine mediators of the association between childhood attention-deficit/hyperactivity disorder (ADHD) and adolescent cigarette use. METHOD: Participants were 142 adolescents diagnosed with ADHD in childhood and 100 adolescents without ADHD. RESULTS: Among probands, cigarette smoking was more frequent, adaptive coping skills (behavioral and cognitive) were fewer, and parental support was lower. Coping and support partially mediated the ADHD association with smoking. Persistence of ADHD and adolescent conduct disorder were also important. CONCLUSIONS: Vulnerability to smoking among probands may be partly due to fewer problem-solving resources. Coping skill deficits and parent-child communication may be important treatment targets for smoking prevention.

Childhood predictors of adolescent substance use in a longitudinal study of children with ADHD.

Children diagnosed with attention-deficit/hyperactivity disorder (ADHD; n = 142) were prospectively monitored into adolescence (13-18 years old) to evaluate their risk for elevated substance use relative to same-aged adolescents without ADHD (n = 100). Probands reported higher levels of alcohol, tobacco, and illicit drug use than did controls. Group differences were apparent for alcohol symptom scores but not for alcohol or marijuana disorder diagnoses. Within probands, severity of childhood inattention symptoms predicted multiple substance use outcomes: childhood oppositional defiant disorder/conduct disorder (ODD/CD) symptoms predicted illicit drug use and CD symptoms. Persistence of ADHD and adolescent CD were each associated with elevated substance use behaviors relative to controls. Further study of the mediating mechanisms that explain risk for early substance use and abuse in children with ADHD is warranted.