Mobile toolbox (MTB) remote measures of executive function and processing speed: development and validation.

OBJECTIVE: The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties. METHOD: MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart. RESULTS: All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability. CONCLUSIONS: Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

The NIH Infant and Toddler Toolbox: A new standardized tool for assessing neurodevelopment in children ages 1-42 months.

The progress of developmental research demands reliable measurement techniques, yet inconsistency persist across laboratories, subfields, and settings. To address this, the NIH Toolbox for Neurological and Behavioral Function (NIHTB)® has been extended to create a universal assessment for developmental and pediatric communities: The NIH Infant and Toddler ("Baby") Toolbox. Currently undergoing norming in a sample of 2550 children demographically representative of the 2020 U.S. census, the Baby Toolbox is slated for release in late 2024. This paper introduces the Baby Toolbox, details its objectives, development, and technological innovations, describes plans for reliability and norming, and invites researchers to consider the Baby Toolbox in the future studies. Ultimately, this initiative stands to enhance cross-study comparability and advance comprehensive developmental evaluation.

Using the Mobile Toolbox in child and adolescent samples: A feasibility study.

Cognitive research with developmental samples requires improved methods that support large-scale, diverse, and open science. This paper offers initial evidence to support the Mobile Toolbox (MTB), a self-administered remote smartphone-based cognitive battery, in youth populations, from a pilot sample of 99 children (Mage = 11.79 years; 36% female; 53% White, 33% Black or African American, 9% Asian, and 15% Hispanic). Completion rates (95%-99%), practice performance (96%-100%), internal consistency (0.60-0.98), and correlations with similar NIHTB measures (0.55-0.77) provide the first evidence to support the MTB in a youth sample, although there were some inconsistencies across measures. Preliminary findings provide promising evidence of the MTB in developmental populations, and further studies are encouraged.

Validity of the multidimensional assessment profile of disruptive behavior in autistic toddlers.

Objective: Early measurement of atypical disruptive behavior within autistic children is critical for later referrals to behavioral screenings, diagnoses, and services. Disruptive behavior in autistic toddlers is often measured using a categorical approach and identifies the presence or absence of behavior. In contrast, dimensional approaches evaluate behavior on a spectrum of typical to atypical by measuring the clinical salience of disruptive behavior. We sought to assess the validity of the Infant/Toddler version of the multidimensional assessment profile of disruptive behavior (MAP-DB-IT), a dimensional approach measurement tool, in a sample of autistic toddlers. Methods: Autistic toddlers (n = 82, M age = 33.2 months, SD = 6.28 months) and their mothers received 8 weeks of caregiver-mediated social communication intervention. Mothers completed the MAP-DB-IT and the Infant Toddler Social Emotional Assessment (ITSEA) across three timepoints: before intervention, immediately after intervention, and at 3 months post-intervention follow-up. The MAP-DB-IT provided scores for three subdomains: temper loss, noncompliance, and aggression (generically or specifically with siblings). Ratings on the MAP-DB-IT were compared to the ITSEA using several analytic strategies such as evaluating (a) the internal consistency of the MAP-DB-IT domain scores; (b) the convergent validity between the two measures; and (c) its convergent change due to intervention and if this varied by child characteristics. Results: The MAP-DB-IT demonstrated excellent internal consistency across all four subdomains. We evaluated convergent validity and found positive correlations between the (a) ITSEA externalizing and MAP-DB-IT aggression domain, (b) ITSEA externalizing and MAP-DB-IT aggression with siblings domain, and (c) ITSEA dysregulation and MAP-DB-IT temper loss domain. Conclusion: The MAP-DB-IT is a valid measurement tool for disruptive behavior in autistic toddlers. Clinicians should consider the use of the MAP-DB-IT for young autistic clients presenting with disruptive behavior to (a) discriminate between early developmentally appropriate tantrums from clinically salient dysregulation, and (b) refer to additional behavioral evaluations and services.

Measurement Invariance in Intellectual and Developmental Disability Research.

Measurement invariance (MI) is a psychometric property of an instrument indicating the degree to which scores from an instrument are comparable across groups. In recent years, there has been a marked uptick in publications using MI in intellectual and developmental disability (IDD) samples. Our goal here is to provide an overview of why MI is important to IDD researchers and to describe some challenges to evaluating it, with an eye towards nudging our subfield into a more thoughtful and measured interpretation of studies using MI.

Developmental Associations between Cognition and Adaptive Behavior in Intellectual and Developmental Disability.

Background: Intellectual and developmental disabilities (IDDs) are associated with both cognitive challenges and difficulties in conceptual, social, and practical areas of living (DSM-5). Individuals with IDD often present with an intellectual disability in addition to a developmental disability such as autism or Down syndrome. Those with IDD may present with deficits in intellectual functioning as well as adaptive functioning that interfere with independence and living skills. The present study sought to examine associations of longitudinal developmental change in domains of cognition (NIH Toolbox Cognition Battery, NIHTB-CB) and adaptive behavior domains (Vineland Adaptive Behavior Scales-3; VABS-3) including Socialization, Communication, and Daily Living Skills (DLS) over a two-year period. Methods: Eligible participants for this multisite longitudinal study included those who were between 6 and 26 years at Visit 1, and who had a diagnosis of, or suspected intellectual disability (ID), including borderline ID. Three groups were recruited, including those with fragile X syndrome, Down syndrome, and other/idiopathic intellectual disability. In order to examine the association of developmental change between cognitive and adaptive behavior domains, bivariate latent change score (BLCS) models were fit to compare change in the three cognitive domains measured by the NIHTB-CB (Fluid, Crystallized, Composite) and the three adaptive behavior domains measured by the VABS-3 (Communication, DLS, and Socialization). Results: Over a two-year period, change in cognition (both Crystalized and Composite) was significantly and positively associated with change in daily living skills. Also, baseline cognition level predicted growth in adaptive behavior, however baseline adaptive behavior did not predict growth in cognition in any model. Conclusions: The present study demonstrated that developmental improvements in cognition and adaptive behavior are associated in children and young adults with IDD, indicating the potential for cross-domain effects of intervention. Notably, improvements in Daily Living Skills on the VABS-3 emerged as a primary area of adaptive behavior that positively related to improvements in cognition. This work provides evidence for the clinical, "real life" meaningfulness of the NIHTB-CB in IDD, and important empirical support for the NIHTB-CB as a fit-for-purpose performance-based outcome measure for this population.

Developmental associations between cognition and adaptive behavior in intellectual and developmental disability.

BACKGROUND: Intellectual and developmental disabilities (IDDs) are associated with both cognitive challenges and difficulties in conceptual, social, and practical areas of living, commonly referred to as adaptive behavior (DSM-5). Although cross-sectional associations between intelligence or cognition and adaptive behavior have been reported in IDD populations, no study to date has examined whether developmental changes in cognition contribute to or track with changes in adaptive behavior. The present study sought to examine associations of longitudinal developmental change in domains of cognition (NIH Toolbox Cognition Battery, NIHTB-CB) and adaptive behavior domains (Vineland Adaptive Behavior Scales-3; VABS-3) including Socialization, Communication, and Daily Living Skills (DLS) over a two year period in a large sample of children, adolescents and young adults with IDD. METHODS: Three groups were recruited, including those with fragile X syndrome, Down syndrome, and other/idiopathic intellectual disability. Eligible participants (n = 263) included those who were between 6 and 26 years (mage = 15.52, sd = 5.17) at Visit 1, and who had a diagnosis of, or suspected intellectual disability (ID), including borderline ID, with a mental age of at least 3.0 years. Participants were given cognitive and adaptive behavior assessments at two time points over a two year period (m = 2.45 years, range = 1.27 to 5.56 years). In order to examine the association of developmental change between cognitive and adaptive behavior domains, bivariate latent change score (BLCS) models were fit to compare change in the three cognitive domains measured by the NIHTB-CB (Fluid Cognition, Crystallized Cognition, Total Cognition) and the three adaptive behavior domains measured by the VABS-3 (Communication, DLS, and Socialization). RESULTS: Over a two year period, change in cognition (both Crystallized and Total Composites) was significantly and positively associated with change in daily living skills. Also, baseline cognition level predicted growth in adaptive behavior, however baseline adaptive behavior did not predict growth in cognition in any model. CONCLUSIONS: The present study demonstrated that developmental changes in cognition and adaptive behavior are associated in children and young adults with IDD, indicating the potential for cross-domain effects of intervention. Notably, improvements in DLS emerged as a primary area of adaptive behavior that positively related to improvements in cognition. This work provides evidence for the clinical, "real life" meaningfulness of changes in cognition detected by the NIHTB-CB in IDD, and provides empirical support for the NIHTB-CB as a fit-for-purpose performance-based outcome measure for this population.

Development and validation of an episodic memory measure in the Mobile Toolbox (MTB): Arranging Pictures.

INTRODUCTION: Arranging Pictures is a new episodic memory test based on the NIH Toolbox (NIHTB) Picture Sequence Memory measure and optimized for self-administration on a personal smartphone within the Mobile Toolbox (MTB). We describe evidence from three distinct validation studies. METHOD: In Study 1, 92 participants self-administered Arranging Pictures on study-provided smartphones in the lab and were administered external measures of similar and dissimilar constructs by trained examiners to assess validity under controlled circumstances. In Study 2, 1,021 participants completed the external measures in the lab and self-administered Arranging Pictures remotely on their personal smartphones to assess validity in real-world contexts. In Study 3, 141 participants self-administered Arranging Pictures remotely twice with a two-week delay on personal iOS smartphones to assess test-retest reliability and practice effects. RESULTS: Internal consistency was good across samples (ρxx = .80 to .85, p < .001). Test-retest reliability was marginal (ICC = .49, p < .001) and there were significant practice effects after a two-week delay (ΔM = 3.21 (95% CI [2.56, 3.88]). As expected, correlations with convergent measures were significant and moderate to large in magnitude (ρ = .44 to .76, p < .001), while correlations with discriminant measures were small (ρ = .23 to .27, p < .05) or nonsignificant. Scores demonstrated significant negative correlations with age (ρ = -.32 to -.21, p < .001). Mean performance was slightly higher in the iOS compared to the Android group (MiOS = 18.80, NiOS = 635; MAndroid = 17.11, NAndroid = 386; t(757.73) = 4.17, p < .001), but device type did not significantly influence the psychometric properties of the measure. Indicators of potential cheating were mixed; average scores were significantly higher in the remote samples (F(2, 850) = 11.415, p < .001), but there were not significantly more perfect scores. CONCLUSION: The MTB Arranging Pictures measure demonstrated evidence of reliability and validity when self-administered on personal device. Future research should examine the potential for cheating in remote settings and the properties of the measure in clinical samples.

Development and Validation of a Vocabulary Measure in the Mobile Toolbox.

OBJECTIVE: We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. METHOD: Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. RESULTS: The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). CONCLUSION: The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.