Mobile toolbox (MTB) remote measures of executive function and processing speed: development and validation.

OBJECTIVE: The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties. METHOD: MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart. RESULTS: All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability. CONCLUSIONS: Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

Challenges in Participant Engagement and Retention Using Mobile Health Apps: Literature Review.

BACKGROUND: Mobile health (mHealth) apps are revolutionizing the way clinicians and researchers monitor and manage the health of their participants. However, many studies using mHealth apps are hampered by substantial participant dropout or attrition, which may impact the representativeness of the sample and the effectiveness of the study. Therefore, it is imperative for researchers to understand what makes participants stay with mHealth apps or studies using mHealth apps. OBJECTIVE: This study aimed to review the current peer-reviewed research literature to identify the notable factors and strategies used in adult participant engagement and retention. METHODS: We conducted a systematic search of PubMed, MEDLINE, and PsycINFO databases for mHealth studies that evaluated and assessed issues or strategies to improve the engagement and retention of adults from 2015 to 2020. We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Notable themes were identified and narratively compared among different studies. A binomial regression model was generated to examine the factors affecting retention. RESULTS: Of the 389 identified studies, 62 (15.9%) were included in this review. Overall, most studies were partially successful in maintaining participant engagement. Factors related to particular elements of the app (eg, feedback, appropriate reminders, and in-app support from peers or coaches) and research strategies (eg, compensation and niche samples) that promote retention were identified. Factors that obstructed retention were also identified (eg, lack of support features, technical difficulties, and usefulness of the app). The regression model results showed that a participant is more likely to drop out than to be retained. CONCLUSIONS: Retaining participants is an omnipresent challenge in mHealth studies. The insights from this review can help inform future studies about the factors and strategies to improve participant retention.

Closing the gap: addressing telehealth disparities across specialties in the sustained pandemic era.

Missed appointments, or no-shows, disrupt healthcare delivery, exacerbating chronic disease management and leading to worse health outcomes. Telehealth has surged as a viable solution to reduce no-shows and improve healthcare accessibility, especially during the COVID-19 pandemic. However, telehealth disparities and its long-term efficacy across various medical specialties remain understudied. To address this, we performed a retrospective analysis of electronic health records from a heterogenous network of hospitals in Illinois, examining telehealth use and no-shows across among 444,752 adult patients with 1,973,098 outpatient encounters across nine specialties during the sustained pandemic phase (i.e., January 1, 2021 to July 1, 2022). Among them, 84,290 (4.27%) were no-shows, and telehealth constituted 202,933 (10.3%) of the total encounters. Telehealth use during the sustained phase varied significantly by specialty type. Overall, telehealth encounters were associated with reduced no-show odds compared to in-person encounters (OR, 0.28; 95% CI, 0.26-0.29). Black and Hispanic patients, as well as those with Medicaid, had higher no-show odds relative to their counterparts, even when using telehealth. Mental health specialty had the highest telehealth usage rate and the highest no-show odds (OR, 2.99; 95% CI, 2.84-3.14) relative to other specialties included in the study. Moreover, specialty type had differential effects on no-shows for telehealth. These results underscore the variability in telehealth use by specialty type and pervasive disparities telehealth use and no-shows. As we move beyond the pandemic, our findings can inform policymakers to tailor policies and incentives to reach different patient groups as well as specialties, with varying needs, to promote equitable telehealth utilization.

Development and validation of an episodic memory measure in the Mobile Toolbox (MTB): Arranging Pictures.

INTRODUCTION: Arranging Pictures is a new episodic memory test based on the NIH Toolbox (NIHTB) Picture Sequence Memory measure and optimized for self-administration on a personal smartphone within the Mobile Toolbox (MTB). We describe evidence from three distinct validation studies. METHOD: In Study 1, 92 participants self-administered Arranging Pictures on study-provided smartphones in the lab and were administered external measures of similar and dissimilar constructs by trained examiners to assess validity under controlled circumstances. In Study 2, 1,021 participants completed the external measures in the lab and self-administered Arranging Pictures remotely on their personal smartphones to assess validity in real-world contexts. In Study 3, 141 participants self-administered Arranging Pictures remotely twice with a two-week delay on personal iOS smartphones to assess test-retest reliability and practice effects. RESULTS: Internal consistency was good across samples (ρxx = .80 to .85, p < .001). Test-retest reliability was marginal (ICC = .49, p < .001) and there were significant practice effects after a two-week delay (ΔM = 3.21 (95% CI [2.56, 3.88]). As expected, correlations with convergent measures were significant and moderate to large in magnitude (ρ = .44 to .76, p < .001), while correlations with discriminant measures were small (ρ = .23 to .27, p < .05) or nonsignificant. Scores demonstrated significant negative correlations with age (ρ = -.32 to -.21, p < .001). Mean performance was slightly higher in the iOS compared to the Android group (MiOS = 18.80, NiOS = 635; MAndroid = 17.11, NAndroid = 386; t(757.73) = 4.17, p < .001), but device type did not significantly influence the psychometric properties of the measure. Indicators of potential cheating were mixed; average scores were significantly higher in the remote samples (F(2, 850) = 11.415, p < .001), but there were not significantly more perfect scores. CONCLUSION: The MTB Arranging Pictures measure demonstrated evidence of reliability and validity when self-administered on personal device. Future research should examine the potential for cheating in remote settings and the properties of the measure in clinical samples.

Shared decision-making and disease management in advanced cancer and chronic kidney disease using patient-reported outcome dashboards.

OBJECTIVES: To assess the use of a co-designed patient-reported outcome (PRO) clinical dashboard and estimate its impact on shared decision-making (SDM) and symptomatology in adults with advanced cancer or chronic kidney disease (CKD). MATERIALS AND METHODS: We developed a clinical PRO dashboard within the Northwestern Medicine Patient-Reported Outcomes system, enhanced through co-design involving 20 diverse constituents. Using a single-group, pretest-posttest design, we evaluated the dashboard's use among patients with advanced cancer or CKD between June 2020 and January 2022. Eligible patients had a visit with a participating clinician, completed at least two dashboard-eligible visits, and consented to follow-up surveys. PROs were collected 72 h prior to visits, including measures for chronic condition management self-efficacy, health-related quality of life (PROMIS measures), and SDM (collaboRATE). Responses were integrated into the EHR dashboard and accessible to clinicians and patients. RESULTS: We recruited 157 participants: 66 with advanced cancer and 91 with CKD. There were significant improvements in SDM from baseline, as assessed by collaboRATE scores. The proportion of participants reporting the highest level of SDM on every collaboRATE item increased by 15 percentage points from baseline to 3 months, and 17 points between baseline and 6-month follow-up. Additionally, there was a clinically meaningful decrease in anxiety levels over study period (T-score baseline: 53; 3-month: 52; 6-month: 50; P < .001), with a standardized response mean (SRM) of -0.38 at 6 months. DISCUSSION: PRO clinical dashboards, developed and shared with patients, may enhance SDM and reduce anxiety among patients with advanced cancer and CKD.

Development and Validation of a Vocabulary Measure in the Mobile Toolbox.

OBJECTIVE: We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. METHOD: Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. RESULTS: The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). CONCLUSION: The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.

A Common Longitudinal Intensive Care Unit data Format (CLIF) to enable multi-institutional federated critical illness research.

Background: Critical illness, or acute organ failure requiring life support, threatens over five million American lives annually. Electronic health record (EHR) data are a source of granular information that could generate crucial insights into the nature and optimal treatment of critical illness. However, data management, security, and standardization are barriers to large-scale critical illness EHR studies. Methods: A consortium of critical care physicians and data scientists from eight US healthcare systems developed the Common Longitudinal Intensive Care Unit (ICU) data Format (CLIF), an open-source database format that harmonizes a minimum set of ICU Data Elements for use in critical illness research. We created a pipeline to process adult ICU EHR data at each site. After development and iteration, we conducted two proof-of-concept studies with a federated research architecture: 1) an external validation of an in-hospital mortality prediction model for critically ill patients and 2) an assessment of 72-hour temperature trajectories and their association with mechanical ventilation and in-hospital mortality using group-based trajectory models. Results: We converted longitudinal data from 94,356 critically ill patients treated in 2020-2021 (mean age 60.6 years [standard deviation 17.2], 30% Black, 7% Hispanic, 45% female) across 8 health systems and 33 hospitals into the CLIF format, The in-hospital mortality prediction model performed well in the health system where it was derived (0.81 AUC, 0.06 Brier score). Performance across CLIF consortium sites varied (AUCs: 0.74-0.83, Brier scores: 0.06-0.01), and demonstrated some degradation in predictive capability. Temperature trajectories were similar across health systems. Hypothermic and hyperthermic-slow-resolver patients consistently had the highest mortality. Conclusions: CLIF facilitates efficient, rigorous, and reproducible critical care research. Our federated case studies showcase CLIF's potential for disease sub-phenotyping and clinical decision-support evaluation. Future applications include pragmatic EHR-based trials, target trial emulations, foundational multi-modal AI models of critical illness, and real-time critical care quality dashboards.

Validity and reliability of the Mobile Toolbox Faces and Names memory test.

Validation of the Mobile Toolbox Faces and Names associative memory test is presented. Ninety-two participants self-administered Faces and Names in-person; 956 self-administered Faces and Names remotely but took convergent measures in person; and 123 self-administered Faces and Names remotely twice, 14 days apart. Internal consistency (.76-.79) and test-retest reliability (ICC = .73) were acceptable. Convergent validity with WMS-IV Verbal Paired Associates was satisfactory (immediate .54; delayed .58). The findings suggest the remotely administered Faces and Names is a reliable instrument.

Face Name Associative Memory Exam and biomarker status in the ARMADA study: Advancing reliable measurement in Alzheimer's disease and cognitive aging.

The Face Name Associative Memory Exam (FNAME) was introduced into the NIH Toolbox as part of the ARMADA study and establishes normative data for diverse participants, ages 64 to 85+, and proposes cutoff scores between biomarker positive versus negative (+/-) groups. The FNAME was administered to 257 participants across the clinical spectrum with 122 having amyloid biomarkers. Linear regression explored the association between demographics and FNAME and between amyloid (+/-) groups. Receiver operating characteristic curves (ROC) identified performance thresholds that best discriminated between biomarker (+/-) individuals. Lower FNAME scores occurred in males, older ages, Black/African Americans, Hispanics, and biomarker-positive participants. ROC analyses demonstrated acceptable accuracy (0.73 to 0.77) but only when combined with clinical status. The diagnostic discrimination of amyloid positivity was acceptable but not excellent, suggesting the FNAME may be a better screening indicator of clinical status rather than amyloid deposition in cognitively normal individuals. Normative data are provided.

Uses of the NIH Toolbox® in Clinical Samples: A Scoping Review.

Background and Objectives: The NIH Toolbox® for the Assessment of Neurologic and Behavioral Function is a compilation of computerized measures designed to assess sensory, motor, emotional, and cognitive functioning of individuals across the life span. The NIH Toolbox was initially developed for use with the general population and was not originally validated in clinical populations. The objective of this scoping review was to assess the extent to which the NIH Toolbox has been used with clinical populations. Methods: Guided by the Joanna Briggs Methods Manual for Scoping Reviews, records were identified through searches of PubMed MEDLINE, PsycINFO, ClinicalTrials.gov, EMBASE, and ProQuest Dissertations and Theses Global (2008-2020). Database searches yielded 5,693 unique titles of original research that used at least one NIH Toolbox assessment in a sample characterized by any clinical diagnosis. Two reviewers screened titles, abstracts, and full texts for inclusion in duplicate. Conflicts at each stage of the review process were resolved by a group discussion. Results: Ultimately, 281 publication records were included in this scoping review (nJournal Articles = 104, nConference Abstracts = 84, nClinical Trial Registrations = 86, and nTheses/Dissertations = 7). The NIH Toolbox Cognition Battery was by far the most used of the 4 batteries in the measurement system (nCognition = 225, nEmotion = 49, nMotor = 29, and nSensation = 16). The most represented clinical category was neurologic disorders (n = 111), followed by psychological disorders (n = 39) and cancer (n = 31). Most (96.8%) of the journal articles and conference abstracts reporting the use of NIH Toolbox measures with clinical samples were published in 2015 or later. As of May 2021, these records had been cited a total of nearly 1,000 times. Discussion: The NIH Toolbox measures have been widely used among individuals with various clinical conditions across the life span. Our results lay the groundwork to support the feasibility and utility of administering the NIH Toolbox measures in research conducted with clinical populations and further suggest that these measures may be of value for implementation in fast-paced clinical settings as part of routine practice.

Translational implications of Th17-skewed inflammation due to genetic deficiency of a cadherin stress sensor.

Desmoglein 1 (Dsg1) is a cadherin restricted to stratified tissues of terrestrial vertebrates, which serve as essential physical and immune barriers. Dsg1 loss-of-function mutations in humans result in skin lesions and multiple allergies, and isolated patient keratinocytes exhibit increased proallergic cytokine expression. However, the mechanism by which genetic deficiency of Dsg1 causes chronic inflammation is unknown. To determine the systemic response to Dsg1 loss, we deleted the 3 tandem Dsg1 genes in mice. Whole transcriptome analysis of embryonic Dsg1-/- skin showed a delay in expression of adhesion/differentiation/keratinization genes at E17.5, a subset of which recovered or increased by E18.5. Comparing epidermal transcriptomes from Dsg1-deficient mice and humans revealed a shared IL-17-skewed inflammatory signature. Although the impaired intercellular adhesion observed in Dsg1-/- mice resembles that resulting from anti-Dsg1 pemphigus foliaceus antibodies, pemphigus skin lesions exhibit a weaker IL-17 signature. Consistent with the clinical importance of these findings, treatment of 2 Dsg1-deficient patients with an IL-12/IL-23 antagonist originally developed for psoriasis resulted in improvement of skin lesions. Thus, beyond impairing the physical barrier, loss of Dsg1 function through gene mutation results in a psoriatic-like inflammatory signature before birth, and treatment with a targeted therapy significantly improved skin lesions in patients.