Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics.

Replicability, the ability to replicate scientific findings, is a prerequisite for scientific discovery and clinical utility. Troublingly, we are in the midst of a replicability crisis. A key to replicability is that multiple measurements of the same item (e.g., experimental sample or clinical participant) under fixed experimental constraints are relatively similar to one another. Thus, statistics that quantify the relative contributions of accidental deviations-such as measurement error-as compared to systematic deviations-such as individual differences-are critical. We demonstrate that existing replicability statistics, such as intra-class correlation coefficient and fingerprinting, fail to adequately differentiate between accidental and systematic deviations in very simple settings. We therefore propose a novel statistic, discriminability, which quantifies the degree to which an individual's samples are relatively similar to one another, without restricting the data to be univariate, Gaussian, or even Euclidean. Using this statistic, we introduce the possibility of optimizing experimental design via increasing discriminability and prove that optimizing discriminability improves performance bounds in subsequent inference tasks. In extensive simulated and real datasets (focusing on brain imaging and demonstrating on genomics), only optimizing data discriminability improves performance on all subsequent inference tasks for each dataset. We therefore suggest that designing experiments and analyses to optimize discriminability may be a crucial step in solving the replicability crisis, and more generally, mitigating accidental measurement error.

Predictors of real-time fMRI neurofeedback performance and improvement - A machine learning mega-analysis.

Real-time fMRI neurofeedback is an increasingly popular neuroimaging technique that allows an individual to gain control over his/her own brain signals, which can lead to improvements in behavior in healthy participants as well as to improvements of clinical symptoms in patient populations. However, a considerably large ratio of participants undergoing neurofeedback training do not learn to control their own brain signals and, consequently, do not benefit from neurofeedback interventions, which limits clinical efficacy of neurofeedback interventions. As neurofeedback success varies between studies and participants, it is important to identify factors that might influence neurofeedback success. Here, for the first time, we employed a big data machine learning approach to investigate the influence of 20 different design-specific (e.g. activity vs. connectivity feedback), region of interest-specific (e.g. cortical vs. subcortical) and subject-specific factors (e.g. age) on neurofeedback performance and improvement in 608 participants from 28 independent experiments. With a classification accuracy of 60% (considerably different from chance level), we identified two factors that significantly influenced neurofeedback performance: Both the inclusion of a pre-training no-feedback run before neurofeedback training and neurofeedback training of patients as compared to healthy participants were associated with better neurofeedback performance. The positive effect of pre-training no-feedback runs on neurofeedback performance might be due to the familiarization of participants with the neurofeedback setup and the mental imagery task before neurofeedback training runs. Better performance of patients as compared to healthy participants might be driven by higher motivation of patients, higher ranges for the regulation of dysfunctional brain signals, or a more extensive piloting of clinical experimental paradigms. Due to the large heterogeneity of our dataset, these findings likely generalize across neurofeedback studies, thus providing guidance for designing more efficient neurofeedback studies specifically for improving clinical neurofeedback-based interventions. To facilitate the development of data-driven recommendations for specific design details and subpopulations the field would benefit from stronger engagement in open science research practices and data sharing.

Can we predict real-time fMRI neurofeedback learning success from pretraining brain activity?

Neurofeedback training has been shown to influence behavior in healthy participants as well as to alleviate clinical symptoms in neurological, psychosomatic, and psychiatric patient populations. However, many real-time fMRI neurofeedback studies report large inter-individual differences in learning success. The factors that cause this vast variability between participants remain unknown and their identification could enhance treatment success. Thus, here we employed a meta-analytic approach including data from 24 different neurofeedback studies with a total of 401 participants, including 140 patients, to determine whether levels of activity in target brain regions during pretraining functional localizer or no-feedback runs (i.e., self-regulation in the absence of neurofeedback) could predict neurofeedback learning success. We observed a slightly positive correlation between pretraining activity levels during a functional localizer run and neurofeedback learning success, but we were not able to identify common brain-based success predictors across our diverse cohort of studies. Therefore, advances need to be made in finding robust models and measures of general neurofeedback learning, and in increasing the current study database to allow for investigating further factors that might influence neurofeedback learning.

Patterns of thought: Population variation in the associations between large-scale network organisation and self-reported experiences at rest.

Contemporary cognitive neuroscience recognises unconstrained processing varies across individuals, describing variation in meaningful attributes, such as intelligence. It may also have links to patterns of on-going experience. This study examined whether dimensions of population variation in different modes of unconstrained processing can be described by the associations between patterns of neural activity and self-reports of experience during the same period. We selected 258 individuals from a publicly available data set who had measures of resting-state functional magnetic resonance imaging, and self-reports of experience during the scan. We used machine learning to determine patterns of association between the neural and self-reported data, finding variation along four dimensions. 'Purposeful' experiences were associated with lower connectivity - in particular default mode and limbic networks were less correlated with attention and sensorimotor networks. 'Emotional' experiences were associated with higher connectivity, especially between limbic and ventral attention networks. Experiences focused on themes of 'personal importance' were associated with reduced functional connectivity within attention and control systems. Finally, visual experiences were associated with stronger connectivity between visual and other networks, in particular the limbic system. Some of these patterns had contrasting links with cognitive function as assessed in a separate laboratory session - purposeful thinking was linked to greater intelligence and better abstract reasoning, while a focus on personal importance had the opposite relationship. Together these findings are consistent with an emerging literature on unconstrained states and also underlines that these states are heterogeneous, with distinct modes of population variation reflecting the interplay of different large-scale networks.

Impact of the resolution of brain parcels on connectome-wide association studies in fMRI.

A recent trend in functional magnetic resonance imaging is to test for association of clinical disorders with every possible connection between selected brain parcels. We investigated the impact of the resolution of functional brain parcels, ranging from large-scale networks to local regions, on a mass univariate general linear model (GLM) of connectomes. For each resolution taken independently, the Benjamini-Hochberg procedure controlled the false-discovery rate (FDR) at nominal level on realistic simulations. However, the FDR for tests pooled across all resolutions could be inflated compared to the FDR within resolution. This inflation was severe in the presence of no or weak effects, but became negligible for strong effects. We thus developed an omnibus test to establish the overall presence of true discoveries across all resolutions. Although not a guarantee to control the FDR across resolutions, the omnibus test may be used for descriptive analysis of the impact of resolution on a GLM analysis, in complement to a primary analysis at a predefined single resolution. On three real datasets with significant omnibus test (schizophrenia, congenital blindness, motor practice), markedly higher rate of discovery were obtained at low resolutions, below 50, in line with simulations showing increase in sensitivity at such resolutions. This increase in discovery rate came at the cost of a lower ability to localize effects, as low resolution parcels merged many different brain regions together. However, with 30 or more parcels, the statistical effect maps were biologically plausible and very consistent across resolutions. These results show that resolution is a key parameter for GLM-connectome analysis with FDR control, and that a functional brain parcellation with 30 to 50 parcels may lead to an accurate summary of full connectome effects with good sensitivity in many situations.

Moving beyond processing- and analysis-related variation in resting-state functional brain imaging.

When fields lack consensus standard methods and accessible ground truths, reproducibility can be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools and processing pipelines. We provide a critical evaluation of the impact of differences across five independently developed minimal preprocessing pipelines for functional magnetic resonance imaging. We show that, even when handling identical data, interpipeline agreement was only moderate, critically shedding light on a factor that limits cross-study reproducibility. We show that low interpipeline agreement can go unrecognized until the reliability of the underlying data is high, which is increasingly the case as the field progresses. Crucially we show that, when interpipeline agreement is compromised, so too is the consistency of insights from brain-wide association studies. We highlight the importance of comparing analytic configurations, because both widely discussed and commonly overlooked decisions can lead to marked variation.

Exploring Post COVID-19 Outbreak Intradaily Mobility Pattern Change in College Students: A GPS-Focused Smartphone Sensing Study.

With the outbreak of the COVID-19 pandemic in 2020, most colleges and universities move to restrict campus activities, reduce indoor gatherings and move instruction online. These changes required that students adapt and alter their daily routines accordingly. To investigate patterns associated with these behavioral changes, we collected smartphone sensing data using the Beiwe platform from two groups of undergraduate students at a major North American university, one from January to March of 2020 (74 participants), the other from May to August (52 participants), to observe the differences in students' daily life patterns before and after the start of the pandemic. In this paper, we focus on the mobility patterns evidenced by GPS signal tracking from the students' smartphones and report findings using several analytical methods including principal component analysis, circadian rhythm analysis, and predictive modeling of perceived sadness levels using mobility-based digital metrics. Our findings suggest that compared to the pre-COVID group, students in the mid-COVID group generally 1) registered a greater amount of midday movement than movement in the morning (8-10 a.m.) and in the evening (7-9 p.m.), as opposed to the other way around; 2) exhibited significantly less intradaily variability in their daily movement; 3) visited less places and stayed at home more everyday, and; 4) had a significant lower correlation between their mobility patterns and negative mood.

The NKI-Rockland Sample: A Model for Accelerating the Pace of Discovery Science in Psychiatry.

The National Institute of Mental Health strategic plan for advancing psychiatric neuroscience calls for an acceleration of discovery and the delineation of developmental trajectories for risk and resilience across the lifespan. To attain these objectives, sufficiently powered datasets with broad and deep phenotypic characterization, state-of-the-art neuroimaging, and genetic samples must be generated and made openly available to the scientific community. The enhanced Nathan Kline Institute-Rockland Sample (NKI-RS) is a response to this need. NKI-RS is an ongoing, institutionally centered endeavor aimed at creating a large-scale (N > 1000), deeply phenotyped, community-ascertained, lifespan sample (ages 6-85 years old) with advanced neuroimaging and genetics. These data will be publically shared, openly, and prospectively (i.e., on a weekly basis). Herein, we describe the conceptual basis of the NKI-RS, including study design, sampling considerations, and steps to synchronize phenotypic and neuroimaging assessment. Additionally, we describe our process for sharing the data with the scientific community while protecting participant confidentiality, maintaining an adequate database, and certifying data integrity. The pilot phase of the NKI-RS, including challenges in recruiting, characterizing, imaging, and sharing data, is discussed while also explaining how this experience informed the final design of the enhanced NKI-RS. It is our hope that familiarity with the conceptual underpinnings of the enhanced NKI-RS will facilitate harmonization with future data collection efforts aimed at advancing psychiatric neuroscience and nosology.