REFORMS: Consensus-based Recommendations for Machine-learning-based Science.

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

hyve, a compositional visualisation engine for brain imaging data.

Visualisations facilitate the interpretation of geometrically structured data and results. However, heterogeneous geometries-such as volumes, surfaces, and networks-have traditionally mandated different software approaches. We introduce hyve, a Python library that uses a compositional functional framework to enable parametric implementation of custom visualisations for different brain geometries. Under this framework, users compose a reusable visualisation protocol from geometric primitives for representing data geometries, input primitives for common data formats and research objectives, and output primitives for producing interactive displays or configurable snapshots. hyve also writes documentation for user-constructed protocols, automates serial production of multiple visualisations, and includes an API for semantically organising an editable multi-panel figure. Through the seamless composition of input, output, and geometric primitives, hyve supports creating visualisations for a range of neuroimaging research objectives.

Impact of analytic decisions on test-retest reliability of individual and group estimates in functional magnetic resonance imaging: a multiverse analysis using the monetary incentive delay task.

Empirical studies reporting low test-retest reliability of individual blood oxygen-level dependent (BOLD) signal estimates in functional magnetic resonance imaging (fMRI) data have resurrected interest among cognitive neuroscientists in methods that may improve reliability in fMRI. Over the last decade, several individual studies have reported that modeling decisions, such as smoothing, motion correction and contrast selection, may improve estimates of test-retest reliability of BOLD signal estimates. However, it remains an empirical question whether certain analytic decisions consistently improve individual and group level reliability estimates in an fMRI task across multiple large, independent samples. This study used three independent samples ( N s: 60, 81, 119) that collected the same task (Monetary Incentive Delay task) across two runs and two sessions to evaluate the effects of analytic decisions on the individual (intraclass correlation coefficient [ICC(3,1)]) and group (Jaccard/Spearman rho ) reliability estimates of BOLD activity of task fMRI data. The analytic decisions in this study vary across four categories: smoothing kernel (five options), motion correction (four options), task parameterizing (three options) and task contrasts (four options), totaling 240 different pipeline permutations. Across all 240 pipelines, the median ICC estimates are consistently low, with a maximum median ICC estimate of .43 - .55 across the three samples. The analytic decisions with the greatest impact on the median ICC and group similarity estimates are the Implicit Baseline contrast, Cue Model parameterization and a larger smoothing kernel. Using an Implicit Baseline in a contrast condition meaningfully increased group similarity and ICC estimates as compared to using the Neutral cue. This effect was largest for the Cue Model parameterization, however, improvements in reliability came at the cost of interpretability. This study illustrates that estimates of reliability in the MID task are consistently low and variable at small samples, and a higher test-retest reliability may not always improve interpretability of the estimated BOLD signal.

The Past, Present, and Future of the Brain Imaging Data Structure (BIDS).

The Brain Imaging Data Structure (BIDS) is a community-driven standard for the organization of data and metadata from a growing range of neuroscience modalities. This paper is meant as a history of how the standard has developed and grown over time. We outline the principles behind the project, the mechanisms by which it has been extended, and some of the challenges being addressed as it evolves. We also discuss the lessons learned through the project, with the aim of enabling researchers in other domains to learn from the success of BIDS.

The response time paradox in functional magnetic resonance imaging analyses.

Response times (RTs) are often the main signal of interest in cognitive psychology but are often ignored in functional MRI (fMRI) analyses. In fMRI analysis the intensity of the signal serves as a proxy for the intensity of local neuronal activity, but changes in either the intensity or the duration of neuronal activity can yield identical fMRI signals. Therefore, if RTs are ignored and pair with neuronal durations, fMRI results claiming intensity differences may be confounded by RTs. We show how ignoring RTs goes beyond this confound, where longer RTs are paired with larger activation estimates, to lesser-known issues where RTs become confounds in group-level analyses and, surprisingly, how the RT confound can induce other artificial group-level associations with variables that are not related to the condition contrast or RTs. We propose a new time-series model to address these issues and encourage increasing focus on what the widespread RT-based signal represents.

A multi-sample evaluation of the measurement structure and function of the modified monetary incentive delay task in adolescents.

Interpreting the neural response elicited during task functional magnetic resonance imaging (fMRI) remains a challenge in neurodevelopmental research. The monetary incentive delay (MID) task is an fMRI reward processing task that is extensively used in the literature. However, modern psychometric tools have not been used to evaluate measurement properties of the MID task fMRI data. The current study uses data for a similar task design across three adolescent samples (N = 346 [Agemean 12.0; 44 % Female]; N = 97 [19.3; 58 %]; N = 112 [20.2; 38 %]) to evaluate multiple measurement properties of fMRI responses on the MID task. Confirmatory factor analysis (CFA) is used to evaluate an a priori theoretical model for the task and its measurement invariance across three samples. Exploratory factor analysis (EFA) is used to identify the data-driven measurement structure across the samples. CFA results suggest that the a priori model is a poor representation of these MID task fMRI data. Across the samples, the data-driven EFA models consistently identify a six-to-seven factor structure with run and bilateral brain region factors. This factor structure is moderately-to-highly congruent across the samples. Altogether, these findings demonstrate a need to evaluate theoretical frameworks for popular fMRI task designs to improve our understanding and interpretation of brain-behavior associations.

Functional neuroimaging as a catalyst for integrated neuroscience.

Functional magnetic resonance imaging (fMRI) enables non-invasive access to the awake, behaving human brain. By tracking whole-brain signals across a diverse range of cognitive and behavioural states or mapping differences associated with specific traits or clinical conditions, fMRI has advanced our understanding of brain function and its links to both normal and atypical behaviour. Despite this headway, progress in human cognitive neuroscience that uses fMRI has been relatively isolated from rapid advances in other subdomains of neuroscience, which themselves are also somewhat siloed from one another. In this Perspective, we argue that fMRI is well-placed to integrate the diverse subfields of systems, cognitive, computational and clinical neuroscience. We first summarize the strengths and weaknesses of fMRI as an imaging tool, then highlight examples of studies that have successfully used fMRI in each subdomain of neuroscience. We then provide a roadmap for the future advances that will be needed to realize this integrative vision. In this way, we hope to demonstrate how fMRI can help usher in a new era of interdisciplinary coherence in neuroscience.

A comparison of neuroelectrophysiology databases.

As data sharing has become more prevalent, three pillars - archives, standards, and analysis tools - have emerged as critical components in facilitating effective data sharing and collaboration. This paper compares four freely available intracranial neuroelectrophysiology data repositories: Data Archive for the BRAIN Initiative (DABI), Distributed Archives for Neurophysiology Data Integration (DANDI), OpenNeuro, and Brain-CODE. The aim of this review is to describe archives that provide researchers with tools to store, share, and reanalyze both human and non-human neurophysiology data based on criteria that are of interest to the neuroscientific community. The Brain Imaging Data Structure (BIDS) and Neurodata Without Borders (NWB) are utilized by these archives to make data more accessible to researchers by implementing a common standard. As the necessity for integrating large-scale analysis into data repository platforms continues to grow within the neuroscientific community, this article will highlight the various analytical and customizable tools developed within the chosen archives that may advance the field of neuroinformatics.

Controversies and progress on standardization of large-scale brain network nomenclature.

Progress in scientific disciplines is accompanied by standardization of terminology. Network neuroscience, at the level of macroscale organization of the brain, is beginning to confront the challenges associated with developing a taxonomy of its fundamental explanatory constructs. The Workgroup for HArmonized Taxonomy of NETworks (WHATNET) was formed in 2020 as an Organization for Human Brain Mapping (OHBM)-endorsed best practices committee to provide recommendations on points of consensus, identify open questions, and highlight areas of ongoing debate in the service of moving the field toward standardized reporting of network neuroscience results. The committee conducted a survey to catalog current practices in large-scale brain network nomenclature. A few well-known network names (e.g., default mode network) dominated responses to the survey, and a number of illuminating points of disagreement emerged. We summarize survey results and provide initial considerations and recommendations from the workgroup. This perspective piece includes a selective review of challenges to this enterprise, including (1) network scale, resolution, and hierarchies; (2) interindividual variability of networks; (3) dynamics and nonstationarity of networks; (4) consideration of network affiliations of subcortical structures; and (5) consideration of multimodal information. We close with minimal reporting guidelines for the cognitive and network neuroscience communities to adopt.

Cognitive tasks, anatomical MRI, and functional MRI data evaluating the construct of self-regulation.

We describe the following shared data from N=103 healthy adults who completed a broad set cognitive tasks, surveys, and neuroimaging measurements to examine the construct of self-regulation. The neuroimaging acquisition involved task-based fMRI, resting fMRI, and structural MRI. Each subject completed the following ten tasks in the scanner across two 90-minute scanning sessions: attention network test (ANT), cued task switching, Columbia card task, dot pattern expectancy (DPX), delay discounting, simple and motor selective stop signal, Stroop, a towers task, and a set of survey questions. Subjects also completed resting state scans. The dataset is shared openly through the OpenNeuro project, and the dataset is formatted according to the Brain Imaging Data Structure (BIDS) standard.

A Comparison of Neuroelectrophysiology Databases.

As data sharing has become more prevalent, three pillars - archives, standards, and analysis tools - have emerged as critical components in facilitating effective data sharing and collaboration. This paper compares four freely available intracranial neuroelectrophysiology data repositories: Data Archive for the BRAIN Initiative (DABI), Distributed Archives for Neurophysiology Data Integration (DANDI), OpenNeuro, and Brain-CODE. The aim of this review is to describe archives that provide researchers with tools to store, share, and reanalyze both human and non-human neurophysiology data based on criteria that are of interest to the neuroscientific community. The Brain Imaging Data Structure (BIDS) and Neurodata Without Borders (NWB) are utilized by these archives to make data more accessible to researchers by implementing a common standard. As the necessity for integrating large-scale analysis into data repository platforms continues to grow within the neuroscientific community, this article will highlight the various analytical and customizable tools developed within the chosen archives that may advance the field of neuroinformatics.

Improving the Rigor of Mechanistic Behavioral Science: The Introduction of the Checklist for Investigating Mechanisms in Behavior-Change Research (CLIMBR).

Diverse fields rely on the development of effective interventions to change human behaviors, such as following prescribed medical regimens, engaging in recommended levels of physical activity, getting vaccinations that promote individual and public health, and getting a healthy amount of sleep. Despite recent advancements in behavioral intervention development and behavior-change science, systematic progress is stalled by the lack of a systematic approach to identifying and targeting mechanisms of action that underlie successful behavior change. Further progress in behavioral intervention science requires that mechanisms be universally prespecified, measurable, and malleable. We developed the CheckList for Investigating Mechanisms in Behavior-change Research (CLIMBR) to guide basic and applied researchers in the planning and reporting of manipulations and interventions relevant to understanding the underlying active ingredients that do-or do not-drive successful change in behavioral outcomes. We report the rationale for creating CLIMBR and detail the processes of its development and refinement based on feedback from behavior-change experts and NIH officials. The final version of CLIMBR is included in full.

Benchmarking explanation methods for mental state decoding with deep learning models.

Deep learning (DL) models find increasing application in mental state decoding, where researchers seek to understand the mapping between mental states (e.g., experiencing anger or joy) and brain activity by identifying those spatial and temporal features of brain activity that allow to accurately identify (i.e., decode) these states. Once a DL model has been trained to accurately decode a set of mental states, neuroimaging researchers often make use of methods from explainable artificial intelligence research to understand the model's learned mappings between mental states and brain activity. Here, we benchmark prominent explanation methods in a mental state decoding analysis of multiple functional Magnetic Resonance Imaging (fMRI) datasets. Our findings demonstrate a gradient between two key characteristics of an explanation in mental state decoding, namely, its faithfulness and its alignment with other empirical evidence on the mapping between brain activity and decoded mental state: explanation methods with high explanation faithfulness, which capture the model's decision process well, generally provide explanations that align less well with other empirical evidence than the explanations of methods with less faithfulness. Based on our findings, we provide guidance for neuroimaging researchers on how to choose an explanation method to gain insight into the mental state decoding decisions of DL models.

On the Long-term Archiving of Research Data.

Accessing research data at any time is what FAIR (Findable Accessible Interoperable Reusable) data sharing aims to achieve at scale. Yet, we argue that it is not sustainable to keep accumulating and maintaining all datasets for rapid access, considering the monetary and ecological cost of maintaining repositories. Here, we address the issue of cold data storage: when to dispose of data for offline storage, how can this be done while maintaining FAIR principles and who should be responsible for cold archiving and long-term preservation.

Modelling human behaviour in cognitive tasks with latent dynamical systems.

Response time data collected from cognitive tasks are a cornerstone of psychology and neuroscience research, yet existing models of these data either make strong assumptions about the data-generating process or are limited to modelling single trials. We introduce task-DyVA, a deep learning framework in which expressive dynamical systems are trained to reproduce sequences of response times observed in data from individual human subjects. Models fitted to a large task-switching dataset captured subject-specific behavioural differences with high temporal precision, including task-switching costs. Through perturbation experiments and analyses of the models' latent dynamics, we find support for a rational account of switch costs in terms of a stability-flexibility trade-off. Thus, our framework can be used to discover interpretable cognitive theories that explain how the brain dynamically gives rise to behaviour.

A dual-task approach to inform the taxonomy of inhibition-related processes.

Response inhibition is key to controlled behavior and is commonly investigated with the stop-signal paradigm. The authors investigated how response inhibition is situated within a taxonomy of control processes by combining multiple forms of control within dual tasks. Response inhibition, as measured by stop-signal reaction time (SSRT), was impaired when combined with shape matching, but not the flanker task, and when combined with cued task switching, but not predictable task switching, suggesting that response inhibition may be weakly or variably impaired when combined with selective attention and set shifting demands, respectively. Response inhibition was also consistently impaired when combined with the N-back or directed forgetting tasks, putative measures of working memory. Impairments of response inhibition by other control demands appeared to be primarily driven by task context, as SSRT slowing was similar for trials where control demands were either high (e.g., task switch) or low (e.g., task stay). These results demonstrate that response inhibition processes are often impaired in the context of other control demands, even on trials where direct engagement of those other control processes is not required. This suggests a taxonomy of control in which response inhibition overlaps with related control processes, especially working memory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

TemplateFlow: FAIR-sharing of multi-scale, multi-species brain models.

Reference anatomies of the brain ('templates') and corresponding atlases are the foundation for reporting standardized neuroimaging results. Currently, there is no registry of templates and atlases; therefore, the redistribution of these resources occurs either bundled within existing software or in ad hoc ways such as downloads from institutional sites and general-purpose data repositories. We introduce TemplateFlow as a publicly available framework for human and non-human brain models. The framework combines an open database with software for access, management, and vetting, allowing scientists to share their resources under FAIR-findable, accessible, interoperable, and reusable-principles. TemplateFlow enables multifaceted insights into brains across species, and supports multiverse analyses testing whether results generalize across standard references, scales, and in the long term, species.

NEMAR: an open access data, tools and compute resource operating on neuroelectromagnetic data.

To preserve scientific data created by publicly and/or philanthropically funded research projects and to make it ready for exploitation using recent and ongoing advances in advanced and large-scale computational modeling methods, publicly available data must use in common, now-evolving standards for formatting, identifying and annotating should share data. The OpenNeuro.org archive, built first as a repository for magnetic resonance imaging data based on the Brain Imaging Data Structure formatting standards, aims to house and share all types of human neuroimaging data. Here, we present NEMAR.org, a web gateway to OpenNeuro data for human neuroelectromagnetic data. NEMAR allows users to search through, visually explore and assess the quality of shared electroencephalography (EEG), magnetoencephalography and intracranial EEG data and then to directly process selected data using high-performance computing resources of the San Diego Supercomputer Center via the Neuroscience Gateway (nsgportal.org, NSG), a freely available web portal to high-performance computing serving a variety of neuroscientific analysis environments and tools. Combined, OpenNeuro, NEMAR and NSG form an efficient, integrated data, tools and compute resource for human neuroimaging data analysis and meta-analysis. Database URL: https://nemar.org.