Open and reproducible neuroimaging: From study inception to publication.

Empirical observations of how labs conduct research indicate that the adoption rate of open practices for transparent, reproducible, and collaborative science remains in its infancy. This is at odds with the overwhelming evidence for the necessity of these practices and their benefits for individual researchers, scientific progress, and society in general. To date, information required for implementing open science practices throughout the different steps of a research project is scattered among many different sources. Even experienced researchers in the topic find it hard to navigate the ecosystem of tools and to make sustainable choices. Here, we provide an integrated overview of community-developed resources that can support collaborative, open, reproducible, replicable, robust and generalizable neuroimaging throughout the entire research cycle from inception to publication and across different neuroimaging modalities. We review tools and practices supporting study inception and planning, data acquisition, research data management, data processing and analysis, and research dissemination. An online version of this resource can be found at https://oreoni.github.io. We believe it will prove helpful for researchers and institutions to make a successful and sustainable move towards open and reproducible science and to eventually take an active role in its future development.

The Open Brain Consent: Informing research participants and obtaining consent to share brain imaging data.

Having the means to share research data openly is essential to modern science. For human research, a key aspect in this endeavor is obtaining consent from participants, not just to take part in a study, which is a basic ethical principle, but also to share their data with the scientific community. To ensure that the participants' privacy is respected, national and/or supranational regulations and laws are in place. It is, however, not always clear to researchers what the implications of those are, nor how to comply with them. The Open Brain Consent (https://open-brain-consent.readthedocs.io) is an international initiative that aims to provide researchers in the brain imaging community with information about data sharing options and tools. We present here a short history of this project and its latest developments, and share pointers to consent forms, including a template consent form that is compliant with the EU general data protection regulation. We also share pointers to an associated data user agreement that is not only useful in the EU context, but also for any researchers dealing with personal (clinical) data elsewhere.

Logical and Methodological Issues Affecting Genetic Studies of Humans Reported in Top Neuroscience Journals.

Genetics and neuroscience are two areas of science that pose particular methodological problems because they involve detecting weak signals (i.e., small effects) in noisy data. In recent years, increasing numbers of studies have attempted to bridge these disciplines by looking for genetic factors associated with individual differences in behavior, cognition, and brain structure or function. However, different methodological approaches to guarding against false positives have evolved in the two disciplines. To explore methodological issues affecting neurogenetic studies, we conducted an in-depth analysis of 30 consecutive articles in 12 top neuroscience journals that reported on genetic associations in nonclinical human samples. It was often difficult to estimate effect sizes in neuroimaging paradigms. Where effect sizes could be calculated, the studies reporting the largest effect sizes tended to have two features: (i) they had the smallest samples and were generally underpowered to detect genetic effects, and (ii) they did not fully correct for multiple comparisons. Furthermore, only a minority of studies used statistical methods for multiple comparisons that took into account correlations between phenotypes or genotypes, and only nine studies included a replication sample or explicitly set out to replicate a prior finding. Finally, presentation of methodological information was not standardized and was often distributed across Methods sections and Supplementary Material, making it challenging to assemble basic information from many studies. Space limits imposed by journals could mean that highly complex statistical methods were described in only a superficial fashion. In summary, methods that have become standard in the genetics literature-stringent statistical standards, use of large samples, and replication of findings-are not always adopted when behavioral, cognitive, or neuroimaging phenotypes are used, leading to an increased risk of false-positive findings. Studies need to correct not just for the number of phenotypes collected but also for the number of genotypes examined, genetic models tested, and subsamples investigated. The field would benefit from more widespread use of methods that take into account correlations between the factors corrected for, such as spectral decomposition, or permutation approaches. Replication should become standard practice; this, together with the need for larger sample sizes, will entail greater emphasis on collaboration between research groups. We conclude with some specific suggestions for standardized reporting in this area.

Beyond differences in means: robust graphical methods to compare two groups in neuroscience.

If many changes are necessary to improve the quality of neuroscience research, one relatively simple step could have great pay-offs: to promote the adoption of detailed graphical methods, combined with robust inferential statistics. Here, we illustrate how such methods can lead to a much more detailed understanding of group differences than bar graphs and t-tests on means. To complement the neuroscientist's toolbox, we present two powerful tools that can help us understand how groups of observations differ: the shift function and the difference asymmetry function. These tools can be combined with detailed visualisations to provide complementary perspectives about the data. We provide implementations in R and MATLAB of the graphical tools, and all the examples in the article can be reproduced using R scripts.

The human voice areas: Spatial organization and inter-individual variability in temporal and extra-temporal cortices.

fMRI studies increasingly examine functions and properties of non-primary areas of human auditory cortex. However there is currently no standardized localization procedure to reliably identify specific areas across individuals such as the standard 'localizers' available in the visual domain. Here we present an fMRI 'voice localizer' scan allowing rapid and reliable localization of the voice-sensitive 'temporal voice areas' (TVA) of human auditory cortex. We describe results obtained using this standardized localizer scan in a large cohort of normal adult subjects. Most participants (94%) showed bilateral patches of significantly greater response to vocal than non-vocal sounds along the superior temporal sulcus/gyrus (STS/STG). Individual activation patterns, although reproducible, showed high inter-individual variability in precise anatomical location. Cluster analysis of individual peaks from the large cohort highlighted three bilateral clusters of voice-sensitivity, or "voice patches" along posterior (TVAp), mid (TVAm) and anterior (TVAa) STS/STG, respectively. A series of extra-temporal areas including bilateral inferior prefrontal cortex and amygdalae showed small, but reliable voice-sensitivity as part of a large-scale cerebral voice network. Stimuli for the voice localizer scan and probabilistic maps in MNI space are available for download.

Imaging learned fear circuitry in awake mice using fMRI.

Functional magnetic resonance imaging (fMRI) of learned behaviour in 'awake rodents' provides the opportunity for translational preclinical studies into the influence of pharmacological and genetic manipulations on brain function. fMRI has recently been employed to investigate learned behaviour in awake rats. Here, this methodology is translated to mice, so that future fMRI studies may exploit the vast number of genetically modified mouse lines that are available. One group of mice was conditioned to associate a flashing light (conditioned stimulus, CS) with foot shock (PG; paired group), and another group of mice received foot shock and flashing light explicitly unpaired (UG; unpaired group). The blood oxygen level-dependent signal (proxy for neuronal activation) in response to the CS was measured 24 h later in awake mice from the PG and UG using fMRI. The amygdala, implicated in fear processing, was activated to a greater degree in the PG than in the UG in response to the CS. Additionally, the nucleus accumbens was activated in the UG in response to the CS. Because the CS signalled an absence of foot shock in the UG, it is possible that this region is involved in processing the safety aspect of the CS. To conclude, the first use of fMRI to visualise brain activation in awake mice that are completing a learned emotional task is reported. This work paves the way for future preclinical fMRI studies to investigate genetic and environmental influences on brain function in transgenic mouse models of disease and aging.

Test-retest reliability of structural brain networks from diffusion MRI.

Structural brain networks constructed from diffusion MRI (dMRI) and tractography have been demonstrated in healthy volunteers and more recently in various disorders affecting brain connectivity. However, few studies have addressed the reproducibility of the resulting networks. We measured the test-retest properties of such networks by varying several factors affecting network construction using ten healthy volunteers who underwent a dMRI protocol at 1.5T on two separate occasions. Each T1-weighted brain was parcellated into 84 regions-of-interest and network connections were identified using dMRI and two alternative tractography algorithms, two alternative seeding strategies, a white matter waypoint constraint and three alternative network weightings. In each case, four common graph-theoretic measures were obtained. Network properties were assessed both node-wise and per network in terms of the intraclass correlation coefficient (ICC) and by comparing within- and between-subject differences. Our findings suggest that test-retest performance was improved when: 1) seeding from white matter, rather than grey; and 2) using probabilistic tractography with a two-fibre model and sufficient streamlines, rather than deterministic tensor tractography. In terms of network weighting, a measure of streamline density produced better test-retest performance than tract-averaged diffusion anisotropy, although it remains unclear which is a more accurate representation of the underlying connectivity. For the best performing configuration, the global within-subject differences were between 3.2% and 11.9% with ICCs between 0.62 and 0.76. The mean nodal within-subject differences were between 5.2% and 24.2% with mean ICCs between 0.46 and 0.62. For 83.3% (70/84) of nodes, the within-subject differences were smaller than between-subject differences. Overall, these findings suggest that whilst current techniques produce networks capable of characterising the genuine between-subject differences in connectivity, future work must be undertaken to improve network reliability.

Early ERPs to faces and objects are driven by phase, not amplitude spectrum information: evidence from parametric, test-retest, single-subject analyses.

One major challenge in determining how the brain categorizes objects is to tease apart the contribution of low-level and high-level visual properties to behavioral and brain imaging data. So far, studies using stimuli with equated amplitude spectra have shown that the visual system relies mostly on localized information, such as edges and contours, carried by phase information. However, some researchers have argued that some event-related potentials (ERP) and blood-oxygen-level-dependent (BOLD) categorical differences could be driven by nonlocalized information contained in the amplitude spectrum. The goal of this study was to provide the first systematic quantification of the contribution of phase and amplitude spectra to early ERPs to faces and objects. We conducted two experiments in which we recorded electroencephalograms (EEG) from eight subjects, in two sessions each. In the first experiment, participants viewed images of faces and houses containing original or scrambled phase spectra combined with original, averaged, or swapped amplitude spectra. In the second experiment, we parametrically manipulated image phase and amplitude in 10% intervals. We performed a range of analyses including detailed single-subject general linear modeling of ERP data, test-retest reliability, and unique variance analyses. Our results suggest that early ERPs to faces and objects are due to phase information, with almost no contribution from the amplitude spectrum. Importantly, our results should not be used to justify uncontrolled stimuli; to the contrary, our results emphasize the need for stimulus control (including the amplitude spectrum), parametric designs, and systematic data analyses, of which we have seen far too little in ERP vision research.

Biases in BCI experiments: Do we really need to balance stimulus properties across categories?

Brain Computer Interfaces (BCIs) consist of an interaction between humans and computers with a specific mean of communication, such as voice, gestures, or even brain signals that are usually recorded by an Electroencephalogram (EEG). To ensure an optimal interaction, the BCI algorithm typically involves the classification of the input signals into predefined task-specific categories. However, a recurrent problem is that the classifier can easily be biased by uncontrolled experimental conditions, namely covariates, that are unbalanced across the categories. This issue led to the current solution of forcing the balance of these covariates across the different categories which is time consuming and drastically decreases the dataset diversity. The purpose of this research is to evaluate the need for this forced balance in BCI experiments involving EEG data. A typical design of neural BCIs involves repeated experimental trials using visual stimuli to trigger the so-called Event-Related Potential (ERP). The classifier is expected to learn spatio-temporal patterns specific to categories rather than patterns related to uncontrolled stimulus properties, such as psycho-linguistic variables (e.g., phoneme number, familiarity, and age of acquisition) and image properties (e.g., contrast, compactness, and homogeneity). The challenges are then to know how biased the decision is, which features affect the classification the most, which part of the signal is impacted, and what is the probability to perform neural categorization per se. To address these problems, this research has two main objectives: (1) modeling and quantifying the covariate effects to identify spatio-temporal regions of the EEG allowing maximal classification performance while minimizing the biasing effect, and (2) evaluating the need to balance the covariates across categories when studying brain mechanisms. To solve the modeling problem, we propose using a linear parametric analysis applied to some observable and commonly studied covariates to them. The biasing effect is quantified by comparing the regions highly influenced by the covariates with the regions of high categorical contrast, i.e., parts of the ERP allowing a reliable classification. The need to balance the stimulus's inner properties across categories is evaluated by assessing the separability between category-related and covariate-related evoked responses. The procedure is applied to a visual priming experiment where the images represent items belonging to living or non-living entities. The observed covariates are the commonly controlled psycho-linguistic variables and some visual features of the images. As a result, we identified that the category of the stimulus mostly affects the late evoked response. The covariates, when not modeled, have a biasing effect on the classification, essentially in the early evoked response. This effect increases with the diversity of the dataset and the complexity of the algorithm used. As the effects of both psycho-linguistic variables and image features appear outside of the spatio-temporal regions of significant categorical contrast, the proper selection of the region of interest makes the classification reliable. Having proved that the covariate effects can be separated from the categorical effect, our framework can be further used to isolate the category-dependent evoked response from the rest of the EEG to study neural processes involved when seeing living vs. non-living entities.

Reliability of ERP and single-trial analyses.

A reliable measure is one we can trust in the long run. Thus, the reliability of measurements is as important as their validity. Here we investigated the reliability of brain electrical visual evoked responses to faces and noise textures. For the first time, we provide reliability measures for the full time course of event-related potentials (ERPs). Our analyses were also performed on a R(2)(t) metric that reflects results from single-trial analyses, therefore providing the first reliability analysis of ERP single-trial analyses. Results show that ERPs and R(2)(t) are highly reliable (cross-correlation ~0.9, lag ~4/6ms, intra-class correlation ~0.9) but also idiosyncratic: ERPs and R(2)(t) are highly reproducible within subjects, who differ reliably from each other and the grand average across subjects. Consequently, grand averages, although highly reliable, can be misleading because they might not reflect the actual brain dynamic of any subjects.

Mindfulness related changes in grey matter: a systematic review and meta-analysis.

Knowing target regions undergoing strfuncti changes caused by behavioural interventions is paramount in evaluating the effectiveness of such practices. Here, using a systematic review approach, we identified 25 peer-reviewed magnetic resonance imaging (MRI) studies demonstrating grey matter changes related to mindfulness meditation. An activation likelihood estimation (ALE) analysis (n = 16) revealed the right anterior ventral insula as the only significant region with consistent effect across studies, whilst an additional functional connectivity analysis indicates that both left and right insulae, and the anterior cingulate gyrus with adjacent paracingulate gyri should also be considered in future studies. Statistical meta-analyses suggest medium to strong effect sizes from Cohen's d ~ 0.8 in the right insula to ~ 1 using maxima across the whole brain. The systematic review revealed design issues with selection, information, attrition and confirmation biases, in addition to weak statistical power. In conclusion, our analyses show that mindfulness meditation practice does induce grey matter changes but also that improvements in methodology are needed to establish mindfulness as a therapeutic intervention.

#EEGManyLabs: Investigating the replicability of influential EEG experiments.

There is growing awareness across the neuroscience community that the replicability of findings about the relationship between brain activity and cognitive phenomena can be improved by conducting studies with high statistical power that adhere to well-defined and standardised analysis pipelines. Inspired by recent efforts from the psychological sciences, and with the desire to examine some of the foundational findings using electroencephalography (EEG), we have launched #EEGManyLabs, a large-scale international collaborative replication effort. Since its discovery in the early 20th century, EEG has had a profound influence on our understanding of human cognition, but there is limited evidence on the replicability of some of the most highly cited discoveries. After a systematic search and selection process, we have identified 27 of the most influential and continually cited studies in the field. We plan to directly test the replicability of key findings from 20 of these studies in teams of at least three independent laboratories. The design and protocol of each replication effort will be submitted as a Registered Report and peer-reviewed prior to data collection. Prediction markets, open to all EEG researchers, will be used as a forecasting tool to examine which findings the community expects to replicate. This project will update our confidence in some of the most influential EEG findings and generate a large open access database that can be used to inform future research practices. Finally, through this international effort, we hope to create a cultural shift towards inclusive, high-powered multi-laboratory collaborations.

From BIDS-Formatted EEG Data to Sensor-Space Group Results: A Fully Reproducible Workflow With EEGLAB and LIMO EEG.

Reproducibility is a cornerstone of scientific communication without which one cannot build upon each other's work. Because modern human brain imaging relies on many integrated steps with a variety of possible algorithms, it has, however, become impossible to report every detail of a data processing workflow. In response to this analytical complexity, community recommendations are to share data analysis pipelines (scripts that implement workflows). Here we show that this can easily be done using EEGLAB and tools built around it. BIDS tools allow importing all the necessary information and create a study from electroencephalography (EEG)-Brain Imaging Data Structure compliant data. From there preprocessing can be carried out in only a few steps using EEGLAB and statistical analyses performed using the LIMO EEG plug-in. Using Wakeman and Henson (2015) face dataset, we illustrate how to prepare data and build different statistical models, a standard factorial design (faces ∗ repetition), and a more modern trial-based regression approach for the stimulus repetition effect, all in a few reproducible command lines.

The genetics-BIDS extension: Easing the search for genetic data associated with human brain imaging.

Metadata are what makes databases searchable. Without them, researchers would have difficulty finding data with features they are interested in. Brain imaging genetics is at the intersection of two disciplines, each with dedicated dictionaries and ontologies facilitating data search and analysis. Here, we present the genetics Brain Imaging Data Structure extension, consisting of metadata files for human brain imaging data to which they are linked, and describe succinctly the genomic and transcriptomic data associated with them, which may be in different databases. This extension will facilitate identifying micro-scale molecular features that are linked to macro-scale imaging repositories, facilitating data aggregation across studies.

Brain classification reveals the right cerebellum as the best biomarker of dyslexia.

BACKGROUND: Developmental dyslexia is a specific cognitive disorder in reading acquisition that has genetic and neurological origins. Despite histological evidence for brain differences in dyslexia, we recently demonstrated that in large cohort of subjects, no differences between control and dyslexic readers can be found at the macroscopic level (MRI voxel), because of large variances in brain local volumes. In the present study, we aimed at finding brain areas that most discriminate dyslexic from control normal readers despite the large variance across subjects. After segmenting brain grey matter, normalizing brain size and shape and modulating the voxels' content, normal readers' brains were used to build a 'typical' brain via bootstrapped confidence intervals. Each dyslexic reader's brain was then classified independently at each voxel as being within or outside the normal range. We used this simple strategy to build a brain map showing regional percentages of differences between groups. The significance of this map was then assessed using a randomization technique. RESULTS: The right cerebellar declive and the right lentiform nucleus were the two areas that significantly differed the most between groups with 100% of the dyslexic subjects (N = 38) falling outside of the control group (N = 39) 95% confidence interval boundaries. The clinical relevance of this result was assessed by inquiring cognitive brain-based differences among dyslexic brain subgroups in comparison to normal readers' performances. The strongest difference between dyslexic subgroups was observed between subjects with lower cerebellar declive (LCD) grey matter volumes than controls and subjects with higher cerebellar declive (HCD) grey matter volumes than controls. Dyslexic subjects with LCD volumes performed worse than subjects with HCD volumes in phonologically and lexicon related tasks. Furthermore, cerebellar and lentiform grey matter volumes interacted in dyslexic subjects, so that lower and higher lentiform grey matter volumes compared to controls differently modulated the phonological and lexical performances. Best performances (observed in controls) corresponded to an optimal value of grey matter and they dropped for higher or lower volumes. CONCLUSION: These results provide evidence for the existence of various subtypes of dyslexia characterized by different brain phenotypes. In addition, behavioural analyses suggest that these brain phenotypes relate to different deficits of automatization of language-based processes such as grapheme/phoneme correspondence and/or rapid access to lexicon entries.

Age-related delay in information accrual for faces: evidence from a parametric, single-trial EEG approach.

BACKGROUND: In this study, we quantified age-related changes in the time-course of face processing by means of an innovative single-trial ERP approach. Unlike analyses used in previous studies, our approach does not rely on peak measurements and can provide a more sensitive measure of processing delays. Young and old adults (mean ages 22 and 70 years) performed a non-speeded discrimination task between two faces. The phase spectrum of these faces was manipulated parametrically to create pictures that ranged between pure noise (0% phase information) and the undistorted signal (100% phase information), with five intermediate steps. RESULTS: Behavioural 75% correct thresholds were on average lower, and maximum accuracy was higher, in younger than older observers. ERPs from each subject were entered into a single-trial general linear regression model to identify variations in neural activity statistically associated with changes in image structure. The earliest age-related ERP differences occurred in the time window of the N170. Older observers had a significantly stronger N170 in response to noise, but this age difference decreased with increasing phase information. Overall, manipulating image phase information had a greater effect on ERPs from younger observers, which was quantified using a hierarchical modelling approach. Importantly, visual activity was modulated by the same stimulus parameters in younger and older subjects. The fit of the model, indexed by R2, was computed at multiple post-stimulus time points. The time-course of the R2 function showed a significantly slower processing in older observers starting around 120 ms after stimulus onset. This age-related delay increased over time to reach a maximum around 190 ms, at which latency younger observers had around 50 ms time lead over older observers. CONCLUSION: Using a component-free ERP analysis that provides a precise timing of the visual system sensitivity to image structure, the current study demonstrates that older observers accumulate face information more slowly than younger subjects. Additionally, the N170 appears to be less face-sensitive in older observers.

Electrophysiological evidence for an early processing of human voices.

BACKGROUND: Previous electrophysiological studies have identified a "voice specific response" (VSR) peaking around 320 ms after stimulus onset, a latency markedly longer than the 70 ms needed to discriminate living from non-living sound sources and the 150 ms to 200 ms needed for the processing of voice paralinguistic qualities. In the present study, we investigated whether an early electrophysiological difference between voice and non-voice stimuli could be observed. RESULTS: ERPs were recorded from 32 healthy volunteers who listened to 200 ms long stimuli from three sound categories - voices, bird songs and environmental sounds - whilst performing a pure-tone detection task. ERP analyses revealed voice/non-voice amplitude differences emerging as early as 164 ms post stimulus onset and peaking around 200 ms on fronto-temporal (positivity) and occipital (negativity) electrodes. CONCLUSION: Our electrophysiological results suggest a rapid brain discrimination of sounds of voice, termed the "fronto-temporal positivity to voices" (FTPV), at latencies comparable to the well-known face-preferential N170.

Parametric study of EEG sensitivity to phase noise during face processing.

BACKGROUND: The present paper examines the visual processing speed of complex objects, here faces, by mapping the relationship between object physical properties and single-trial brain responses. Measuring visual processing speed is challenging because uncontrolled physical differences that co-vary with object categories might affect brain measurements, thus biasing our speed estimates. Recently, we demonstrated that early event-related potential (ERP) differences between faces and objects are preserved even when images differ only in phase information, and amplitude spectra are equated across image categories. Here, we use a parametric design to study how early ERP to faces are shaped by phase information. Subjects performed a two-alternative force choice discrimination between two faces (Experiment 1) or textures (two control experiments). All stimuli had the same amplitude spectrum and were presented at 11 phase noise levels, varying from 0% to 100% in 10% increments, using a linear phase interpolation technique. Single-trial ERP data from each subject were analysed using a multiple linear regression model. RESULTS: Our results show that sensitivity to phase noise in faces emerges progressively in a short time window between the P1 and the N170 ERP visual components. The sensitivity to phase noise starts at about 120-130 ms after stimulus onset and continues for another 25-40 ms. This result was robust both within and across subjects. A control experiment using pink noise textures, which had the same second-order statistics as the faces used in Experiment 1, demonstrated that the sensitivity to phase noise observed for faces cannot be explained by the presence of global image structure alone. A second control experiment used wavelet textures that were matched to the face stimuli in terms of second- and higher-order image statistics. Results from this experiment suggest that higher-order statistics of faces are necessary but not sufficient to obtain the sensitivity to phase noise function observed in response to faces. CONCLUSION: Our results constitute the first quantitative assessment of the time course of phase information processing by the human visual brain. We interpret our results in a framework that focuses on image statistics and single-trial analyses.