A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more.

Quality control (QC) assessment is a vital part of FMRI processing and analysis, and a typically under-discussed aspect of reproducibility. This includes checking datasets at their very earliest stages (acquisition and conversion) through their processing steps (e.g., alignment and motion correction) to regression modeling (correct stimuli, no collinearity, valid fits, enough degrees of freedom, etc.) for each subject. There are a wide variety of features to verify throughout any single subject processing pipeline, both quantitatively and qualitatively. We present several FMRI preprocessing QC features available in the AFNI toolbox, many of which are automatically generated by the pipeline-creation tool, afni_proc.py. These items include: a modular HTML document that covers full single subject processing from the raw data through statistical modeling; several review scripts in the results directory of processed data; and command line tools for identifying subjects with one or more quantitative properties across a group (such as triaging warnings, making exclusion criteria or creating informational tables). The HTML itself contains several buttons that efficiently facilitate interactive investigations into the data, when deeper checks are needed beyond the systematic images. The pages are linkable, so that users can evaluate individual items across a group, for increased sensitivity to differences (e.g., in alignment or regression modeling images). Finally, the QC document contains rating buttons for each "QC block", as well as comment fields for each, to facilitate both saving and sharing the evaluations. This increases the specificity of QC, as well as its shareability, as these files can be shared with others and potentially uploaded into repositories, promoting transparency and open science. We describe the features and applications of these QC tools for FMRI.

Hyperbolic trade-off: The importance of balancing trial and subject sample sizes in neuroimaging.

Here we investigate the crucial role of trials in task-based neuroimaging from the perspectives of statistical efficiency and condition-level generalizability. Big data initiatives have gained popularity for leveraging a large sample of subjects to study a wide range of effect magnitudes in the brain. On the other hand, most task-based FMRI designs feature a relatively small number of subjects, so that resulting parameter estimates may be associated with compromised precision. Nevertheless, little attention has been given to another important dimension of experimental design, which can equally boost a study's statistical efficiency: the trial sample size. The common practice of condition-level modeling implicitly assumes no cross-trial variability. Here, we systematically explore the different factors that impact effect uncertainty, drawing on evidence from hierarchical modeling, simulations and an FMRI dataset of 42 subjects who completed a large number of trials of cognitive control task. We find that, due to an approximately symmetric hyperbola-relationship between trial and subject sample sizes in the presence of relatively large cross-trial variability, 1) trial sample size has nearly the same impact as subject sample size on statistical efficiency; 2) increasing both the number of trials and subjects improves statistical efficiency more effectively than focusing on subjects alone; 3) trial sample size can be leveraged alongside subject sample size to improve the cost-effectiveness of an experimental design; 4) for small trial sample sizes, trial-level modeling, rather than condition-level modeling through summary statistics, may be necessary to accurately assess the standard error of an effect estimate. We close by making practical suggestions for improving experimental designs across neuroimaging and behavioral studies.

Trial and error: A hierarchical modeling approach to test-retest reliability.

The concept of test-retest reliability indexes the consistency of a measurement across time. High reliability is critical for any scientific study, but specifically for the study of individual differences. Evidence of poor reliability of commonly used behavioral and functional neuroimaging tasks is mounting. Reports on low reliability of task-based fMRI have called into question the adequacy of using even the most common, well-characterized cognitive tasks with robust population-level effects, to measure individual differences. Here, we lay out a hierarchical framework that estimates reliability as a correlation divorced from trial-level variability, and show that reliability tends to be underestimated under the conventional intraclass correlation framework through summary statistics based on condition-level modeling. In addition, we examine how reliability estimation between the two statistical frameworks diverges and assess how different factors (e.g., trial and subject sample sizes, relative magnitude of cross-trial variability) impact reliability estimates. As empirical data indicate that cross-trial variability is large in most tasks, this work highlights that a large number of trials (e.g., greater than 100) may be required to achieve precise reliability estimates. We reference the tools TRR and 3dLMEr for the community to apply trial-level models to behavior and neuroimaging data and discuss how to make these new measurements most useful for future studies.

Beyond linearity in neuroimaging: Capturing nonlinear relationships with application to longitudinal studies.

The ubiquitous adoption of linearity for quantitative predictors in statistical modeling is likely attributable to its advantages of straightforward interpretation and computational feasibility. The linearity assumption may be a reasonable approximation especially when the variable is confined within a narrow range, but it can be problematic when the variable's effect is non-monotonic or complex. Furthermore, visualization and model assessment of a linear fit are usually omitted because of challenges at the whole brain level in neuroimaging. By adopting a principle of learning from the data in the presence of uncertainty to resolve the problematic aspects of conventional polynomial fitting, we introduce a flexible and adaptive approach of multilevel smoothing splines (MSS) to capture any nonlinearity of a quantitative predictor for population-level neuroimaging data analysis. With no prior knowledge regarding the underlying relationship other than a parsimonious assumption about the extent of smoothness (e.g., no sharp corners), we express the unknown relationship with a sufficient number of smoothing splines and use the data to adaptively determine the specifics of the nonlinearity. In addition to introducing the theoretical framework of MSS as an efficient approach with a counterbalance between flexibility and stability, we strive to (a) lay out the specific schemes for population-level nonlinear analyses that may involve task (e.g., contrasting conditions) and subject-grouping (e.g., patients vs controls) factors; (b) provide modeling accommodations to adaptively reveal, estimate and compare any nonlinear effects of a predictor across the brain, or to more accurately account for the effects (including nonlinear effects) of a quantitative confound; (c) offer the associated program 3dMSS to the neuroimaging community for whole-brain voxel-wise analysis as part of the AFNI suite; and (d) demonstrate the modeling approach and visualization processes with a longitudinal dataset of structural MRI scans.

To pool or not to pool: Can we ignore cross-trial variability in FMRI?

In this work, we investigate the importance of explicitly accounting for cross-trial variability in neuroimaging data analysis. To attempt to obtain reliable estimates in a task-based experiment, each condition is usually repeated across many trials. The investigator may be interested in (a) condition-level effects, (b) trial-level effects, or (c) the association of trial-level effects with the corresponding behavior data. The typical strategy for condition-level modeling is to create one regressor per condition at the subject level with the underlying assumption that responses do not change across trials. In this methodology of complete pooling, all cross-trial variability is ignored and dismissed as random noise that is swept under the rug of model residuals. Unfortunately, this framework invalidates the generalizability from the confine of specific trials (e.g., particular faces) to the associated stimulus category ("face"), and may inflate the statistical evidence when the trial sample size is not large enough. Here we propose an adaptive and computationally tractable framework that meshes well with the current two-level pipeline and explicitly accounts for trial-by-trial variability. The trial-level effects are first estimated per subject through no pooling. To allow generalizing beyond the particular stimulus set employed, the cross-trial variability is modeled at the population level through partial pooling in a multilevel model, which permits accurate effect estimation and characterization. Alternatively, trial-level estimates can be used to investigate, for example, brain-behavior associations or correlations between brain regions. Furthermore, our approach allows appropriate accounting for serial correlation, handling outliers, adapting to data skew, and capturing nonlinear brain-behavior relationships. By applying a Bayesian multilevel model framework at the level of regions of interest to an experimental dataset, we show how multiple testing can be addressed and full results reported without arbitrary dichotomization. Our approach revealed important differences compared to the conventional method at the condition level, including how the latter can distort effect magnitude and precision. Notably, in some cases our approach led to increased statistical sensitivity. In summary, our proposed framework provides an effective strategy to capture trial-by-trial responses that should be of interest to a wide community of experimentalists.

The Haskins pediatric atlas: a magnetic-resonance-imaging-based pediatric template and atlas.

BACKGROUND: Spatial normalization plays an essential role in multi-subject MRI and functional MRI (fMRI) experiments by facilitating a common space in which group analyses are performed. Although many prominent adult templates are available, their use for pediatric data is problematic. Generalized templates for pediatric populations are limited or constructed using older methods that result in less ideal normalization. OBJECTIVE: The Haskins pediatric templates and atlases aim to provide superior registration and more precise accuracy in labeling of anatomical and functional regions essential for all fMRI studies involving pediatric populations. MATERIALS AND METHODS: The Haskins pediatric templates and atlases were generated with nonlinear methods using structural MRI from 72 children (age range 7-14 years, median 10 years), allowing for a detailed template with corresponding parcellations of labeled atlas regions. The accuracy of these templates and atlases was assessed using multiple metrics of deformation distance and overlap. RESULTS: When comparing the deformation distances from normalizing pediatric data between this template and both the adult templates and other pediatric templates, we found significantly less deformation distance for the Haskins pediatric template (P<0.0001). Further, the correct atlas classification was higher using the Haskins pediatric template in 74% of regions (P<0.0001). CONCLUSION: The Haskins pediatric template results in more accurate correspondence across subjects because of lower deformation distances. This correspondence also provides better accuracy in atlas locations to benefit structural and functional imaging analyses of pediatric populations.

A series of five population-specific Indian brain templates and atlases spanning ages 6-60 years.

Anatomical brain templates are commonly used as references in neurological MRI studies, for bringing data into a common space for group-level statistics and coordinate reporting. Given the inherent variability in brain morphology across age and geography, it is important to have templates that are as representative as possible for both age and population. A representative-template increases the accuracy of alignment, decreases distortions as well as potential biases in final coordinate reports. In this study, we developed and validated a new set of T1w Indian brain templates (IBT) from a large number of brain scans (total n = 466) acquired across different locations and multiple 3T MRI scanners in India. A new tool in AFNI, make_template_dask.py, was created to efficiently make five age-specific IBTs (ages 6-60 years) as well as maximum probability map (MPM) atlases for each template; for each age-group's template-atlas pair, there is both a "population-average" and a "typical" version. Validation experiments on an independent Indian structural and functional-MRI dataset show the appropriateness of IBTs for spatial normalization of Indian brains. The results indicate significant structural differences when comparing the IBTs and MNI template, with these differences being maximal along the Anterior-Posterior and Inferior-Superior axes, but minimal Left-Right. For each age-group, the MPM brain atlases provide reasonably good representation of the native-space volumes in the IBT space, except in a few regions with high intersubject variability. These findings provide evidence to support the use of age and population-specific templates in human brain mapping studies.

Variability in the analysis of a single neuroimaging dataset by many teams.

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Beware (Surprisingly Common) Left-Right Flips in Your MRI Data: An Efficient and Robust Method to Check MRI Dataset Consistency Using AFNI.

Knowing the difference between left and right is generally assumed throughout the brain MRI research community. However, we note widespread occurrences of left-right orientation errors in MRI open database repositories where volumes have contained systematic left-right flips between subject EPIs and anatomicals, due to having incorrect or missing file header information. Here we present a simple method in AFNI for determining the consistency of left and right within a pair of acquired volumes for a particular subject; the presence of EPI-anatomical inconsistency, for example, is a sign that dataset header information likely requires correction. The method contains both a quantitative evaluation as well as a visualizable verification. We test the functionality using publicly available datasets. Left-right flipping is not immediately obvious in most cases, so we also present visualization methods for looking at this problem (and other potential problems), using examples from both FMRI and DTI datasets.

Fast detection and reduction of local transient artifacts in resting-state fMRI.

Image quality control (QC) is a critical and computationally intensive component of functional magnetic resonance imaging (fMRI). Artifacts caused by physiologic signals or hardware malfunctions are usually identified and removed during data processing offline, well after scanning sessions are complete. A system with the computational efficiency to identify and remove artifacts during image acquisition would permit rapid adjustment of protocols as issues arise during experiments. To improve the speed and accuracy of QC and functional image correction, we developed Fast Anatomy-Based Image Correction (Fast ANATICOR) with newly implemented nuisance models and an improved pipeline. We validated its performance on a dataset consisting of normal scans and scans containing known hardware-driven artifacts. Fast ANATICOR's increased processing speed may make real-time QC and image correction feasible as compared with the existing offline method.

Exploring treatment of medial tibial stress syndrome via posture and the MyoKinesthetic system.

INTRODUCTION: A standard treatment protocol for medial tibial stress syndrome (MTSS) has not been identified. Clinical practice focuses on local evaluation and treatment neglecting a global approach. The MyoKinesthetic™ (MYK) System includes a full-body postural assessment to identify compensatory patterns that may lead to MTSS. The purpose of this study was to assess the effects of the MYK System in treating patients diagnosed with MTSS. METHOD: A multi-site exploratory study was used to assess the effects of the MYK System on perceived pain and disability in patients diagnosed with MTSS. Eighteen physically active patients (6 female, 12 male), ages 18-25 years (19.89 ±â€¯1.32) were treated with the MYK System. RESULTS: Paired T-tests were utilized to assess change. The change in patient reported pain was statistically significant (t(17) = 10.48, p < .001, Cohen's d = 2.48) and represented an average decrease of 96% in patient reported pain. The change in disablement was statistically significant (t(17) = 7.39, p < .001, Cohen's d = 1.74) and represented an average decrease of 88.2% in patient reported disablement. DISCUSSION: Participants treated with the MYK System experienced significant improvements and appear to surpass traditional interventions without the need of rest. CONCLUSION: Implementation of the MYK System to treat MTSS led to significant decreases in patient reported pain and dysfunction. A full-scale clinical investigation of the MYK System is warranted to determine its effects compared to traditional treatment options.

Untangling the relatedness among correlations, part III: Inter-subject correlation analysis through Bayesian multilevel modeling for naturalistic scanning.

While inter-subject correlation (ISC) analysis is a powerful tool for naturalistic scanning data, drawing appropriate statistical inferences is difficult due to the daunting task of accounting for the intricate relatedness in data structure as well as handling the multiple testing issue. Although the linear mixed-effects (LME) modeling approach (Chen et al., 2017a) is capable of capturing the relatedness in the data and incorporating explanatory variables, there are a few challenging issues: 1) it is difficult to assign accurate degrees of freedom for each testing statistic, 2) multiple testing correction is potentially over-penalizing due to model inefficiency, and 3) thresholding necessitates arbitrary dichotomous decisions. Here we propose a Bayesian multilevel (BML) framework for ISC data analysis that integrates all regions of interest into one model. By loosely constraining the regions through a weakly informative prior, BML dissolves multiplicity through conservatively pooling the effect of each region toward the center and improves collective fitting and overall model performance. In addition to potentially achieving a higher inference efficiency, BML improves spatial specificity and easily allows the investigator to adopt a philosophy of full results reporting. A dataset of naturalistic scanning is utilized to illustrate the modeling approach with 268 parcels and to showcase the modeling capability, flexibility and advantages in results reporting. The associated program will be available as part of the AFNI suite for general use.

Fighting or embracing multiplicity in neuroimaging? neighborhood leverage versus global calibration.

Neuroimaging faces the daunting challenge of multiple testing - an instance of multiplicity - that is associated with two other issues to some extent: low inference efficiency and poor reproducibility. Typically, the same statistical model is applied to each spatial unit independently in the approach of massively univariate modeling. In dealing with multiplicity, the general strategy employed in the field is the same regardless of the specifics: trust the local "unbiased" effect estimates while adjusting the extent of statistical evidence at the global level. However, in this approach, modeling efficiency is compromised because each spatial unit (e.g., voxel, region, matrix element) is treated as an isolated and independent entity during massively univariate modeling. In addition, the required step of multiple testing "correction" by taking into consideration spatial relatedness, or neighborhood leverage, can only partly recoup statistical efficiency, resulting in potentially excessive penalization as well as arbitrariness due to thresholding procedures. Moreover, the assigned statistical evidence at the global level heavily relies on the data space (whole brain or a small volume). The present paper reviews how Stein's paradox (1956) motivates a Bayesian multilevel (BML) approach that, rather than fighting multiplicity, embraces it to our advantage through a global calibration process among spatial units. Global calibration is accomplished via a Gaussian distribution for the cross-region effects whose properties are not a priori specified, but a posteriori determined by the data at hand through the BML model. Our framework therefore incorporates multiplicity as integral to the modeling structure, not a separate correction step. By turning multiplicity into a strength, we aim to achieve five goals: 1) improve the model efficiency with a higher predictive accuracy, 2) control the errors of incorrect magnitude and incorrect sign, 3) validate each model relative to competing candidates, 4) reduce the reliance and sensitivity on the choice of data space, and 5) encourage full results reporting. Our modeling proposal reverberates with recent proposals to eliminate the dichotomization of statistical evidence ("significant" vs. "non-significant"), to improve the interpretability of study findings, as well as to promote reporting the full gamut of results (not only "significant" ones), thereby enhancing research transparency and reproducibility.

An integrative Bayesian approach to matrix-based analysis in neuroimaging.

Understanding the correlation structure associated with brain regions is a central goal in neuroscience, as it informs about interregional relationships and network organization. Correlation structure can be conveniently captured in a matrix that indicates the relationships among brain regions, which could involve electroencephalogram sensors, electrophysiology recordings, calcium imaging data, or functional magnetic resonance imaging (FMRI) data-We call this type of analysis matrix-based analysis, or MBA. Although different methods have been developed to summarize such matrices across subjects, including univariate general linear models (GLMs), the available modeling strategies tend to disregard the interrelationships among the regions, leading to "inefficient" statistical inference. Here, we develop a Bayesian multilevel (BML) modeling framework that simultaneously integrates the analyses of all regions, region pairs (RPs), and subjects. In this approach, the intricate relationships across regions as well as across RPs are quantitatively characterized. The adoption of the Bayesian framework allows us to achieve three goals: (a) dissolve the multiple testing issue typically associated with seeking evidence for the effect of each RP under the conventional univariate GLM; (b) make inferences on effects that would be treated as "random" under the conventional linear mixed-effects framework; and (c) estimate the effect of each brain region in a manner that indexes their relative "importance". We demonstrate the BML methodology with an FMRI dataset involving a cognitive-emotional task and compare it to the conventional GLM approach in terms of model efficiency, performance, and inferences. The associated program MBA is available as part of the AFNI suite for general use.

Equitable Thresholding and Clustering: A Novel Method for Functional Magnetic Resonance Imaging Clustering in AFNI.

This article describes a hybrid method to threshold functional magnetic resonance imaging (FMRI) group statistical maps derived from voxel-wise second-level statistical analyses. The proposed "Equitable Thresholding and Clustering" (ETAC) approach seeks to reduce the dependence of clustering results on arbitrary parameter values by using multiple subtests, each equivalent to a standard FMRI clustering analysis, to make decisions about which groups of voxels are potentially significant. The union of these subtest results decides which voxels are accepted. The approach adjusts the cluster-thresholding parameter of each subtest in an equitable way, so that the individual false-positive rates (FPRs) are balanced across subtests to achieve a desired final FPR (e.g., 5%). ETAC utilizes resampling methods to estimate the FPR and thus does not rely on parametric assumptions about the spatial correlation of FMRI noise. The approach was validated with pseudotask timings in resting-state brain data. In addition, a task FMRI data collection was used to compare ETACs true positive detection power versus a standard cluster detection method, demonstrating that ETAC is able to detect true results and control false positives while reducing reliance on arbitrary analysis parameters.

Handling Multiplicity in Neuroimaging Through Bayesian Lenses with Multilevel Modeling.

Here we address the current issues of inefficiency and over-penalization in the massively univariate approach followed by the correction for multiple testing, and propose a more efficient model that pools and shares information among brain regions. Using Bayesian multilevel (BML) modeling, we control two types of error that are more relevant than the conventional false positive rate (FPR): incorrect sign (type S) and incorrect magnitude (type M). BML also aims to achieve two goals: 1) improving modeling efficiency by having one integrative model and thereby dissolving the multiple testing issue, and 2) turning the focus of conventional null hypothesis significant testing (NHST) on FPR into quality control by calibrating type S errors while maintaining a reasonable level of inference efficiency. The performance and validity of this approach are demonstrated through an application at the region of interest (ROI) level, with all the regions on an equal footing: unlike the current approaches under NHST, small regions are not disadvantaged simply because of their physical size. In addition, compared to the massively univariate approach, BML may simultaneously achieve increased spatial specificity and inference efficiency, and promote results reporting in totality and transparency. The benefits of BML are illustrated in performance and quality checking using an experimental dataset. The methodology also avoids the current practice of sharp and arbitrary thresholding in the p-value funnel to which the multidimensional data are reduced. The BML approach with its auxiliary tools is available as part of the AFNI suite for general use.

A tail of two sides: Artificially doubled false positive rates in neuroimaging due to the sidedness choice with t-tests.

One-sided t-tests are widely used in neuroimaging data analysis. While such a test may be applicable when investigating specific regions and prior information about directionality is present, we argue here that it is often mis-applied, with severe consequences for false positive rate (FPR) control. Conceptually, a pair of one-sided t-tests conducted in tandem (e.g., to test separately for both positive and negative effects), effectively amounts to a two-sided t-test. However, replacing the two-sided test with a pair of one-sided tests without multiple comparisons correction essentially doubles the intended FPR of statements made about the same study; that is, the actual family-wise error (FWE) of results at the whole brain level would be 10% instead of the 5% intended by the researcher. Therefore, we strongly recommend that, unless otherwise explicitly justified, two-sided t-tests be applied instead of two simultaneous one-sided t-tests.

Comparison of heritability estimates on resting state fMRI connectivity phenotypes using the ENIGMA analysis pipeline.

We measured and compared heritability estimates for measures of functional brain connectivity extracted using the Enhancing Neuroimaging Genetics through Meta-Analysis (ENIGMA) rsfMRI analysis pipeline in two cohorts: the genetics of brain structure (GOBS) cohort and the HCP (the Human Connectome Project) cohort. These two cohorts were assessed using conventional (GOBS) and advanced (HCP) rsfMRI protocols, offering a test case for harmonization of rsfMRI phenotypes, and to determine measures that show consistent heritability for in-depth genome-wide analysis. The GOBS cohort consisted of 334 Mexican-American individuals (124M/210F, average age = 47.9 ± 13.2 years) from 29 extended pedigrees (average family size = 9 people; range 5-32). The GOBS rsfMRI data was collected using a 7.5-min acquisition sequence (spatial resolution = 1.72 × 1.72 × 3 mm3 ). The HCP cohort consisted of 518 twins and family members (240M/278F; average age = 28.7 ± 3.7 years). rsfMRI data was collected using 28.8-min sequence (spatial resolution = 2 × 2 × 2 mm3 ). We used the single-modality ENIGMA rsfMRI preprocessing pipeline to estimate heritability values for measures from eight major functional networks, using (1) seed-based connectivity and (2) dual regression approaches. We observed significant heritability (h2 = 0.2-0.4, p < .05) for functional connections from seven networks across both cohorts, with a significant positive correlation between heritability estimates across two cohorts. The similarity in heritability estimates for resting state connectivity measurements suggests that the additive genetic contribution to functional connectivity is robustly detectable across populations and imaging acquisition parameters. The overarching genetic influence, and means to consistently detect it, provides an opportunity to define a common genetic search space for future gene discovery studies.

Intraclass correlation: Improved modeling approaches and applications for neuroimaging.

Intraclass correlation (ICC) is a reliability metric that gauges similarity when, for example, entities are measured under similar, or even the same, well-controlled conditions, which in MRI applications include runs/sessions, twins, parent/child, scanners, sites, and so on. The popular definitions and interpretations of ICC are usually framed statistically under the conventional ANOVA platform. Here, we provide a comprehensive overview of ICC analysis in its prior usage in neuroimaging, and we show that the standard ANOVA framework is often limited, rigid, and inflexible in modeling capabilities. These intrinsic limitations motivate several improvements. Specifically, we start with the conventional ICC model under the ANOVA platform, and extend it along two dimensions: first, fixing the failure in ICC estimation when negative values occur under degenerative circumstance, and second, incorporating precision information of effect estimates into the ICC model. These endeavors lead to four modeling strategies: linear mixed-effects (LME), regularized mixed-effects (RME), multilevel mixed-effects (MME), and regularized multilevel mixed-effects (RMME). Compared to ANOVA, each of these four models directly provides estimates for fixed effects and their statistical significances, in addition to the ICC estimate. These new modeling approaches can also accommodate missing data and fixed effects for confounding variables. More importantly, we show that the MME and RMME approaches offer more accurate characterization and decomposition among the variance components, leading to more robust ICC computation. Based on these theoretical considerations and model performance comparisons with a real experimental dataset, we offer the following general-purpose recommendations. First, ICC estimation through MME or RMME is preferable when precision information (i.e., weights that more accurately allocate the variances in the data) is available for the effect estimate; when precision information is unavailable, ICC estimation through LME or the RME is the preferred option. Second, even though the absolute agreement version, ICC(2,1), is presently more popular in the field, the consistency version, ICC(3,1), is a practical and informative choice for whole-brain ICC analysis that achieves a well-balanced compromise when all potential fixed effects are accounted for. Third, approaches for clear, meaningful, and useful result reporting in ICC analysis are discussed. All models, ICC formulations, and related statistical testing methods have been implemented in an open source program 3dICC, which is publicly available as part of the AFNI suite. Even though our work here focuses on the whole-brain level, the modeling strategy and recommendations can be equivalently applied to other situations such as voxel, region, and network levels.