Generalizability of polygenic prediction models: how is the R2 defined on test data?

BACKGROUND: Polygenic risk scores (PRS) quantify an individual's genetic predisposition for different traits and are expected to play an increasingly important role in personalized medicine. A crucial challenge in clinical practice is the generalizability and transferability of PRS models to populations with different ancestries. When assessing the generalizability of PRS models for continuous traits, the R 2 is a commonly used measure to evaluate prediction accuracy. While the R 2 is a well-defined goodness-of-fit measure for statistical linear models, there exist different definitions for its application on test data, which complicates interpretation and comparison of results. METHODS: Based on large-scale genotype data from the UK Biobank, we compare three definitions of the R 2 on test data for evaluating the generalizability of PRS models to different populations. Polygenic models for several phenotypes, including height, BMI and lipoprotein A, are derived based on training data with European ancestry using state-of-the-art regression methods and are evaluated on various test populations with different ancestries. RESULTS: Our analysis shows that the choice of the R 2  definition can lead to considerably different results on test data, making the comparison of R 2  values from the literature problematic. While the definition as the squared correlation between predicted and observed phenotypes solely addresses the discriminative performance and always yields values between 0 and 1, definitions of the R 2 based on the mean squared prediction error (MSPE) with reference to intercept-only models assess both discrimination and calibration. These MSPE-based definitions can yield negative values indicating miscalibrated predictions for out-of-target populations. We argue that the choice of the most appropriate definition depends on the aim of PRS analysis - whether it primarily serves for risk stratification or also for individual phenotype prediction. Moreover, both correlation-based and MSPE-based definitions of R 2 can provide valuable complementary information. CONCLUSIONS: Awareness of the different definitions of the R 2 on test data is necessary to facilitate the reporting and interpretation of results on PRS generalizability. It is recommended to explicitly state which definition was used when reporting R 2 values on test data. Further research is warranted to develop and evaluate well-calibrated polygenic models for diverse populations.

New insights into black carbon light absorption enhancement: A comprehensive analysis of two differential behaviors.

High uncertainty in optical properties of black carbon (BC) involving heterogeneous chemistry has recently attracted increasing attention in the field of atmospheric climatology. To fill the gap in BC optical knowledge so as to estimate more accurate climate effects and serve the response to global warming, it is beneficial to conduct site-level studies on BC light absorption enhancement (Eabs) characteristics. Real-time surface gas and particulate pollutant observations during the summer and winter over Wuhan were utilized for the analysis of Eabs simulated by minimum R squared (MRS), considering two distinct atmospheric conditions (2015 and 2017). In general, differences in aerosol emissions led to Eabs differential behaviors. The summer average of Eabs (1.92 ± 0.55) in 2015 was higher than the winter average (1.27 ± 0.42), while the average (1.11 ± 0.20) in 2017 summer was lower than that (1.67 ± 0.69) in winter. Eabs and RBC (representing the mass ratio of non-refractory constituents to elemental carbon) constraints suggest that Eabs increased with the increase in RBC under the ambient condition enriched by secondary inorganic aerosol (SIA), with a maximum growth rate of 70.6% in 2015 summer. However, Eabs demonstrated a negative trend against RBC in 2017 winter due to the more complicated mixing state. The result arose from the opposite impact of hygroscopic SIA and absorbing OC/irregular distributed coatings on amplifying the light absorbency of BC. Furthermore, sensitivity analysis revealed a robust positive correlation (R > 0.9) between aerosol chemical compositions (including sulfate, nitrate, ammonium and secondary organic carbon), which could be significantly perturbed by only a small fraction of absorbing materials or restructuring BC through gaps filling. The above findings not only deepen the understanding of BC, but also provide useful information for the scientific decision-making in government to mitigate particulate pollution and obtain more precise BC radiative forcing.

Local and Overall Deviance R-Squared Measures for Mixtures of Generalized Linear Models.

In generalized linear models (GLMs), measures of lack of fit are typically defined as the deviance between two nested models, and a deviance-based R2 is commonly used to evaluate the fit. In this paper, we extend deviance measures to mixtures of GLMs, whose parameters are estimated by maximum likelihood (ML) via the EM algorithm. Such measures are defined both locally, i.e., at cluster-level, and globally, i.e., with reference to the whole sample. At the cluster-level, we propose a normalized two-term decomposition of the local deviance into explained, and unexplained local deviances. At the sample-level, we introduce an additive normalized decomposition of the total deviance into three terms, where each evaluates a different aspect of the fitted model: (1) the cluster separation on the dependent variable, (2) the proportion of the total deviance explained by the fitted model, and (3) the proportion of the total deviance which remains unexplained. We use both local and global decompositions to define, respectively, local and overall deviance R2 measures for mixtures of GLMs, which we illustrate-for Gaussian, Poisson and binomial responses-by means of a simulation study. The proposed fit measures are then used to assess, and interpret clusters of COVID-19 spread in Italy in two time points.

Effect size measure for mediation analysis with a multicategorical predictor.

Many currently available effect size measures for mediation have limitations when the predictor is nominal with three or more categories. The mediation effect size measure υ was adopted for this situation. A simulation study was conducted to investigate the performance of its estimators. We manipulated several factors in data generation (number of groups, sample size per group, and effect sizes of paths) and effect size estimation [different R-squared (R 2) shrinkage estimators]. Results showed that the Olkin-Pratt extended adjusted R 2 estimator had the least bias and the smallest MSE in estimating υ across conditions. We also applied different estimators of υ in a real data example. Recommendations and guidelines were provided about the use of this estimator.

r2mlm: An R package calculating R-squared measures for multilevel models.

Multilevel models are used ubiquitously in the social and behavioral sciences and effect sizes are critical for contextualizing results. A general framework of R-squared effect size measures for multilevel models has only recently been developed. Rights and Sterba (2019) distinguished each source of explained variance for each possible kind of outcome variance. Though researchers have long desired a comprehensive and coherent approach to computing R-squared measures for multilevel models, the use of this framework has a steep learning curve. The purpose of this tutorial is to introduce and demonstrate using a new R package - r2mlm - that automates the intensive computations involved in implementing the framework and provides accompanying graphics to visualize all multilevel R-squared measures together. We use accessible illustrations with open data and code to demonstrate how to use and interpret the R package output.

R-squared Measures for Multilevel Models with Three or More Levels.

Applications of multilevel models (MLMs) with three or more levels have increased alongside expanding software capability and dataset availability. Though researchers often express interest in R-squared measures as effect sizes for MLMs, R-squareds previously proposed for MLMs with three or more levels cover a limited subset of choices for how to quantify explained variance in these models. Additionally, analytic relationships between total and level-specific versions of MLM R-squared measures have not been clarified, despite such relationships becoming increasingly important to understand when there are more levels. Furthermore, the impact of predictor centering strategy on R-squared computation and interpretation has not been explicated for MLMs with any number of levels. To fill these gaps, we extend the Rights and Sterba two-level MLM R-squared framework to three or more levels, providing a general set of measures that includes preexisting three-level measures as special cases and yields additional results not obtainable from existing measures. We mathematically and pedagogically relate total and level-specific R-squareds, and show how all total and level-specific R-squared measures in our framework can be computed under any centering strategy. Finally, we provide and empirically demonstrate software (available in the r2mlm R package) to compute measures and graphically depict results.

Parameter estimation of the Solow-Swan fundamental differential equation.

Background: The Solow-Swan model describes the long-term growth of the capital to labor ratio by the fundamental differential equation of Solow-Swan theory. In conventional approaches, this equation was fitted to data using additional information, such as the rates of population growth, capital depreciation, or saving. However, this was not the best possible fit. Objectives: Using the method of least squares, what is the best possible fit of the fundamental equation to the time-series of the capital to labor ratios? Are the best-fit parameters economically sound? Method: For the data, we used the Penn-World Table in its 2021 version and compared six countries and three definitions of the capital to labor ratio. For optimization, we used a custom-made variant of the method of simulated annealing. We also compared different optimization methods and calibrations. Results: When comparing different methods of optimization, our custom-made tool provided reliable parameter estimates. In terms of R-squared they improved upon the parameter estimates of the conventional approach. Except for the USA, the best-fit values of the exponent were unplausible, as they suggested a too large elasticity of output. However, using a different calibration resulted in more plausible values of the best-fit exponent also for France and Pakistan, but not for Argentina and Japan. Conclusion: Our results have shown a discrepancy between the best-fit parameters obtained from optimization and the parameter values that are deemed plausible in economy. We propose a research program to resolve this issue by investigating if suitable calibrations may generate economically plausible best-fit parameter values.

Defining R-squared measures for mixed-effects location scale models.

Ecological momentary assessment and other modern data collection technologies facilitate research on both within-subject and between-subject variability of health outcomes and behaviors. For such intensively measured longitudinal data, Hedeker et al extended the usual two-level mixed-effects model to a two-level mixed-effects location scale (MELS) model to accommodate covariates' influence as well as random subject effects on both mean (location) and variability (scale) of the outcome. However, there is a lack of existing standardized effect size measures for the MELS model. To fill this gap, our study extends Rights and Sterba's framework of R 2 $$ {R}^2 $$ measures for multilevel models, which is based on model-implied variances, to MELS models. Our proposed framework applies to two different specifications of the random location effects, namely, through covariate-influenced random intercepts and through random intercepts combined with random slopes of observation-level covariates. We also provide an R function, R2MELS, that outputs summary tables and visualization for values of our R 2 $$ {R}^2 $$ measures. This framework is validated through a simulation study, and data from a health behaviors study and a depression study are used as examples to demonstrate this framework. These R 2 $$ {R}^2 $$ measures can help researchers provide greater interpretation of their findings using MELS models.

Devil in the details: how can we avoid potential pitfalls of CATS regression when our data do not follow a Poisson distribution?

BACKGROUND: Community assembly by trait selection (CATS) allows for the detection of environmental filtering and estimation of the relative role of local and regional (meta-community-level) effects on community composition from trait and abundance data without using environmental data. It has been shown that Poisson regression of abundances against trait data results in the same parameter estimates. Abundance data do not necessarily follow a Poisson distribution, and in these cases, other generalized linear models should be fitted to obtain unbiased parameter estimates. AIMS: This paper discusses how the original algorithm for calculating the relative role of local and regional effects has to be modified if Poisson model is not appropriate. RESULTS: It can be shown that the use of the logarithm of regional relative abundances as an offset is appropriate only if a log-link function is applied. Otherwise, the link function should be applied to the product of local total abundance and regional relative abundances. Since this product may be outside the domain of the link function, the use of log-link is recommended, even if it is not the canonical link. An algorithm is also suggested for calculating the offset when data are zero-inflated. The relative role of local and regional effects is measured by Kullback-Leibler R2. The formula for this measure presented by Shipley (2014) is valid only if the abundances follow a Poisson distribution. Otherwise, slightly different formulas have to be applied. Beyond theoretical considerations, the proposed refinements are illustrated by numerical examples. CATS regression could be a useful tool for community ecologists, but it has to be slightly modified when abundance data do not follow a Poisson distribution. This paper gives detailed instructions on the necessary refinement.

Model Selection and Average Proportion Explained Variance in Exploratory Factor Analysis.

A procedure for evaluating the average R-squared index for a given set of observed variables in an exploratory factor analysis model is discussed. The method can be used as an effective aid in the process of model choice with respect to the number of factors underlying the interrelationships among studied measures. The approach is developed within the framework of exploratory structural equation modeling and is readily applicable with popular statistical software. The outlined procedure is illustrated using a numerical example.

Implementation and Performance Evaluation of a Bivariate Cut-HDMR Metamodel for Semiconductor Packaging Design Problems with a Large Number of Input Variables.

A metamodeling technique based on Bivariate Cut High Dimensional Model Representation (Bivariate Cut HDMR) is implemented for a semiconductor packaging design problem with 10 design variables. Bivariate Cut-HDMR constructs a metamodel by considering only up to second-order interactions. The implementation uses three uniformly distributed sample points (s = 3) with quadratic spline interpolation to construct the component functions of Bivariate Cut-HDMR, which can be used to make a direct comparison with a metamodel based on Central Composite Design (CCD). The performance of Bivariate Cut-HDMR is evaluated by two well-known error metrics: R-squared and Relative Average Absolute Error (RAAE). The results are compared with the performance of CCD. Bivariate Cut HDMR does not compromise the accuracy compared to CCD, although the former uses only one-fifth of sample points (201 sample points) required by the latter (1045 sample points). The sampling schemes and the predictions of cut-planes and boundary-planes are discussed to explain possible reasons for the outstanding performance of Bivariate Cut HDMR.

R-squared change in structural equation models with latent variables and missing data.

Researchers frequently wish to make incremental validity claims, suggesting that a construct of interest significantly predicts a given outcome when controlling for other overlapping constructs and potential confounders. Once the significance of such an effect has been established, it is good practice to also assess and report its magnitude. In OLS regression, this is easily accomplished by calculating the change in R-squared, ΔR2, between one's full model and a reduced model that omits all but the target predictor(s) of interest. Because observed variable regression methods ignore measurement error, however, their estimates are prone to bias and inflated type I error rates. As a result, researchers are increasingly encouraged to switch from observed variable modeling conducted in the regression framework to latent variable modeling conducted in the structural equation modeling (SEM) framework. Standard SEM software packages provide overall R2 measures for each outcome, yet calculation of ΔR2 is not intuitive in models with latent variables. Omitting all indicators of a latent factor in a reduced model will alter the overidentifying constraints imposed on the model, affecting parameter estimation and fit. Furthermore, omitting variables in a reduced model may affect estimation under missing data, particularly when conditioning on those variables is essential to meeting the MAR assumption. In this article, I describe four approaches to calculating ΔR2 in SEMs with latent variables and missing data, compare their performance via simulation, describe a set of extensions to the methods, and provide a set of R functions for calculating ΔR2 in SEM.

Effect size measures for longitudinal growth analyses: Extending a framework of multilevel model R-squareds to accommodate heteroscedasticity, autocorrelation, nonlinearity, and alternative centering strategies.

Developmental researchers commonly utilize multilevel models (MLMs) to describe and predict individual differences in change over time. In such growth model applications, researchers have been widely encouraged to supplement reporting of statistical significance with measures of effect size, such as R-squareds (R2 ) that convey variance explained by terms in the model. An integrative framework for computing R-squareds in MLMs with random intercepts and/or slopes was recently introduced by Rights and Sterba and it subsumed pre-existing MLM R-squareds as special cases. However, this work focused on cross-sectional applications, and hence did not address how the computation and interpretation of MLM R-squareds are affected by modeling considerations typically arising in longitudinal settings: (a) alternative centering choices for time (e.g., centering-at-a-constant vs. person-mean-centering), (b) nonlinear effects of predictors such as time, (c) heteroscedastic level-1 errors and/or (d) autocorrelated level-1 errors. This paper addresses these gaps by extending the Rights and Sterba R-squared framework to longitudinal contexts. We: (a) provide a full framework of total and level-specific R-squared measures for MLMs that utilize any type of centering, and contrast these with Rights and Sterba's measures assuming cluster-mean-centering, (b) explain and derive which measures are applicable for MLMs with nonlinear terms, and extend the R-squared computation to accommodate (c) heteroscedastic and/or (d) autocorrelated errors. Additionally, we show how to use differences in R-squared (ΔR2 ) measures between growth models (adding, for instance, time-varying covariates as level-1 predictors or time-invariant covariates as level-2 predictors) to obtain effects sizes for individual terms. We provide R software (r2MLMlong) and a running pedagogical example analyzing growth in adolescent self-efficacy to illustrate these methodological developments. With these developments, researchers will have greater ability to consider effect size when analyzing and predicting change using MLMs.

A note on estimating the Cox-Snell R2 from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome.

In 2019 we published a pair of articles in Statistics in Medicine that describe how to calculate the minimum sample size for developing a multivariable prediction model with a continuous outcome, or with a binary or time-to-event outcome. As for any sample size calculation, the approach requires the user to specify anticipated values for key parameters. In particular, for a prediction model with a binary outcome, the outcome proportion and a conservative estimate for the overall fit of the developed model as measured by the Cox-Snell R2 (proportion of variance explained) must be specified. This proposal raises the question of how to identify a plausible value for R2 in advance of model development. Our articles suggest researchers should identify R2 from closely related models already published in their field. In this letter, we present details on how to derive R2 using the reported C statistic (AUROC) for such existing prediction models with a binary outcome. The C statistic is commonly reported, and so our approach allows researchers to obtain R2 for subsequent sample size calculations for new models. Stata and R code is provided, and a small simulation study.

Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.

Balancing fit and parsimony to improve Q-matrix validation.

The Q-matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q-matrix might contain some misspecifications, disrupting classification accuracy. Empirical Q-matrix validation methods such as the general discrimination index (GDI) and Wald have shown promising results in addressing this problem. However, a cut-off point is used in both methods, which might be suboptimal. To address this limitation, the Hull method is proposed and evaluated in the present study. This method aims to find the optimal balance between fit and parsimony, and it is flexible enough to be used either with a measure of item discrimination (the proportion of variance accounted for, PVAF) or a coefficient of determination (pseudo-R2 ). Results from a simulation study showed that the Hull method consistently showed the best performance and shortest computation time, especially when used with the PVAF. The Wald method also performed very well overall, while the GDI method obtained poor results when the number of attributes was high. The absence of a cut-off point provides greater flexibility to the Hull method, and it places it as a comprehensive solution to the Q-matrix specification problem in applied settings. This proposal is illustrated using real data.

New Recommendations on the Use of R-Squared Differences in Multilevel Model Comparisons.

When comparing multilevel models (MLMs) differing in fixed and/or random effects, researchers have had continuing interest in using R-squared differences to communicate effect size and importance of included terms. However, there has been longstanding confusion regarding which R-squared difference measures should be used for which kind of MLM comparisons. Furthermore, several limitations of recent studies on R-squared differences in MLM have led to misleading or incomplete recommendations for practice. These limitations include computing measures that are by definition incapable of detecting a particular type of added term, considering only a subset of the broader class of available R-squared difference measures, and incorrectly defining what a given R-squared difference measure quantifies. The purpose of this paper is to elucidate and resolve these issues. To do so, we define a more general set of total, within-cluster, and between-cluster R-squared difference measures than previously considered in MLM comparisons and give researchers concrete step-by-step procedures for identifying which measure is relevant to which model comparison. We supply simulated and analytic demonstrations of limitations of previous MLM studies on R-squared differences and show how application of our step-by-step procedures and general set of measures overcomes each. Additionally, we provide and illustrate graphical tools and software allowing researchers to automatically compute and visualize our set of measures in an integrated manner. We conclude with recommendations, as well as extensions involving (a) how our framework relates to and can be used to obtain pseudo-R-squareds, and (b) how our framework can accommodate both simultaneous and hierarchical model-building approaches.

Dataset for landscape pattern analysis from a climatic perspective.

Revealing the driving forces of changes in landscape pattern is a key question of landscape ecology and landscape analysis. Temperature and precipitation as climatic variables have a dominant role in triggering vegetation changes; thus, a database, which contain their interaction, can support the understanding of spatio-temporal changes in vegetation patterns even on a large scale. The dataset provided in this article contain the R-squared values of bivariate linear regression analysis between the Normalized Difference Vegetation Index (target variable; as a general quantitative descriptor of surface greenness) of the TERRA satellite's MODIS sensor and the climatic variables of the CarpatClim database (predictor variables; maximum monthly temperature, aridification index, evapotranspiration and precipitation). Environmental variables are also included to support further analysis: terrain height, macro regions, land cover classes. The dataset has a spatial projection (i.e. maps) and covers the area of Hungary. Tabular version provides the possibility of traditional statistical analysis, while maps allow the investigation to involve the spatial characteristics of absolute and relative position of the data points. This data article is related to the paper "NDVI dynamics as reflected in climatic variables: spatial and temporal trends - a case study of Hungary" (Szabo et al., 2019).

Evaluation of factors influencing secondary organic carbon (SOC) estimation by CO and EC tracer methods.

Secondary organic carbon (SOC) is known to account for a substantial fraction of fine-mode carbonaceous aerosol. Owing to a limited understanding of SOC formation processes and the absence of direct measurement methods, SOC concentrations are mostly estimated using a tracer-based method utilizing either elemental carbon (EC) or carbon monoxide (CO) as tracers. The performance of these tracer-based methods depends heavily on accurate determination of the (OC/Tracer)pri value. The minimum R squared (MRS) method is currently recognized as a relatively reasonable tool to determine (OC/Tracer)pri. This study estimated SOC based on the MRS method with EC and CO as tracers, followed by the Monte Carlo method to analyze quantitatively the effects of measurement uncertainty, emission scenario and sample size on SOC estimates. We report here four major findings: i) the concentration of O3 was used as an indicator to atmospheric secondary reaction potential, and it was found that the mass proportion of SOC in total OC estimated by CO as the tracer is more consistent with the seasonality of actual secondary reaction potential; ii) the estimation results are highly sensitive to the measurement uncertainty in different emission scenarios, which leads us to conclude that the CO tracer method is more robust than the EC tracer method due to large inherent uncertainties in current EC measurements; iii) oversimplification of emission scenarios has substantial impacts on the estimated SOC value, and careful evaluation of the interdependence between sources should be performed to minimize this bias; and iv) the estimation bias of SOC can be reduced by increasing the sample size, and the tracer method can be expected to generate robust results for sample sizes over 1000. These findings are important in terms of providing a reference to choose appropriate tracers, emission scenarios and sample sizes for robust estimation of SOC in future studies.

Ipsilesional Mu Rhythm Desynchronization and Changes in Motor Behavior Following Post Stroke BCI Intervention for Motor Rehabilitation.

Loss of motor function is a common deficit following stroke insult and often manifests as persistent upper extremity (UE) disability which can affect a survivor's ability to participate in activities of daily living. Recent research suggests the use of brain-computer interface (BCI) devices might improve UE function in stroke survivors at various times since stroke. This randomized crossover-controlled trial examines whether intervention with this BCI device design attenuates the effects of hemiparesis, encourages reorganization of motor related brain signals (EEG measured sensorimotor rhythm desynchronization), and improves movement, as measured by the Action Research Arm Test (ARAT). A sample of 21 stroke survivors, presenting with varied times since stroke and levels of UE impairment, received a maximum of 18-30 h of intervention with a novel electroencephalogram-based BCI-driven functional electrical stimulator (EEG-BCI-FES) device. Driven by spectral power recordings from contralateral EEG electrodes during cued attempted grasping of the hand, the user's input to the EEG-BCI-FES device modulates horizontal movement of a virtual cursor and also facilitates concurrent stimulation of the impaired UE. Outcome measures of function and capacity were assessed at baseline, mid-therapy, and at completion of therapy while EEG was recorded only during intervention sessions. A significant increase in r-squared values [reflecting Mu rhythm (8-12 Hz) desynchronization as the result of attempted movements of the impaired hand] presented post-therapy compared to baseline. These findings suggest that intervention corresponds with greater desynchronization of Mu rhythm in the ipsilesional hemisphere during attempted movements of the impaired hand and this change is related to changes in behavior as a result of the intervention. BCI intervention may be an effective way of addressing the recovery of a stroke impaired UE and studying neuromechanical coupling with motor outputs. Clinical Trial Registration: ClinicalTrials.gov, identifier NCT02098265.