Distributed Inference for Spatial Extremes Modeling in High Dimensions.

Extreme environmental events frequently exhibit spatial and temporal dependence. These data are often modeled using max stable processes (MSPs) that are computationally prohibitive to fit for as few as a dozen observations. Supposed computationally-efficient approaches like the composite likelihood remain computationally burdensome with a few hundred observations. In this paper, we propose a spatial partitioning approach based on local modeling of subsets of the spatial domain that delivers computationally and statistically efficient inference. Marginal and dependence parameters of the MSP are estimated locally on subsets of observations using censored pairwise composite likelihood, and combined using a modified generalized method of moments procedure. The proposed distributed approach is extended to estimate inverted MSP models, and to estimate spatially varying coefficient models to deliver computationally efficient modeling of spatial variation in marginal parameters. We demonstrate consistency and asymptotic normality of estimators, and show empirically that our approach leads to statistically efficient estimation of model parameters. We illustrate the flexibility and practicability of our approach through simulations and the analysis of streamflow data from the U.S. Geological Survey.

Adaptive bias-variance trade-off in advantage estimator for actor-critic algorithms.

Actor-critic methods are leading in many challenging continuous control tasks. Advantage estimators, the most common critics in the actor-critic framework, combine state values from bootstrapping value functions and sample returns. Different combinations balance the bias introduced by state values and the variance returned by samples to reduce estimation errors. The bias and variance constantly fluctuate throughout training, leading to different optimal combinations. However, existing advantage estimators usually use fixed combinations that fail to account for the trade-off between minimizing bias and variance to find the optimal estimate. Our previous work on adaptive advantage estimation (AAE) analyzed the sources of bias and variance and offered two indicators. This paper further explores the relationship between the indicators and their optimal combination through typical numerical experiments. These analyses develop a general form of adaptive combinations of state values and sample returns to achieve low estimation errors. Empirical results on simulated robotic locomotion tasks show that our proposed estimators achieve similar or superior performance compared to previous generalized advantage estimators (GAE).

Using Decomposed Error for Reproducing Implicit Understanding of Algorithms.

Reproducibility is important for having confidence in evolutionary machine learning algorithms. Although the focus of reproducibility is usually to recreate an aggregate prediction error score using fixed random seeds, this is not sufficient. Firstly, multiple runs of an algorithm, without a fixed random seed, should ideally return statistically equivalent results. Secondly, it should be confirmed whether the expected behaviour of an algorithm matches its actual behaviour, in terms of how an algorithm targets a reduction in prediction error. Confirming the behaviour of an algorithm is not possible when using a total error aggregate score. Using an error decomposition framework as a methodology for improving the reproducibility of results in evolutionary computation addresses both of these factors. By estimating decomposed error using multiple runs of an algorithm and multiple training sets, the framework provides a greater degree of certainty about the prediction error. Also, decomposing error into bias, variance due to the algorithm (internal variance), and variance due to the training data (external variance) more fully characterises evolutionary algorithms. This allows the behaviour of an algorithm to be confirmed. Applying the framework to a number of evolutionary algorithms shows that their expected behaviour can be different to their actual behaviour. Identifying a behaviour mismatch is important in terms of understanding how to further refine an algorithm as well as how to effectively apply an algorithm to a problem.

Analysis of a Binary Outcome Dichotomized from an Underlying Continuous Variable in Clinical Trials.

Binary-valued outcome is often seen in many clinical trials across therapeutic areas. It is not uncommon that such binary endpoints are derived from a continuous variable. For example, in diabetes clinical trials, the proportion of patients with HbA1c< 7% is often investigated as one of the key objectives, where HbA1c is a continuous-valued variable reflecting the averaged blood glucose value from the previous three months. Most of the time, if not all, the mean of those binary endpoints were estimated directly through the binary variable defined by the corresponding cutoff. Alternatively, by the nature of the derivation, that quantity could also be estimated by leveraging the density of the underlying continuous variable and computing the area under the density curve up to a threshold. This paper provides a few methods in relation to density estimation. Extensive simulation studies were conducted based on real clinical trial data to compare these estimation approaches against the direct estimation of the proportions. Simulation results showed that the density estimation approaches in general benefited from a smaller mean squared error in early phase studies where the sample size is limited. The density estimation approach is certainly expected to introduce bias, however, a favorable bias-variance trade-off may make these approaches attractive in early phase studies.

A Method for Increasing the Robustness of Stable Feature Selection for Biomarker Discovery in Molecular Medicine Developed Using Serum Small Extracellular Vesicle Associated miRNAs and the Barrett's Oesophagus Disease Spectrum.

The biomarker development field within molecular medicine remains limited by the methods that are available for building predictive models. We developed an efficient method for conservatively estimating confidence intervals for the cross validation-derived prediction errors of biomarker models. This new method was investigated for its ability to improve the capacity of our previously developed method, StaVarSel, for selecting stable biomarkers. Compared with the standard cross validation method, StaVarSel markedly improved the estimated generalisable predictive capacity of serum miRNA biomarkers for the detection of disease states that are at increased risk of progressing to oesophageal adenocarcinoma. The incorporation of our new method for conservatively estimating confidence intervals into StaVarSel resulted in the selection of less complex models with increased stability and improved or similar predictive capacities. The methods developed in this study have the potential to improve progress from biomarker discovery to biomarker driven translational research.

The Big Data Paradox in Clinical Practice.

The big data paradox is a real-world phenomenon whereby as the number of patients enrolled in a study increases, the probability that the confidence intervals from that study will include the truth decreases. This occurs in both observational and experimental studies, including randomized clinical trials, and should always be considered when clinicians are interpreting research data. Furthermore, as data quantity continues to increase in today's era of big data, the paradox is becoming more pernicious. Herein, I consider three mechanisms that underlie this paradox, as well as three potential strategies to mitigate it: (1) improving data quality; (2) anticipating and modeling patient heterogeneity; (3) including the systematic error, not just the variance, in the estimation of error intervals.

Learning in Medicine: The Importance of Statistical Thinking.

In many fields, including medicine and biology, there has been in the last years an increasing diffusion and availability of complex data from different sources. Examples include biological experiments or data from health care providers. These data encompass information that can potentially enhance therapeutic advancement and constitute the core of modern system medicine. When analyzing these complex data, it is important to appropriately quantify uncertainty, avoiding using only algorithmic and automated approaches, which are not always appropriate. Improper application of algorithmic approaches, which ignore domain knowledge, could result in filling the literature with imprecise and/or misleading conclusions. In this chapter, we highlight the importance of statistical thinking when leveraging complex data and models to enhance science progress. In particular, we discuss the reproducibility and replicability issues, the importance of uncertainty quantification, and some common pitfalls in the analysis of big data.

On the Use of the Pearson Correlation Coefficient for Model Evaluation in Genome-Wide Prediction.

The large number of markers in genome-wide prediction demands the use of methods with regularization and model comparison based on some hold-out test prediction error measure. In quantitative genetics, it is common practice to calculate the Pearson correlation coefficient (r2 ) as a standardized measure of the predictive accuracy of a model. Based on arguments from the bias-variance trade-off theory in statistical learning, we show that shrinkage of the regression coefficients (i.e., QTL effects) reduces the prediction mean squared error (MSE) by introducing model bias compared with the ordinary least squares method. We also show that the LASSO and the adaptive LASSO (ALASSO) can reduce the model bias and prediction MSE by adding model variance. In an application of ridge regression, the LASSO and ALASSO to a simulated example based on results for 9,723 SNPs and 3,226 individuals, the best model selected was with the LASSO when r2 was used as a measure. However, when model selection was based on test MSE and coefficient of determination R2 the ALASSO proved to be the best method. Hence, use of r2 may lead to selection of the wrong model and therefore also nonoptimal ranking of phenotype predictions and genomic breeding values. Instead, we propose use of the test MSE for model selection and R2 as a standardized measure of the accuracy.

Reconciling modern machine-learning practice and the classical bias-variance trade-off.

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in modern machine-learning practice. The bias-variance trade-off implies that a model should balance underfitting and overfitting: Rich enough to express underlying structure in data and simple enough to avoid fitting spurious patterns. However, in modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered overfitted, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double-descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine-learning models delineates the limits of classical analyses and has implications for both the theory and the practice of machine learning.

Rejoinder: Remaining Challenges in Investigating Grade-Retention Effectiveness.

This rejoinder, in response to the commentaries of Steiner, Park, and Kim (this issue) and Reshetnyak, Cham, and Hughes (this issue), discusses remaining challenges in grade retention research. First, a same-age comparison assumes that the instruments used in different grades measure ability equally well. We discuss the importance of evaluating the properties of the scaling process to address whether this assumption has been met. Second, we discuss issues in the selection of covariates to be included in the weights. Third, we discuss the unconfoundedness assumption and the problem of remaining imbalance. Finally, we provide an empirical illustration showing that studying grade-retention effectiveness comes with multiple methodological decisions that are rooted in a bias-variance trade-off.

Combining parametric, semi-parametric, and non-parametric survival models with stacked survival models.

For estimating conditional survival functions, non-parametric estimators can be preferred to parametric and semi-parametric estimators due to relaxed assumptions that enable robust estimation. Yet, even when misspecified, parametric and semi-parametric estimators can possess better operating characteristics in small sample sizes due to smaller variance than non-parametric estimators. Fundamentally, this is a bias-variance trade-off situation in that the sample size is not large enough to take advantage of the low bias of non-parametric estimation. Stacked survival models estimate an optimally weighted combination of models that can span parametric, semi-parametric, and non-parametric models by minimizing prediction error. An extensive simulation study demonstrates that stacked survival models consistently perform well across a wide range of scenarios by adaptively balancing the strengths and weaknesses of individual candidate survival models. In addition, stacked survival models perform as well as or better than the model selected through cross-validation. Finally, stacked survival models are applied to a well-known German breast cancer study.

Nonlinear Calibration Model Choice between the Four and Five-Parameter Logistic Models.

Both the four-parameter logistic (4PL) and the five-parameter logistic (5PL) models are widely used in nonlinear calibration. In this paper, we study the choice between 5PL and 4PL both by the accuracy and precision of the estimated concentrations and by the power to detect an association between a binary disease outcome and the estimated concentrations. Our results show that when the true curve is symmetric around its inflection point, the efficiency loss from using 5PL is negligible under the prevalent experimental design. When the true curve is asymmetric, 4PL may sometimes offer better performance due to bias-variance trade-off. We provide a practical guideline for choosing between 5PL and 4PL and illustrate its application with a real dataset from the HIV Vaccine Trials Network laboratory.