A Comparison of Additional Benefit Assessment Methods for Time-to-Event Endpoints Using Hazard Ratio Point Estimates or Confidence Interval Limits by Means of a Simulation Study.

BACKGROUND: For time-to-event endpoints, three additional benefit assessment methods have been developed aiming at an unbiased knowledge about the magnitude of clinical benefit of newly approved treatments. The American Society of Clinical Oncology (ASCO) defines a continuous score using the hazard ratio point estimate (HR-PE). The European Society for Medical Oncology (ESMO) and the German Institute for Quality and Efficiency in Health Care (IQWiG) developed methods with an ordinal outcome using lower and upper limits of the 95% HR confidence interval (HR-CI), respectively. We describe all three frameworks for additional benefit assessment aiming at a fair comparison across different stakeholders. Furthermore, we determine which ASCO score is consistent with which ESMO/IQWiG category. METHODS: In a comprehensive simulation study with different failure time distributions and treatment effects, we compare all methods using Spearman's correlation and descriptive measures. For determination of ASCO values consistent with categories of ESMO/IQWiG, maximizing weighted Cohen's Kappa approach was used. RESULTS: Our research depicts a high positive relationship between ASCO/IQWiG and a low positive relationship between ASCO/ESMO. An ASCO score smaller than 17, 17 to 20, 20 to 24, and greater than 24 corresponds to ESMO categories. Using ASCO values of 21 and 38 as cutoffs represents IQWiG categories. LIMITATIONS: We investigated the statistical aspects of the methods and hence implemented slightly reduced versions of all methods. CONCLUSIONS: IQWiG and ASCO are more conservative than ESMO, which often awards the maximal category independent of the true effect and is at risk of overcompensating with various failure time distributions. ASCO has similar characteristics as IQWiG. Delayed treatment effects and underpowered/overpowered studies influence all methods in some degree. Nevertheless, ESMO is the most liberal one. HIGHLIGHTS: For the additional benefit assessment, the American Society of Clinical Oncology (ASCO) uses the hazard ratio point estimate (HR-PE) for their continuous score. In contrast, the European Society for Medical Oncology (ESMO) and the German Institute for Quality and Efficiency in Health Care (IQWiG) use the lower and upper 95% HR confidence interval (HR-CI) to specific thresholds, respectively. ESMO generously assigns maximal scores, while IQWiG is more conservative.This research provides the first comparison between IQWiG and ASCO and describes all three frameworks for additional benefit assessment aiming for a fair comparison across different stakeholders. Furthermore, thresholds for ASCO consistent with ESMO and IQWiG categories are determined, enabling a comparison of the methods in practice in a fair manner.IQWiG and ASCO are the more conservative methods, while ESMO awards high percentages of maximal categories, especially with various failure time distributions. ASCO has similar characteristics as IQWiG. Delayed treatment effects and under/-overpowered studies influence all methods. Nevertheless, ESMO is the most liberal one. An ASCO score smaller than 17, 17 to 20, 20 to 24, and greater than 24 correspond to the categories of ESMO. Using ASCO values of 21 and 38 as cutoffs represents categories of IQWiG.

Confidence intervals for odds ratio from multistage randomized phase II trials.

A multi-stage randomized trial design can significantly improve efficiency by allowing early termination of the trial when the experimental arm exhibits either low or high efficacy compared to the control arm during the study. However, proper inference methods are necessary because the underlying distribution of the target statistic changes due to the multi-stage structure. This article focuses on multi-stage randomized phase II trials with a dichotomous outcome, such as treatment response, and proposes exact conditional confidence intervals for the odds ratio. The usual single-stage confidence intervals are invalid when used in multi-stage trials. To address this issue, we propose a linear ordering of all possible outcomes. This ordering is conditioned on the total number of responders in each stage and utilizes the exact conditional distribution function of the outcomes. This approach enables the estimation of an exact confidence interval accounting for the multi-stage designs.

Confidence intervals for validation statistics with data truncation in genomic prediction.

BACKGROUND: Validation by data truncation is a common practice in genetic evaluations because of the interest in predicting the genetic merit of a set of young selection candidates. Two of the most used validation methods in genetic evaluations use a single data partition: predictivity or predictive ability (correlation between pre-adjusted phenotypes and estimated breeding values (EBV) divided by the square root of the heritability) and the linear regression (LR) method (comparison of "early" and "late" EBV). Both methods compare predictions with the whole dataset and a partial dataset that is obtained by removing the information related to a set of validation individuals. EBV obtained with the partial dataset are compared against adjusted phenotypes for the predictivity or EBV obtained with the whole dataset in the LR method. Confidence intervals for predictivity and the LR method can be obtained by replicating the validation for different samples (or folds), or bootstrapping. Analytical confidence intervals would be beneficial to avoid running several validations and to test the quality of the bootstrap intervals. However, analytical confidence intervals are unavailable for predictivity and the LR method. RESULTS: We derived standard errors and Wald confidence intervals for the predictivity and statistics included in the LR method (bias, dispersion, ratio of accuracies, and reliability). The confidence intervals for the bias, dispersion, and reliability depend on the relationships and prediction error variances and covariances across the individuals in the validation set. We developed approximations for large datasets that only need the reliabilities of the individuals in the validation set. The confidence intervals for the ratio of accuracies and predictivity were obtained through the Fisher transformation. We show the adequacy of both the analytical and approximated analytical confidence intervals and compare them versus bootstrap confidence intervals using two simulated examples. The analytical confidence intervals were closer to the simulated ones for both examples. Bootstrap confidence intervals tend to be narrower than the simulated ones. The approximated analytical confidence intervals were similar to those obtained by bootstrapping. CONCLUSIONS: Estimating the sampling variation of predictivity and the statistics in the LR method without replication or bootstrap is possible for any dataset with the formulas presented in this study.

Principal Component Analysis Enhanced with Bootstrapped Confidence Interval for the Classification of Parkinsonian Patients Using Gaussian Mixture Model and Gait Initiation Parameters.

Parkinson's disease is one of the major neurodegenerative diseases that affects the postural stability of patients, especially during gait initiation. There is actually an increasing demand for the development of new non-pharmacological tools that can easily classify healthy/affected patients as well as the degree of evolution of the disease. The experimental characterization of gait initiation (GI) is usually done through the simultaneous acquisition of about 20 variables, resulting in very large datasets. Dimension reduction tools are therefore suitable, considering the complexity of the physiological processes involved. The principal Component Analysis (PCA) is very powerful at reducing the dimensionality of large datasets and emphasizing correlations between variables. In this paper, the Principal Component Analysis (PCA) was enhanced with bootstrapping and applied to the study of the GI to identify the 3 majors sets of variables influencing the postural control disability of Parkinsonian patients during GI. We show that the combination of these methods can lead to a significant improvement in the unsupervised classification of healthy/affected patients using a Gaussian mixture model, since it leads to a reduced confidence interval on the estimated parameters. The benefits of this method for the identification and study of the efficiency of potential treatments is not addressed in this paper but could be addressed in future works.

A nonparametric relative treatment effect for direct comparisons of censored paired survival outcomes.

A frequently addressed issue in clinical trials is the comparison of censored paired survival outcomes, for example, when individuals were matched based on their characteristics prior to the analysis. In this regard, a proper incorporation of the dependence structure of the paired censored outcomes is required and, up to now, appropriate methods are only rarely available in the literature. Moreover, existing methods are not motivated by the strive for insights by means of an easy-to-interpret parameter. Hence, we seek to develop a new estimand-driven method to compare the effectiveness of two treatments in the context of right-censored survival data with matched pairs. With the help of competing risks techniques, the so-called relative treatment effect is estimated. This estimand describes the probability that individuals under Treatment 1 have a longer lifetime than comparable individuals under Treatment 2. We derive hypothesis tests and confidence intervals based on a studentized version of the estimator, where resampling-based inference is established by means of a randomization method. In a simulation study, we demonstrate for numerous sample sizes and different amounts of censoring that the developed test exhibits a good power. Finally, we apply the methodology to a well-known benchmark data set from a trial with patients suffering from diabetic retinopathy.

A Comparison of Statistical Methods to Construct Confidence Intervals and Fiducial Intervals for Measures of Health Disparities.

Health disparities are differences in health status across different socioeconomic groups. Classical methods, e.g., the Delta method, have been used to estimate the standard errors of estimated measures of health disparities and to construct confidence intervals for these measures. However, the confidence intervals constructed using the classical methods do not have good coverage properties for situations involving sparse data. In this article, we introduce three new methods to construct fiducial intervals for measures of health disparities based on approximate fiducial quantities. Through a comprehensive simulation study, We compare the empirical coverage properties of the proposed fiducial intervals against two Monte Carlo simulation-based methods-utilizing either a truncated Normal distribution or the Gamma distribution-as well as the classical method. The findings of the simulation study advocate for the adoption of the Monte Carlo simulation-based method with the Gamma distribution when a unified approach is sought for all health disparity measures.

Point estimation, confidence intervals, and P-values for optimal adaptive two-stage designs with normal endpoints.

Due to the dependency structure in the sampling process, adaptive trial designs create challenges in point and interval estimation and in the calculation of P-values. Optimal adaptive designs, which are designs where the parameters governing the adaptivity are chosen to maximize some performance criterion, suffer from the same problem. Various analysis methods which are able to handle this dependency structure have already been developed. In this work, we aim to give a comprehensive summary of these methods and show how they can be applied to the class of designs with planned adaptivity, of which optimal adaptive designs are an important member. The defining feature of these kinds of designs is that the adaptive elements are completely prespecified. This allows for explicit descriptions of the calculations involved, which makes it possible to evaluate different methods in a fast and accurate manner. We will explain how to do so, and present an extensive comparison of the performance characteristics of various estimators between an optimal adaptive design and its group-sequential counterpart.

Confidence intervals of location for marine mammal calls via time-differences-of-arrival: Sensitivity analysis.

Confidence intervals of location (CIL) of calling marine mammals, derived from time-differences-of-arrival (TDOA) between receivers, depend on errors of TDOAs, receiver location, clocks, and sound speeds. Simulations demonstrate a time-differences-of-arrival-beamforming-locator (TDOA-BL) yields CIL in error by O(10-100) km for experimental scenarios because it is not designed to account for relevant errors. The errors are large and sometimes exceed the distances of detection. Another locator designed for all errors, sequential bound estimation, yields CIL always containing the true location. TDOA-BL have and are being used to understand potential effects of environmental stress on marine mammals; a use worth reconsidering.

Exact interval estimation for the linear combination of binomial proportions.

The weighted sum of binomial proportions and the interaction effect are two important cases of the linear combination of binomial proportions. Existing confidence intervals for these two parameters are approximate. We apply the h-function method to a given approximate interval and obtain an exact interval. The process is repeated multiple times until the final-improved interval (exact) cannot be shortened. In particular, for the weighted sum of two proportions, we derive two final-improved intervals based on the (approximate) adjusted score and fiducial intervals. After comparing several currently used intervals, we recommend these two final-improved intervals for practice. For the weighted sum of three proportions and the interaction effect, the final-improved interval based on the adjusted score interval should be used. Three real datasets are used to detail how the approximate intervals are improved.

Probabilistic precision calculations for the planning of studies assessing negative binomial rates.

PURPOSE: Outcome variables that are assumed to follow a negative binomial distribution are frequently used in both clinical and epidemiological studies. Epidemiological studies, particularly those performed by pharmaceutical companies often aim to describe a population rather than compare treatments. Such descriptive studies are often analysed using confidence intervals. While precision calculations and sample size calculations are not always performed in these settings, they have the important role of setting expectations of what results the study may generate. Current methods for precision calculations for the negative binomial rate are based on plugging in parameter values into the confidence interval formulae. This method has the downside of ignoring the randomness of the confidence interval limits. To enable better practice for precision calculations, methods are needed that address the randomness. METHODS: Using the well-known delta-method we develop a method for calculating the precision probability, that is, the probability of achieving a certain width. We assess the performance of the method in smaller samples through simulations. RESULTS: The method for the precision probability performs well in small to medium sample sizes, and the usefulness of the method is demonstrated through an example. CONCLUSIONS: We have developed a simple method for calculating the precision probability for negative binomial rates. This method can be used when planning epidemiological studies in for example, asthma, while correctly taking the randomness of confidence intervals into account.

Towards More Robust Evaluation of the Predictive Performance of Physiologically Based Pharmacokinetic Models: Using Confidence Intervals to Support Use of Model-Informed Dosing in Clinical Care.

BACKGROUND AND OBJECTIVE: With the rise in the use of physiologically based pharmacokinetic (PBPK) modeling over the past decade, the use of PBPK modeling to underpin drug dosing for off-label use in clinical care has become an attractive option. In order to use PBPK models for high-impact decisions, thorough qualification and validation of the model is essential to gain enough confidence in model performance. Currently, there is no agreed method for model acceptance, while clinicians demand a clear measure of model performance before considering implementing PBPK model-informed dosing. We aim to bridge this gap and propose the use of a confidence interval for the predicted-to-observed geometric mean ratio with predefined boundaries. This approach is similar to currently accepted bioequivalence testing procedures and can aid in improved model credibility and acceptance. METHODS: Two different methods to construct a confidence interval are outlined, depending on whether individual observations or aggregate data are available from the clinical comparator data sets. The two testing procedures are demonstrated for an example evaluation of a midazolam PBPK model. In addition, a simulation study is performed to demonstrate the difference between the twofold criterion and our proposed method. RESULTS: Using midazolam adult pharmacokinetic data, we demonstrated that creating a confidence interval yields more robust evaluation of the model than a point estimate, such as the commonly used twofold acceptance criterion. Additionally, we showed that the use of individual predictions can reduce the number of required test subjects. Furthermore, an easy-to-implement software tool was developed and is provided to make our proposed method more accessible. CONCLUSIONS: With this method, we aim to provide a tool to further increase confidence in PBPK model performance and facilitate its use for directly informing drug dosing in clinical care.

An evaluation of confidence intervals for a cumulative proportion to enable decisions at interim reviews of single-arm trials.

BACKGROUND: Clinical trials often include interim analyses of the proportion of participants experiencing an event by a fixed time-point. A pre-specified proportion excluded from a corresponding confidence interval (CI) may lead an independent monitoring committee to recommend stopping the trial. Frequently this cumulative proportion is estimated by the Kaplan-Meier estimator with a Wald approximate CI, which may have coverage issues with small samples. METHODS: We reviewed four alternative CI methods for cumulative proportions (Beta Product Confidence Procedure (BPCP), BPCP Mid P, Rothman-Wilson, Thomas-Grunkemeier) and two CI methods for simple proportions (Clopper-Pearson, Wilson). We conducted a simulation study comparing CI methods across true event proportions for 12 scenarios differentiated by sample sizes and censoring patterns. We re-analyzed interim data from A5340, a HIV cure trial considering the proportion of participants experiencing virologic failure. RESULTS: Our simulation study highlights the lower and upper tail error probabilities for each CI method. Across scenarios, we found differences in the performance of lower versus upper bounds. No single method is always preferred. The upper bound of a Wald approximate CI performed reasonably with some error inflation, whereas the lower bound of the BPCP Mid P method performed well. For a trial design similar to A5340, we recommend BPCP Mid P. CONCLUSIONS: The design of future single-arm interim analyses of event proportions should consider the most appropriate CI method based on the relevant bound, anticipated sample size and event proportion. Our paper summarizes available methods, demonstrates performance in a simulation study, and includes code for implementation.

Editorial Commentary: The Statistical Fragility Index of Medical Trials Is Low By Design: Critical Evaluation of Confidence Intervals Is Required.

The Fragility Index (FI) provides the number of patients whose outcome would need to have changed for the results of a clinical trial to no longer be statistically significant. Although it's a well-intended and easily interpreted metric, its calculation is based on reversing a significant finding and therefore its interpretation is only relevant in the domain of statistical significance. Its interpretation is only relevant in the domain of statistical significance. A well-designed clinical trial includes an a priori sample size calculation that aims to find the bare minimum of patients needed to obtain statistical significance. Such trials are fragile by design! Examining the robustness of clinical trials requires an estimation of uncertainty, rather than a misconstrued, dichotomous focus on statistical significance. Confidence intervals (CIs) provide a range of values that are compatible with a study's data and help determine the precision of results and the compatibility of the data with different hypotheses. The width of the CI speaks to the precision of the results, and the extent to which the values contained within have potential to be clinically important. Finally, one should not assume that a large FI indicates robust findings. Poorly executed trials are prone to bias, leading to large effects, and therefore, small P values, and a large FI. Let's move our future focus from the FI toward the CI.

P-values and confidence intervals of linear rank tests for left-truncated data under truncated binomial design.

The paper provides computations comparing the accuracy of the saddlepoint approximation approach and the normal approximation method in approximating the mid-p-value of Wilcoxon and log-rank tests for the left-truncated data using a truncated binomial design. The paper uses real data examples to apply the comparison, along with some simulated studies. Confidence intervals are provided by the inversion of the tests under consideration.

Analysis of two binomial proportions in noninferiority confirmatory trials.

In this article, we propose considering an approximate exact score (AES) test for noninferiority comparisons and we derive its test-based confidence interval for the difference between two independent binomial proportions. This test was published in the literature, but not its associated confidence interval. The p-value for this test is obtained by using exact binomial probabilities with the nuisance parameter being replaced by its restricted maximum likelihood estimate. Calculated type I errors revealed that the AES method has important advantages for noninferiority comparisons over popular asymptotic methods for adequately powered confirmatory clinical trials, at 80% or 90% statistical power. For unbalanced sample sizes of the compared groups, type I errors for the asymptotic score method were shown to be higher than the nominal level in a systematic pattern over a range of true proportions, but the AES method did not suffer from such a problem. On average, the true type I error of the AES method was closer to the nominal level than all considered methods in the empirical comparisons. In rare cases, type I errors of the AES test exceeded the nominal level, but only by a small amount. Presented examples showed that the AES method can be more attractive in practice than practical exact methods. In addition, p-value and confidence interval of the AES method can be obtained in <30 s of computer time for most confirmatory trials. Theoretical arguments, combined with empirical evidence and fast computation time should make the AES method attractive in statistical practice.

The autocorrelated Bayesian sampler: A rational process for probability judgments, estimates, confidence intervals, choices, confidence judgments, and response times.

Normative models of decision-making that optimally transform noisy (sensory) information into categorical decisions qualitatively mismatch human behavior. Indeed, leading computational models have only achieved high empirical corroboration by adding task-specific assumptions that deviate from normative principles. In response, we offer a Bayesian approach that implicitly produces a posterior distribution of possible answers (hypotheses) in response to sensory information. But we assume that the brain has no direct access to this posterior, but can only sample hypotheses according to their posterior probabilities. Accordingly, we argue that the primary problem of normative concern in decision-making is integrating stochastic hypotheses, rather than stochastic sensory information, to make categorical decisions. This implies that human response variability arises mainly from posterior sampling rather than sensory noise. Because human hypothesis generation is serially correlated, hypothesis samples will be autocorrelated. Guided by this new problem formulation, we develop a new process, the Autocorrelated Bayesian Sampler (ABS), which grounds autocorrelated hypothesis generation in a sophisticated sampling algorithm. The ABS provides a single mechanism that qualitatively explains many empirical effects of probability judgments, estimates, confidence intervals, choice, confidence judgments, response times, and their relationships. Our analysis demonstrates the unifying power of a perspective shift in the exploration of normative models. It also exemplifies the proposal that the "Bayesian brain" operates using samples not probabilities, and that variability in human behavior may primarily reflect computational rather than sensory noise. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Estimation of median survival time and its 95% confidence interval using SAS PROC LIFETEST.

Estimation of median survival and its 95% confidence interval depends on the choice of the survival function, standard error, and a method for constructing the confidence interval. This paper outlines several available possibilities in SAS® (version 9.4) PROC LIFETEST and compares them on theoretical grounds and using simulated data, with criteria: ability to estimate the 95% CI, coverage probability, interval width, and appropriateness for practical use. Data are generated with varying hazard patterns, N, % censoring, and censoring patterns (early, uniform, late, last visit). LIFETEST was run using the Kaplan-Meier and Nelson-Aalen estimators and the transformations available (linear, log, logit, complementary log-log, and arcsine square root). Using the Kaplan-Meier estimator with the logarithmic transformation as well as with the logit leads to a high frequency of LIFETEST not being able to estimate the 95% CI. The combination of Kaplan-Meier with the linear transformation is associated with poor coverage achieved. For small samples, late/last visit censoring has a negative effect on being able to estimate the 95% CI. Heavy early censoring can lead to low coverage of the 95% CI of median survival for sample sizes up to and including N = 40. The two combinations that are optimal for being able to estimate the 95% CI and having adequate coverage are the Kaplan-Meier estimator with complementary log-log transformation, and the Nelson-Aalen estimator with linear transformation. The former fares best on the third criterion (smaller width) and is also the SAS® default and validates the choice of default.

A Generalized Bootstrap Procedure of the Standard Error and Confidence Interval Estimation for Inverse Probability of Treatment Weighting.

The inverse probability of treatment weighting (IPTW) approach is commonly used in propensity score analysis to infer causal effects in regression models. Due to oversized IPTW weights and errors associated with propensity score estimation, the IPTW approach can underestimate the standard error of causal effect. To remediate this, bootstrap standard errors have been recommended to replace the IPTW standard error, but the ordinary bootstrap (OB) procedure might still result in underestimation of the standard error because of its inefficient resampling scheme and untreated oversized weights. In this paper, we develop a generalized bootstrap (GB) procedure for estimating the standard error and confidence intervals of the IPTW approach. Compared with the OB procedure and other three procedures in comparison, the GB procedure has the highest precision and yields conservative standard error estimates. As a result, the GB procedure produces short confidence intervals with highest coverage rates. We demonstrate the effectiveness of the GB procedure via two simulation studies and a dataset from the National Educational Longitudinal Study-1988 (NELS-88).

Monte Carlo confidence intervals for the indirect effect with missing data.

Missing data is a common occurrence in mediation analysis. As a result, the methods used to construct confidence intervals around the indirect effect should consider missing data. Previous research has demonstrated that, for the indirect effect in data with complete cases, the Monte Carlo method performs as well as nonparametric bootstrap confidence intervals (see MacKinnon et al., Multivariate Behavioral Research, 39(1), 99-128, 2004; Preacher & Selig, Communication Methods and Measures, 6(2), 77-98, 2012; Tofighi & MacKinnon, Structural Equation Modeling: A Multidisciplinary Journal, 23(2), 194-205, 2015). In this manuscript, we propose a simple, fast, and accurate two-step approach for generating confidence intervals for the indirect effect, in the presence of missing data, based on the Monte Carlo method. In the first step, an appropriate method, for example, full-information maximum likelihood or multiple imputation, is used to estimate the parameters and their corresponding sampling variance-covariance matrix in a mediation model. In the second step, the sampling distribution of the indirect effect is simulated using estimates from the first step. A confidence interval is constructed from the resulting sampling distribution. A simulation study with various conditions is presented. Implications of the results for applied research are discussed.