Estimating the rate of overdiagnosis with prostate cancer screening: evidence from the Finnish component of the European Randomized Study of Screening for Prostate Cancer.

PURPOSE: Screening for prostate cancer may have limited impact on decreasing prostate cancer-related mortality. A major disadvantage is overdiagnosis, whereby lesions are identified that would not have become evident during the man's lifetime if screening had not taken place. The present study aims to estimate the rate of overdiagnosis using Finnish data from the European randomized trial of prostate cancer screening. METHODS: We used data from 80,149 men randomized to a screening or a control group, distinguishing four birth cohorts. We used the "catch-up method" to identify when the difference in the cumulative incidence of prostate cancer between the screening and control groups had stabilized, implying that the screening has no further effect. We define the overdiagnosis rate to be the relative excess cumulative incidence in the screened group at that point. As an independent method, we also examined the diagnosis rates of T1c tumors as an indicator of early tumors detected by PSA. RESULTS: The estimates of overdiagnosis rates from the catch-up method using the full period of available follow-up ranged between cohorts from 2.3% to 15.4%, and the T1c analysis gave very similar results. CONCLUSION: Some overdiagnosis has occurred, but there is uncertainty about its extent. A long follow-up is required to demonstrate the full impact of screening. We evaluated the overdiagnosis rates at a population level, associated with being offered screening, taking account of contamination (screening among the controls). The overall evaluation of screening should incorporate mortality benefit, cost-effectiveness, and quality of life.

A systematic survey of randomised trials that stopped early for reasons of futility.

BACKGROUND: Randomised trial protocols may incorporate interim analyses, with the potential to stop the study for futility if early data show insufficient promise of a treatment benefit. Previously, we have shown that this approach will theoretically lead to mis-estimation of the treatment effect. We now wished to ascertain the importance of this phenomenon in practice. METHODS: We reviewed the methods and results in a set of trials that had stopped for futility, identified through an extensive literature search. We recorded clinical areas, interventions, study design, outcomes, trial setting, sponsorship, planned and actual treatment effects, sample sizes; power; and if there was a data safety monitoring board, or a published protocol. We identified: if interim analyses were pre-specified, and how many analyses actually occurred; what pre-specified criteria might define futility; if a futility analysis formed the basis for stopping; who made the decision to stop; and the conditional power of each study, i.e. the probability of statistically significant results if the study were to continue to its complete sample size. RESULTS: We identified 52 eligible trials, covering many clinical areas. Most trials had multiple centres, tested drugs, and 40% were industry sponsored. There were 75% where at least one interim analysis was planned a priori; a majority had only one interim analysis, typically with about half the target total sample size. A majority of trials did not pre-define a stopping rule, and a variety of reasons were given for stopping. Few studies calculated and reported low conditional power to justify the early stop. When conditional power could be calculated, it was typically low, especially under the current trend hypothesis. However, under the original design hypothesis, a few studies had relatively high conditional power. Data collection often continued after the interim analysis. CONCLUSIONS: Although other factors will typically be involved, we conclude that, from the perspective of conditional power, stopping early for futility was probably reasonable in most cases, but documentation of the basis for stopping was often missing or vague. Interpretation of truncated trials would be enhanced by improved reporting of stopping protocols, and of their actual execution.

Randomised trials with provision for early stopping for benefit (or harm): The impact on the estimated treatment effect.

Stopping rules for clinical trials are primarily intended to control Type I error rates if interim analyses are planned, but less is known about the impact that potential stopping has on estimating treatment benefit. In this paper, we derive analytic expressions for (1) the over-estimation of benefit in studies that stop early, (2) the under-estimation of benefit in completed studies, and (3) the overall bias in studies with a stopping rule. We also examine the probability of stopping early and the situation in meta-analyses. Numerical evaluations show that the greatest concern is with over-estimation of benefit in stopped studies, especially if the probability of stopping early is small. The overall bias is usually less than 10% of the true benefit, and under-estimation in completed studies is also typically small. The probability of stopping depends on the true treatment effect and sample size. The magnitude of these effects depends on the particular rule adopted, but we show that the maximum overall bias is the same for all stopping rules. We also show that an essentially unbiased meta-analysis estimate of benefit can be recovered, even if some component studies have stopping rules. We illustrate these methods using data from three clinical trials. The results confirm our earlier empirical work on clinical trials. Investigators may consult our numerical results for guidance on potential mis-estimation and bias in the treatment effect if a stopping rule is adopted. Particular concern is warranted in studies that actually stop early, where interim results may be quite misleading.

Quantifying the bias in the estimated treatment effect in randomized trials having interim analyses and a rule for early stopping for futility.

In this paper, we consider the potential bias in the estimated treatment effect obtained from clinical trials, the protocols of which include the possibility of interim analyses and an early termination of the study for reasons of futility. In particular, by considering the conditional power at an interim analysis, we derive analytic expressions for various parameters of interest: (i) the underestimation or overestimation of the treatment effect in studies that stop for futility; (ii) the impact of the interim analyses on the estimation of treatment effect in studies that are completed, i.e. that do not stop for futility; (iii) the overall estimation bias in the estimated treatment effect in a single study with such a stopping rule; and (iv) the probability of stopping at an interim analysis. We evaluate these general expressions numerically for typical trial scenarios. Results show that the parameters of interest depend on a number of factors, including the true underlying treatment effect, the difference that the trial is designed to detect, the study power, the number of planned interim analyses and what assumption is made about future data to be observed after an interim analysis. Because the probability of stopping early is small for many practical situations, the overall bias is often small, but a more serious issue is the potential for substantial underestimation of the treatment effect in studies that actually stop for futility. We also consider these ideas using data from an illustrative trial that did stop for futility at an interim analysis. Copyright © 2017 John Wiley & Sons, Ltd.

Inference on cancer screening exam accuracy using population-level administrative data.

This paper develops a model for cancer screening and cancer incidence data, accommodating the partially unobserved disease status, clustered data structures, general covariate effects, and dependence between exams. The true unobserved cancer and detection status of screening participants are treated as latent variables, and a Markov Chain Monte Carlo algorithm is used to estimate the Bayesian posterior distributions of the diagnostic error rates and disease prevalence. We show how the Bayesian approach can be used to draw inferences about screening exam properties and disease prevalence while allowing for the possibility of conditional dependence between two exams. The techniques are applied to the estimation of the diagnostic accuracy of mammography and clinical breast examination using data from the Ontario Breast Screening Program in Canada.

Effect of dependent errors in the assessment of diagnostic or screening test accuracy when the reference standard is imperfect.

When no gold standard is available to evaluate a diagnostic or screening test, as is often the case, an imperfect reference standard test must be used instead. Furthermore, the errors of the test and its reference standard may not be independent. Some authors have opined that positively dependent errors will lead to overestimation of test performance. Although positive dependence does increase agreement between the test and the reference standard, it is not clear if test accuracy will necessarily be overestimated in this situation, and the case of negatively associated test errors is even less clear. To examine this issue in more detail, we derive the apparent sensitivity, specificity, and overall accuracy of a test relative to an imperfect reference standard and the bias in these parameters. We demonstrate that either positive or negative bias can occur if the reference standard is imperfect. The type and magnitude of bias depend on several components: the disease prevalence, the true test sensitivity and specificity, the covariance between the false-negative test errors among the true disease cases, and the covariance between the false-positive test errors among the true noncases. If, for example, sensitivity and specificity are 0.8 for both the test and reference standard and the errors have a moderate positive dependence, test sensitivity is then underestimated at low prevalence but overestimated at high prevalence, while the opposite occurs for specificity. We illustrate these ideas through general numerical calculations and an empirical example of screening for breast cancer with magnetic resonance imaging and mammography.

Optimal allocation of participants for the estimation of selection, preference and treatment effects in the two-stage randomised trial design.

Outcomes in clinical trials may be affected by the choice of treatment that participants might make, if they were indeed allowed to choose (a so-called selection effect), and by whether they actually receive their preferred treatment (a preference effect). Selection and preference effects can be important, but they cannot be estimated in the conventional trial design. An alternative approach is the two-stage randomised trial, in which participants are first randomly divided into two subgroups. In one subgroup, participants are randomly assigned to treatments, while in the other, participants are allowed to choose their own treatment. This approach yields estimates of the direct treatment effect, and of the preference and selection effects. The latter two provide insight that goes considerably beyond what is possible in the standard randomised trial. In this paper, we determine the optimal proportion of participants who should be allocated to the choice subgroup. The precision of the estimated selection, preference and treatment effects are functions of: the total sample size; the proportion of participants allocated to choose their treatment; the variances of the outcome; the proportions of participants who select each treatment in the choice group; and the selection, preference and treatment effects themselves. We develop general expressions for the optimum proportion of participants in the choice group, depending on which effects are of primary interest. We illustrate the results with trial data comparing alternative clinical management strategies for women with abnormal results on cervical screening.

Relative efficiencies of alternative preference-based designs for randomised trials.

Recent work has shown that outcomes in clinical trials can be affected by which treatment the trial participants would select if they were allowed to do so, and if they do or do not actually receive that treatment. These influences are known as selection and preference effects, respectively. Unfortunately, they cannot be evaluated in conventional, parallel group trials because patient preferences remain unknown. However, several alternative designs have been proposed, to measure and take account of patient preferences. In this paper, we discuss three preference-based designs (the two-stage, fully randomised, and partially randomised designs). In conventional trials, only the treatment effect is estimable, while the preference-based designs have the potential to estimate some or all of the selection and preference effects. The relative efficiency of these designs is affected by several factors, including the proportion of participants who are undecided about treatments, or who are unable or unwilling to state a preference; the relative preference rate between the treatments being compared, among patients who do have a preference; and the ratio of patients randomised to each treatment. We also discuss the advantages and disadvantages of these designs under different scenarios.

Developing common metrics of mechanical exposures across aetiological studies of low back pain in working populations for use in meta-analysis.

OBJECTIVES: One of the challenges of conducting meta-analyses on the relationship between workplace mechanical exposures and low back pain is that mechanical exposures are reported in a wide variety of ways. We aimed to develop common metrics to apply in the translation of literature-based workplace mechanical exposures for use in meta-analyses, and to test the metrics' measurement properties. METHODS: We developed a set of 7-point scales to capture the intensity of important aspects of mechanical exposures that may be related to the development of low back pain in workers. The scales represented three dimensions of mechanical exposures at work: (1) trunk posture, (2) weight lifted or force exerted and (3) spinal loading, and estimated both peak and cumulative loads. Measurement properties of the scales were tested through a survey of experts in biomechanics and ergonomics who were asked to rate literature-based workplace exposure definitions using the scales and provide estimates of their confidence in their ratings. RESULTS: For each dimension the ratings for peak loads tended to be higher than the cumulative load ratings. The inter-rater reliability for the scales ranged from 0.3 to 0.5; we would need to average the ratings of at least four expert raters to have an acceptable level of reliability (>0.7). Inter-expert reliability was positively related to the experts' level of confidence in their ratings. In most cases the ranking of intensity ratings from the experts matched the ranking of exposure intensity from the original articles. CONCLUSIONS: This study provides insight into estimating the intensity of literature-based mechanical exposure metrics using a common set of scales which can be applied across epidemiologic studies. These metrics may be useful to quantify the relationship between workplace mechanical exposure and low back pain in a systematic review and meta-analysis.

Wound irrigation does not affect health-related quality of life after open fractures: results of a randomized controlled trial.

AIMS: The Fluid Lavage in Open Fracture Wounds (FLOW) trial was a multicentre, blinded, randomized controlled trial that used a 2 × 3 factorial design to evaluate the effect of irrigation solution (soap versus normal saline) and irrigation pressure (very low versus low versus high) on health-related quality of life (HRQL) in patients with open fractures. In this study, we used this dataset to ascertain whether these factors affect whether HRQL returns to pre-injury levels at 12-months post-injury. PATIENTS AND METHODS: Participants completed the Short Form-12 (SF-12) and the EuroQol-5 Dimensions (EQ-5D) at baseline (pre-injury recall), at two and six weeks, and at three, six, nine and 12-months post-fracture. We calculated the Physical Component Score (PCS) and the Mental Component Score (MCS) of the SF-12 and the EQ-5D utility score, conducted an analysis using a multi-level generalized linear model, and compared differences between the baseline and 12-month scores. RESULTS: We found no clinically important differences between irrigating solutions or pressures for the SF-12 PCS, SF-12 MCS and EQ-5D. Irrespective of treatment, participants had not returned to their pre-injury function at 12-months for any of the three outcomes (p < 0.001). CONCLUSION: Neither the composition of the irrigation solution nor irrigation pressure applied had an effect on HRQL. Irrespective of treatment, patients had not returned to their pre-injury HRQL at 12 months post-fracture. Cite this article: Bone Joint J 2018;100-B:88-94.

Effect sizes can be calculated for studies reporting ranges for outcome variables in systematic reviews.

OBJECTIVE: To develop a method by which studies reporting the ranges (or maxima and minima) of observed outcomes can be included in systematic reviews, along with other studies reporting in the more usual way by using standard deviations (SDs). STUDY DESIGN AND SETTING: An approach is proposed to allow a numerical conversion of a reported range from a continuous outcome into an equivalent SD. RESULTS: The SD is estimated from the observed range times an appropriate conversion factor. Two examples (the first concerning a patient education program on adherence to drug treatment for rheumatoid arthritis, and the second investigating if cognitive behavior therapy could improve adherence to antiretroviral therapy and then lead to suppression of human immunodeficiency virus replication) demonstrate the calculations. CONCLUSION: This note provides a simple method to allow studies that report outcome variability in terms of ranges to be included in systematic reviews by conversion to equivalent SDs. The method is entirely valid if the outcome variable is normally distributed, but for nonnormal data, some caution may be needed.

Studies reporting ROC curves of diagnostic and prediction data can be incorporated into meta-analyses using corresponding odds ratios.

OBJECTIVE: To develop an approach by which studies describing the accuracy of diagnostic tests or clinical predictions can be combined in a meta-analysis, even though studies may report their results using different summary measures. STUDY DESIGN: A method is proposed to allow algebraic and numerical conversion of values of the Receiver Operating Characteristic Area Under the Curve (AUC) summary statistic into corresponding odds ratios (OR). A similar conversion is demonstrated for the standard errors (SEs) of these summary statistics. RESULTS: The conversion of the AUC values into OR values was achieved using a logit-threshold model. The delta method was used to convert the associated SEs. An example concerning predictions of mortality in the intensive care unit illustrates the calculations. CONCLUSION: This paper provides an accessible method that permits the meta-analyst to overcome some of the difficulties implied by incomplete and inconsistent reporting of research studies in this area. It allows all studies to be included on the same metric, which in turn more easily permits exploration of issues such as heterogeneity. The method can readily be used for meta-analyses of diagnostic or screening tests, or for prediction data.

Estimation, power and sample size calculations for stochastic cost and effectiveness analysis.

Various methods have been proposed to address uncertainty in economic evaluations of healthcare programmes. One approach suggested in the literature is to estimate separate confidence intervals for the incremental costs and effects of a new health programme in comparison with an existing programme. These intervals are then combined to generate a rectangular confidence region in the cost-effectiveness plane that implicitly defines a corresponding confidence interval for the incremental cost-effectiveness ratio (ICER). The same approach has been used to calculate sample sizes and study power. This application of the rectangle method is consistent with the adoption of ICERs and a threshold as a decision rule, this being the most commonly used approach in empirical applications of cost-effectiveness analysis, as well as the one recommended by agencies that assess medical technology around the world. In this paper, we first outline the rectangle method, and then propose a modification that recognises that separate inferences are being drawn on the cost and effectiveness domains, and that corrects for multiple statistical comparisons. The confidence rectangle is otherwise too small, the corresponding confidence interval for the ICER is too narrow and sample sizes are under-estimated. Our modification corrects these problems. A further difficulty is that the placement of the confidence rectangle around the null value is somewhat arbitrary, and does not correspond to a unique value of ICERs. As a result, different values of sample size and power for the estimation of ICERs can be obtained, depending on the null values of the cost and effectiveness. We conclude that it is important to clearly identify the analytic goal in terms of estimating differential costs, differential effects or a combination of the two using the ICER index. These ideas are illustrated using numerical examples.

Beyond the treatment effect: Evaluating the effects of patient preferences in randomised trials.

The treatments under comparison in a randomised trial should ideally have equal value and acceptability - a position of equipoise - to study participants. However, it is unlikely that true equipoise exists in practice, because at least some participants may have preferences for one treatment or the other, for a variety of reasons. These preferences may be related to study outcomes, and hence affect the estimation of the treatment effect. Furthermore, the effects of preferences can sometimes be substantial, and may even be larger than the direct effect of treatment. Preference effects are of interest in their own right, but they cannot be assessed in the standard parallel group design for a randomised trial. In this paper, we describe a model to represent the impact of preferences on trial outcomes, in addition to the usual treatment effect. In particular, we describe how outcomes might differ between participants who would choose one treatment or the other, if they were free to do so. Additionally, we investigate the difference in outcomes depending on whether or not a participant receives his or her preferred treatment, which we characterise through a so-called preference effect. We then discuss several study designs that have been proposed to measure and exploit data on preferences, and which constitute alternatives to the conventional parallel group design. Based on the model framework, we determine which of the various preference effects can or cannot be estimated with each design. We also illustrate these ideas with some examples of preference designs from the literature.

A new preference-based analysis for randomized trials can estimate treatment acceptability and effect in compliant patients.

BACKGROUND AND OBJECTIVES: Development of a new method of analysis to evaluate the acceptability of (or preferences for) the treatments in a randomized trial, and the benefit of treatment among compliers. MATERIALS AND METHODS: We characterize trial participants through the groups who would: accept either treatment if offered (compliers); refuse one treatment but accept the other if it is offered to them (two groups of preferers); or prefer one treatment and insist on it if it is not offered to them initially (two groups of insisters). RESULTS: We show that in our framework, one can always estimate the proportions of patients in these five preference groups. However, constraints are required to estimate the corresponding outcome rates, and thus estimate the treatment effect in the compliers. We propose two possible sets of constraints and illustrate them by numerical examples. CONCLUSIONS: The traditional intention-to-treat analysis avoids biases associated with the alternative per-protocol or as-treated approaches, but it provides imperfect information about the expected treatment effect among patients who are committed to taking the treatment. Many physicians and patients want to know the expected benefit if they adhere to the therapy. Our preference-based analysis provides an estimate of treatment benefit among such patients.

Patients undergoing knee surgery provided accurate ratings of preoperative quality of life and function 2 weeks after surgery.

OBJECTIVE: To evaluate patients' ability to recall their preoperative self-reported quality of life, function, and general health 2 weeks postoperatively. STUDY DESIGN AND SETTING: We randomized consecutive patients undergoing arthroscopic knee surgery to group I (assessments at 4 weeks preoperatively, on the day of surgery, and 2 weeks and 12 months postoperatively) or group II (assessments at 2 weeks and 12 months postoperatively). At each visit patients completed disease-specific, knee-specific, and generic health rating scales. At a median of 2 weeks postoperative (range, 0.6 to 14 weeks), patients completed questionnaires according to their recollection of their health 2 weeks before surgery. RESULTS: Agreement between actual and recalled data was excellent for disease-specific (ICC(WOMET)=0.88 (95% CI 0.82-0.91), ICC(ACL-QOL)=0.86 (95% CI 0.75-0.91)), knee-specific (ICC(IKDC)=0.92 (95% CI 0.90-0.94), ICC(KOOS)=0.93 (95% CI 0.91 to 0.95), and general physical health (ICC(SF-36(PCS))=0.81 (95% CI 0.75-0.86)) instruments. Agreement for general mental health was moderate (ICC(SF-36(MCS))=0.67 (95% CI 0.58-0.75). Greater error associated with recalled ratings contributed to small increases in sample size requirements or small decreases in power to detect differences between groups. CONCLUSION: Patients recalled their preoperative quality of life, function, and general health at 2 weeks postoperative with sufficiently high accuracy to warrant substituting prospectively collected baseline data with recalled ratings.

Estimation of screening sensitivity and sojourn time from an organized screening program.

Regular screening with mammography is widely recommended to reduce breast cancer mortality. However, whether breast screening does more harm than good has long been debated. Since a full evaluation of the effect on mortality could take 10-15 years in order to provide a reliable estimate of the eventual benefits and harms, it is unrealistic to expect each new modification of a screening technique to be evaluated in this way. Therefore, one needs to rapidly estimate suitable measures of the screening effect. In this paper, two measures of interest, the length of the pre-clinical state and the screening false negative rate, are discussed. A procedure is proposed to model the pre-clinical disease state duration, the false negative rate of the screening exam, and the underlying incidence rate in the screened population. We applied the model to data from the Ontario Breast Screening Program in Canada. Our results suggest that the mean preclinical duration is longer than 2 years. We also find only small marginal gains by screening every two instead of three years. The most important objective of a screening program should be to encourage first-time screening attendance.

Pre-trial evaluation of the potential for unblinding in drug trials: a prototype example.

Blinding is an important design feature of randomised trials that may reduce bias in the results, compared to the situation where blinding is not possible or is not maintained. The literature provides some guidance for the evaluation of blinding in ongoing or completed studies, but the question of pre-trial assessment of the potential for unblinding has not been addressed. This paper describes the design and analysis of a prototype experiment for the pre-trial assessment of blinding in a drug trial. This work was motivated by a trial using antibiotic therapy, in which the investigators were concerned about the possibility of subjects being able to differentiate active medication from placebo, and thus become unblinded to their treatment assignment. A small experiment was mounted in which participants had to divide a random mixture of tablets into two groups. Statistical methods were developed to calculate the probability of a given number of similar tablets being classified into the same group by chance, with a modification to allow for some participants having constrained their responses to have equal numbers of tablets in each group. Differentiation of tablets by taste (the initial concern of the investigators) was not statistically different from chance. A smaller set of data on differentiation by appearance (a possibility not originally considered) had borderline statistical significance. After reviewing all these results, the investigators decided to proceed with the study without modifying the tablets, in part because subjects in the study would be unlikely to compare the two types of medication side-by-side. Our results suggest that blinding might sometimes be compromised in unexpected ways. Whenever possible, we suggest that similar and larger such experiments be carried out before the trial to assess whether blinding might be compromised. The methods proposed here could easily be adapted to evaluate the results of such experiments.

The Ontario cohort study of running-related injuries.

A cohort of 1680 runners was enrolled through two community road race events and monitored during a 12-month follow-up period for the occurrence of musculoskeletal injuries. Forty-eight percent of the runners experienced at least one injury, and 54% of these injuries were new; the remainder were recurrences of previous injuries. The risk of injury was associated with increased running mileage but was relatively unassociated with other aspects of training, such as usual pace, usual running surface, hill running, or intense training. Injury rates were equal for all age-sex groups and were independent of years of running experience. Runners injured in the previous year had approximately a 50% higher risk for a new injury during follow-up.

The ecologic method in the study of environmental health. I. Overview of the method.

This paper summarizes the salient features of the ecologic method, with emphasis on its application in the study of environmental health. Various types of ecologic design are described, with examples. Finally, the main advantages and disadvantages are indicated. A companion paper discusses the methodology of ecologic designs in more detail and describes a census of data sets with potential suitability for the ecologic study of water quality and human health.