The effects of communicating uncertainty on public trust in facts and numbers.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the "posttruth" era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims. We conducted five experiments-including one preregistered replication with a national sample and one field experiment on the BBC News website (total n = 5,780)-to examine whether communicating epistemic uncertainty about facts across different topics (e.g., global warming, immigration), formats (verbal vs. numeric), and magnitudes (high vs. low) influences public trust. Results show that whereas people do perceive greater uncertainty when it is communicated, we observed only a small decrease in trust in numbers and trustworthiness of the source, and mostly for verbal uncertainty communication. These results could help reassure all communicators of facts and science that they can be more open and transparent about the limits of human knowledge.

Communicating health risks in science publications: time for everyone to take responsibility.

Research that is poorly communicated or presented is as potentially damaging as research that is poorly conducted or fraudulent. Recent examples illustrate how the problem often lies with researchers, not press officers or journalists. The quest for publication and 'impact' must not outweigh the importance of accurate representation of science; herein, we suggest steps that researchers, journalists and press officers can take to help ensure this.

Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data.

Many meta-analyses combine results from only a small number of studies, a situation in which the between-study variance is imprecisely estimated when standard methods are applied. Bayesian meta-analysis allows incorporation of external evidence on heterogeneity, providing the potential for more robust inference on the effect size of interest. We present a method for performing Bayesian meta-analysis using data augmentation, in which we represent an informative conjugate prior for between-study variance by pseudo data and use meta-regression for estimation. To assist in this, we derive predictive inverse-gamma distributions for the between-study variance expected in future meta-analyses. These may serve as priors for heterogeneity in new meta-analyses. In a simulation study, we compare approximate Bayesian methods using meta-regression and pseudo data against fully Bayesian approaches based on importance sampling techniques and Markov chain Monte Carlo (MCMC). We compare the frequentist properties of these Bayesian methods with those of the commonly used frequentist DerSimonian and Laird procedure. The method is implemented in standard statistical software and provides a less complex alternative to standard MCMC approaches. An importance sampling approach produces almost identical results to standard MCMC approaches, and results obtained through meta-regression and pseudo data are very similar. On average, data augmentation provides closer results to MCMC, if implemented using restricted maximum likelihood estimation rather than DerSimonian and Laird or maximum likelihood estimation. The methods are applied to real datasets, and an extension to network meta-analysis is described. The proposed method facilitates Bayesian meta-analysis in a way that is accessible to applied researchers. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Development of a transparent interactive decision interrogator to facilitate the decision-making process in health care.

BACKGROUND: Decisions about the use of new technologies in health care are often based on complex economic models. Decision makers frequently make informal judgments about evidence, uncertainty, and the assumptions that underpin these models. OBJECTIVES: Transparent interactive decision interrogator (TIDI) facilitates more formal critique of decision models by decision makers such as members of appraisal committees of the National Institute for Health and Clinical Excellence in the UK. By allowing them to run advanced statistical models under different scenarios in real time, TIDI can make the decision process more efficient and transparent, while avoiding limitations on pre-prepared analysis. METHODS: TIDI, programmed in Visual Basic for applications within Excel, provides an interface for controlling all components of a decision model developed in the appropriate software (e.g., meta-analysis in WinBUGS and the decision model in R) by linking software packages using RExcel and R2WinBUGS. TIDI's graphical controls allow the user to modify assumptions and to run the decision model, and results are returned to an Excel spreadsheet. A tool displaying tornado plots helps to evaluate the influence of individual parameters on the model outcomes, and an interactive meta-analysis module allows the user to select any combination of available studies, explore the impact of bias adjustment, and view results using forest plots. We demonstrate TIDI using an example of a decision model in antenatal care. CONCLUSION: Use of TIDI during the NICE appraisal of tumor necrosis factor-alpha inhibitors (in psoriatic arthritis) successfully demonstrated its ability to facilitate critiques of the decision models by decision makers.

The adverse effects of bisphosphonates in breast cancer: A systematic review and network meta-analysis.

BACKGROUND: Bisphosphonate drugs can be used to improve the outcomes of women with breast cancer. Whilst many meta-analyses have quantified their potential benefits for patients, attempts at comprehensive quantification of potential adverse effects have been limited. We undertook a meta-analysis with novel methodology to identify and quantify these adverse effects. METHODS: We systematically reviewed randomised controlled trials in breast cancer where at least one of the treatments was a bisphosphonate (zoledronic acid, ibandronate, pamidronate, alendronate or clodronate). Neoadjuvant, adjuvant and metastatic settings were examined. Primary outcomes were adverse events of any type or severity (excluding death). We carried out pairwise and network meta-analyses to estimate the size of any adverse effects potentially related to bisphosphonates. In order to ascertain whether adverse effects differed by individual factors such as age, or interacted with other common adjuvant breast cancer treatments, we examined individual-level patient data for one large trial, AZURE. FINDINGS: We identified 56 trials that reported adverse data, which included a total of 29,248 patients (18,301 receiving bisphosphonate drugs versus 10,947 not). 24 out of the 103 different adverse outcomes analysed showed a statistically and practically significant increase in patients receiving a bisphosphonate drug compared with those not (2 additional outcomes that appeared statistically significant came only from small studies with low event counts and no clinical suspicion so are likely artifacts). Most of these 24 are already clinically recognised: 'flu-like symptoms, fever, headache and chills; increased bone pain, arthralgia, myalgia, back pain; cardiac events, thromboembolic events; hypocalcaemia and osteonecrosis of the jaw; as well as possibly stiffness and nausea. Oral clodronate appeared to increase the risk of vomiting and diarrhoea (which may also be increased by other bisphosphonates), and there may be some hepatotoxicity. Four additional potential adverse effects emerged for bisphosphonate drugs in this analysis which have not classically be recognised: fatigue, neurosensory problems, hypertonia/muscle spasms and possibly dysgeusia. Several symptoms previously reported as potential side effects in the literature were not significantly increased in this analysis: constipation, insomnia, respiratory problems, oedema or thirst/dry mouth. Individual patient-level data and subgroup analysis revealed little variation in side effects between women of different ages or menopausal status, those with metastatic versus non-metastatic cancer, or between women receiving different concurrent breast cancer therapies. CONCLUSIONS: This meta-analysis has produced estimates for the absolute frequencies of a range of side effects significantly associated with bisphosphonate drugs when used by breast cancer patients. These results show good agreement with previous literature on the subject but are the first systematic quantification of side effects and their severities. However, the analysis is limited by the availability and quality of data on adverse events, and the potential for bias introduced by a lack of standards for reporting of such events. We therefore present a table of adverse effects for bisphosphonates, identified and quantified to the best of our ability from a large number of trials, which we hope can be used to improve the communication of the potential harms of these drugs to patients and their healthcare providers.

Redevelopment of the Predict: Breast Cancer website and recommendations for developing interfaces to support decision-making.

OBJECTIVES: To develop a new interface for the widely used prognostic breast cancer tool: Predict: Breast Cancer. To facilitate decision-making around post-surgery breast cancer treatments. To derive recommendations for communicating the outputs of prognostic models to patients and their clinicians. METHOD: We employed a user-centred design process comprised of background research and iterative testing of prototypes with clinicians and patients. Methods included surveys, focus groups and usability testing. RESULTS: The updated interface now caters to the needs of a wider audience through the addition of new visualisations, instantaneous updating of results, enhanced explanatory information and the addition of new predictors and outputs. A programme of future research was identified and is now underway, including the provision of quantitative data on the adverse effects of adjuvant breast cancer treatments. Based on our user-centred design process, we identify six recommendations for communicating the outputs of prognostic models including the need to contextualise statistics, identify and address gaps in knowledge, and the critical importance of engaging with prospective users when designing communications. CONCLUSIONS: For prognostic algorithms to fulfil their potential to assist with decision-making they need carefully designed interfaces. User-centred design puts patients and clinicians needs at the forefront, allowing them to derive the maximum benefit from prognostic models.

Summarizing historical information on controls in clinical trials.

BACKGROUND: Historical information is always relevant when designing clinical trials, but it might also be incorporated in the analysis. It seems appropriate to exploit past information on comparable control groups. PURPOSE: Phase IV and proof-of-concept trials are used to discuss aspects of summarizing historical control data as prior information in a new trial. The importance of a fair assessment of the similarity of control parameters is emphasized. METHODS: The methodology is meta-analytic-predictive. Heterogeneity of control parameters is expressed via the between-trial variation, which is the key parameter determining the prior effective sample size and its upper bound (prior maximum sample size). RESULTS: For a Phase IV trial (930 control patients in 11 historical trials) between-trial heterogeneity was fairly small, resulting in a prior effective sample size of approximately 90 patients. For a proof-of-concept trial (363 patients in four historical trials) heterogeneity was moderate to substantial, resulting in a prior effective sample size of approximately 20. For another proof-of-concept trial (14 patients in one historical trial), assuming substantial heterogeneity implied a prior effective sample size of 7. The prior effective sample size can only be large if the amount of historical data is large and between-trial heterogeneity is small. The prior effective sample size is bounded by the prior maximum sample size (ratio of within- to between-trial variance), irrespective of the amount of historical data. LIMITATIONS: The meta-analytic-predictive approach assumes exchangeability of control parameters across trials. Due to the difficulty to quantify between-trial variability, sensitivity of conclusions regarding assumptions and type of inference should be assessed. CONCLUSIONS: The use of historical control information is a valuable option and may lead to more efficient clinical trials. The proposed approach is attractive for nonconfirmatory trials, but under certain circumstances extensions to the confirmatory setting could be envisaged as well.

A note on the power prior.

The power prior by Ibrahim and Chen (Statist. Sci. 2000; 15:46-60) is one of several methods to incorporate historical data in the analysis of a clinical trial. The power prior raises the likelihood of the historical data to the power parameter a(0) which quantifies the discounting of the historical data due to heterogeneity between trials. It is shown that the standard method of estimating the power parameter from the historical and current data is inappropriate, and we therefore suggest to use a modified power prior approach or to consider alternative methods instead.

Communicating uncertainty about facts, numbers and science.

Uncertainty is an inherent part of knowledge, and yet in an era of contested expertise, many shy away from openly communicating their uncertainty about what they know, fearful of their audience's reaction. But what effect does communication of such epistemic uncertainty have? Empirical research is widely scattered across many disciplines. This interdisciplinary review structures and summarizes current practice and research across domains, combining a statistical and psychological perspective. This informs a framework for uncertainty communication in which we identify three objects of uncertainty-facts, numbers and science-and two levels of uncertainty: direct and indirect. An examination of current practices provides a scale of nine expressions of direct uncertainty. We discuss attempts to codify indirect uncertainty in terms of quality of the underlying evidence. We review the limited literature about the effects of communicating epistemic uncertainty on cognition, affect, trust and decision-making. While there is some evidence that communicating epistemic uncertainty does not necessarily affect audiences negatively, impact can vary between individuals and communication formats. Case studies in economic statistics and climate change illustrate our framework in action. We conclude with advice to guide both communicators and future researchers in this important but so far rather neglected field.

Use of the false discovery rate when comparing multiple health care providers.

OBJECTIVE: Comparisons of the performance of multiple health care providers are often based on hypothesis tests, those with resulting P-values below some critical threshold being identified as potentially extreme. Because of the multiple testing involved, the classical P-value threshold of, say, 0.05 may not be considered strict enough, as it will tend to lead to too many "false positives." However, we argue that the commonly used Bonferroni-corrected threshold is in general too strict for the problem in hand. The purpose of this article is to demonstrate a suitable alternative thresholding procedure that is already well established in other fields. STUDY DESIGN AND SETTING: The suggested procedure involves control of an error measure called the "false discovery rate" (FDR). We present a worked example involving a comparison of risk-adjusted mortality rates following heart surgery in New York State hospitals during 2000-2002. It is shown that the FDR critical threshold lines can be drawn on a "funnel plot," providing a simple graphical presentation of the results. RESULTS: The FDR procedure identified more providers as potentially extreme than the Bonferroni correction, while maintaining control of an intuitively sensible error measure. CONCLUSION: Control of the FDR offers a simple guideline to determining where to draw critical thresholds when comparing multiple health care providers.

The pros and cons of funnel plots as an aid to risk communication and patient decision making.

Funnel plots, which simultaneously display a sample statistic and the corresponding sample size for multiple cases, have a range of applications. In medicine, they are used to display treatment outcome rates and caseload volume by institution, which can inform strategic decisions about health care delivery. We investigated lay people's understanding of such plots and explored their suitability as an aid to individual treatment decisions. In two studies, 172 participants answered objective questions about funnel plots representing the surgical outcomes (survival or mortality rates) of institutions varying in caseload, and indicated their preferred institutions. Accuracy for extracting objective information was high, unless question phrasing was inconsistent with the plot's survival/mortality framing, or participants had low numeracy levels. Participants integrated caseload-volume and outcome-rate data when forming preferences, but were influenced by reference lines on the plot to make inappropriate discriminations between institutions with similar outcome rates. With careful choice of accompanying language, funnel plots can be readily understood and are therefore a useful tool for communicating risk. However, they are less effective as a decision aid for individual patient's treatment decisions, and we recommend refinements to the standard presentation of the plots if they are to be used for that purpose.

Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction.

BACKGROUND: There has been extensive discussion of the apparent conflict between meta-analyses and a mega-trial investigating the benefits of intravenous magnesium following myocardial infarction, in which the early trial results have been said to be 'too good to be true'. METHODS: We apply Bayesian methods of meta-analysis to the trials available before and after the publication of the ISIS-4 results. We show how scepticism can be formally incorporated into an analysis as a Bayesian prior distribution, and how Bayesian meta-analysis models allow appropriate exploration of hypotheses that the treatment effect depends on the size of the trial or the risk in the control group. RESULTS: Adoption of a sceptical prior would have led early enthusiasm for magnesium to be suitably tempered, but only if combined with a random effects meta-analysis, rather than the fixed effect analysis that was actually conducted. CONCLUSIONS: We argue that neither a fixed effect nor a random effects analysis is appropriate when the mega-trial is included. The Bayesian framework provides many possibilities for flexible exploration of clinical hypotheses, but there can be considerable sensitivity to apparently innocuous assumptions.

Information visualisation for science and policy: engaging users and avoiding bias.

Visualisations and graphics are fundamental to studying complex subject matter. However, beyond acknowledging this value, scientists and science-policy programmes rarely consider how visualisations can enable discovery, create engaging and robust reporting, or support online resources. Producing accessible and unbiased visualisations from complicated, uncertain data requires expertise and knowledge from science, policy, computing, and design. However, visualisation is rarely found in our scientific training, organisations, or collaborations. As new policy programmes develop [e.g., the Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES)], we need information visualisation to permeate increasingly both the work of scientists and science policy. The alternative is increased potential for missed discoveries, miscommunications, and, at worst, creating a bias towards the research that is easiest to display.

Simple and multistate survival curves: can people learn to use them?

UNLABELLED: OBJECTIVE AND SAMPLE: This investigation assessed the comprehension of survival curves in a community sample of 88 young and middle-aged adults when several aspects of good practice for graphical communication were implemented, and it compared comprehension for alternative presentation formats. DESIGN, METHOD, and MEASUREMENTS: After reading worked examples of using survival curves that provided explanation and answers, participants answered questions on survival data for pairs of treatments. Study 1 compared presenting survival curves for both treatments on the same figure against presentation via 2 separate figures. Study 2 compared presenting data for 3 possible outcome states via a single "multistate" figure for each treatment against presenting each outcome on a separate figure (with both treatments on the same figure). Both studies compared alternative forms of questioning (e.g., "number alive" versus "number dead"). Numeracy levels (self-rated and objective measures) were also assessed. RESULTS: Comprehension was generally good--exceeding 90% correct answers on half the questions--and was similar across alternative graphical formats. Lower accuracy was observed for questions requiring a calculation but was significantly lower only when the requirement for calculation was not explicit (13%-28% decrements in performance). In study 1, this effect was most acute for those with lower levels of numeracy. Subjective (self-rated) numeracy and objective (measured) numeracy were both moderate positive predictors of overall task accuracy (r ≈ 0.3). CONCLUSIONS: A high degree of accuracy in extracting information from survival curves is possible, as long as any calculations that are required are made explicit (e.g., finding differences between 2 survival rates). Therefore, practitioners need not avoid using survival curves in discussions with patients, although clear and explicit explanations are important.

Routine antenatal anti-D prophylaxis in women who are Rh(D) negative: meta-analyses adjusted for differences in study design and quality.

BACKGROUND: To estimate the effectiveness of routine antenatal anti-D prophylaxis for preventing sensitisation in pregnant Rhesus negative women, and to explore whether this depends on the treatment regimen adopted. METHODS: Ten studies identified in a previous systematic literature search were included. Potential sources of bias were systematically identified using bias checklists, and their impact and uncertainty were quantified using expert opinion. Study results were adjusted for biases and combined, first in a random-effects meta-analysis and then in a random-effects meta-regression analysis. RESULTS: In a conventional meta-analysis, the pooled odds ratio for sensitisation was estimated as 0.25 (95% CI 0.18, 0.36), comparing routine antenatal anti-D prophylaxis to control, with some heterogeneity (I²â€Š =  19%). However, this naïve analysis ignores substantial differences in study quality and design. After adjusting for these, the pooled odds ratio for sensitisation was estimated as 0.31 (95% CI 0.17, 0.56), with no evidence of heterogeneity (I²  =  0%). A meta-regression analysis was performed, which used the data available from the ten anti-D prophylaxis studies to inform us about the relative effectiveness of three licensed treatments. This gave an 83% probability that a dose of 1250 IU at 28 and 34 weeks is most effective and a 76% probability that a single dose of 1500 IU at 28-30 weeks is least effective. CONCLUSION: There is strong evidence for the effectiveness of routine antenatal anti-D prophylaxis for prevention of sensitisation, in support of the policy of offering routine prophylaxis to all non-sensitised pregnant Rhesus negative women. All three licensed dose regimens are expected to be effective.

A collaboratively-derived science-policy research agenda.

The need for policy makers to understand science and for scientists to understand policy processes is widely recognised. However, the science-policy relationship is sometimes difficult and occasionally dysfunctional; it is also increasingly visible, because it must deal with contentious issues, or itself becomes a matter of public controversy, or both. We suggest that identifying key unanswered questions on the relationship between science and policy will catalyse and focus research in this field. To identify these questions, a collaborative procedure was employed with 52 participants selected to cover a wide range of experience in both science and policy, including people from government, non-governmental organisations, academia and industry. These participants consulted with colleagues and submitted 239 questions. An initial round of voting was followed by a workshop in which 40 of the most important questions were identified by further discussion and voting. The resulting list includes questions about the effectiveness of science-based decision-making structures; the nature and legitimacy of expertise; the consequences of changes such as increasing transparency; choices among different sources of evidence; the implications of new means of characterising and representing uncertainties; and ways in which policy and political processes affect what counts as authoritative evidence. We expect this exercise to identify important theoretical questions and to help improve the mutual understanding and effectiveness of those working at the interface of science and policy.