Using routine intelligence to target inspection of healthcare providers in England.

BACKGROUND: The Healthcare Commission, the national regulator for the National Health Service in England, has to assess providers (NHS trusts) on compliance with core standards in a way that targets appropriate local inspection resources. OBJECTIVES: To develop and evaluate a system for targeting inspections in 2006 of 44 standards in 567 healthcare organisations. METHODS: A wide range of available information was structured as a series of indicators (called items) that mapped to the standards. Each item was scored on a common scale (a modified Z-score), and these scores were aggregated to indicate risks of undeclared non-compliance for all trusts and standards. In addition, local qualitative intelligence was coded and scored. RESULTS: The information sets used comprised 463 875 observations structured in 1689 specific items, drawn from 83 different data streams. Follow-up inspections were undertaken on the 10% of trusts with the highest-risk scores (where the trust had declared compliance with a standard) and an additional 10% of trusts randomly selected from the remainder. The success of the targeting was measured by the number of trust declarations that were "qualified" following inspection. In the risk-based sample, the proportion of inspected standards that were qualified (26%) was significantly higher than in the random sample (13%). The success rate for targeting varied between standards and care sectors. CONCLUSION: This innovative approach to using information to target inspection activity achieved its overall aims. The method worked better for some standards and in some settings than for others, and is being improved in the light of experience gained. Such applications are increasingly important as modern regulators strive to be targeted and proportionate in their activities.

How to interpret your dot: decoding the message of clinical performance indicators.

Comparative performance reports continue to proliferate, so it is increasingly important that healthcare workers can interpret the graphically displayed results correctly. This article acquaints readers with key concepts for thinking clearly and critically about such displays: (1) articulating the question a display answers, along with reflecting on questions the display might appear to, but does not, answer; (2) establishing that provider comparisons are made fairly, ever mindful of methodological assumptions and limitations; (3) accounting for systematic differences among performers that are unexplained by specified predictors, that is, random effect methods that yield 'shrunken' estimates; (4) understanding funnel plots used to summarize complex analyses and how one may vary the interrogative focus so that 'outlier' values most likely signal extraordinary performance. Finally, these concepts are given broader context in a view of the ultimate aim of the evaluative enterprise.

Monitoring the introduction of a surgical intervention with long-term consequences.

Surgical innovations are often introduced for their expected long-term benefits, but the decision to abandon the existing treatment must be based on the available short-term data and rational judgment. We present a framework for monitoring the introduction of a surgical intervention with long-term consequences and failure-time endpoints. The framework is based on Bayesian methods, and formally combines study data, clinical opinion, and external evidence to construct a posterior survival function from which intuitive summary statistics can be extracted to aid decision making. It incorporates learning effects and is adaptable to a wide variety of settings. The methods are illustrated on survival data from a cohort of 325 consecutive neonates treated for simple transposition of the great arteries with either the Senning or the Switch operation during the period 1978-1998.

Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons.

Random effects models are used in many applications in medical statistics, including meta-analysis, cluster randomized trials and comparisons of health care providers. This paper provides a tutorial on the practical implementation of a flexible random effects model based on methodology developed in Bayesian non-parametrics literature, and implemented in freely available software. The approach is applied to the problem of hospital comparisons using routine performance data, and among other benefits provides a diagnostic to detect clusters of providers with unusual results, thus avoiding problems caused by masking in traditional parametric approaches. By providing code for Winbugs we hope that the model can be used by applied statisticians working in a wide variety of applications.

Intracortically distributed neurovascular coupling relationships within and between human somatosensory cortices.

The coupling of neuronal cellular activity to its blood supply is of critical importance to the physiology of the human brain and has been under discussion for more than a century. Linearity in this relationship has been demonstrated in some animal studies, but evidence is lacking in humans. In this study, we compared scalp evoked potentials and the functional magnetic resonance imaging (fMRI) blood oxygen level-dependent (BOLD) signal from healthy human volunteers with changes in the intensity of a somatosensory stimulus. By weighting the fMRI images according to the evoked potential amplitude at corresponding intensities, we tested for positive and negative covariation between these 2 data sets and the extent to which these were linear. Hemodynamic changes in primary somatosensory cortex covaried positively with neuronal activity in a predominantly linear manner, with a small quadratic contribution. Simultaneously, other cortical areas corresponding to the nonstimulated limbs were found to covary negatively and linearly in the hemispheres ipsilateral and contralateral to the stimulus. These concurrent and bilateral cortical dynamics, as well as the intraregional features of this neurovascular coupling, are both more complex than had been considered to date, with considerable implications.

Handling over-dispersion of performance indicators.

OBJECTIVES: A problem can arise when a performance indicator shows substantially more variability than would be expected by chance alone, since ignoring such "over-dispersion" could lead to a large number of institutions being inappropriately classified as "abnormal". A number of options for handling this phenomenon are investigated, ranging from improved risk stratification to fitting a statistical model that robustly estimates the degree of over-dispersion. DESIGN: Retrospective analysis of publicly available data on survival following coronary artery bypass grafts, emergency readmission rates, and teenage pregnancies. SETTING: NHS trusts in England. RESULTS: Funnel plots clearly show the influence of the method chosen for dealing with over-dispersion on the "banding" a trust receives. Both multiplicative and additive approaches are feasible and give intuitively reasonable results, but the additive random effects formulation appears to have a stronger conceptual foundation. CONCLUSION: A random effects model may offer a reasonable solution. This method has now been adopted by the UK Healthcare Commission in their derivation of star ratings.

Issues in data monitoring and interim analysis of trials.

OBJECTIVES: To address issues about data monitoring committees (DMCs) for randomised controlled trials (RCTs). DATA SOURCES: Electronic databases. Handsearching of selected books. Personal contacts with experts in the field. REVIEW METHODS: Systematic literature reviews of DMCs and small group processes in decision-making; sample surveys of: reports of RCTs, recently completed and ongoing RCTs and policies of major organisations involved in RCTs; case studies of four DMCs; and interviews with experienced DMC members. All focused on 23 prestated questions. RESULTS: Although still a minority, RCTs increasingly have DMCs. There is wide agreement that nearly all trials need some form of data monitoring. Central to the role of the DMC is monitoring accumulating evidence related to benefit and toxicity; variation in emphasis has been reflected in the plethora of names. DMCs for trials performed for regulatory purposes should be aware of any special requirements and regulatory consequences. Advantages were identified for both larger and smaller DMCs. There is general agreement that a DMC should be independent and multidisciplinary. Consumer and ethicist membership is controversial. The chair is recognised as being particularly influential, and likely to be most effective if he or she is experienced, understands both statistical and clinical issues, and is facilitating in style and impartial. There is no evidence available to judge suggested approaches to training. The review suggested that costs should be covered, but other rewards must be so minimal as to not affect decision-making. It is usual to have a minimum frequency of DMC meetings, with evidence that face-to-face meetings are preferable. It is common to have open sessions and a closed session. A report to a DMC should cover benefits and risks in a balanced way, summarised in an accessible style, avoiding excessive detail, and be as current as possible. Disadvantages of blinded analyses seem to outweigh advantages. Information about comparable studies should be included, although interaction with the DMCs of similar ongoing trials is controversial. A range of formal statistical approaches can be used, although this is only one of a number of considerations. DMCs usually reach decisions by consensus, but other approaches are sometimes used. The general, but not unanimous, view is that DMCs should be advisory rather than executive on the basis that it is the trial organisers who are ultimately responsible for the conduct of the trial. CONCLUSIONS: Some form of data monitoring should be considered for all RCTs, with reasons given where there is no DMC or when any member is not independent. An early DMC meeting is helpful, determining roles and responsibilities; planned operations can be agreed with investigators and sponsors/funders. A template for a DMC charter is suggested. Competing interests should be declared. DMC size (commonly three to eight people) is chosen to optimise performance. Members are usually independent and drawn from appropriate backgrounds, and some, particularly the chair, are experienced. A minimum frequency of meetings is usually agreed, with flexibility for more if needed. The DMC should understand and agree the statistical approach (and guidelines) chosen, with both the DMC statistician and analysis statistician competent to apply the method. A DMC's primary purpose is to ensure that continuing a trial according to its protocol is ethical, taking account of both individual and collective ethics. A broader remit in respect of wider ethical issues is controversial; arguably, these are primarily the responsibility of research ethics committees, trial steering committees and investigators. The DMC should know the range of recommendations or decisions open to it, in advance. A record should be kept describing the key issues discussed and the rationale for decisions taken. Errors are likely to be reduced if a DMC makes a thorough review of the evidence and has a clear understanding of how it should function, there is active participation by all members, differences are resolved through discussion and there is systematic consideration of the various decision options. DMCs should be encouraged to comment on draft final trial reports. These should include information about the data monitoring process and detail the DMC membership. It is recommended that groups responsible for data monitoring be given the standard name 'Data Monitoring Committee' (DMC). Areas for further research include: widening DMC membership beyond clinicians, trialists and statisticians; initiatives to train DMC members; methods of DMC decision-making; 'open' data monitoring; DMCs covering a portfolio of trials rather than single trials; DMC size and membership, incorporating issues of group dynamics; empirical study of the workings of DMCs and their decision-making, and which trials should or should not have a DMC.

Infant wellbeing at 2 years of age in the Growth Restriction Intervention Trial (GRIT): multicentred randomised controlled trial.

BACKGROUND: Although delivery is widely used for preterm babies failing to thrive in utero, the effect of altering delivery timing has never been assessed in a randomised controlled trial. We aimed to compare the effect of delivering early with delaying birth for as long as possible. METHODS: 548 pregnant women were recruited by 69 hospitals in 13 European countries. Participants had fetal compromise between 24 and 36 weeks, an umbilical-artery doppler waveform recorded, and clinical uncertainty about whether immediate delivery was indicated. Before birth, 588 babies were randomly assigned to immediate delivery (n=296) or delayed delivery until the obstetrician was no longer uncertain (n=292). The main outcome was death or disability at or beyond 2 years of age. Disability was defined as a Griffiths developmental quotient of 70 or less or the presence of motor or perceptual severe disability. Analysis was by intention-to-treat. This trial has been assigned the International Standard Randomised Controlled Trial Number ISRCTN41358726. FINDINGS: Primary outcomes were available on 290 (98%) immediate and 283 (97%) deferred deliveries. Overall rate of death or severe disability at 2 years was 55 (19%) of 290 immediate births, and 44 (16%) of 283 delayed births. With adjustment for gestational age and umbilical-artery doppler category, the odds ratio (95% CrI) was 1.1 (0.7-1.8). Most of the observed difference was in disability in babies younger than 31 weeks of gestation at randomisation: 14 (13%) immediate versus five (5%) delayed deliveries. No important differences in the median Griffiths developmental quotient in survivors was seen. INTERPRETATION: The lack of difference in mortality suggests that obstetricians are delivering sick preterm babies at about the correct moment to minimise mortality. However, they could be delivering too early to minimise brain damage. These results do not lend support to the idea that obstetricians can deliver before terminal hypoxaemia to improve brain development.

Dopaminergic effects on electrophysiological and functional MRI measures of human cortical stimulus-response power laws.

Power laws have been widely used to formulate relationships between objective intensity of stimulation and subjective intensity of sensation. We investigated the effects of dopaminergic drug treatment (sulpiride) on the relationship between somatosensory stimulus intensity and cortical response measured electrophysiologically by somatosensory-evoked potentials (SEP) and functional magnetic resonance imaging (fMRI). The intensity of stimulation was related by a simple power law to both electrophysiological and fMRI measures of cortical response, with overlapping confidence intervals for both power law exponents. Sulpiride did not modulate the power law exponent, but significantly attenuated the "gain" of both stimulus-response functions. Using path analysis we decomposed dopaminergic effects on fMRI data into an indirect component (16%), predictable by drug effects on SEP, and a direct component (84%), not explained electrophysiologically. Results indicate that sulpiride has comparable effects on power law parameters estimated from SEP and fMRI, but fMRI has superior sensitivity to detect drug effects on somatosensory cortical recruitment by graded stimulation.

Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts.

In this paper we discuss the use of charts derived from the sequential probability ratio test (SPRT): the cumulative sum (CUSUM) chart, RSPRT (resetting SPRT), and FIR (fast initial response) CUSUM. The theoretical development of the methods is described and some considerations one might address when designing a chart, explored, including the approximation of average run lengths (ARLs), the importance of detecting improvements in a process as well as detecting deterioration and estimation of the process parameter following a signal. Two examples are used to demonstrate the practical issues of quality control in the medical setting, the first a running example and the second a fully worked example at the end of the paper. The first example relates to 30-day mortality for patients of a single cardiac surgeon over the period 1994-1998, the second to patient deaths in the practice of a single GP, Harold Shipman. The charts' performances relative to each other are shown to be sensitive to the definition of the 'out of control' state of the process being monitored. In light of this, it is stressed that a suitable means by which to compare charts is chosen in any specific application.

Approximate cross-validatory predictive checks in disease mapping models.

When fitting complex hierarchical disease mapping models, it can be important to identify regions that diverge from the assumed model. Since full leave-one-out cross-validatory assessment is extremely time-consuming when using Markov chain Monte Carlo (MCMC) estimation methods, Stern and Cressie consider an importance sampling approximation. We show that this can be improved upon through replication of both random effects and data. Our approach is simple to apply, entirely generic, and may aid the criticism of any Bayesian hierarchical model.

Bayesian random effects meta-analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales.

When conducting a meta-analysis of clinical trials with binary outcomes, a normal approximation for the summary treatment effect measure in each trial is inappropriate in the common situation where some of the trials in the meta-analysis are small, or the observed risks are close to 0 or 1. This problem can be avoided by making direct use of the binomial distribution within trials. A fully Bayesian method has already been developed for random effects meta-analysis on the log-odds scale using the BUGS implementation of Gibbs sampling. In this paper we demonstrate how this method can be extended to perform analyses on both the absolute and relative risk scales. Within each approach we exemplify how trial-level covariates, including underlying risk, can be considered. Data from 46 trials of the effect of single-dose ibuprofen on post-operative pain are analysed and the results contrasted with those derived from classical and Bayesian summary statistic methods. The clinical interpretation of the odds ratio scale is not straightforward. The advantages and flexibility of a fully Bayesian approach to meta-analysis of binary outcome data, considered on an absolute risk or relative risk scale, are now available.

Monitoring of large randomised clinical trials: a new approach with Bayesian methods.

BACKGROUND: In judging whether or not to continue enrolling patients into a randomised clinical trial, most data-monitoring and ethics committees (DMECs) rely on the p value for the difference in effect between the study groups. In the 1990s, two randomised controlled trials-one in patients with lung cancer and one in those with head and neck cancer-were instead monitored by Bayesian methods. We assessed the value of this approach in the monitoring of these clinical trials. METHODS: Before the trials opened, participating clinicians were asked their opinions on the expected difference between the study treatment (continuous hyperfractionated accelerated radiotherapy [CHART]) and conventional radiotherapy. These opinions were used to form an "enthusiastic" and a "sceptical" prior distribution. These prior distributions were combined with the trial data at each of the annual DMEC meetings. If, during monitoring, a result in favour of CHART was seen, the DMEC was to decide whether the results were sufficiently convincing to persuade a sceptic that CHART was worthwhile. Conversely, if there was apparently no or little difference, the DMEC was asked whether they thought the results sufficiently convincing to persuade an enthusiast that CHART was not worthwhile. FINDINGS: At each of the annual meetings, the DMEC concluded that there was insufficient evidence to convert either sceptics or enthusiasts, and that the trials should therefore remain open to recruitment. Neither trial was closed to recruitment earlier than planned. However if a conventional (p-value-based) stopping rule had been used, the lung-cancer trial would probably have been stopped. INTERPRETATION: This Bayesian approach to monitoring is simple to implement and straightforward for members of the DMEC to understand. In our opinion, it is more intuitively appealing than conventional approaches.

Bayesian methods for cluster randomized trials with continuous responses.

Bayesian methods for cluster randomized trials extend the random-effects formulation by allowing both the use of external evidence on parameters and straightforward relaxation of the standard normality and constant variance assumptions. Care is required in specifying prior distributions on variance components, and a number of different options are explored with implied prior distributions for other parameters given in closed form. Markov chain Monte Carlo (MCMC) methods permit the fitting of very general models and the introduction of parameter uncertainty into power calculations. We illustrate these ideas using a published example in which general practices were randomized to intervention or control, and show that different choices of supposedly 'non-informative' prior distributions can have substantial influence on conclusions. We also illustrate the use of forward simulation methods in power calculations with uncertainty on multiple inputs. Bayesian methods have the potential to be very useful but guidance is required as to appropriate strategies for robust analysis. Our current experience leads us to recommend a standard 'non-informative' prior distribution for the within-cluster sampling variance, and an independent prior on the intraclass correlation coefficient (ICC). The latter may exploit background evidence or, as a reference analysis, be a uniform ICC or a 'uniform shrinkage' prior.

Prospective application of Bayesian monitoring and analysis in an "open" randomized clinical trial.

We describe the prospective application of Bayesian monitoring and analysis in an ongoing large multi-centre, randomized trial in which interim results are released to investigators. Substantial variability in prior opinion led us to reject the use of elicited clinical priors for monitoring, in favour of archetypal prior distributions representing reasonable scepticism and enthusiasm. Likelihoods for odds ratios for different covariate values are derived from a logistic regression model, which allows us to incorporate information from prognostic factors without resorting to specialized software. Priors, likelihoods and posterior distributions are regularly reported to both an independent Data Monitoring Committee and the trial investigators.

Bayesian methods in health technology assessment: a review.

BACKGROUND: Bayesian methods may be defined as the explicit quantitative use of external evidence in the design, monitoring, analysis, interpretation and reporting of a health technology assessment. In outline, the methods involve formal combination through the use of Bayes's theorem of: 1. a prior distribution or belief about the value of a quantity of interest (for example, a treatment effect) based on evidence not derived from the study under analysis, with 2. a summary of the information concerning the same quantity available from the data collected in the study (known as the likelihood), to yield 3. an updated or posterior distribution of the quantity of interest. These methods thus directly address the question of how new evidence should change what we currently believe. They extend naturally into making predictions, synthesising evidence from multiple sources, and designing studies: in addition, if we are willing to quantify the value of different consequences as a 'loss function', Bayesian methods extend into a full decision-theoretic approach to study design, monitoring and eventual policy decision-making. Nonetheless, Bayesian methods are a controversial topic in that they may involve the explicit use of subjective judgements in what is conventionally supposed to be a rigorous scientific exercise. OBJECTIVES: This report is intended to provide: 1. a brief review of the essential ideas of Bayesian analysis 2. a full structured review of applications of Bayesian methods to randomised controlled trials, observational studies, and the synthesis of evidence, in a form which should be reasonably straightforward to update 3. a critical commentary on similarities and differences between Bayesian and conventional approaches 4. criteria for assessing the reporting of a Bayesian analysis 5. a comprehensive list of published 'three-star' examples, in which a proper prior distribution has been used for the quantity of primary interest 6. tutorial case studies of a variety of types 7. recommendations on how Bayesian methods and approaches may be assimilated into health technology assessments in a variety of contexts and by a variety of participants in the research process. METHODS: The BIDS ISI database was searched using the terms 'Bayes' or 'Bayesian'. This yielded almost 4000 papers published in the period 1990-98. All resultant abstracts were reviewed for relevance to health technology assessment; about 250 were so identified, and used as the basis for forward and backward searches. In addition EMBASE and MEDLINE databases were searched, along with websites of prominent authors, and available personal collections of references, finally yielding nearly 500 relevant references. A comprehensive review of all references describing use of 'proper' Bayesian methods in health technology assessment (those which update an informative prior distribution through the use of Bayes's theorem) has been attempted, and around 30 such papers are reported in structured form. There has been very limited use of proper Bayesian methods in practice, and relevant studies appear to be relatively easily identified. RESULTS: Bayesian methods in the health technology assessment context 1. Different contexts may demand different statistical approaches. Prior opinions are most valuable when the assessment forms part of a series of similar studies. A decision-theoretic approach may be appropriate where the consequences of a study are reasonably predictable. 2. The prior distribution is important and not unique, and so a range of options should be examined in a sensitivity analysis. Bayesian methods are best seen as a transformation from initial to final opinion, rather than providing a single 'correct' inference. 3. The use of a prior is based on judgement, and hence a degree of subjectivity cannot be avoided. However, subjective priors tend to show predictable biases, and archetypal priors may be useful for identifying a reasonable range of prior opinion.

Use of person-years of CIN III exposure as a surrogate outcome measure in cervical cancer screening trials.

BACKGROUND: The reluctance to perform randomised trials on still unresolved issues of the cervical screening programme is largely due to the view that using invasive cancer as an end-point would require huge and lengthy studies. However, by assuming that the incidence of invasive cancer is related to the number of women with CIN III, an estimate of person-years of CIN III can act as a surrogate outcome. METHODS: By having a reliable model for the development of CIN III and the errors involved in taking the smears and biopsies, the person-years of CIN III can be estimated from a smear and biopsy history. Methods for comparing resulting distributions of person-years of CIN III are discussed. Sensitivity analyses on the error rates of smear and biopsy results, and on the incidence of onset and regression of CIN III are performed. RESULTS: Estimates of person-years of CIN III were calculated for women with mildly abnormal smears in two screening programmes. 11% of women in the Cambridge programme and 21% in the Aberdeen programme were estimated to have been exposed to CIN III for more than 12 months. The greater estimated person-years of CIN III in the Aberdeen study reflects the more conservative treatment policy which was operating there. DISCUSSION: The use of person-years of CIN III as a surrogate outcome can provide a practical and meaningful assessment of strategies for cervical cancer screening. Using CIN III, in place of invasive disease, considerably reduces the study duration and sample size required.