Is the principle of a stable Heinrich ratio a myth? A multimethod analysis.

BACKGROUND: Safety improvements are sometimes based on the premise that introducing measures to combat minor or no-harm incidents proportionately reduces the incidence of major incidents involving harm. This is in line with the principle of the Heinrich ratio, which asserts that there is a relatively fixed ratio between the incidence of no-harm incidents, minor incidents and major incidents. This principle has been advocated as a means of targeting and evaluating new safety initiatives. RESEARCH METHODOLOGY: Both thought experimentation and analysis of empirical data were used to examine the plausibility of this principle. A descriptive statistical analysis was carried out using triangle plots to display the relative frequencies of the occurrence of safety incidents classified as minor, moderate or severe. FINDINGS: Thought experiments indicated that the principle of a fixed Heinrich ratio has a dubious logical foundation. Analysis of emergency department attendance and studies of medication errors demonstrated marked variation in the relative ratios of different outcomes. Triangle plots of UK road traffic accident data revealed a hitherto unrecognized systematic pattern of change that contradicts the principle of the Heinrich ratio. INTERPRETATION: This study of the principle of a fixed Heinrich ratio invalidates it: introducing measures to reduce the incidence of minor incidents will not inevitably reduce the incidence of major incidents pro rata. Any safety policies based on the assumption that the Heinrich ratio is true need to be rethought.

Dead reckoning: can we trust estimates of mortality rates in clinical databases?

OBJECTIVES: Databases almost invariably contain some errors and improvements to the quality of recorded data are costly. We sought to assess the extent to which given levels of error in a clinical database can lead to misleading mortality rates being derived. METHODS: We deliberately seeded a large database concerning congenital heart surgery involving over 17,600 operations, which we assumed to be error free, with errors at known rates of 0-20%. The effects of three different types of random error were explored: data omission, outcome miscoding (alive or dead) and the miscoding of procedures. For each error type, we compared the mortality rates calculated from the 'seeded' database to those calculated from the pristine database. RESULTS: Outcome miscoding typically results in overestimated mortality rates which for low-risk procedures may well give estimates over double the true value. Random data omission has relatively little effect. If procedure types are miscoded, procedure-specific mortality estimates for high-risk operations tend to be underestimates and those for low-risk operations overestimates. A mathematical model developed to examine these effects accurately forecasted the results of such error-seeding experiments. Software to implement this model is available free of charge on the Internet. CONCLUSION: Even small levels of data error can substantially affect the accuracy of mortality rate estimates, especially for low-risk operations. Such inaccuracy could lead to misleading analysis of institutional and individual surgeons' results. Our results suggest that caution is warranted in interpreting the mortality estimates derived from clinical databases. Our analysis goes beyond the realms of surgical mortality and concerns all adverse events whose frequency is rare.

MADCAP: a graphical method for assessing risk scoring systems.

OBJECTIVE: We set out to develop a method for assessing the performance of clinical risk models over the spectrum of risks and to assess the performance of the EuroSCORE risk model used in cardiac surgery. METHODS: We developed a graphical method for assessing the performance of clinical risk models over the spectrum of risks. To illustrate the technique, we analysed retrospective data concerning 9268 patients that underwent cardiac surgery and for whom both the additive EuroSCORE prediction of risk of morality and vital status at 30 days were available. RESULTS: The graphical tool developed, called MADCAP (Mean Adjusted Deaths Compared Against Predictions), can be used to highlight systematic features of the performance of a clinical risk model. Its use in the current study indicates that the additive version of the EuroSCORE model seems to underestimate risk amongst low-risk cases (0% and 1%). Otherwise the score systematically favours risk avoiding behaviour as the risk model underestimates mortality for 2--6% prediction but not at 7% and above. CONCLUSION: The robustness of case-mix adjusted audit is dependent on the performance of the risk scoring system over the entire spectrum of risk. If we are to use risk adjustment of mortality rates when comparing outcomes obtained by different units or individual surgeons, it is essential that we continually review the performance of the risk adjustment method. The MADCAP method presented here provides a useful tool to this end.

Quality assurance in congenital heart surgery.

OBJECTIVE: The aim of this study was to develop a graphical method of risk-stratified outcome analysis in paediatric cardiac surgery to provide a means of continuous, prospective performance monitoring and allow real-time detection of change in outcomes. METHODS: Risk-adjusted survival following open-heart surgery was prospectively measured over a 15-month period (n=460). Outcomes were charted using variable life-adjusted display (VLAD) charts, which indicate the cumulative difference in observed minus expected survival against the cumulative number of cases performed. Risk stratification was based on RACHS-1 (risk adjustment in congenital heart surgery) risk category and age at surgery, using our previously published risk model. The probability of deviation in performance from the expected baseline level was determined using a mathematical model. RESULTS: By the end of the series, observed survival (443/460=96.3%) exceeded that predicted by the risk model (434.5/460=94.5%), equivalent to a one-third reduction in expected mortality. Mathematical modelling indicated a 1-5% likelihood that this difference would have occurred by random variation alone, suggesting the outcomes represented genuine improvement. CONCLUSIONS: VLAD charts provide an effective, easily visualised display of surgical performance and can be applied to paediatric cardiac surgery. Early detection of change, whether improvement or deterioration, is important for ongoing quality assurance within a cardiac surgery programme.

Monitoring the effectiveness of anticoagulation control.

PURPOSE: To assess the quality of anticoagulation control at an out-patient clinic and to investigate patterns of deviation from therapeutic ranges. DESIGN/METHODOLOGY/APPROACH: Records for 36,157 clinic visits (2050 patients) were studied. The quality of anticoagulation control was assessed by comparing the measurement of pro-thrombin time recorded at each clinic visit, expressed as an international normalised ratio (INR), with the target therapeutic range for that patient, also recorded at the time of the clinic visit. Each INR measurement was classified according to the relevant patient's therapeutic range and the signed difference between the INR measurement and the centre of the therapeutic range was calculated. For each patient the percentage of their INR measurements that lay within their therapeutic range was calculated. FINDINGS: Of the measurements, 52.3 per cent were within the relevant therapeutic range. The proportion of individual patients' INR measurements within range varied greatly (median 52 per cent, inter-quartile range 40-65 per cent). The quality of anticoagulation control, as measured by the proportion of patients within their therapeutic range, changed little with patient follow-up time. ORIGINALITY/VALUE: The quality of anticoagulation control reported is comparable with that at other centres. The vast majority of patients spend periods outside the therapeutic range for their condition. There may be considerable room for improvement.

Delivering stepped care: an analysis of implementation in routine practice.

BACKGROUND: In the United Kingdom, clinical guidelines recommend that services for depression and anxiety should be structured around a stepped care model, where patients receive treatment at different 'steps,' with the intensity of treatment (i.e., the amount and type) increasing at each step if they fail to benefit at previous steps. There are very limited data available on the implementation of this model, particularly on the intensity of psychological treatment at each step. Our objective was to describe patient pathways through stepped care services and the impact of this on patient flow and management. METHODS: We recorded service design features of four National Health Service sites implementing stepped care (e.g., the types of treatments available and their links with other treatments), together with the actual treatments received by individual patients and their transitions between different treatment steps. We computed the proportions of patients accessing, receiving, and transiting between the various steps and mapped these proportions visually to illustrate patient movement. RESULTS: We collected throughput data on 7,698 patients referred. Patient pathways were highly complex and very variable within and between sites. The ratio of low (e.g., self-help) to high-intensity (e.g., cognitive behaviour therapy) treatments delivered varied between sites from 22:1, through 2.1:1, 1.4:1 to 0.5:1. The numbers of patients allocated directly to high-intensity treatment varied from 3% to 45%. Rates of stepping up from low-intensity treatment to high-intensity treatment were less than 10%. CONCLUSIONS: When services attempt to implement the recommendation for stepped care in the National Institute for Health and Clinical Excellence guidelines, there were significant differences in implementation and consequent high levels of variation in patient pathways. Evaluations driven by the principles of implementation science (such as targeted planning, defined implementation strategies, and clear activity specification around service organisation) are required to improve evidence on the most effective, efficient, and acceptable stepped care systems.

A technical note concerning emergency bed demand.

A common method for modelling emergency referrals to a hospital in-patient unit is to assume that they follow a Poisson distribution. If lengths of stay are independent and identically distributed, then the distribution of bed demand can be derived by analysis of a queueing system with an infinite number of servers. Undemanding analysis can be used to express the mean and variance of bed demand in terms of simple formulae involving mean referral rates and the distribution of lengths of stay.

Monitoring risk-adjusted outcomes in congenital heart surgery: does the appropriateness of a risk model change with time?

BACKGROUND: Risk adjustment of outcomes in pediatric congenital heart surgery is challenging due to the great diversity in diagnoses and procedures. We have previously shown that variable life-adjusted display (VLAD) charts provide an effective graphic display of risk-adjusted outcomes in this specialty. A question arises as to whether the risk model used remains appropriate over time. METHODS: We used a recently developed graphic technique to evaluate the performance of an existing risk model among those patients at a single center during 2000 to 2003 originally used in model development. We then compared the distribution of predicted risk among these patients with that among patients in 2004 to 2006. Finally, we constructed a VLAD chart of risk-adjusted outcomes for the latter period. RESULTS: Among 1083 patients between April 2000 and March 2003, the risk model performed well at predicted risks above 3%, underestimated mortality at 2% to 3% predicted risk, and overestimated mortality below 2% predicted risk. There was little difference in the distribution of predicted risk among these patients and among 903 patients between June 2004 and October 2006. Outcomes for the more recent period were appreciably better than those expected according to the risk model. This finding cannot be explained by any apparent bias in the risk model combined with changes in case-mix. CONCLUSIONS: Risk models can, and hopefully do, become out of date. There is scope for complacency in the risk-adjusted audit if the risk model used is not regularly recalibrated to reflect changing standards and expectations.

Challenging the role of calibration, validation and sensitivity analysis in relation to models of health care processes.

Many applications of Operational Research in the context of health care involve processes of calibration, validation and sensitivity analysis. Indeed these processes seem to have such an elevated status that their absence is often regarded as a marker that a study is somehow substandard. Undoubtedly this may be the case, however there may also be circumstances where it is perfectly reasonable not to use such methods. This paper concerns general principles underlying mathematical modelling, particularly in contexts where data for calibration are either poor quality or non-existent. The discussion challenges the view that modelling should necessarily be subject to formulaic calibration, validation and sensitivity analysis processes in an attempt to achieve or establish 'accuracy'. Some models are used purely to deduce the logical consequences of a set of beliefs and in this context, the need for validation is at best questionable. If calibration and sensitivity analysis are to be carried out, there is a need to be clear about what the objective is in such analyses.

Modelling of errors in databases.

A lot of time and energy are expended assembling national databases containing information about health care processes and outcomes. Unfortunately, given the complexity of the data gathering procedures involved, errors occur. This inevitably leads to problems when it comes to the analysis of data from such sources. Indeed, sometimes it is very much a matter of faith that summary statistics represent a true reflection of the facts. On the assumption that one knows the rates at which different forms of errors occur, mathematical modelling methods can be used to obtain estimates of the effects of such errors on the estimates that would be derived for summary statistics associated with an erroneous data base.

Restructuring routine elective services to reduce overall capacity requirements within a local health economy.

The U.K. Government has introduced a new class of health service providers called Treatment Centres that provide routine elective services but that do not deal with emergency cases or non-routine elective patients. The introduction of these centres provides a possible mechanism for improving the efficiency of service delivery in terms of overall capacity requirements. In this paper we discuss a mathematical modelling approach that has been used to examine circumstances under which such benefits might be realised. As an illustration of the analysis, we present results obtained using data concerning urological services, for which there would seem to be benefits associated with the introduction of a TC in only a limited range of circumstances.

Quantifying the impact of non-adherence to drug therapy: a technical note concerning an application of a branch and bound algorithm.

Pharmacokinetic models typically rely on a key assumption that patients take their medication as prescribed, whereas this is often not the case. We present a branch and bound algorithm that can be used to estimate the time-varying probability that, given a specified pattern of nonadherence to a prescribed regimen, a patient receives no therapeutic benefit from treatment. Use of this algorithm is a much faster method for obtaining this probability than exhaustive computation of the relevant probability distribution. The use of this algorithm to assess, in quantitative terms, the impact of non-adherence on the effectiveness of treatment provides a rational basis for evaluating the potential harm to patients.

Evolution of the health economics of cervical cancer vaccination.

This paper reviews the history of modelling for cervical cancer vaccination. We provide an interpretation and summary of conclusions pertaining to the usefulness of different models, the predicted epidemiological impact of vaccination and the cost-effectiveness of adolescent, catch-up and sex-specific vaccination strategies. To date, model results predict a critical role for vaccination in reducing the burden of cervical disease, with cost-effectiveness being consistently shown across studies using a common threshold of US $50,000 per QALY, but further clinical and epidemiological data are required to confirm these findings. Through this paper, we aim to provide useful insights for decision-makers as they examine how to best evaluate the potential impact of vaccines against cervical cancer and determine how to best incorporate vaccination into practice.

A technical note concerning non-adherence to drug therapy: exact expressions for the mean and variance of drug concentration.

There is considerable evidence that prescribed drugs are not taken as intended. We present a stochastic pharmacokinetic mathematical model that can be used to assess the implications of non-adherence to prescribed drug regimens, on the part of either patients or health professionals. In the context of an orally administered drug, explicit equations are derived for the time varying mean and variance of the concentration of the drug in the serum, depending on the probability of each scheduled administration being omitted. The analysis presented here can be used to assess whether a given level of non-adherence is likely to have a clinical impact. The methods used can easily be adapted for use in the context of other routes of drug administration.

Comparison of detailed and succinct cohort modelling approaches in a multi-regional evaluation of cervical cancer vaccination.

Mathematical models have been used extensively in the evaluation of chronic diseases and in exploring the health economics of vaccination. In this study, we examine the value of having two different cohort models based on similar assumptions, one comprehensive and one simplified, which can be used to evaluate the impact of cervical cancer vaccination. To compare models, we ran cost-effectiveness analyses in four geographical regions (Italy, the UK, Taiwan and Canada). We show that the models produce comparable results and therefore can be used independently. However, as they require different complexities of data inputs, they are more suited to different circumstances depending on the level of data inputs available or the complexity of the research question asked.

A consensus process for identifying a prioritised list of study questions.

A consensus process has been developed to assist the UK National Confidential Enquiry into Patient Outcome and Death in identifying questions to be addressed in its studies. The process utilises the knowledge and experience of a panel of experts via a facilitated brainstorming exercise and employs a robust voting system to produce a list of candidate questions ordered in terms of the preferences expressed by individual panel members. The process which is described has been used successfully to assist the design of two national studies and it is anticipated that the process will be adopted for many studies in the future.

Providing feedback to hospital doctors about prescribing errors; a pilot study.

OBJECTIVE: To assess the feasibility and acceptability of obtaining data on prescribing error rates in routine practice, and presenting feedback on such errors to medical staff. SETTING: One clinical directorate of a London teaching trust. METHODS: Ward pharmacists recorded all prescribing errors identified in newly written medication orders on one day each fortnight between February and May 2005. We examined prescribing errors reported on the trust's medication incident database for the same period. MAIN OUTCOME MEASURES: Prescribing errors identified and recorded by ward pharmacists, prescribing errors reported as incident reports; prescribing error rates per clinical specialty; lead consultants' views on receiving feedback on errors for their specialty. RESULTS: During eight data collection days, 4,995 new medication orders were examined. Of these, 462 (9.2%; 95% confidence interval 8.5 -10.1%) contained at least one prescribing error. There were 474 errors in total. Pharmacists indicated that they would have reported 19 (4%) of the prescribing errors as medication incidents. Eight prescribing errors were reported for the entire four-month study period on non-data collection days. Feedback was presented to lead clinicians of 10 clinical specialties. This included graphical summaries showing how the specialty compared with others, and a list of errors identified. This information was well-received by clinicians. CONCLUSION: Prescribing errors identified by ward pharmacists can be systematically fed back at the level of the clinical specialty; this is acceptable to the consultants involved. Incident report data is subject to gross under-reporting. Routinely providing feedback for each consultant team or for individual prescribers will require more focussed data collection.

A stochastic model to evaluate options for antenatal genetic screening.

Haemoglobinopathies are a group of genetic disorders which are particularly prevalent among certain risk groups such as ethnic groups. Antenatal screening of potential haemoglobinopathy carriers allows early diagnosis for affected fetuses, leading to therapeutic intervention or termination. However, it has drawbacks such as screening costs and possible miscarriage as a result of fetal testing. This paper describes a model that allows the outcomes of a screening programme to be estimated for different risk groups. The model has been implemented as a computer package that can be used to inform decisions made by health care planners.

Mathematical methods to assist with hospital operation and planning.

Within health Operational Research, the use of 'computer package' methods such as simulation and system dynamics is becoming so prevalent that it feels somewhat old hat to use analytical methods to develop explicit mathematical formulae or even to explore the mathematical structure of problems. This paper will discuss the use of such 'back of the envelope' analysis illustrating its usefulness. It will be shown that not only does this approach yield considerable insight, but also that it can give rise to powerful and practical solution methods. Examples of this will be discussed in relation to issues such as bed needs estimation, admissions and facilities planning. The author is Director of the Clinical Operational Research Unit (CORU) which was established in 1983, receiving core funding from the UK Department of Health. The concept of a full time university-based research unit dedicated to applying expertise in Operational Research (OR) to problems in health care provides a relatively rare research resource. Yet, the scope for such research, applied to an increasing range of health care activity, is enormous. Issues such as treatment evaluation, performance measures, clinical governance, evidence based medicine and health service delivery are all amenable to OR. Further, OR often provides an immensely cost effective alternative to traditional methods of clinical research based on randomised controlled trials or large scale epidemiological studies. The nature of OR, and one of its main strengths, is that it encompasses a wide range of analytical and scientific methods. Mathematical modeling, statistics, computer-based methods, trial design and analysis all contribute to health OR and, under both of its Directors since 1983, a conscious effort has been made within CORU to foster a diversity of research methodologies. Particular emphasis is put on developing new mathematical methods and computer software. This is somewhat at odds with what seems to be a growing trend in health OR towards researchers specialising in just one or two areas of methodology; thus Tom does queueing theory, Dick does simulation and Harry does System Dynamics. Of course there are exceptions, but for whatever reason, the trend towards specialisation seems real. In this paper, benefits of a more diverse approach to health OR is advocated, particularly the use of 'back of envelope' mathematical methods as an alternative to the use of proprietary software packages. Three case studies are described to illustrate this approach.

Assessing mortality rates from dubious data--when to stop doing statistics and start doing mathematics.

When trying to assess surgical outcomes at a particular centre, it is important to take account of case mix in terms of the types of operation performed. This is because those centres that undertake a disproportionately high number of complex operations might well be expected to have higher mortality rates than other centres whose case mix is more routine. From a statistical viewpoint, such case-mix adjustment is relatively straightforward if there are reliable risk estimates for different operation types. However this may not be the case and the risk estimates may have to be derived from several different sources which may not themselves be in agreement. Here, standard case-mix adjustment methods are no longer applicable and alternative analysis methods need to be used to make use of such unreliable risk estimates.