Revising ethical guidance for the evaluation of programmes and interventions not initiated by researchers.

Public health and service delivery programmes, interventions and policies (collectively, 'programmes') are typically developed and implemented for the primary purpose of effecting change rather than generating knowledge. Nonetheless, evaluations of these programmes may produce valuable learning that helps determine effectiveness and costs as well as informing design and implementation of future programmes. Such studies might be termed 'opportunistic evaluations', since they are responsive to emergent opportunities rather than being studies of interventions that are initiated or designed by researchers. However, current ethical guidance and registration procedures make little allowance for scenarios where researchers have played no role in the development or implementation of a programme, but nevertheless plan to conduct a prospective evaluation. We explore the limitations of the guidance and procedures with respect to opportunistic evaluations, providing a number of examples. We propose that one key missing distinction in current guidance is moral responsibility: researchers can only be held accountable for those aspects of a study over which they have control. We argue that requiring researchers to justify an intervention, programme or policy that would occur regardless of their involvement prevents or hinders research in the public interest without providing any further protections to research participants. We recommend that trial consent and ethics procedures allow for a clear separation of responsibilities for the intervention and the evaluation.

Ranking Hospitals Based on Preventable Hospital Death Rates: A Systematic Review With Implications for Both Direct Measurement and Indirect Measurement Through Standardized Mortality Rates.

Policy Points The use of standardized mortality rates (SMRs) to profile hospitals presumes differences in preventable deaths, and at least one health system has suggested measuring preventable death rates of hospitals for comparison across time or in league tables. The influence of reliability on the optimal review number per case note or hospital for such a program has not been explored. Estimates for preventable death rates using implicit case note reviews by clinicians are quite low, suggesting that SMRs will not work well to rank hospitals, and any misspecification of the risk-adjustment models will produce a high risk of mislabelling outliers. Most studies achieve only fair to moderate reliability of the direct assessment of whether a death is preventable, and thus it is likely that substantial numbers of reviews of deaths would be required to distinguish preventable from nonpreventable deaths as part of learning from individual cases, or for profiling hospitals. Furthermore, population- and hospital system-specific data on the variation in preventable deaths or adverse events across the hospitals and providers to be compared are required in order to design a measurement procedure and the number of reviews needed to distinguish between the patients or hospitals. CONTEXT: There is interest in monitoring avoidable or preventable deaths measured directly or indirectly through standardized mortality rates (SMRs). While there have been numerous studies in recent years on adverse events, including preventable deaths, using implicit case note reviews by clinicians, no systematic reviews have aimed to summarize the estimates or the variations in methodologies used to derive these estimates. We reviewed studies that use implicit case note reviews to estimate the range of preventable death rates observed, the measurement characteristics of those estimates, and the measurement procedures used to generate them. We comment on the implications for monitoring SMRs and illustrate a way to calculate the number of reviews needed to establish a reliable estimate of the preventability of one death or the hospital preventable death rate. METHODS: We conducted a systematic review of the literature supplemented by a reanalysis of authors' previously published and unpublished data and measurement design calculations. We conducted initial searches in PubMed, MEDLINE (OvidSP), and ISI Web of Knowledge in June 2010 and updated them in June 2012 and December 2017. Eligibility criteria included studies of hospital-wide admissions from general and acute medical wards where preventable death rates are provided or can be estimated and that can provide interobserver variations. FINDINGS: Twenty-three studies were included from 1985 to 2017. Recent larger studies suggest consistently low rates of preventable deaths (interquartile range of 3.0%-6.0% since 2008). Reliability of a single review for distinguishing between individual cases with regard to the preventability of death had a Kappa statistic of 0.10-0.50 for deaths and 0.21-0.76 for adverse events. A Kappa of 0.35 would require an average of 8 to 17 reviews of a single case to be precise enough to have confidence in high-stakes decisions to change care procedures or impose sanctions within a hospital as a result. No study estimated the variation in preventable deaths across hospitals, although we were able to reanalyze one study to obtain an estimate. Based on this estimate, 200 to 300 total case note reviews per hospital could be required to reliably distinguish between hospitals. The studies displayed considerable heterogeneity: 13/23 studies defined preventable death with a threshold of greater than or equal to four in a six-category Likert scale and 11/24 involved a two-stage screening process with nurses at the first stage and physicians at the second. Fifteen studies provided expert clinical review support for reviewer disagreements, advice, and quality control. A "generalist/internist" was the modal physician specialty for reviewers and they received one to three days of generic tools orientation and case note review practice. Methods did not consider the influence of human or environmental factors. CONCLUSIONS: The literature provides limited information about the measurement characteristics of preventable deaths, suggesting that substantial numbers of reviews may be needed to create reliable estimates of preventable deaths at the individual or hospital level. Any operational program would require population-specific estimates of reliability. Preventable death rates are low, which is likely to make it difficult to use SMRs based on all deaths to validly profile hospitals. The literature provides little information to guide improvements in the measurement procedures.

An approach to prioritization of medical devices in low-income countries: an example based on the Republic of South Sudan.

BACKGROUND: Efficient and evidence-based medical device and equipment prioritization is of particular importance in low-income countries due to constraints in financing capacity, physical infrastructure and human resource capabilities. METHODS: This paper outlines a medical device prioritization method developed in first instance for the Republic of South Sudan. The simple algorithm offered here is a starting point for procurement and selection of medical devices and can be regarded as a screening test for those that require more labour intensive health economic modelling. CONCLUSIONS: A heuristic method, such as the one presented here, is appropriate for reaching many medical device prioritization decisions in low-income settings. Further investment and purchasing decisions that cannot be reached so simply require more complex health economic modelling approaches.

Peri-operative pulse oximetry in low-income countries: a cost-effectiveness analysis.

OBJECTIVE: To evaluate the cost-effectiveness of pulse oximetry--compared with no peri-operative monitoring--during surgery in low-income countries. METHODS: We considered the use of tabletop and portable, hand-held pulse oximeters among patients of any age undergoing major surgery in low-income countries. From earlier studies we obtained baseline mortality and the effectiveness of pulse oximeters to reduce mortality. We considered the direct costs of purchasing and maintaining pulse oximeters as well as the cost of supplementary oxygen used to treat hypoxic episodes identified by oximetry. Health benefits were measured in disability-adjusted life-years (DALYs) averted and benefits and costs were both discounted at 3% per year. We used recommended cost-effectiveness thresholds--both absolute and relative to gross domestic product (GDP) per capita--to assess if pulse oximetry is a cost-effective health intervention. To test the robustness of our results we performed sensitivity analyses. FINDINGS: In 2013 prices, tabletop and hand-held oximeters were found to have annual costs of 310 and 95 United States dollars (US$), respectively. Assuming the two types of oximeter have identical effectiveness, a single oximeter used for 22 procedures per week averted 0.83 DALYs per annum. The tabletop and hand-held oximeters cost US$ 374 and US$ 115 per DALY averted, respectively. For any country with a GDP per capita above US$ 677 the hand-held oximeter was found to be cost-effective if it prevented just 1.7% of anaesthetic-related deaths or 0.3% of peri-operative mortality. CONCLUSION: Pulse oximetry is a cost-effective intervention for low-income settings.

Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies.

BACKGROUND: This protocol concerns the assessment of cost-effectiveness of hospital health information technology (HIT) in four hospitals. Two of these hospitals are acquiring ePrescribing systems incorporating extensive decision support, while the other two will implement systems incorporating more basic clinical algorithms. Implementation of an ePrescribing system will have diffuse effects over myriad clinical processes, so the protocol has to deal with a large amount of information collected at various 'levels' across the system. METHODS/DESIGN: The method we propose is use of Bayesian ideas as a philosophical guide.Assessment of cost-effectiveness requires a number of parameters in order to measure incremental cost utility or benefit - the effectiveness of the intervention in reducing frequency of preventable adverse events; utilities for these adverse events; costs of HIT systems; and cost consequences of adverse events averted. There is no single end-point that adequately and unproblematically captures the effectiveness of the intervention; we therefore plan to observe changes in error rates and adverse events in four error categories (death, permanent disability, moderate disability, minimal effect). For each category we will elicit and pool subjective probability densities from experts for reductions in adverse events, resulting from deployment of the intervention in a hospital with extensive decision support. The experts will have been briefed with quantitative and qualitative data from the study and external data sources prior to elicitation. Following this, there will be a process of deliberative dialogues so that experts can "re-calibrate" their subjective probability estimates. The consolidated densities assembled from the repeat elicitation exercise will then be used to populate a health economic model, along with salient utilities. The credible limits from these densities can define thresholds for sensitivity analyses. DISCUSSION: The protocol we present here was designed for evaluation of ePrescribing systems. However, the methodology we propose could be used whenever research cannot provide a direct and unbiased measure of comparative effectiveness.

Methodological issues in economic evaluations of emergency transport systems in low-income and middle-income countries.

A recent systematic review identified few papers on the economic evaluation of systems for emergency transport of acutely ill or injured patients. In addition, we found no articles dealing with the methodological challenges posed by such studies in low-income or middle-income countries. We therefore carried out an analysis of issues that are of particular salience to this important topic. This is an intellectual study in which we develop models, identify their limitations, suggest potential extensions to the models and discuss priorities for empirical studies to populate models. First, we develop a general model to calculate changes in survival contingent on the reduced time to treatment that an emergency transport system is designed to achieve. Second, we develop a model to estimate transfer times over an area that will be served by a proposed transfer system. Third, we discuss difficulties in obtaining parameters with which to populate the models. Fourth, we discuss costs, both direct and indirect, of an emergency transfer service. Fifth, we discuss the issue that outcomes other than survival should be considered and that the effects of a service are a weighted sum over all the conditions and severities for which the service caters. Lastly, based on the above work, we identify priorities for research. To our knowledge, this is the first study to identify and frame issues in the health economics of acute transfer systems and to develop models to calculate survival rates from basic parameters, such as time delay/survival relationships, that vary by intervention type and context.

Removal of all ovarian tissue versus conserving ovarian tissue at time of hysterectomy in premenopausal patients with benign disease: study using routine data and data linkage.

OBJECTIVE:  To conduct a nationwide study of associations between removal of all ovarian tissue versus conservation of at least one ovary at the time of hysterectomy and important health outcomes (ischaemic heart disease, cancer, and all cause mortality). STUDY DESIGN AND SETTING:  Retrospective analysis of the English Hospital Episode Statistics database linked to national registers of deprivation indices and of deaths. PARTICIPANTS:  113 679 patients aged 35-45 who had had a hysterectomy for benign conditions between April 2004 and March 2014. EXPOSURES:  Bilateral ovarian removal versus no removal or unilateral ovarian removal (ovarian conservation). MAIN OUTCOME MEASURES:  Hospital admissions for ischaemic heart disease, cancer, or attempted suicide; deaths, overall and from heart disease, cancer, or suicide. Statistical adjustments were made using Cox regression and propensity score matching for potential confounders. RESULTS:  A third of patients had bilateral ovarian removal. Patients in the ovarian conservation group were less likely to be admitted for ischaemic heart disease after hysterectomy than were those in the bilateral removal group (adjusted hazard ratio 0.85, 95% confidence interval 0.77 to 0.93; P=0.001). They were also less likely to have a cancer related post-hysterectomy admission (adjusted hazard ratio 0.83, 0.78 to 0.89; P<0.001). A significant difference in all cause mortality was also seen: 0.60% (456/76 581) of patients with ovarian conservation compared with 1.01% (376/37 098) of patients with bilateral removal. Again, this difference in favour of ovarian conservation was significant (adjusted hazard ratio 0.64, 0.55 to 0.73; P<0.001). Fewer deaths related specifically to heart disease (adjusted hazard ratio 0.50, 0.28 to 0.90; P=0.02) and to cancer (0.54, 0.45 to 0.65; P<0.001) occurred in the ovarian conservation group than in the bilateral removal group. No significant difference between groups was found relating to suicide (attempted or completed). The results after propensity score matching were essentially unchanged. CONCLUSION:  Patients who had ovarian conservation had a significantly lower hazard of all cause mortality compared with those who had bilateral ovarian removal and also had lower death rates from ischaemic heart disease and cancer. Consistent with this observation, admissions to hospital for both ischaemic heart disease and cancer were also lower in the ovarian conservation group than in the bilateral removal group. Although removal of both ovaries protects against subsequent development of ovarian cancer, premenopausal women should be advised that this benefit comes at the cost of an increased risk of cardiovascular disease and of other (more prevalent) cancers and higher overall mortality.

Reviewing deaths in British and US hospitals: a study of two scales for assessing preventability.

BACKGROUND: Standardised mortality ratios do not provide accurate measures of preventable mortality. This has generated interest in using case notes to assess the preventable component of mortality. But, different methods of measurement have not been compared. We compared the reliability of two scales for assessing preventability and the correspondence between them. METHODS: Medical specialists reviewed case notes of patients who had died in hospital, using two instruments: a five-point Likert scale and a continuous (0-100) scale of preventability. To enhance generalisability, we used two different hospital datasets with different types of acute medical patients across different epochs, and in two jurisdictions (UK and USA). We investigated the reliability of measurement and correspondence of preventability estimates across the two scales. Ordinal mixed effects regression methods were used to analyse the Likert scale and to calibrate it against the continuous scale. We report the estimates of the probability a death could have been prevented, accounting for reviewer inconsistency. RESULTS: Correspondence between the two scales was strong; the Likert categories explained most of the variation (76% UK, 73% USA) in the continuous scale. Measurement reliability was low, but similar across the two instruments in each dataset (intraclass correlation: 0.27, UK; 0.23, USA). Adjusting for the inconsistency of reviewer judgements reduced the proportion of cases with high preventability, such that the proportion of all deaths judged probably or definitely preventable on the balance of probability was less than 1%. CONCLUSIONS: The correspondence is high between a Likert and a continuous scale, although the low reliability of both would suggest careful measurement design would be needed to use either scale. Few to no cases are above the threshold when using a balance of probability approach to determining a preventable death, and in any case, there is little evidence supporting anything more than an ordinal correspondence between these reviewer estimates of probability and the true probability. Thus, it would be more defensible to use them as an ordinal measure of the quality of care received by patients who died in the hospital.

Use of traditional medicine in middle-income countries: a WHO-SAGE study.

It is frequently stated in the scientific literature, official reports and the press that 80% of Asian and African populations use traditional medicine (TM) to meet their healthcare needs; however, this statistic was first reported in 1983. This study aimed to update knowledge of the prevalence of TM use and the characteristics of those who access it, to inform health policy-makers as countries seek to fulfil the WHO TM strategy 2014-23 and harness TM for population health. Prevalence of reported use of TM was studied in 35 334 participants of the WHO-SAGE, surveyed 2007-10. TM users were compared with users of modern healthcare in univariate and multivariate analyses. Characteristics examined included age, sex, geography (urban/rural), income quintile, education, self-reported health and presence of specific chronic conditions. This study found TM use was highest in India, 11.7% of people reported that their most frequent source of care during the previous 3 years was TM; 19.0% reported TM use in the previous 12 months. In contrast <3% reported TM as their most frequent source of care in China, Ghana, Mexico, Russia and South Africa; and <2% reported using TM in the previous year in Ghana, Mexico, Russia and South Africa. In univariate analyses, poorer, less educated and rural participants were more likely to be TM-users. In the China multivariate analysis, rurality, poor self-reported health and presence of arthritis were associated with TM use; whereas diagnosed diabetes, hypertension and cataracts were less prevalent in TM users. In Ghana and India, lower income, depression and hypertension were associated with TM use. In conclusion, TM use is less frequent than commonly reported. It may be unnecessary, and perhaps futile, to seek to employ TM for population health needs when populations are increasingly using modern medicine.

Evidence synthesis and decision modelling to support complex decisions: stockpiling neuraminidase inhibitors for pandemic influenza usage.

Objectives: The stockpiling of neuraminidase inhibitor (NAI) antivirals as a defence against pandemic influenza is a significant public health policy decision that must be made despite a lack of conclusive evidence from randomised controlled trials regarding the effectiveness of NAIs on important clinical end points such as mortality. The objective of this study was to determine whether NAIs should be stockpiled for treatment of pandemic influenza on the basis of current evidence. Methods: A decision model for stockpiling was designed. Data on previous pandemic influenza epidemiology was combined with data on the effectiveness of NAIs in reducing mortality obtained from a recent individual participant meta-analysis using observational data. Evidence synthesis techniques and a bias modelling method for observational data were used to incorporate the evidence into the model. The stockpiling decision was modelled for adults (≥16 years old) and the United Kingdom was used as an example. The main outcome was the expected net benefits of stockpiling in monetary terms. Health benefits were estimated from deaths averted through stockpiling. Results: After adjusting for biases in the estimated effectiveness of NAIs, the expected net benefit of stockpiling in the baseline analysis was £444 million, assuming a willingness to pay of £20,000/QALY ($31,000/QALY). The decision would therefore be to stockpile NAIs. There was a greater probability that the stockpile would not be utilised than utilised. However, the rare but catastrophic losses from a severe pandemic justified the decision to stockpile. Conclusions: Taking into account the available epidemiological data and evidence of effectiveness of NAIs in reducing mortality, including potential biases, a decision maker should stockpile anti-influenza medication in keeping with the postulated decision rule.

Scientific hypotheses can be tested by comparing the effects of one treatment over many diseases in a systematic review.

OBJECTIVES: To describe the use of systematic reviews or overviews (systematic reviews of systematic reviews) to synthesize quantitative evidence of intervention effects across multiple indications (multiple-indication reviews) and to highlight issues pertaining to such reviews. STUDY DESIGN AND SETTING: MEDLINE was searched from 2003 to January 2014. We selected multiple-indication reviews of interventions of allopathic medicine that included evidence from randomized controlled trials. We categorized the subject areas evaluated by these reviews and examined their methodology. Utilities and caveats of multiple-indication reviews are illustrated with examples drawn from published literature. RESULTS: We retrieved 52 multiple-indication reviews covering a wide range of interventions. The method has been used to detect unintended effects, improve precision by pooling results across indications, and examine scientific hypotheses across disease classes. CONCLUSION: Systematic reviews of interventions are typically used to evaluate the effects of treatments, one indication at a time. Here, we argue that, with due attention to methodological caveats, much can be learned by comparing the effects of a given treatment across many related indications.

Bayesian cohort and cross-sectional analyses of the PINCER trial: a pharmacist-led intervention to reduce medication errors in primary care.

BACKGROUND: Medication errors are an important source of potentially preventable morbidity and mortality. The PINCER study, a cluster randomised controlled trial, is one of the world's first experimental studies aiming to reduce the risk of such medication related potential for harm in general practice. Bayesian analyses can improve the clinical interpretability of trial findings. METHODS: Experts were asked to complete a questionnaire to elicit opinions of the likely effectiveness of the intervention for the key outcomes of interest--three important primary care medication errors. These were averaged to generate collective prior distributions, which were then combined with trial data to generate bayesian posterior distributions. The trial data were analysed in two ways: firstly replicating the trial reported cohort analysis acknowledging pairing of observations, but excluding non-paired observations; and secondly as cross-sectional data, with no exclusions, but without acknowledgement of the pairing. Frequentist and bayesian analyses were compared. FINDINGS: Bayesian evaluations suggest that the intervention is able to reduce the likelihood of one of the medication errors by about 50 (estimated to be between 20% and 70%). However, for the other two main outcomes considered, the evidence that the intervention is able to reduce the likelihood of prescription errors is less conclusive. CONCLUSIONS: Clinicians are interested in what trial results mean to them, as opposed to what trial results suggest for future experiments. This analysis suggests that the PINCER intervention is strongly effective in reducing the likelihood of one of the important errors; not necessarily effective in reducing the other errors. Depending on the clinical importance of the respective errors, careful consideration should be given before implementation, and refinement targeted at the other errors may be something to consider.

Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study.

Risk-adjustment schemes are used to monitor hospital performance, on the assumption that excess mortality not explained by case mix is largely attributable to suboptimal care. We have developed a model to estimate the proportion of the variation in standardised mortality ratios (SMRs) that can be accounted for by variation in preventable mortality. The model was populated with values from the literature to estimate a predictive value of the SMR in this context-specifically the proportion of those hospitals with SMRs among the highest 2.5% that fall among the worst 2.5% for preventable mortality. The extent to which SMRs reflect preventable mortality rates is highly sensitive to the proportion of deaths that are preventable. If 6% of hospital deaths are preventable (as suggested by the literature), the predictive value of the SMR can be no greater than 9%. This value could rise to 30%, if 15% of deaths are preventable. The model offers a 'reality check' for case mix adjustment schemes designed to isolate the preventable component of any outcome rate.

Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers.

BACKGROUND: We developed a method to estimate the expected cost-effectiveness of a service intervention at the design stage and 'road-tested' the method on an intervention to improve patient handover of care between hospital and community. METHOD: The development of a nine-step evaluation framework: 1. Identification of multiple endpoints and arranging them into manageable groups; 2. Estimation of baseline overall and preventable risk; 3. Bayesian elicitation of expected effectiveness of the planned intervention; 4. Assigning utilities to groups of endpoints; 5. Costing the intervention; 6. Estimating health service costs associated with preventable adverse events; 7. Calculating health benefits; 8. Cost-effectiveness calculation; 9. Sensitivity and headroom analysis. RESULTS: Literature review suggested that adverse events follow 19% of patient discharges, and that one-third are preventable by improved handover (ie, 6.3% of all discharges). The intervention to improve handover would reduce the incidence of adverse events by 21% (ie, from 6.3% to 4.7%) according to the elicitation exercise. Potentially preventable adverse events were classified by severity and duration. Utilities were assigned to each category of adverse event. The costs associated with each category of event were obtained from the literature. The unit cost of the intervention was €16.6, which would yield a Quality Adjusted Life Year (QALY) gain per discharge of 0.010. The resulting cost saving was €14.3 per discharge. The intervention is cost-effective at approximately €214 per QALY under the base case, and remains cost-effective while the effectiveness is greater than 1.6%. CONCLUSIONS: We offer a usable framework to assist in ex ante health economic evaluations of health service interventions.