Reflecting Parameter Uncertainty in Addition to Variability in Constrained Healthcare Resource Discrete Event Simulations: Worth Going the Extra Mile or a Road to Nowhere?

OBJECTIVES: Probabilistic sensitivity analysis (PSA) has been shown to reduce bias in outcomes of health economic models. However, only 1 existing study has been identified that incorporates PSA within a resource-constrained discrete event simulation (DES) model. This article aims to assess whether it is feasible and appropriate to use PSA to characterize parameter uncertainty in DES models that are primarily constructed to explore the impact of constrained resources. METHODS: PSA is incorporated into a new case study of an Emergency Department DES. Structured expert elicitation is used to derive the variability and uncertainty input distributions associated with length of time taken to complete key activities within the Emergency Department. Potential challenges of implementation and analysis are explored. RESULTS: The results of a trial of the model, which used the best estimates of the elicited means and variability around the time taken to complete activities, provided a reasonable fit to the data for length of time within the Emergency Department. However, there was substantial and skewed uncertainty around the activity times estimated from the elicitation exercise. This led to patients taking almost 3 weeks to leave the Emergency Department in some PSA runs, which would not occur in practice. CONCLUSIONS: Structured expert elicitation can be used to derive plausible estimates of activity times and their variability, but experts' uncertainty can be substantial. For parameters that have an impact on interactions within a resource-constrained simulation model, PSA can lead to implausible model outputs; hence, other methods may be needed.

Priorities for research to support local authority action on health and climate change: a study in England.

BACKGROUND: Evidence is needed to support local action to reduce the adverse health impacts of climate change and maximise the health co-benefits of climate action. Focused on England, the study identifies priority areas for research to inform local decision making. METHODS: Firstly, potential priority areas for research were identified from a brief review of UK policy documents, and feedback invited from public and policy stakeholders. This included a survey of Directors of Public Health (DsPH) in England, the local government officers responsible for public health. Secondly, rapid reviews of research evidence examined whether there was UK evidence relating to the priorities identified in the survey. RESULTS: The brief policy review pointed to the importance of evidence in two broad areas: (i) community engagement in local level action on the health impacts of climate change and (ii) the economic (cost) implications of such action. The DsPH survey (n = 57) confirmed these priorities. With respect to community engagement, public understanding of climate change's health impacts and the public acceptability of local climate actions were identified as key evidence gaps. With respect to economic implications, the gaps related to evidence on the health and non-health-related costs and benefits of climate action and the short, medium and longer-term budgetary implications of such action, particularly with respect to investments in the built environment. Across both areas, the need for evidence relating to impacts across income groups was highlighted, a point also emphasised by the public involvement panel. The rapid reviews confirmed these evidence gaps (relating to public understanding, public acceptability, economic evaluation and social inequalities). In addition, public and policy stakeholders pointed to other barriers to action, including financial pressures, noting that better evidence is insufficient to enable effective local action. CONCLUSIONS: There is limited evidence to inform health-centred local action on climate change. More evidence is required on public perspectives on, and the economic dimensions of, local climate action. Investment in locally focused research is urgently needed if local governments are to develop and implement evidence-based policies to protect public health from climate change and maximise the health co-benefits of local action.

Challenges around quantifying uncertainty in a holistic approach to hard-to-heal wound management: Health economic perspective.

Treatment of hard-to-heal wounds involves a holistic approach for choosing between available treatment options. However, evidence for informing these choices is sparse, introducing uncertainty into decisions about the optimum treatment pathways that reflect the vast heterogeneity in this patient population. This paper discusses the existing clinical and health economic literature in order to provide insight into sources of uncertainty in the evaluation of the holistic approach to management of the hard-to-heal wounds, and how this uncertainty can be appropriately reflected in research. We identified three key sources of uncertainty in the evaluation of chronic wound treatments, namely heterogeneity in aetiology and patient populations, heterogeneity in treatment pathways, and challenges around capturing all relevant outcomes. Reflecting these complexities requires sophisticated modelling of treatment sequencing and long-term outcomes. The paper discusses how the scope specification, scenario analyses, and sensitivity analyses can be used to fully characterise analytical uncertainty.

Eliciting uncertainty for complex parameters in model-based economic evaluations: quantifying a temporal change in the treatment effect.

BACKGROUND: In model-based economic evaluations, the effectiveness parameter is often informed by studies with a limited duration of follow-up, requiring extrapolation of the treatment effect over a longer time horizon. Extrapolation from short-term data alone may not adequately capture uncertainty in that extrapolation. This study aimed to use structured expert elicitation to quantify uncertainty associated with extrapolation of the treatment effect observed in a clinical trial. METHODS: A structured expert elicitation exercise was conducted for an applied study of a podiatry intervention designed to reduce the rate of falls and fractures in the elderly. A bespoke web application was used to elicit experts' beliefs about two outcomes (rate of falls and odds of fracture) as probability distributions (priors), for two treatment options (intervention and treatment as usual) at multiple time points. These priors were used to derive the temporal change in the treatment effect of the intervention, to extrapolate outcomes observed in a trial. The results were compared with extrapolation without experts' priors. RESULTS: The study recruited thirty-eight experts (geriatricians, general practitioners, physiotherapists, nurses, and academics) from England and Wales. The majority of experts (32/38) believed that the treatment effect would depreciate over time and expressed greater uncertainty than that extrapolated from a trial-based outcome alone. The between-expert variation in predicted outcomes was relatively small. CONCLUSIONS: This study suggests that uncertainty in extrapolation can be informed using structured expert elicitation methods. Using structured elicitation to attach values to complex parameters requires key assumptions and simplifications to be considered.

Cefiderocol for treating severe aerobic Gram-negative bacterial infections: technology evaluation to inform a novel subscription-style payment model.

Background: To limit the use of antimicrobials without disincentivising the development of novel antimicrobials, there is interest in establishing innovative models that fund antimicrobials based on an evaluation of their value as opposed to the volumes used. The aim of this project was to evaluate the population-level health benefit of cefiderocol in the NHS in England, for the treatment of severe aerobic Gram-negative bacterial infections when used within its licensed indications. The results were used to inform the National Institute for Health and Care Excellence guidance in support of commercial discussions regarding contract value between the manufacturer and NHS England. Methods: The health benefit of cefiderocol was first derived for a series of high-value clinical scenarios. These represented uses that were expected to have a significant impact on patients' mortality risks and health-related quality of life. The clinical effectiveness of cefiderocol relative to its comparators was estimated by synthesising evidence on susceptibility of the pathogens of interest to the antimicrobials in a network meta-analysis. Patient-level costs and health outcomes of cefiderocol under various usage scenarios compared with alternative management strategies were quantified using decision modelling. Results were reported as incremental net health effects expressed in quality-adjusted life-years, which were scaled to 20-year population values using infection number forecasts based on data from Public Health England. The outcomes estimated for the high-value clinical scenarios were extrapolated to other expected uses for cefiderocol. Results: Among Enterobacterales isolates with the metallo-beta-lactamase resistance mechanism, the base-case network meta-analysis found that cefiderocol was associated with a lower susceptibility relative to colistin (odds ratio 0.32, 95% credible intervals 0.04 to 2.47), but the result was not statistically significant. The other treatments were also associated with lower susceptibility than colistin, but the results were not statistically significant. In the metallo-beta-lactamase Pseudomonas aeruginosa base-case network meta-analysis, cefiderocol was associated with a lower susceptibility relative to colistin (odds ratio 0.44, 95% credible intervals 0.03 to 3.94), but the result was not statistically significant. The other treatments were associated with no susceptibility. In the base case, patient-level benefit of cefiderocol was between 0.02 and 0.15 quality-adjusted life-years, depending on the site of infection, the pathogen and the usage scenario. There was a high degree of uncertainty surrounding the benefits of cefiderocol across all subgroups. There was substantial uncertainty in the number of infections that are suitable for treatment with cefiderocol, so population-level results are presented for a range of scenarios for the current infection numbers, the expected increases in infections over time and rates of emergence of resistance. The population-level benefits varied substantially across the base-case scenarios, from 896 to 3559 quality-adjusted life-years over 20 years. Conclusion: This work has provided quantitative estimates of the value of cefiderocol within its areas of expected usage within the NHS. Limitations: Given existing evidence, the estimates of the value of cefiderocol are highly uncertain. Future work: Future evaluations of antimicrobials would benefit from improvements to NHS data linkages; research to support appropriate synthesis of susceptibility studies; and application of routine data and decision modelling to assess enablement value. Study registration: No registration of this study was undertaken. Funding: This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment Policy Research Programme (NIHR award ref: NIHR135591), conducted through the Policy Research Unit in Economic Methods of Evaluation in Health and Social Care Interventions, PR-PRU-1217-20401, and is published in full in Health Technology Assessment; Vol. 28, No. 28. See the NIHR Funding and Awards website for further award information.

This project tested new methods for estimating the value to the NHS of an antimicrobial, cefiderocol, so its manufacturer could be paid fairly even if very little drug is used in order to reduce the risk of bacteria becoming resistant to the product. Clinicians said that the greatest benefit of cefiderocol is when used for complicated urinary tract infections and pneumonia acquired within hospitals caused by two types of bacteria (called Enterobacterales and Pseudomonas aeruginosa), with a resistance mechanism called metallo-beta-lactamase. Because there were no relevant clinical trial data, we estimated how effective cefiderocol and alternative treatments were by doing a systematic literature review of studies that grew bacteria from infections in the laboratory and tested the drugs on them. We linked this to data estimating the long-term health and survival of patients. Some evidence was obtained by asking clinicians detailed questions about what they thought the effects would be based on their experience and the available evidence. We included the side effects of the alternative treatments, some of which can cause kidney damage. We estimated how many infections there would be in the UK, whether they would increase over time and how resistance to treatments may change over time. Clinicians told us that they would also use cefiderocol to treat intra-abdominal and bloodstream infections, and some infections caused by another bacteria called Stenotrophomonas. We estimated how many of these infections there would be, and assumed the same health benefits as for other types of infections. The total value to the NHS was calculated using these estimates. We also considered whether we had missed any additional elements of value. We estimated that the value to the NHS was £18­71 million over 20 years. This reflects the maximum the NHS could pay for use of cefiderocol if the health lost as a result of making these payments rather than funding other NHS services is not to exceed the health benefits of using this antimicrobial. However, these estimates are uncertain due to limitations with the evidence used to produce them and assumptions that had to be made.

Constructing Relative Effect Priors for Research Prioritization and Trial Design: A Meta-epidemiological Analysis.

BACKGROUND: Bayesian methods have potential for efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Furthermore, value of information (VOI) methods estimate the value of reducing decision uncertainty, aiding transparent research prioritization. These methods require a prior distribution describing current uncertainty in key parameters, such as relative treatment effect (RTE). However, at the time of designing and commissioning research, there may be no data to base the prior on. The aim of this article is to present methods to construct priors for RTEs based on a collection of previous RCTs. METHODS: We developed 2 Bayesian hierarchical models that captured variability in RTE between studies within disease area accounting for study characteristics. We illustrate the methods using a data set of 743 published RCTs across 9 disease areas to obtain predictive distributions for RTEs for a range of disease areas. We illustrate how the priors from such an analysis can be used in a VOI analysis for an RCT in bladder cancer and compare the results with those using an uninformative prior. RESULTS: For most disease areas, the predicted RTE favored new interventions over comparators. The predicted effects and uncertainty differed across the 9 disease areas. VOI analysis showed that the expected value of research is much lower with our empirically derived prior compared with an uninformative prior. CONCLUSIONS: This study demonstrates a novel approach to generating informative priors that can be used to aid research prioritization and trial design. The methods can also be used to combine RCT evidence with expert opinion. Further work is needed to create a rich database of RCT evidence that can be used to form off-the-shelf priors. HIGHLIGHTS: Bayesian methods have potential to aid the efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Value-of-information (VOI) methods can be used to aid research prioritization by calculating the value of current decision uncertainty.These methods require a distribution describing current uncertainty in key parameters, that is, "prior distributions."This article demonstrates a methodology to estimate prior distributions for relative treatment effects (odds and hazard ratios) estimated from a collection of previous RCTs.These results may be combined with expert elicitation to facilitate 1) value-of-information methods to prioritize research or 2) Bayesian methods for research design.

Cost Effectiveness of Digital Interventions for Generalised Anxiety Disorder: A Model-Based Analysis.

BACKGROUND: Digital interventions (DIs) are increasingly being used in mental health care, despite limited evidence regarding their value for money. This study aimed to evaluate the cost effectiveness of DIs for generalised anxiety disorder (GAD), in comparison with alternative care options, from the perspective of the UK health care system. METHODS: An open-source decision analytic cohort model was used to extrapolate the results of a network meta-analysis over a patient's lifetime and estimate the costs and outcomes (quality-adjusted life-years) of DIs and their comparators. The net monetary benefit (NMB) and probability of cost effectiveness was estimated for each comparator, and we conducted a Value of Information analysis to evaluate the scale and drivers of uncertainty. RESULTS: DIs were associated with lower NMB compared with medication and with group therapy, but greater NMB compared with non-therapeutic controls and with usual care. DIs that were supported by a clinician, an assistant or a lay person had higher delivery costs than purely patient-self-directed DIs, yielding a greater NMB when opportunity cost was above £3000/QALY. There was considerable uncertainty in the findings driven largely by uncertainty in the estimated treatment effects. The value of further research to establish the effectiveness of DIs for GAD was substantial, at least £12.9 billion. CONCLUSIONS: The high uncertainty about these results does not allow for recommendations based on the cost effectiveness of DIs. However, the analysis highlights areas for future research, and demonstrates that apparent cost savings associated with DIs can be offset by reduced effectiveness.

Reference Case Methods for Expert Elicitation in Health Care Decision Making.

BACKGROUND: The evidence used to inform health care decision making (HCDM) is typically uncertain. In these situations, the experience of experts is essential to help decision makers reach a decision. Structured expert elicitation (referred to as elicitation) is a quantitative process to capture experts' beliefs. There is heterogeneity in the existing elicitation methodology used in HCDM, and it is not clear if existing guidelines are appropriate for use in this context. In this article, we seek to establish reference case methods for elicitation to inform HCDM. METHODS: We collated the methods available for elicitation using reviews and critique. In addition, we conducted controlled experiments to test the accuracy of alternative methods. We determined the suitability of the methods choices for use in HCDM according to a predefined set of principles for elicitation in HCDM, which we have also generated. We determined reference case methods for elicitation in HCDM for health technology assessment (HTA). RESULTS: In almost all methods choices available for elicitation, we found a lack of empirical evidence supporting recommendations. Despite this, it is possible to define reference case methods for HTA. The reference methods include a focus on gathering experts with substantive knowledge of the quantities being elicited as opposed to those trained in probability and statistics, eliciting quantities that the expert might observe directly, and individual elicitation of beliefs, rather than solely consensus methods. It is likely that there are additional considerations for decision makers in health care outside of HTA. CONCLUSIONS: The reference case developed here allows the use of different methods, depending on the decision-making setting. Further applied examples of elicitation methods would be useful. Experimental evidence comparing methods should be generated.

Digital interventions in mental health: evidence syntheses and economic modelling.

BACKGROUND: Economic evaluations provide evidence on whether or not digital interventions offer value for money, based on their costs and outcomes relative to the costs and outcomes of alternatives. OBJECTIVES: (1) Evaluate and summarise published economic studies about digital interventions across different technologies, therapies, comparators and mental health conditions; (2) synthesise clinical evidence about digital interventions for an exemplar mental health condition; (3) construct an economic model for the same exemplar mental health condition using the previously synthesised clinical evidence; and (4) consult with stakeholders about how they understand and assess the value of digital interventions. METHODS: We completed four work packages: (1) a systematic review and quality assessment of economic studies about digital interventions; (2) a systematic review and network meta-analysis of randomised controlled trials on digital interventions for generalised anxiety disorder; (3) an economic model and value-of-information analysis on digital interventions for generalised anxiety disorder; and (4) a series of knowledge exchange face-to-face and digital seminars with stakeholders. RESULTS: In work package 1, we reviewed 76 economic evaluations: 11 economic models and 65 within-trial analyses. Although the results of the studies are not directly comparable because they used different methods, the overall picture suggests that digital interventions are likely to be cost-effective, compared with no intervention and non-therapeutic controls, whereas the value of digital interventions compared with face-to-face therapy or printed manuals is unclear. In work package 2, we carried out two network meta-analyses of 20 randomised controlled trials of digital interventions for generalised anxiety disorder with a total of 2350 participants. The results were used to inform our economic model, but when considered on their own they were inconclusive because of the very wide confidence intervals. In work package 3, our decision-analytic model found that digital interventions for generalised anxiety disorder were associated with lower net monetary benefit than medication and face-to-face therapy, but greater net monetary benefit than non-therapeutic controls and no intervention. Value for money was driven by clinical outcomes rather than by intervention costs, and a value-of-information analysis suggested that uncertainty in the treatment effect had the greatest value (£12.9B). In work package 4, stakeholders identified several areas of benefits and costs of digital interventions that are important to them, including safety, sustainability and reducing waiting times. Four factors may influence their decisions to use digital interventions, other than costs and outcomes: increasing patient choice, reaching underserved populations, enabling continuous care and accepting the 'inevitability of going digital'. LIMITATIONS: There was substantial uncertainty around effect estimates of digital interventions compared with alternatives. This uncertainty was driven by the small number of studies informing most comparisons, the small samples in some of these studies and the studies' high risk of bias. CONCLUSIONS: Digital interventions may offer good value for money as an alternative to 'doing nothing' or 'doing something non-therapeutic' (e.g. monitoring or having a general discussion), but their added value compared with medication, face-to-face therapy and printed manuals is uncertain. Clinical outcomes rather than intervention costs drive 'value for money'. FUTURE WORK: There is a need to develop digital interventions that are more effective, rather than just cheaper, than their alternatives. STUDY REGISTRATION: This study is registered as PROSPERO CRD42018105837. FUNDING: This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 26, No. 1. See the NIHR Journals Library website for further project information.

Digital interventions are activities accessed via technology platforms (e.g. computers, smartphones and virtual reality) that can improve users' mental health and reduce addiction problems. To assess whether or not digital interventions offer 'value for money', we needed to compare their costs and outcomes with the costs and outcomes of alternatives, such as face-to-face therapy and medication. This was done through economic evaluations. This project consisted of four work packages. In work package 1, we reviewed 76 published economic evaluations of digital interventions for different mental health and addiction problems. We could not directly compare their results because of differences in the methods that were used, but the overall picture suggested that digital interventions could offer good value for money as an alternative to 'doing nothing' or simply monitoring someone or giving them general information. The picture was unclear when digital interventions were compared with face-to-face therapy. In work package 2, we pooled research studies that evaluated the outcomes of digital interventions in reducing anxiety and worry; the results were inconclusive because we were uncertain about the differences in outcomes between digital interventions and alternatives. In work package 3, an economic model suggested that value for money in digital interventions is driven by how good they are and not by how much they cost. In work package 4, we presented our methods and results to service users, mental health professionals and researchers who wanted to know more about the value of digital interventions for specific groups (e.g. children and older adults) and for outcomes other than reducing symptoms (e.g. reducing waiting times for treatment and improving attendance for therapy). Finally, the stakeholders highlighted four factors that may influence their decisions to use digital interventions, other than costs and outcomes: increasing choice, reaching underserved populations, enabling continuous care and accepting the 'inevitability of going digital'.

Study protocol: a mixed-methods study to evaluate which health visiting models in England are most promising for mitigating the harms of adverse childhood experiences.

INTRODUCTION: Exposure to adverse childhood experiences (ACEs) is associated with poorer health outcomes throughout life. In England, health visiting is a long-standing, nationally implemented service that aims to prevent and mitigate the impact of adversity in early childhood, including for children exposed to ACEs. A range of health visiting service delivery practices exist across England (from the minimum five recommended contacts to tailored intensive interventions), but there is a lack of evidence on who receives what services, how this varies across local authorities (LAs) and the associated outcomes. METHODS AND ANALYSIS: This study will integrate findings from analysis of individual-level, deidentified administrative data related to hospital admissions (Hospital Episode Statistics (HES)) and health visiting contacts (Community Services Data Set (CSDS)), aggregate LA-level data, in-depth case studies in up to six LAs (including interviews with mothers), a national survey of health visiting services, and workshops with stakeholders and experts by experience. We will use an empirical-to-conceptual approach to develop a typology of health visiting service delivery in England, starting with a data-driven classification generated from latent class analysis of CSDS-HES data, which will be refined based on all other available qualitative and quantitative data. We will then evaluate which models of health visiting are most promising for mitigating the impact of ACEs on child and maternal outcomes using CSDS-HES data for a cohort of children born on 1 April 2015 to 31 March 2019. ETHICS AND DISSEMINATION: The University College London Institute of Education Research Ethics Committee approved this study. Results will be submitted for publication in a peer-reviewed journal and summaries will be provided to key stakeholders including the funders, policy-makers, local commissioners and families.

Economic analysis of the prevalence and clinical and economic burden of medication error in England.

OBJECTIVES: To provide national estimates of the number and clinical and economic burden of medication errors in the National Health Service (NHS) in England. METHODS: We used UK-based prevalence of medication errors (in prescribing, dispensing, administration and monitoring) in primary care, secondary care and care home settings, and associated healthcare resource use, to estimate annual number and burden of errors to the NHS. Burden (healthcare resource use and deaths) was estimated from harm associated with avoidable adverse drug events (ADEs). RESULTS: We estimated that 237 million medication errors occur at some point in the medication process in England annually, 38.4% occurring in primary care; 72% have little/no potential for harm and 66 million are potentially clinically significant. Prescribing in primary care accounts for 34% of all potentially clinically significant errors. Definitely avoidable ADEs are estimated to cost the NHS £98 462 582 per year, consuming 181 626 bed-days, and causing/contributing to 1708 deaths. This comprises primary care ADEs leading to hospital admission (£83.7 million; causing 627 deaths), and secondary care ADEs leading to longer hospital stay (£14.8 million; causing or contributing to 1081 deaths). CONCLUSIONS: Ubiquitous medicines use in health care leads unsurprisingly to high numbers of medication errors, although most are not clinically important. There is significant uncertainty around estimates due to the assumption that avoidable ADEs correspond to medication errors, data quality, and lack of data around longer-term impacts of errors. Data linkage between errors and patient outcomes is essential to progress understanding in this area.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study.

BACKGROUND: Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making. OBJECTIVES: The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed. METHODS: A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken. RESULTS: For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution. LIMITATIONS: The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are. CONCLUSIONS: The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required. FUNDING: This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

BACKGROUND: Decisions in health care aim to maximise health, requiring judgements about treatments. The evidence used to make these judgements is typically uncertain. In these situations, the experience of experts is essential. Structured expert elicitation collects beliefs from experts. There are different guidelines available for structured expert elicitation; however, it is not clear if any of these be can be used in health-care decision-making, for example in considering if a treatment should be made available in the NHS. This project aimed to develop a guidance for structured expert elicitation to inform health-care decision-making. METHODS: Reviews and experimental techniques were used to gather a list of methods to conduct structured expert elicitation. The suitability of these choices in health-care decision-making was then determined by comparing these with a set of standards that support the use of structured expert elicitation in health-care decision-making. RESULTS: Different guidelines prefer different approaches to conduct structured expert elicitation. There is a lack of evidence available to determine which of these methods is most appropriate across the whole of health-care decision-making. It is possible to define reference protocol methods that could be used in a particular type of health-care decision-making, health technology assessment. This includes gathering experts with knowledge of the clinical area, asking experts about things that they observe in clinical practice and asking experts individually for their beliefs. For decision-makers working outside health technology assessment, for example at a local level, or for treatments that are not yet available to patients, these choices may not be appropriate. CONCLUSIONS: This flexibility of this guidance is a useful feature. It is possible for different decision-makers in health care to interpret the reference protocol for their own circumstances.

Digital Interventions for Generalized Anxiety Disorder (GAD): Systematic Review and Network Meta-Analysis.

Background: Generalized anxiety disorder is the most common mental health condition based on weekly prevalence. Digital interventions have been used as alternatives or as supplements to conventional therapies to improve access, patient choice, and clinical outcomes. Little is known about their comparative effectiveness for generalized anxiety disorder. Methods: We conducted a systematic review and network meta-analysis of randomized controlled trials comparing digital interventions with medication, non-digital interventions, non-therapeutic controls, and no intervention. Results: We included 21 randomized controlled trials with a total of 2,350 participants from generalized anxiety disorder populations. Pooled outcomes using analysis of Covariance and rankograms based on the surface under the cumulative ranking curves indicated that antidepressant medication and group therapy had a higher probability than digital interventions of being the "best" intervention. Supported digital interventions were not necessarily "better" than unsupported (pure self-help) ones. Conclusions: Due to very wide confidence intervals, network meta-analysis results were inconclusive as to whether digital interventions are better than no intervention and non-therapeutic active controls, or whether they confer an additional benefit to standard therapy. Future research needs to compare digital interventions with one-to-one therapy and with manualized non-digital self-help and to include antidepressant medication as a treatment comparator and effect modifier.

Systematic Review and Critique of Methods for Economic Evaluation of Digital Mental Health Interventions.

OBJECTIVES: Investment in digital interventions for mental health conditions is growing rapidly, offering the potential to elevate systems that are currently overstretched. Despite a growing literature on economic evaluation of digital mental health interventions (DMHIs), including several systematic reviews, there is no conclusive evidence regarding their cost-effectiveness. This paper reviews the methodology used to determine their cost-effectiveness and assesses whether this meets the requirements for decision-making. In doing so we consider the challenges specific to the economic evaluation of DMHIs, and identify where consensus and possible further research is warranted. METHODS: A systematic review was conducted to identify all economic evaluations of DMHIs published between 1997 and December 2018. The searches included databases of published and unpublished research, reference lists and citations of all included studies, forward citations on all identified protocols and conference abstracts, and contacting authors researchers in the field. The identified studies were critiqued against a published set of requirements for decision-making in healthcare, identifying methodological challenges and areas where consensus is required. RESULTS: The review identified 67 papers evaluating DMHIs. The majority of the evaluations were conducted alongside trials, failing to capture all relevant available evidence and comparators, and long-term impact of mental health disorders. The identified interventions are complex and heterogeneous. As a result, there are a number of challenges specific to their evaluation, including estimation of all costs and outcomes, conditional on analysis viewpoint, and identification of relevant comparators. A taxonomy for DMHIs may be required to inform what interventions can reasonably be pooled and compared. CONCLUSIONS: This study represents the first attempt to understand the appropriateness of the methodologies used to evaluate the value for money of DMHIs, helping work towards consensus and methods' harmonisation on these complex interventions.

Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties.

In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.

Glycemic control in inpatients with diabetes following august changeover of trainee doctors in England.

The first Wednesday of August is the day of changeover of trainee doctors in England. It is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors, mismanagement, and mortality. The aim of this study was to analyze the impact of the August changeover of trainee doctors on inpatient glycemic control in a single English hospital. This is currently unknown in England. Overall, 16,870 patient-day capillary glucose reading measures in 2730 inpatients with diabetes were analyzed for 4 weeks before and after the changeover period for the years 2012, 2013, and 2014. Only inpatients hospitalized for longer than 1 day were included. Contrary to expectations, inpatient glycemic control did not worsen in the first 4 weeks after changeover compared to the preceding 4 weeks before changeover in the 3-year period. This may be due to forethought and planning by the deanery foundation school and the inpatient diabetes team in this hospital.

Markov Model Predicts Changes in STH Prevalence during Control Activities Even with a Reduced Amount of Baseline Information.

BACKGROUND: Estimating the reduction in levels of infection during implementation of soil-transmitted helminth (STH) control programmes is important to measure their performance and to plan interventions. Markov modelling techniques have been used with some success to predict changes in STH prevalence following treatment in Viet Nam. The model is stationary and to date, the prediction has been obtained by calculating the transition probabilities between the different classes of intensity following the first year of drug distribution and assuming that these remain constant in subsequent years. However, to run this model longitudinal parasitological data (including intensity of infection) are required for two consecutive years from at least 200 individuals. Since this amount of data is not often available from STH control programmes, the possible application of the model in control programme is limited. The present study aimed to address this issue by adapting the existing Markov model to allow its application when a more limited amount of data is available and to test the predictive capacities of these simplified models. METHOD: We analysed data from field studies conducted with different combination of three parameters: (i) the frequency of drug administration; (ii) the drug distributed; and (iii) the target treatment population (entire population or school-aged children only). This analysis allowed us to define 10 sets of standard transition probabilities to be used to predict prevalence changes when only baseline data are available (simplified model 1). We also formulated three equations (one for each STH parasite) to calculate the predicted prevalence of the different classes of intensity from the total prevalence. These equations allowed us to design a simplified model (SM2) to obtain predictions when the classes of intensity at baseline were not known. To evaluate the performance of the simplified models, we collected data from the scientific literature on changes in STH prevalence during the implementation of 26 control programmes in 16 countries. Using the baseline data observed, we applied the simplified models and predicted the onward prevalence of STH infection at each time-point for which programme data were available. We then compared the output from the model with the observed data from the programme. RESULTS: The comparison between the model-predicted prevalence and the observed values demonstrated a good accuracy of the predictions. In 77% of cases the original model predicted a prevalence within five absolute percentage points from the observed figure, for the simplified model one in 69% of cases and for the simplified model two in 60% of cases. We consider that the STH Markov model described here could be an important tool for programme managers to monitor the progress of their control programmes and to select the appropriate intervention. We also developed, and made freely available online, a software tool to enable the use of the STH Markov model by personnel with limited knowledge of mathematical models.

Soil-transmitted helminthiasis: the relationship between prevalence and classes of intensity of infection.

BACKGROUND: Recently, WHO has developed a predictive model to evaluate the impact of preventive chemotherapy programs to control the morbidity of soil-transmitted helminths (STHs). To make predictions, this model needs baseline information about the proportion of infections classified as low, moderate and high intensity, for each of the three STH species. However, epidemiological data available are often limited to prevalence estimates. METHODS: We reanalyzed available data from 19 surveys in 10 countries and parameterized the relationship between prevalence of STH infections and the proportion of moderate and heavy intensity infections. RESULTS: The equations derived allow feeding the WHO model with estimates of the proportion of the different classes of infection intensity when only prevalence data is available. CONCLUSIONS: The prediction capacities of the STH model using the equations developed in the present study, should be tested by comparing it with the changes on STH epidemiological data observed in control programs operating for several years.