Implementing a sandbox approach in health technology assessment: benefits and recommendations.

OBJECTIVES: The sandbox approach, developed in the financial technologies sector, creates an environment to collaboratively develop and test innovative new products, methods and regulatory approaches, separated from business as usual. It has been used in health care to encourage innovation in response to emerging challenges, but, until recently, has not been used in health technology assessment (HTA). This article summarizes our learnings from using the sandbox approach to address three challenges facing HTA organizations and to identify implications for the use of this approach in HTA. METHODS: We identified three challenging contemporary HTA-related topics to explore in a sandbox environment, away from the pressures and interests of "live" assessments. We convened a pool of 120 stakeholders and experts to participate in various sandbox activities and ultimately co-develop solutions to help HTA organizations respond to the identified challenges. RESULTS: Important general learnings about the potential benefits and implementation of a sandbox approach in HTA were identified. Consequently, we developed recommendations to guide its use, including how to implement an HTA sandbox in an effective way and the types of challenges for which it may be best suited. CONCLUSIONS: For many HTA organizations, it is difficult to carefully consider emerging challenges and innovate their processes due to risks associated with decision errors and resource limitations. The sandbox approach could reduce these barriers. The potential benefits of addressing HTA challenges in a collaborative "safe space" are considerable.

Linking Reimbursement to Patient Benefits for Advanced Therapy Medicinal Products and Other High-Cost Innovations: Policy Recommendations for Outcomes-Based Agreements in Europe.

OBJECTIVES: Health technology assessment (HTA) of advanced therapy medicinal products (ATMPs), such as high-cost and one-time cell and gene therapies, is particularly challenging. Outcomes-based agreements (OBAs) are a potential solution to mitigate the risks while providing access to patients but are not widely used across Europe. This study aimed to develop policy recommendations to support the acceptability and implementation of OBAs in Europe. METHODS: A policy sandbox approach was used to engage with stakeholders and explore how HTA organizations can support reimbursement decisions regarding OBAs for ATMPs. A panel of 38 experts from across the European region was convened in 2 workshops, representing payers, HTA organizations, patients, registries, and an industry trade body. RESULTS: Policy recommendations were developed to support the appropriate consideration of OBAs for reimbursing highly uncertain technologies, such as ATMPs. If a positive HTA recommendation cannot be made at the proposed price, then a simple price discount reflecting the uncertainty is preferred over complex solutions such as OBAs. If an OBA is pursued, it should be designed collaboratively with all stakeholders to understand data collection feasibility and minimize burden to patients and providers. Payers are encouraged to approach OBAs as a tool for informed decision making, including a readiness to make negative reimbursement decisions based on unfavorable evidence. CONCLUSIONS: The study presents a policy framework for using OBAs in reimbursement decisions. OBAs must be carefully designed, focusing on appropriateness and the burden of implementation. The relevant authorities should be committed to making decisions in light of the resulting evidence.

Consolidated Health Economic Evaluation Reporting Standards for Interventions That Use Artificial Intelligence (CHEERS-AI).

OBJECTIVES: Economic evaluations (EEs) are commonly used by decision makers to understand the value of health interventions. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS 2022) provide reporting guidelines for EEs. Healthcare systems will increasingly see new interventions that use artificial intelligence (AI) to perform their function. We developed Consolidated Health Economic Evaluation Reporting Standards for Interventions that use AI (CHEERS-AI) to ensure EEs of AI-based health interventions are reported in a transparent and reproducible manner. METHODS: Potential CHEERS-AI reporting items were informed by 2 published systematic literature reviews of EEs and a contemporary update. A Delphi study was conducted using 3 survey rounds to elicit multidisciplinary expert views on 26 potential items, through a 9-point Likert rating scale and qualitative comments. An online consensus meeting was held to finalize outstanding reporting items. A digital health patient group reviewed the final checklist from a patient perspective. RESULTS: A total of 58 participants responded to survey round 1, 42, and 31 of whom responded to rounds 2 and 3, respectively. Nine participants joined the consensus meeting. Ultimately, 38 reporting items were included in CHEERS-AI. They comprised the 28 original CHEERS 2022 items, plus 10 new AI-specific reporting items. Additionally, 8 of the original CHEERS 2022 items were elaborated on to ensure AI-specific nuance is reported. CONCLUSIONS: CHEERS-AI should be used when reporting an EE of an intervention that uses AI to perform its function. CHEERS-AI will help decision makers and reviewers to understand important AI-specific details of an intervention, and any implications for the EE methods used and cost-effectiveness conclusions.

Diagnostics and treatments of COVID-19: two-year update to a living systematic review of economic evaluations.

Objectives: As the initial crisis of the COVID-19 pandemic recedes, healthcare decision makers are likely to want to make rational evidence-guided choices between the many interventions now available. We sought to update a systematic review to provide an up-to-date summary of the cost-effectiveness evidence regarding tests for SARS-CoV-2 and treatments for COVID-19. Methods: Key databases, including MEDLINE, EconLit and Embase, were searched on 3 July 2023, 2 years on from the first iteration of this review in July 2021. We also examined health technology assessment (HTA) reports and the citations of included studies and reviews. Peer-reviewed studies reporting full health economic evaluations of tests or treatments in English were included. Studies were quality assessed using an established checklist, and those with very serious limitations were excluded. Data from included studies were extracted into predefined tables. Results: The database search identified 8,287 unique records, of which 54 full texts were reviewed, 28 proceeded for quality assessment, and 15 were included. Three further studies were included through HTA sources and citation checking. Of the 18 studies ultimately included, 17 evaluated treatments including corticosteroids, antivirals and immunotherapies. In most studies, the comparator was standard care. Two studies in lower-income settings evaluated the cost effectiveness of rapid antigen tests and critical care provision. There were 17 modelling analyses and 1 trial-based evaluation. Conclusion: A large number of economic evaluations of interventions for COVID-19 have been published since July 2021. Their findings can help decision makers to prioritise between competing interventions, such as the repurposed antivirals and immunotherapies now available to treat COVID-19. However, some evidence gaps remain present, including head-to-head analyses, disease-specific utility values, and consideration of different disease variants. Systematic Review Registration: [https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021272219], identifier [PROSPERO 2021 CRD42021272219].

Economic evaluations of artificial intelligence-based healthcare interventions: a systematic literature review of best practices in their conduct and reporting.

Objectives: Health economic evaluations (HEEs) help healthcare decision makers understand the value of new technologies. Artificial intelligence (AI) is increasingly being used in healthcare interventions. We sought to review the conduct and reporting of published HEEs for AI-based health interventions. Methods: We conducted a systematic literature review with a 15-month search window (April 2021 to June 2022) on 17th June 2022 to identify HEEs of AI health interventions and update a previous review. Records were identified from 3 databases (Medline, Embase, and Cochrane Central). Two reviewers screened papers against predefined study selection criteria. Data were extracted from included studies using prespecified data extraction tables. Included studies were quality assessed using the National Institute for Health and Care Excellence (NICE) checklist. Results were synthesized narratively. Results: A total of 21 studies were included. The most common type of AI intervention was automated image analysis (9/21, 43%) mainly used for screening or diagnosis in general medicine and oncology. Nearly all were cost-utility (10/21, 48%) or cost-effectiveness analyses (8/21, 38%) that took a healthcare system or payer perspective. Decision-analytic models were used in 16/21 (76%) studies, mostly Markov models and decision trees. Three (3/16, 19%) used a short-term decision tree followed by a longer-term Markov component. Thirteen studies (13/21, 62%) reported the AI intervention to be cost effective or dominant. Limitations tended to result from the input data, authorship conflicts of interest, and a lack of transparent reporting, especially regarding the AI nature of the intervention. Conclusion: Published HEEs of AI-based health interventions are rapidly increasing in number. Despite the potentially innovative nature of AI, most have used traditional methods like Markov models or decision trees. Most attempted to assess the impact on quality of life to present the cost per QALY gained. However, studies have not been comprehensively reported. Specific reporting standards for the economic evaluation of AI interventions would help improve transparency and promote their usefulness for decision making. This is fundamental for reimbursement decisions, which in turn will generate the necessary data to develop flexible models better suited to capturing the potentially dynamic nature of AI interventions.

Prognostic Factors of COVID-19: An Umbrella Review Endorsed by the International Society for Pharmacoepidemiology.

During the coronavirus disease 2019 (COVID-19) pandemic, the urgency for updated evidence to inform public health and clinical care placed systematic literature reviews (SLRs) at the cornerstone of research. We aimed to summarize evidence on prognostic factors for COVID-19 outcomes through published SLRs and to critically assess quality elements in the findings' interpretation. An umbrella review was conducted via electronic databases from January 2020 to April 2022. All SLRs (and meta-analyses) in English were considered. Data screening and extraction were conducted by two independent reviewers. AMSTAR 2 tool was used to assess SLR quality. The study was registered with PROSPERO (CRD4202232576). Out of 4,564 publications, 171 SLRs were included of which 3 were umbrella reviews. Our primary analysis included 35 SLRs published in 2022, which incorporated studies since the beginning of the pandemic. Consistent findings showed that, for adults, older age, obesity, heart disease, diabetes, and cancer were more strongly predictive of risk of hospitalization, intensive care unit admission, and mortality due to COVID-19. Male sex was associated with higher risk of short-term adverse outcomes, but female sex was associated with higher risk of long COVID. For children, socioeconomic determinants that may unravel COVID-19 disparities were rarely reported. This review highlights key prognostic factors of COVID-19, which can help clinicians and health officers identify high-risk groups for optimal care. Findings can also help optimize confounding adjustment and patient phenotyping in comparative effectiveness research. A living SLR approach may facilitate dissemination of new findings. This paper is endorsed by the International Society for Pharmacoepidemiology.

Health technology assessment of tests for SARS-CoV-2 and treatments for COVID-19: A proposed approach and best-practice recommendations.

OBJECTIVES: To develop best-practice guidance for health technology assessment (HTA) agencies when appraising diagnostic tests for SARS-CoV-2 and treatments for COVID-19. METHODS: We used a policy sandbox approach to develop best-practice guidance for HTA agencies to approach known challenges associated with assessing tests and treatments for COVID-19. The guidance was developed by a multi-stakeholder workshop of twenty-one participants representing HTA agencies, clinical and patient experts, academia, industry, and a payer, from across Europe and North America. The workshop was supported by extensive background work to identify the key challenges, including: targeted reviews of existing COVID-related methods guidance for assessing interventions and clinical guidelines, engagement with clinical experts, a survey and workshop of HTA agencies, a systematic review of published economic evaluations, and a workshop of health economic modelers. RESULTS: We suggest HTA agencies should consider using other types of evidence (e.g., real world) where high-quality randomized controlled trials may be lacking and healthcare systems would value timely HTA outputs. A "living" HTA approach may be useful, given the context of an evolving disease, scientific understanding and evidence base, allowing for decisions to be efficiently revisited in response to new information; particularly, if supported by a common "disease model" for COVID-19. Innovative ways of engaging with the public and clinicians, and early engagement with regulators and payers, are recommended. CONCLUSIONS: HTA agencies should consider the elements of this guidance that are most suited to their existing processes to enable them to assess the effectiveness and value of interventions for COVID-19.

Living Health Technology Assessment: Issues, Challenges and Opportunities.

Health technology assessments (HTAs) are typically performed as one-off evaluations and can potentially become out-of-date due to the availability of new data, new comparators, or other factors. Recently, living approaches have been applied to systematic reviews and network meta-analyses to enable evidence syntheses to be updated more easily. In this paper, we provide a definition for 'Living HTA' where such a living approach could be applied to the entire HTA process. Living HTA could involve performing regular or scheduled updates using a traditional manual approach, or indeed in a semi-automated manner leveraging recent technological innovations that automate parts of the HTA process. The practical implementation of living HTA using both approaches (i.e., manual approach and using semi-automation) is described along with the likely issues and challenges with planning and implementing a living HTA process. The time, resources and additional considerations outlined may prohibit living HTA from becoming the norm for every evaluation; however, scenarios where living HTA would be particularly beneficial are discussed.

Advancing the use of real world evidence in health technology assessment: insights from a multi-stakeholder workshop.

Introduction: Real-world evidence (RWE) in health technology assessment (HTA) holds significant potential for informing healthcare decision-making. A multistakeholder workshop was organised by the European Health Data and Evidence Network (EHDEN) and the GetReal Institute to explore the status, challenges, and opportunities in incorporating RWE into HTA, with a focus on learning from regulatory initiatives such as the European Medicines Agency (EMA) Data Analysis and Real World Interrogation Network (DARWIN EU®). Methods: The workshop gathered key stakeholders from regulatory agencies, HTA organizations, academia, and industry for three panel discussions on RWE and HTA integration. Insights and recommendations were collected through panel discussions and audience polls. The workshop outcomes were reviewed by authors to identify key themes, challenges, and recommendations. Results: The workshop discussions revealed several important findings relating to the use of RWE in HTA. Compared with regulatory processes, its adoption in HTA to date has been slow. Barriers include limited trust in RWE, data quality concerns, and uncertainty about best practices. Facilitators include multidisciplinary training, educational initiatives, and stakeholder collaboration, which could be facilitated by initiatives like EHDEN and the GetReal Institute. Demonstrating the impact of "driver projects" could promote RWE adoption in HTA. Conclusion: To enhance the integration of RWE in HTA, it is crucial to address known barriers through comprehensive training, stakeholder collaboration, and impactful exemplar research projects. By upskilling users and beneficiaries of RWE and those that generate it, promoting collaboration, and conducting "driver projects," can strengthen the HTA evidence base for more informed healthcare decisions.

Outcome-based reimbursement in Central-Eastern Europe and Middle-East.

Outcome-based reimbursement models can effectively reduce the financial risk to health care payers in cases when there is important uncertainty or heterogeneity regarding the clinical value of health technologies. Still, health care payers in lower income countries rely mainly on financial based agreements to manage uncertainties associated with new therapies. We performed a survey, an exploratory literature review and an iterative brainstorming in parallel about potential barriers and solutions to outcome-based agreements in Central and Eastern Europe (CEE) and in the Middle East (ME). A draft list of recommendations deriving from these steps was validated in a follow-up workshop with payer experts from these regions. 20 different barriers were identified in five groups, including transaction costs and administrative burden, measurement issues, information technology and data infrastructure, governance, and perverse policy outcomes. Though implementing outcome-based reimbursement models is challenging, especially in lower income countries, those challenges can be mitigated by conducting pilot agreements and preparing for predictable barriers. Our guidance paper provides an initial step in this process. The generalizability of our recommendations can be improved by monitoring experiences from pilot reimbursement models in CEE and ME countries and continuing the multistakeholder dialogue at national levels.

Variations in comorbidity burden in people with type 2 diabetes over disease duration: A population-based analysis of real world evidence.

Background: The prevalence of type 2 diabetes (T2DM) is increasing, but increasing longevity among persons with diagnosed diabetes may be is associated with more extensive and diverse types of morbidity. The extent and breadth of morbidity and how this varies across sub-groups is unclear and could have important clinical and public health implications. We aimed to estimate comorbidity profiles in people with T2DM and variations across sub-groups and over time. Methods: We identified approximately 224,000 people with T2DM in the Discover-NOW dataset, a real-world primary care database from 2000 to 2020 covering 2.5 million people across North-West London, England, linked to hospital records. We generated a mixed prevalence and incidence study population through repeated annual cross sections, and included a broad set of 35 comorbidities covering traditional T2DM conditions, emerging T2DM conditions and other common conditions.We estimated annual age-standardised prevalence of comorbidities, over the course of the disease in people with T2DM and several sub-groups. Findings: Multimorbidity (two or more chronic conditions) is common in people with T2DM and increasing, but the comorbidity profiles of people with T2DM vary substantially. Nearly 30% of T2DM patients had three or more comorbidities at diagnosis, increasing to 60% of patients ten years later. Two of the five commonest comorbidities at diagnosis were traditional T2DM conditions (hypertension (37%) and ischaemic heart disease (10%)) the other three were not (depression (15%), back pain (25%) and osteoarthritis (11%)). The prevalence of each increased during the course of the disease, with more than one in three patients having back pain and one in four having depression ten years post diagnosis.People with five or more comorbidities at diagnosis had higher prevalence of each of the 35 comorbidities. Hypertension (73%) was the commonest comorbidity at diagnosis in this group; followed by back pain (69%), depression (67%), asthma (45%) and osteoarthritis (36%). People with obesity at diagnosis had substantially different comorbidity profiles to those without, and the five commonest comorbidities were 50% more common in this group. Interpretation: Preventative and clinical interventions alongside care pathways for people with T2DM should transition to reflect the diverse set of causes driving persistent morbidity. This would benefit both patients and healthcare systems alike. Funding: The study was funded by the National Institute for Health and Care Excellence (NICE).

Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers.

As part of the HTx (Next Generation Health Technology Assessment) project, this study was aimed at identifying the main barriers for application of real-world evidence (RWE) for the purposes of health technology assessment in the Central and Eastern European countries. A mixed methods approach was employed to identify the main barriers: a scoping review of the literature and a series of discussions with stakeholders. Based on the applied approaches, we attempted to summarize the main barriers and challenges related to transferability of RWE in five main groups: technical, regulatory, clinical, scientific and perceptional barriers. Further research should pursue the development of detailed, consensus-based guidelines to improve the harmonization and standardization of RWE.

A Review of Economic Models Submitted to NICE's Technology Appraisal Programme, for Treatments of T1DM & T2DM.

Background: In the UK, 4.7 million people are currently living with diabetes. This is projected to increase to 5 million by 2025. The direct and indirect costs of T1DM and T2DM are rising, and direct costs already account for approximately 10% of the National Health Service (NHS) budget. Objective: The aim of this review is to assess the economic models used in the context of NICE's Technology Appraisals (TA) Programme of T1DM and T2DM treatments, as well as to examine their compliance with the American Diabetes Association's (ADA) guidelines on computer modelling. Methods: A review of the economic models used in NICE's TA programme of T1DM and T2DM treatments was undertaken. Relevant TAs were identified through searching the NICE website for published appraisals completed up to April 2021. The review also examined the associated Evidence Review Group (ERG) reports and Final Appraisal Documents (FAD), which are publicly accessible. ERG reports were scrutinised to identify major issues pertaining to the economic modelling. The FAD documents were then examined to assess how these issues reflected on NICE recommendations. Results: Overall, 10 TAs pertaining to treatments of T1DM and T2DM were identified. Two TAs were excluded as they did not use economic models. Seven of the 8 included TAs related to a novel class of oral antidiabetic drugs (OADs), gliflozins, and one to continuous subcutaneous insulin infusion (CSII) devices. There is a lack of recent, robust data informing risk equations to enable the derivation of transition probabilities. Despite uncertainty surrounding its clinical relevance, bodyweight/BMI is a key driver in many T2DM-models. HbA1c's reliability as a predictor of hard outcomes is uncertain, chiefly for macrovascular complications. The external validity of T1DM is even less clear. There is an inevitable trade-off between the sophistication of models' design, their transparency and practicality. Conclusion: Economic models are essential tools to support decision-making in relation to market access and ascertain diabetes technologies' cost effectiveness. However, key structural and methodological issues exist. Models' shortcomings should be acknowledged and contextualised within the framework of technology appraisals. Diabetes medications and other technologies should also be subject to regular and consistent re-appraisal to inform disinvestment decisions. Artificial intelligence could potentially enhance models' transparency and practicality.

Diagnostics and Treatments of COVID-19: A Living Systematic Review of Economic Evaluations.

OBJECTIVES: As healthcare systems continue to respond to the COVID-19 pandemic, cost-effectiveness evidence will be needed to identify which tests and treatments for COVID-19 offer value for money. We sought to review economic evaluations of diagnostic tests and treatments for COVID-19, critically appraising the methodological approaches used and reporting cost-effectiveness estimates, using a "living" systematic review approach. METHODS: Key databases (including MEDLINE, EconLit, Embase) were last searched on July 12, 2021. Gray literature and model repositories were also searched. Only full economic evaluations published in English were included. Studies were quality assessed and data were extracted into standard tables. Results were narratively summarized. The review was completed by 2 reviewers independently, with disagreements resolved through discussion with a senior reviewer. RESULTS: Overall, 3540 records were identified, with 13 meeting the inclusion criteria. After quality assessment, 6 were excluded because of very severe limitations. Of the 7 studies included, 5 were cost-utility analyses and 2 were cost-effectiveness analyses. All were model-based analyses. A total of 5 evaluated treatments (dexamethasone, remdesivir, hypothetical) and 2 evaluated hypothetical testing strategies. Cost-effectiveness estimates were sensitive to the treatment effect on survival and hospitalization, testing speed and accuracy, disease severity, and price. CONCLUSIONS: Presently, there are few economic evaluations for COVID-19 tests and treatments. They suggest treatments that confer a survival benefit and fast diagnostic tests may be cost effective. Nevertheless, studies are subject to major evidence gaps and take inconsistent analytical approaches. The evidence may improve for planned updates of this "living" review.

Out of Date or Best Before? A Commentary on the Relevance of Economic Evaluations Over Time.

The impact of time on the applicability and relevance of historical economic evaluations can be considerable. Ignoring this may lead to the use of weak or invalid evidence to inform important research questions or resource allocation decisions, as historical economic evaluations may have reached different conclusions compared to if a similar study had been conducted more recently. There are multiple factors that contribute towards evidence becoming outdated including changes to the relevant decision problem (e.g. comparators), changes to parameters (such as costs, utilities and resource use) and methodological updates (e.g. recommendations on uncertainty analysis). Researchers reviewing economic evaluations need to consider whether changes over time would influence the study design and results if the evaluation were repeated, to the extent that it is no longer helpful or informative. In this paper, we summarise these key issues and make recommendations about how and whether researchers can future proof their economic evaluations.

Assessing Technologies for COVID-19: What are the Challenges for Health Technology Assessment Agencies? Findings From a Survey and Roundtable Workshop.

BACKGROUND: To date, health technology assessment (HTA) agencies have not been at the forefront of decision making regarding the adoption of interventions for coronavirus disease 2019 (COVID-19). Instead, policymakers have prioritised rapid action in response to the pandemic emergency, with no assessment of value for money. As COVID-19 vaccination coverage increases and healthcare systems begin to recover, HTA agencies will be expected to assess technologies for COVID-19. OBJECTIVE: We aimed to identify the key challenges when assessing therapeutic and diagnostic technologies for COVID-19, from the perspective of HTA agencies, and identify whether there is a case for novel HTA methods and/or processes to address them. METHODS: We used a mixed-methods approach, by conducting an online survey of HTA agencies, to collect data about the challenges faced when assessing or planning to assess diagnostic and therapeutic technologies for COVID-19. The online survey was followed by a 'roundtable' workshop of HTA agencies' representatives to discuss the results and to elaborate on their responses. RESULTS: We received 21 completed surveys (response rate of 45%) and 11 of the respondents joined the roundtable discussion. Five themes emerged from the responses: assessing clinical effectiveness (44%), assessing cost effectiveness (19%), practical (19%), political (11%), and decision making (11%) challenges. At the roundtable, attendees elaborated on the challenges and identified two additional themes: how HTA agencies have responded to the pandemic to date, and how their role might change over time. CONCLUSION: HTA agencies face both methodological and logistical challenges when assessing or planning to assess technologies for COVID-19. An interim best-practice HTA framework to address the key challenges would be valuable.

Challenges in synthesising cost-effectiveness estimates.

Economic evaluations help decision-makers faced with tough decisions on how to allocate resources. Systematic reviews of economic evaluations are useful as they allow readers to assess whether interventions have been demonstrated to be cost effective, the uncertainty in the evidence base, and key limitations or gaps in the evidence base. The synthesis of systematic reviews of economic evaluations commonly takes a narrative approach whereas a meta-analysis is common step for reviews of clinical evidence (e.g. effectiveness or adverse event outcomes). As they are common objectives in other reviews, readers may query why a synthesis has not been attempted for economic outcomes. However, a meta-analysis of incremental cost-effectiveness ratios, costs, or health benefits (including quality-adjusted life years) is fraught with issues largely due to heterogeneity across study designs and methods and further practical challenges. Therefore, meta-analysis is rarely feasible or robust. This commentary outlines these issues, supported by examples from the literature, to support researchers and reviewers considering systematic review of economic evidence.

Cost-Effectiveness Evaluations of Psychological Therapies for Schizophrenia and Bipolar Disorder: A Systematic Review.

OBJECTIVES: This review aims to assess the cost-effectiveness of psychological interventions for schizophrenia/bipolar disorder (BD), to determine the robustness of current evidence and identify gaps in the available evidence. METHODS: Electronic searches (PsycINFO, MEDLINE, Embase) identified economic evaluations relating incremental cost to outcomes in the form of an incremental cost-effectiveness ratio published in English since 2000. Searches were concluded in November 2018. Inclusion criteria were: adults with schizophrenia/BD; any psychological/psychosocial intervention (e.g., psychological therapy and integrated/collaborative care); probability of cost-effectiveness at explicitly defined thresholds reported. Comparators could be routine practice, no intervention, or alternative psychological therapies. Screening, data extraction, and critical appraisal were performed using pre-specified criteria and forms. Results were summarized qualitatively. The protocol was registered on the PROSPERO database (CRD42017056579). RESULTS: Of 3,864 studies identified, 12 met the criteria for data extraction. All were integrated clinical and economic randomized controlled trials. The most common intervention was cognitive behavioral therapy (CBT, 6/12 studies). The most common measure of health benefit was the quality-adjusted life-year (6/12). Follow-up ranged from 6 months to 5 years. Interventions were found to be cost-effective in most studies (9/12): the probability of cost-effectiveness ranged from 35-99.5 percent. All studies had limitations and demonstrated uncertainty (particularly related to incremental costs). CONCLUSIONS: Most studies concluded psychological interventions for schizophrenia/BD are cost-effective, including CBT, although there was notable uncertainty. Heterogeneity across studies makes it difficult to reach strong conclusions. There is a particular need for more evidence in the population with BD and for longer-term evidence across both populations.