Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models.

BACKGROUND: Current generation large language models (LLMs) such as Generative Pre-Trained Transformer 4 (GPT-4) have achieved human-level performance on many tasks including the generation of computer code based on textual input. This study aimed to assess whether GPT-4 could be used to automatically programme two published health economic analyses. METHODS: The two analyses were partitioned survival models evaluating interventions in non-small cell lung cancer (NSCLC) and renal cell carcinoma (RCC). We developed prompts which instructed GPT-4 to programme the NSCLC and RCC models in R, and which provided descriptions of each model's methods, assumptions and parameter values. The results of the generated scripts were compared to the published values from the original, human-programmed models. The models were replicated 15 times to capture variability in GPT-4's output. RESULTS: GPT-4 fully replicated the NSCLC model with high accuracy: 100% (15/15) of the artificial intelligence (AI)-generated NSCLC models were error-free or contained a single minor error, and 93% (14/15) were completely error-free. GPT-4 closely replicated the RCC model, although human intervention was required to simplify an element of the model design (one of the model's fifteen input calculations) because it used too many sequential steps to be implemented in a single prompt. With this simplification, 87% (13/15) of the AI-generated RCC models were error-free or contained a single minor error, and 60% (9/15) were completely error-free. Error-free model scripts replicated the published incremental cost-effectiveness ratios to within 1%. CONCLUSION: This study provides a promising indication that GPT-4 can have practical applications in the automation of health economic model construction. Potential benefits include accelerated model development timelines and reduced costs of development. Further research is necessary to explore the generalisability of LLM-based automation across a larger sample of models.

Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models.

BACKGROUND: The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results. METHODS: We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis. RESULTS: The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results. CONCLUSIONS: This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.

Ertugliflozin Compared to Other Anti-hyperglycemic Agents as Monotherapy and Add-on Therapy in Type 2 Diabetes: A Systematic Literature Review and Network Meta-Analysis.

INTRODUCTION: Ertugliflozin is a new sodium-glucose co-transporter-2 inhibitor (SGLT2i) for the treatment of type 2 diabetes mellitus. As there are no head-to-head trials comparing the efficacy of SGLT2is, the primary objective of this analysis was to indirectly compare ertugliflozin to other SGLT2i in patient populations with inadequately controlled glycated hemoglobin (HbA1c > 7.0%) and previously treated with either diet/exercise, metformin alone or metformin plus a dipeptidyl peptidase-4 inhibitor (DPP4i). METHODS: A systematic literature review (SLR) identified randomized controlled trials (RCTs) reporting outcomes at 24-26 weeks of treatment. Comparators to ertugliflozin were the SGLT2is canagliflozin, dapagliflozin and empagliflozin, with non-SGLT2i comparators also evaluated third-line [insulin and glucagon-like peptide-1 receptor agonists (GLP-1 RAs)]. Outcomes were change from baseline in HbA1c, weight and systolic blood pressure (SBP) as well as HbA1c < 7% and key safety events. Bayesian network meta-analysis was used to synthesize evidence. Results are presented as the median of the mean difference (MD) or as odds ratios with 95% credible intervals (CrI). RESULTS: In patients uncontrolled on diet/exercise, the efficacy of ertugliflozin 5 mg monotherapy was not significantly different from that of other low-dose SGLT2is in terms of HbA1c reduction, while ertugliflozin 15 mg was more effective than dapagliflozin 10 mg (MD - 0.36%, CrI - 0.65, - 0.08) and empagliflozin 25 mg (MD - 0.31%, CrI - 0.58, - 0.04). As add-on therapy to metformin, ertugliflozin 5 mg was more effective in lowering HbA1c than dapagliflozin 5 mg (MD - 0.22%, CrI - 0.42, - 0.02), and ertugliflozin 15 mg was more effective than dapagliflozin 10 mg (MD - 0.26%, CrI - 0.46, - 0.06) and empagliflozin 25 mg (MD - 0.23%, CrI - 0.44, - 0.03). Among patients uncontrolled on combination therapy metformin plus a DPP4i, no relevant RCTs with insulin were identified from the SLR. One study with a GLP-1 RA was included in a sensitivity analysis due to limited data. There were no differences between ertugliflozin 5 or 15 mg and other SGLT2is, with the exception of dapagliflozin 10 mg, which was significantly less effective when added to sitagliptin and metformin. Overall, there were no other significant differences for remaining efficacy and safety outcomes except for a lower SBP for canagliflozin 300 mg compared to ertugliflozin 15 mg in the diet/exercise population. CONCLUSIONS: Indirect comparisons for HbA1c reduction found that ertugliflozin 5 mg was more effective than dapagliflozin 5 mg when added to metformin monotherapy, whereas ertugliflozin 15 mg was more effective than dapagliflozin 10 mg and empagliflozin 25 mg when added to diet/exercise and to metformin monotherapy. The HbA1c reduction associated with ertugliflozin was no different than that associated with canagliflozin across all populations. FUNDING: Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA, and Pfizer Inc., New York, NY, USA.

An efficacy comparison of anti-vascular growth factor agents and laser photocoagulation in diabetic macular edema: a network meta-analysis incorporating individual patient-level data.

BACKGROUND: This was an updated network meta-analysis (NMA) of anti-vascular endothelial growth factor (VEGF) agents and laser photocoagulation in patients with diabetic macular edema (DME). Unlike previous NMA that used meta-regression to account for potential confounding by systematic variation in treatment effect modifiers across studies, this update incorporated individual patient-level data (IPD) regression to provide more robust adjustment. METHODS: An updated review was conducted to identify randomised controlled trials for inclusion in a Bayesian NMA. The network included intravitreal aflibercept (IVT-AFL) 2 mg bimonthly (2q8) after 5 initial doses, ranibizumab 0.5 mg as-needed (PRN), ranibizumab 0.5 mg treat-and-extend (T&E), and laser photocoagulation. Outcomes included in the analysis were change in best-corrected visual acuity (BCVA), measured using an Early Treatment Diabetic Retinopathy Study (ETDRS) chart, and patients with ≥10 and ≥ 15 ETDRS letter gains/losses at 12 months. Analyses were performed using networks restricted to IPD-only and IPD and aggregate data with (i) no covariable adjustment, (ii) covariable adjustment for baseline BVCA assuming common interaction effects (against reference treatment), and (iii) covariable adjustments specific to each treatment comparison (restricted to IPD-only network). RESULTS: Thirteen trials were included in the analysis. IVT-AFL 2q8 was superior to laser in all analyses. IVT-AFL 2q8 showed strong evidence of superiority (95% credible interval [CrI] did not cross null) versus ranibizumab 0.5 mg PRN for mean change in BCVA (mean difference 5.20, 95% CrI 1.90-8.52 ETDRS letters), ≥15 ETDRS letter gain (odds ratio [OR] 2.30, 95% CrI 1.12-4.20), and ≥10 ETDRS letter loss (OR 0.25, 95% CrI 0.05-0.74) (IPD and aggregate random-effects model with baseline BCVA adjustment). IVT-AFL 2q8 was not superior to ranibizumab 0.5 mg T&E for mean change in BCVA (mean difference 5.15, 95% CrI -0.26-10.61 ETDRS letters) (IPD and aggregate random-effects model). CONCLUSIONS: This NMA, which incorporated IPD to improve analytic robustness, showed evidence of superiority of IVT-AFL 2q8 to laser and ranibizumab 0.5 mg PRN. These results were irrespective of adjustment for baseline BCVA.

Comparative efficacy of fixed-dose combinations of long-acting muscarinic antagonists and long-acting ß2-agonists: a systematic review and network meta-analysis.

BACKGROUND: A number of long-acting muscarinic antagonist (LAMA)/long-acting ß2-agonist (LABA) fixed-dose combinations (FDCs) for treatment of moderate-to-very severe chronic obstructive pulmonary disease (COPD) have recently become available, but none have been directly compared in head-to-head randomized controlled trials (RCTs). The purpose of this study was to assess the relative clinical benefit of all currently available LAMA/LABA FDCs using a Bayesian network meta-analysis (NMA). METHODS: A systematic literature review identified RCTs investigating the efficacy, safety and quality of life associated with licensed LAMA/LABA FDCs for the treatment of moderate-to-very severe COPD. RCTs were screened for inclusion in the NMA using prespecified eligibility criteria. Data were extracted for outcomes of interest, including change in trough forced expiratory volume in 1 second (tFEV1) from baseline, St. George Respiratory Questionnaire (SGRQ) percentage of responders, Transition Dyspnea Index (TDI) percentage of responders, change in SGRQ score from baseline, change in TDI focal score from baseline, moderate-to-severe exacerbations, all-cause discontinuation, and discontinuation due to adverse events. RESULTS: Following screening, a total of 27 trials from 26 publications with 30,361 subjects were eligible for inclusion in the NMA. Nonsignificant results were seen in most analyses comparing efficacy, exacerbations and discontinuation rates of included LAMA/LABA FDCs (i.e. aclidinium/formoterol 400/12 µg, glycopyrronium/indacaterol 110/50 µg, tiotropium + olodaterol 5/5 µg, umeclidinium/vilanterol 62.5/25 µg). Meta-regression controlling for post-bronchodilator percentage of tFEV1 predicted at baseline as well as meta-regression adjusting for concomitant use of inhaled corticosteroids at baseline was performed to assess the magnitude of effect modification and produced similar results as observed in the base case analysis. CONCLUSION: All LAMA/LABA FDCs were found to have similar efficacy and safety. Definitive assessment of the relative efficacy of different treatments can only be performed through direct comparison in head-to-head RCTs. In the absence of such data, this indirect comparison may be of value in clinical and health economic decision-making.