Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases.

OBJECTIVE: This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. METHODS: We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. RESULTS: The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. CONCLUSION: Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Methods for Handling Unobserved Covariates in a Bayesian Update of a Cost-effectiveness Model.

Health economic decision models often involve a wide-ranging and complicated synthesis of evidence from a number of sources, making design and implementation of such models resource-heavy. When new data become available and reassessment of treatment recommendations is warranted, it may be more efficient to perform a Bayesian update of an existing model than to construct a new model. If the existing model depends on many, possibly correlated, covariates, then an update may produce biased estimates of model parameters if some of these covariates are completely absent from the new data. Motivated by the need to update a cost-effectiveness analysis comparing diagnostic strategies for coronary heart disease, this study develops methods to overcome this obstacle by either introducing additional data or using results from previous studies. We outline a framework to handle unobserved covariates, and use our motivating example to illustrate both the flexibility of the proposed methods and some potential difficulties in applying them.