Calibration plots for multistate risk predictions models.

INTRODUCTION: There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. METHODS: We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. RESULTS: The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability. CONCLUSIONS: We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

Making predictions under interventions: a case study from the PREDICT-CVD cohort in New Zealand primary care.

Background: Most existing clinical prediction models do not allow predictions under interventions. Such predictions allow predicted risk under different proposed strategies to be compared and are therefore useful to support clinical decision making. We aimed to compare methodological approaches for predicting individual level cardiovascular risk under three interventions: smoking cessation, reducing blood pressure, and reducing cholesterol. Methods: We used data from the PREDICT prospective cohort study in New Zealand to calculate cardiovascular risk in a primary care setting. We compared three strategies to estimate absolute risk under intervention: (a) conditioning on hypothetical interventions in non-causal models; (b) combining existing prediction models with causal effects estimated using observational causal inference methods; and (c) combining existing prediction models with causal effects reported in published literature. Results: The median absolute cardiovascular risk among smokers was 3.9%; our approaches predicted that smoking cessation reduced this to a median between a non-causal estimate of 2.5% and a causal estimate of 2.8%, depending on estimation methods. For reducing blood pressure, the proposed approaches estimated a reduction of absolute risk from a median of 4.9% to a median between 3.2% and 4.5% (both derived from causal estimation). Reducing cholesterol was estimated to reduce median absolute risk from 3.1% to between 2.2% (non-causal estimate) and 2.8% (causal estimate). Conclusions: Estimated absolute risk reductions based on non-causal methods were different to those based on causal methods, and there was substantial variation in estimates within the causal methods. Researchers wishing to estimate risk under intervention should be explicit about their causal modelling assumptions and conduct sensitivity analysis by considering a range of possible approaches.

Applying Team Science to Collaborative Digital Health Research: Learnings from the Wearable Clinic.

Collaboration across disciplinary boundaries is vital to address the complex challenges and opportunities in Digital Health. We present findings and experiences of applying the principles of Team Science to a digital health research project called 'The Wearable Clinic'. Challenges faced were a lack of shared understanding of key terminology and concepts, and differences in publication cultures between disciplines. We also encountered more profound discrepancies, relating to definitions of "success" in a research project. We recommend that collaborative digital health research projects select a formal Team Science methodology from the outset.

Comparing Predictive Performance of Time Invariant and Time Variant Clinical Prediction Models in Cardiac Surgery.

Clinical prediction models are increasingly used across healthcare to support clinical decision making. Existing methods and models are time-invariant and thus ignore the changes in populations and healthcare practice that occur over time. We aimed to compare the performance of time-invariant with time-variant models in UK National Adult Cardiac Surgery Audit data from Manchester University NHS Foundation Trust between 2009 and 2019. Data from 2009-2011 were used for initial model fitting, and data from 2012-2019 for validation and updating. We fitted four models to the data: a time-invariant logistic regression model (not updated), a logistic model which was updated every year and validated it in each subsequent year, a logistic regression model where the intercept is a function of calendar time (not updated), and a continually updating Bayesian logistic model which was updated with each new observation and continuously validated. We report predictive performance over the complete validation cohort and for each year in the validation data. Over the complete validation data, the Bayesian model had the best predictive performance.

Identification of heart failure hospitalization from NHS Digital data: comparison with expert adjudication.

AIMS: Population-wide, person-level, linked electronic health record data are increasingly used to estimate epidemiology, guide resource allocation, and identify events in clinical trials. The accuracy of data from NHS Digital (now part of NHS England) for identifying hospitalization for heart failure (HHF), a key HF standard, is not clear. This study aimed to evaluate the accuracy of NHS Digital data for identifying HHF. METHODS AND RESULTS: Patients experiencing at least one HHF, as determined by NHS Digital data, and age- and sex-matched patients not experiencing HHF, were identified from a prospective cohort study and underwent expert adjudication. Three code sets commonly used to identify HHF were applied to the data and compared with expert adjudication (I50: International Classification of Diseases-10 codes beginning I50; OIS: Clinical Commissioning Groups Outcomes Indicator Set; and NICOR: National Institute for Cardiovascular Outcomes Research, used as the basis for the National Heart Failure Audit in England and Wales). Five hundred four patients underwent expert adjudication, of which 10 (2%) were adjudicated to have experienced HHF. Specificity was high across all three code sets in the first diagnosis position {I50: 96.2% [95% confidence interval (CI) 94.1-97.7%]; NICOR: 93.3% [CI 90.8-95.4%]; OIS: 95.6% [CI 93.3-97.2%]} but decreased substantially as the number of diagnosis positions expanded. Sensitivity [40.0% (CI 12.2-73.8%)] and positive predictive value (PPV) [highest with I50: 17.4% (CI 8.1-33.6%)] were low in the first diagnosis position for all coding sets. PPV was higher for the National Heart Failure Audit criteria, albeit modestly [36.4% (CI 16.6-62.2%)]. CONCLUSIONS: NHS Digital data were not able to accurately identify HHF and should not be used in isolation for this purpose.

Compatibility in Missing Data Handling Across the Prediction Model Pipeline: A Simulation Study.

Careful handling of missing data is crucial to ensure that clinical prediction models are developed, validated, and implemented in a robust manner. We determined the bias in estimating predictive performance of different combinations of approaches for handling missing data across validation and implementation. We found four strategies that are compatible across the model pipeline and have provided recommendations for handling missing data between model validation and implementation under different missingness mechanisms.

Patient experiences of an online consultation system: a qualitative study in English primary care post-COVID-19.

BACKGROUND: Online consultation systems (OCSs) allow patients to contact their healthcare teams online. Since 2020 they have been rapidly rolled out in primary care following policy initiatives and the COVID-19 pandemic. In-depth research of patients' experiences using OCSs is lacking. AIM: Explore patients' experiences of using an OCS. DESIGN AND SETTING: Qualitative study in English GP practices using the Patchs OCS (www.Patchs.ai) from March 2020 to July 2022. METHOD: Thematic analysis of 25 patient interviews and 21 467 written comments from 11 851 patients who used the OCS from nine and 240 GP practices, respectively. RESULTS: Patients cited benefits of using the OCS as speed, flexibility, and efficiency. Nevertheless, some patients desired a return to traditional consultation methods. GP practices often did not clearly advertise the OCS or use it as patients expected, which caused frustration. Patients reported advantages of having a written record of consultations and the opportunity to communicate detailed queries in free text. Views differed on how the OCS influenced clinical safety and discussions of sensitive topics. Patients who struggled to communicate in traditional consultations often preferred using the OCS, and male patients reported being more likely to use it. CONCLUSION: Globally, this is the largest in-depth study of patient experiences of an OCS. It contributes new knowledge that the patient experience of using OCSs can be influenced by previously unreported patient characteristics and the conditions they consult about. Further, it contributes recommendations on the design and implementation of the OCS in practice.

A scoping review finds a growing trend in studies validating multimorbidity patterns and identifies five broad types of validation methods.

OBJECTIVES: Multimorbidity, the presence of two or more long-term conditions, is a growing public health concern. Many studies use analytical methods to discover multimorbidity patterns from data. We aimed to review approaches used in published literature to validate these patterns. STUDY DESIGN AND SETTING: We systematically searched PubMed and Web of Science for studies published between July 2017 and July 2023 that used analytical methods to discover multimorbidity patterns. RESULTS: Out of 31,617 studies returned by the searches, 172 were included. Of these, 111 studies (64%) conducted validation, the number of studies with validation increased from 53.13% (17 out of 32 studies) to 71.25% (57 out of 80 studies) in 2017-2019 to 2022-2023, respectively. Five types of validation were identified: assessing the association of multimorbidity patterns with clinical outcomes (n = 79), stability across subsamples (n = 26), clinical plausibility (n = 22), stability across methods (n = 7) and exploring common determinants (n = 2). Some studies used multiple types of validation. CONCLUSION: The number of studies conducting a validation of multimorbidity patterns is clearly increasing. The most popular validation approach is assessing the association of multimorbidity patterns with clinical outcomes. Methodological guidance on the validation of multimorbidity patterns is needed.

Trends for opioid prescribing and the impact of the COVID-19 pandemic in patients with rheumatic and musculoskeletal diseases between 2006 and 2021.

OBJECTIVE: To investigate opioid prescribing trends and assess the impact of the COVID-19 pandemic on opioid prescribing in rheumatic and musculoskeletal diseases (RMDs). METHODS: Adult patients with RA, PsA, axial spondyloarthritis (AxSpA), SLE, OA and FM with opioid prescriptions between 1 January 2006 and 31 August 2021 without cancer in UK primary care were included. Age- and gender-standardized yearly rates of new and prevalent opioid users were calculated between 2006 and 2021. For prevalent users, monthly measures of mean morphine milligram equivalents (MME)/day were calculated between 2006 and 2021. To assess the impact of the pandemic, we fitted regression models to the monthly number of prevalent opioid users between January 2015 and August 2021. The time coefficient reflects the trend pre-pandemic and the interaction term coefficient represents the change in the trend during the pandemic. RESULTS: The study included 1 313 519 RMD patients. New opioid users for RA, PsA and FM increased from 2.6, 1.0 and 3.4/10 000 persons in 2006 to 4.5, 1.8 and 8.7, respectively, in 2018 or 2019. This was followed by a fall to 2.4, 1.2 and 5.9, respectively, in 2021. Prevalent opioid users for all RMDs increased from 2006 but plateaued or dropped beyond 2018, with a 4.5-fold increase in FM between 2006 and 2021. In this period, MME/day increased for all RMDs, with the highest for FM (≥35). During COVID-19 lockdowns, RA, PsA and FM showed significant changes in the trend of prevalent opioid users. The trend for FM increased pre-pandemic and started decreasing during the pandemic. CONCLUSION: The plateauing or decreasing trend of opioid users for RMDs after 2018 may reflect the efforts to tackle rising opioid prescribing in the UK. The pandemic led to fewer people on opioids for most RMDs, providing reassurance that there was no sudden increase in opioid prescribing during the pandemic.

Using hypergraphs to quantify importance of sets of diseases by healthcare resource utilisation: A retrospective cohort study.

Rates of Multimorbidity (also called Multiple Long Term Conditions, MLTC) are increasing in many developed nations. People with multimorbidity experience poorer outcomes and require more healthcare intervention. Grouping of conditions by health service utilisation is poorly researched. The study population consisted of a cohort of people living in Wales, UK aged 20 years or older in 2000 who were followed up until the end of 2017. Multimorbidity clusters by prevalence and healthcare resource use (HRU) were modelled using hypergraphs, mathematical objects relating diseases via links which can connect any number of diseases, thus capturing information about sets of diseases of any size. The cohort included 2,178,938 people. The most prevalent diseases were hypertension (13.3%), diabetes (6.9%), depression (6.7%) and chronic obstructive pulmonary disease (5.9%). The most important sets of diseases when considering prevalence generally contained a small number of diseases, while the most important sets of diseases when considering HRU were sets containing many diseases. The most important set of diseases taking prevalence and HRU into account was diabetes & hypertension and this combined measure of importance featured hypertension most often in the most important sets of diseases. We have used a single approach to find the most important sets of diseases based on co-occurrence and HRU measures, demonstrating the flexibility of the hypergraph approach. Hypertension, the most important single disease, is silent, underdiagnosed and increases the risk of life threatening co-morbidities. Co-occurrence of endocrine and cardiovascular diseases was common in the most important sets. Combining measures of prevalence with HRU provides insights which would be helpful for those planning and delivering services.

Digital health and care: emerging from pandemic times.

In 2020, we published an editorial about the massive disruption of health and care services caused by the COVID-19 pandemic and the rapid changes in digital service delivery, artificial intelligence and data sharing that were taking place at the time. Now, 3 years later, we describe how these developments have progressed since, reflect on lessons learnt and consider key challenges and opportunities ahead by reviewing significant developments reported in the literature. As before, the three key areas we consider are digital transformation of services, realising the potential of artificial intelligence and wise data sharing to facilitate learning health systems. We conclude that the field of digital health has rapidly matured during the pandemic, but there are still major sociotechnical, evaluation and trust challenges in the development and deployment of new digital services.

The Complexity of Transferring Remote Monitoring and Virtual Care Technology Between Countries: Lessons From an International Workshop.

International deployment of remote monitoring and virtual care (RMVC) technologies would efficiently harness their positive impact on outcomes. Since Canada and the United Kingdom have similar populations, health care systems, and digital health landscapes, transferring digital health innovations between them should be relatively straightforward. Yet examples of successful attempts are scarce. In a workshop, we identified 6 differences that may complicate RMVC transfer between Canada and the United Kingdom and provided recommendations for addressing them. These key differences include (1) minority groups, (2) physical geography, (3) clinical pathways, (4) value propositions, (5) governmental priorities and support for digital innovation, and (6) regulatory pathways. We detail 4 broad recommendations to plan for sustainability, including the need to formally consider how highlighted country-specific recommendations may impact RMVC and contingency planning to overcome challenges; the need to map which pathways are available as an innovator to support cross-country transfer; the need to report on and apply learnings from regulatory barriers and facilitators so that everyone may benefit; and the need to explore existing guidance to successfully transfer digital health solutions while developing further guidance (eg, extending the nonadoption, abandonment, scale-up, spread, sustainability framework for cross-country transfer). Finally, we present an ecosystem readiness checklist. Considering these recommendations will contribute to successful international deployment and an increased positive impact of RMVC technologies. Future directions should consider characterizing additional complexities associated with global transfer.

Trajectories in chronic disease accrual and mortality across the lifespan in Wales, UK (2005-2019), by area deprivation profile: linked electronic health records cohort study on 965,905 individuals.

Background: Understanding and quantifying the differences in disease development in different socioeconomic groups of people across the lifespan is important for planning healthcare and preventive services. The study aimed to measure chronic disease accrual, and examine the differences in time to individual morbidities, multimorbidity, and mortality between socioeconomic groups in Wales, UK. Methods: Population-wide electronic linked cohort study, following Welsh residents for up to 20 years (2000-2019). Chronic disease diagnoses were obtained from general practice and hospitalisation records using the CALIBER disease phenotype register. Multi-state models were used to examine trajectories of accrual of 132 diseases and mortality, adjusted for sex, age and area-level deprivation. Restricted mean survival time was calculated to measure time spent free of chronic disease(s) or mortality between socioeconomic groups. Findings: In total, 965,905 individuals aged 5-104 were included, from a possible 2.9 m individuals following a 5-year clearance period, with an average follow-up of 13.2 years (12.7 million person-years). Some 673,189 (69.7%) individuals developed at least one chronic disease or died within the study period. From ages 10 years upwards, the individuals living in the most deprived areas consistently experienced reduced time between health states, demonstrating accelerated transitions to first and subsequent morbidities and death compared to their demographic equivalent living in the least deprived areas. The largest difference were observed in 10 and 20 year old males developing multimorbidity (-0.45 years (99% CI: -0.45, -0.44)) and in 70 year old males dying after developing multimorbidity (-1.98 years (99% CI: -2.01, -1.95)). Interpretation: This study adds to the existing literature on health inequalities by demonstrating that individuals living in more deprived areas consistently experience accelerated time to diagnosis of chronic disease and death across all ages, accounting for competing risks. Funding: UK Medical Research Council, Health Data Research UK, and Administrative Data Research Wales.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques.

INTRODUCTION: This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis. METHODS: We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring. RESULTS: Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors. DISCUSSION: We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study.

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

A Novel Automated Approach to Mutation-Cancer Relation Extraction by Incorporating Heterogeneous Knowledge.

Automatic extraction of relations between gene mutations and cancer entities occurring in the cancer literature using text mining can rapidly provide vital information to support precision cancer medicine. However, mutation-cancer relation extraction is more challenging than general relation extraction from free text, since it is often not possible without cancer-specific background knowledge and thus the model replies on a deeper understanding of complex surrounding tokens. We propose a deep learning model that jointly extracts mutations and their associated cancers. Background knowledge comes from two different knowledge bases which store different types of information about mutations. Given the different ways in which knowledge is stored in these two resources, we propose two separate methods for embedding knowledge, namely sentence-based knowledge integration and attribute-aware knowledge integration. The evaluation demonstrated that our model outperforms a number of baseline models and gains 96.00%, 92.57% and 94.57% F1 scores on three public datasets, EMU BCa, EMU PCa, and BRONCO, thus illustrating the effectiveness of our knowledge integration approach. The auxiliary experiments show that our models can utilize more informative text from the KBs and link the mutations to their corresponding cancer disease although the input text provides insufficient context.

Heightened COVID-19 Mortality in People With Severe Mental Illness Persists After Vaccination: A Cohort Study of Greater Manchester Residents.

BACKGROUND AND HYPOTHESIS: Previous studies show that people with severe mental illness (SMI) are at higher risk of COVID-19 mortality, however limited evidence exists regarding risk postvaccination. We investigated COVID-19 mortality among people with schizophrenia and other SMIs before, during and after the UK vaccine roll-out. STUDY DESIGN: Using the Greater Manchester (GM) Care Record to access routinely collected health data linked with death records, we plotted COVID-19 mortality rates over time in GM residents with schizophrenia/psychosis, bipolar disorder (BD), and/or recurrent major depressive disorder (MDD) from February 2020 to September 2021. Multivariable logistic regression was used to compare mortality risk (risk ratios; RRs) between people with SMI (N = 193 435) and age-sex matched controls (N = 773 734), adjusted for sociodemographic factors, preexisting comorbidities, and vaccination status. STUDY RESULTS: Mortality risks were significantly higher among people with SMI compared with matched controls, particularly among people with schizophrenia/psychosis (RR 3.18, CI 2.94-3.44) and/or BD (RR 2.69, CI 2.16-3.34). In adjusted models, the relative risk of COVID-19 mortality decreased, though remained significantly higher than matched controls for people with schizophrenia (RR 1.61, CI 1.45-1.79) and BD (RR 1.92, CI 1.47-2.50), but not recurrent MDD (RR 1.08, CI 0.99-1.17). People with SMI continued to show higher mortality rate ratios relative to controls throughout 2021, during vaccination roll-out. CONCLUSIONS: People with SMI, notably schizophrenia and BD, were at greater risk of COVID-19 mortality compared to matched controls. Despite population vaccination efforts that have prioritized people with SMI, disparities still remain in COVID-19 mortality for people with SMI.

Using discrete-choice experiments to elicit preferences for digital wearable health technology for self-management of chronic kidney disease.

OBJECTIVES: Wearable digital health technologies (DHTs) have the potential to improve chronic kidney disease (CKD) management through patient engagement. This study aimed to investigate and elicit preferences of individuals with CKD toward wearable DHTs designed to support self-management of their condition. METHODS: Using the results of our review of the published literature and after conducting qualitative patient interviews, five-choice attributes were identified and included in a discrete-choice experiment. The design consisted of 10-choice tasks, each comprising two hypothetical technologies and one opt-out scenario. We collected data from 113 adult patients with CKD stages 3-5 not on dialysis and analyzed their responses via a latent class model to explore preference heterogeneity. RESULTS: Two patient segments were identified. In all preference segments, the most important attributes were the device appearance, format, and type of information provided. Patients within the largest preference class (70 percent) favored information provided in any format except the audio, while individuals in the other class preferred information in text format. In terms of the style of engagement with the device, both classes wanted a device that provides options rather than telling them what to do. CONCLUSIONS: Our analysis indicates that user preferences differ between patient subgroups, supporting the case for offering a different design of the device for different patients' strata, thus moving away from a one-size-fits-all service provision. Furthermore, we showed how to leverage the information from user preferences early in the R&D process to inform and support the provision of nuanced person-centered wearable DHTs.