Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.

BACKGROUND: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. METHODS: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. RESULTS: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73-0.78]) and validation (0.64 [interquartile range 0.60-0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59-0.90]) than distantly related cohorts (0.59 [interquartile range 0.43-0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. CONCLUSIONS: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Why clinical trials may not help patients make treatment decisions: results from focus group discussions with 22 patients.

Aim: Despite broad interest in advancing personalized medicine, most evidence is currently derived from average results of clinical trials that may obscure heterogeneity of trial participants. Little is known currently about how patients view heterogeneity in trials and whether they can participate in methodological discussions about this concept. Materials & methods: In structured discussions with three focus groups involving 22 participants, we assessed how representatives of patient communities have used research to guide individual treatment decisions. Discussion themes were organized into a framework describing patient decision-making in four steps: decisions patients make in the course of care; information used to make decisions; sources for information; and quality of information. Results/conclusion: Patients prioritize information that reflects their own characteristics, preferences and values. They struggle applying clinical research to their own case.

Clinical Prediction Models for Valvular Heart Disease.

Background While many clinical prediction models (CPMs) exist to guide valvular heart disease treatment decisions, the relative performance of these CPMs is largely unknown. We systematically describe the CPMs available for patients with valvular heart disease with specific attention to performance in external validations. Methods and Results A systematic review identified 49 CPMs for patients with valvular heart disease treated with surgery (n=34), percutaneous interventions (n=12), or no intervention (n=3). There were 204 external validations of these CPMs. Only 35 (71%) CPMs have been externally validated. Sixty-five percent (n=133) of the external validations were performed on distantly related populations. There was substantial heterogeneity in model performance and a median percentage change in discrimination of -27.1% (interquartile range, -49.4%--5.7%). Nearly two-thirds of validations (n=129) demonstrate at least a 10% relative decline in discrimination. Discriminatory performance of EuroSCORE II and Society of Thoracic Surgeons (2009) models (accounting for 73% of external validations) varied widely: EuroSCORE II validation c-statistic range 0.50 to 0.95; Society of Thoracic Surgeons (2009) Models validation c-statistic range 0.50 to 0.86. These models performed well when tested on related populations (median related validation c-statistics: EuroSCORE II, 0.82 [0.76, 0.85]; Society of Thoracic Surgeons [2009], 0.72 [0.67, 0.79]). There remain few (n=9) external validations of transcatheter aortic valve replacement CPMs. Conclusions Many CPMs for patients with valvular heart disease have never been externally validated and isolated external validations appear insufficient to assess the trustworthiness of predictions. For surgical valve interventions, there are existing predictive models that perform reasonably well on related populations. For transcatheter aortic valve replacement (CPMs additional external validations are needed to broadly understand the trustworthiness of predictions.

Quality of hemodialysis services in a poor population, Sistan and Baluchestan province, Iran: A descriptive, prospective study.

Managing patients with chronic kidney disease causes enormous financial burden on the Ministry of Health and Medical Education. In addition, there is a lack of feedback and adequate information in general. This study aimed at investigating quality-of-care indicators among hemodialysis (HD) patients. This descriptive, prospective study was conducted on 144 HD patients in Zabol and Iranshahr dialysis centers from March 21 to December 22, 2015. Measurement indicators included hemoglobin level, dialysis adequacy, albumin level, vascular access, and calcium and phosphorus levels. The mean hemoglobin and dialysis adequacy level at baseline were 10.58 ± 1.6 g/dL and 1.09 ± 0.18, respectively. At the end of the study, 49.6% of participants achieved target hemoglobin level. However, only 18.6% of patients achieved target dialysis adequacy at the end of the study. Dialysis adequacy was calculated by using an standard software for calculating the KT/V that provided by Iran ministry of health for all dialysis centers. The prevalence rate of use of central venous catheter was 43.2% at the end of the study. The majority of patients (59%) had albumin within normal limits and also achieved target in terms of calcium (52%) and phosphorus (59%) levels at the end of the study. Despite partial improvement in several indicators, none achieved target values which indicate the need for greater attention to quality-of-care indicators for correct planning, cost reduction, and efficiency improvement.

Patient Variability Seldom Assessed in Cost-effectiveness Studies.

BACKGROUND: Cost-effectiveness analysis (CEA) estimates can vary substantially across patient subgroups when patient characteristics influence preferences, outcome risks, treatment effectiveness, life expectancy, or associated costs. However, no systematic review has reported the frequency of subgroup analysis in CEA, what type of heterogeneity they address, and how often heterogeneity influences whether cost-effectiveness ratios exceed or fall below conventional thresholds. METHODS: We reviewed the CEA literature cataloged in the Tufts Medical Center CEA Registry, a repository describing cost-utility analyses published through 2016. After randomly selecting 200 of 642 articles published in 2014, we ascertained whether each study reported subgroup results and collected data on the defining characteristics of these subgroups. We identified whether any of the CEA subgroup results crossed conventional cost-effectiveness benchmarks (e.g., $100,000 per QALY) and compared characteristics of studies with and without subgroup-specific findings. RESULTS: Thirty-eight studies (19%) reported patient subgroup results. Articles reporting subgroup analyses were more likely to be US-based, government funded (v. drug industry- or nonprofit foundation-funded) studies, with a focus on primary or secondary (v. tertiary) prevention (P < 0.05 for comparisons). One or more patient characteristics were used to stratify CEA results 68 times within the 38 studies, with most stratifications using one characteristic (n = 47), most commonly age (n = 35). Among the 23 stratifications reported alongside average ratios in US studies, 13 produced subgroup ratios that crossed a conventional CEA ratio benchmark. CONCLUSIONS: Most CEAs do not report any subgroup results, and those that do most often stratify only by patient age. Over half of the subgroup analyses reported could lead to different value-based decision making for at least some patients.

Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015.

BACKGROUND: Clinical predictive models (CPMs) estimate the probability of clinical outcomes and hold the potential to improve decision-making and individualize care. The Tufts Predictive Analytics and Comparative Effectiveness (PACE) CPM Registry is a comprehensive database of cardiovascular disease (CVD) CPMs. The Registry was last updated in 2012, and there continues to be substantial growth in the number of available CPMs. METHODS: We updated a systematic review of CPMs for CVD to include articles published from January 1990 to March 2015. CVD includes coronary artery disease (CAD), congestive heart failure (CHF), arrhythmias, stroke, venous thromboembolism (VTE), and peripheral vascular disease (PVD). The updated Registry characterizes CPMs based on population under study, model performance, covariates, and predicted outcomes. RESULTS: The Registry includes 747 articles presenting 1083 models, including both prognostic (n = 1060) and diagnostic (n = 23) CPMs representing 183 distinct index condition/outcome pairs. There was a threefold increase in the number of CPMs published between 2005 and 2014, compared to the prior 10-year interval from 1995 to 2004. The majority of CPMs were derived from either North American (n = 455, 42%) or European (n = 344, 32%) populations. The database contains 265 CPMs predicting outcomes for patients with coronary artery disease, 196 CPMs for population samples at risk for incident CVD, and 158 models for patients with stroke. Approximately two thirds (n = 701, 65%) of CPMs report a c-statistic, with a median reported c-statistic of 0.77 (IQR, 0.05). Of the CPMs reporting validations, only 333 (57%) report some measure of model calibration. Reporting of discrimination but not calibration is improving over time (p for trend < 0.0001 and 0.39 respectively). CONCLUSIONS: There is substantial redundancy of CPMs for a wide spectrum of CVD conditions. While the number of CPMs continues to increase, model performance is often inadequately reported and calibration is infrequently assessed. More work is needed to understand the potential impact of this literature.