Performance metrics for models designed to predict treatment effect.

BACKGROUND: Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs). METHODS: Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest. RESULTS: As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics". CONCLUSION: The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Predicting dysphagia onset in patients with ALS: the ALS dysphagia risk score.

Purpose: For patients diagnosed with ALS, dysphagia can result in aspiration, malnutrition, and mortality. The purpose of this study was to develop a clinical prediction model capable of identifying patients with ALS at imminent risk for developing swallowing complications. Methods: A retrospective cohort study using the Pooled Resource Open-Access ALS Clinical Trials Database (PRO-ACT) was conducted. After dividing the PRO-ACT database into development and validation cohorts with dysphagia defined from the ALS Functional Rating Scale (ALSFRS), a multivariable Cox proportional hazards regression model estimated the probability of dysphagia at 3, 6, and 12-months with subsequent evaluation of model discrimination and calibration. Results: With 2057 participants in the development cohort and 1891 in the validation cohort, the Cox model included 7 clinical variables: spinal-onset; bulbar, fine and gross motor ALSFRS subscale scores; respiratory impairment; functional progression rate; and time from diagnosis. The cumulative incidence of dysphagia was 18% at 3-months, 29% at 6-months, and 45% at 12-months. The mean predicted probability of dysphagia development ranged from 4.5% in the bottommost risk decile to 40% in the topmost decile at 3 months, 10%-72% at 6 months, and 25%-93% at 12 months. In the validation cohort, the model had good discrimination and calibration with an optimism corrected c-statistic of 0.70 and calibration slope of 0.96. Conclusions: The ALS dysphagia risk score can be used to identify patients with ALS at high risk for self-reported dysphagia development who would benefit from a comprehensive swallowing assessment and proactive dysphagia management strategies.

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting.

BACKGROUND: Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices. METHODS: Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research. RESULTS: Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model. CONCLUSIONS: Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential. REGISTRATION: This review was not registered.

A clinical predictive score for mood disorder risk in low-income primary care settings.

BACKGROUND: Despite availability of validated screening tests for mood disorders, busy general practitioners (GPs) often lack the time to use them routinely. This study aimed to develop a simplified clinical predictive score to help screen for presence of current mood disorder in low-income primary care settings. METHODS: In a cross-sectional study, 197 patients seen at 10 primary care centers in Santiago, Chile completed self-administered screening tools for mood disorders: the Patient Health questionnaire (PHQ-9) and the Mood Disorder Questionnaire (MDQ). To determine participants' current-point mood disorder status, trained clinicians applied a gold-standard diagnostic interview (SCID-I). A simplified clinical predictive model (CM) was developed based on clinical features and selected questions from the screening tools. Using CM, a clinical predictive score (PS) was developed. Full PHQ-9 and GP assessment were compared with PS. RESULTS: Using multivariate logistic regression, clinical and demographic variables predictive of current mood disorder were identified for a simplified 8-point predictive score (PS). PS had better discrimination than GP assessment (auROC-statistic=0.80 [95% CI 0.72, 0.85] vs. 0.58 [95% CI 0.52, 0.62] p-value <0.0001), but not as good as the full PHQ-9 (0.89 [95% CI 0.85, 0.93], p-value=0.03). Compared with GP assessment, PS increased sensitivity by 50% at a fixed specificity of 90%. Administered in a typical primary care clinical population, it correctly predicted almost 80% of cases. LIMITATIONS: Further research must verify external validity of the PS. CONCLUSION: An easily administered clinical predictive score determined, with reasonable accuracy, the current risk of mood disorders in low-income primary care settings.

Association of necrotizing enterocolitis with anemia and packed red blood cell transfusions in preterm infants.

OBJECTIVE: To determine association of anemia and red blood cell (RBC) transfusions with necrotizing enterocolitis (NEC) in preterm infants. STUDY DESIGN: A total of 111 preterm infants with NEC ≥ stage 2a were compared with 222 matched controls. In all, 28 clinical variables, including hematocrit (Hct) and RBC transfusions were recorded. Propensity scores and multivariate logistic regression models were created to examine effects on the risk of NEC. RESULT: Controlling for other factors, lower Hct was associated with increased odds of NEC (odds ratio (OR)=1.10, P=0.01). RBC transfusion has a temporal relationship with NEC onset. Transfusion within 24 h (OR=7.60, P=0.001) and 48 h (OR=5.55, P=0.001) has a higher odds of developing NEC but this association is not significant by 96 h (OR=2.13, P=0.07), post-transfusion. CONCLUSION: Anemia may increase the risk of developing NEC in preterm infants. RBC transfusions are temporally related to NEC. Prospective studies are needed to better evaluate the potential influence of transfusions on the development of NEC.

Development of predictive models for airflow obstruction in alpha-1-antitrypsin deficiency.

Alpha-1-antitrypsin deficiency is a genetic condition associated with severe, early-onset chronic obstructive pulmonary disease (COPD). However, there is significant variability in lung function impairment among persons with the protease inhibitor ZZ genotype. Early identification of persons at highest risk of developing lung disease could be beneficial in guiding monitoring and treatment decisions. Using a multicenter, family-based study sample (2002-2005) of 372 persons with the protease inhibitor ZZ genotype, the authors developed prediction models for forced expiratory volume in 1 second (FEV(1)) and the presence of severe COPD using demographic, clinical, and genetic variables. Half of the data sample was used for model development, and the other half was used for model validation. In the training sample, variables found to be predictive of both FEV(1) and severe COPD were age, sex, pack-years of smoking, bronchodilator responsiveness, chronic bronchitis symptoms, and index case status. In the validation sample, the predictive model for FEV(1) explained 50% of the variance in FEV(1), and the model for severe COPD exhibited excellent discrimination (c statistic = 0.88).

Subgroup analyses in randomized controlled trials: the need for risk stratification in kidney transplantation.

Although randomized controlled trials (RCT) are the gold standard for establishing causation in clinical research, their aggregated results can be misleading when applied to individual patients. A treatment may be beneficial in some patients, but its harms may outweigh benefits in others. While conventional one-variable-at-a-time subgroup analyses have well-known limitations, multivariable risk-based analyses can help uncover clinically significant heterogeneity in treatment effects that may be otherwise obscured. Trials in kidney transplantation have yielded the finding that a reduction in acute rejection does not translate into a similar benefit in prolonging graft survival and improving graft function. This paradox might be explained by the variation in risk for acute rejection among included kidney transplant recipients varying the likelihood of benefit or harm from intense immunosuppressive regimens. Analyses that stratify patients by their immunological risk may resolve these otherwise puzzling results. Reliable risk models should be developed to investigate benefits and harms in rationally designed risk-based subgroups of patients in existing RCT data sets. These risk strata would need to be validated in future prospective clinical trials examining long-term effects on patient and graft survival. This approach may allow better individualized treatment choices for kidney transplant recipients.

Balancing the benefits of primary angioplasty against the benefits of thrombolytic therapy for acute myocardial infarction: the importance of timing.

CONTEXT: A meta-analysis found that primary percutaneous transluminal coronary angioplasty (PTCA) was more effective than thrombolytic therapy in reducing mortality from acute myocardial infarction. However, fewer than 20% of U.S. hospitals have facilities to perform PTCA and many clinicians must choose between immediate thrombolytic therapy and delayed PTCA. COUNT: The number of minutes of PTCA-related delay that would nullify its benefits. CALCULATION: For 10 published randomized trials, we calculated the following: PTCA-related delay = median "door-to-balloon" time--median "door-to-needle" time Survival benefit = 30-day mortality after thrombolytic therapy--30-day mortality after PTCA The relationship between delay and benefit was assessed with linear regression. RESULTS: The reported PTCA-related delay ranged from 7 to 59 minutes, while the absolute survival benefit ranged from -2.2% (favoring thrombolytic therapy) to 7.4% (favoring PTCA). Across trials, the survival benefit decreased as the PTCA-related delay increased: For each additional 10-minute delay, the benefit was predicted to decrease 1.7% (P < 0.001). Linear regression showed that at a PTCA-related delay of 50 minutes, PTCA and thrombolytic therapy yielded equivalent reductions in mortality. CONCLUSIONS: In clinical trials with short PTCA-related delays, PTCA produced better outcomes, while trials with longer delays favored thrombolytic therapy. A more precise estimate of the time interval to equipoise between the two therapies needs to be modeled with patient-level data. At experienced cardiac centers, PTCA is probably still preferable, even with delays longer than 50 minutes.

Are randomized controlled trials sufficient evidence to guide clinical practice in type II (non-insulin-dependent) diabetes mellitus?

Randomized controlled trials (RCTs) are often considered the standard for defining the practice of evidence-based medicine. Taken alone, they are, however, often insufficient to guide clinical care. Randomized controlled trials are clearly the best method to determine whether interventions are efficacious. They have, however, numerous limitations which make them difficult to carry out or limit applicability to routine clinical practice. Although observational studies also have inherent limitations, they provide data which can help to further explain the results of randomized controlled trials. The use of observational studies to frame randomized trials can allow better application of randomized controlled trial results to individual patients and can thus help to optimize delivery of care, inform clinical practice and determine the need for further such trials.

The potential use of ECG-based prognostic instruments in clinical trials and cost-effectiveness analyses of new therapies in acute cardiac ischemia.

The dramatic improvements in outcomes in acute cardiac ischemia because of therapeutic advances has led to "diminishing returns" with increasingly intensive therapies. This article explores the potential of electrocardiograph (ECG)-based prognostic instruments to identify patients likely to benefit from intense regimens, even in the absence of overall average benefit in the population, with 2 clinical examples: 1) Reperfusion therapy in acute myocardial infarction (AMI); and 2) anticoagulation/antiplatelet therapy in unstable angina. Based on previously developed, ECG-based prognostic instruments we explored the distribution of potential benefits in individual patients from increasingly intense therapy in both AMI and unstable angina. Predictions were obtained on community-based patient samples with both AMI and unstable angina to examine the distribution of effectiveness and cost-effectiveness. For both AMI and unstable angina, much of the benefit of intensifying therapy can be obtained by targeting a subgroup of patients that can be identified in multivariable dimensions by clinical and ECG characteristics. Treatment of these patients with more potent agents (such as hirudin or the glycoprotein inhibitors in unstable angina) is likely to be both effective and cost-effective. However, treatment of "low benefit" patients is unlikely to be effective or cost-effective, and some candidates for therapy are more likely to be harmed, than to benefit, by the more intensive regimens. Multivariable stratification can improve clinical and economic outcomes in acute cardiac ischemia, particularly when such models help identify "high benefit" patients early in their clinical course. Additionally, using validated models in the planning and execution of clinical trials of new therapies can improve the power of the trial and help target the therapies to patients most likely to benefit.

Malignant lymphoma of the maxillary sinus.

Cancer of the paranasal sinuses and nasal cavity is an uncommon disease accounting for less than 0.2% of all malignancies. Lymphoma of the paranasal sinuses is a rare clinical entity representing less than 100 cases a year. In most cases, tumors arising in the paranasal sinuses are of the non-Hodgkin's variety. Most of these tumors are asymptomatic in the early stages and are diagnosed only after invasion of adjacent structures with local bone destruction. Computed tomography is essential for staging. The prognosis and treatment of lymphoma of the paranasal sinuses are based on the anatomic staging and histologic subtype. The prognosis is favorable for patients with localized disease. The authors describe a malignant lymphoma of the maxillary sinus in a 44-year-old man.