Performance metrics for models designed to predict treatment effect.

BACKGROUND: Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs). METHODS: Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest. RESULTS: As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics". CONCLUSION: The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Development and validation of a nomogram to predict recurrence and melanoma-specific mortality in patients with negative sentinel lymph nodes.

BACKGROUND: Patients with melanoma and negative sentinel nodes (SNs) have varying outcomes, dependent on several prognostic factors. Considering all these factors in a prediction model might aid in identifying patients who could benefit from a personalized treatment strategy. The objective was to construct and validate a nomogram for recurrence and melanoma-specific mortality (MSM) in patients with melanoma and negative SNs. METHODS: A total of 3220 patients with negative SNs were identified from a cohort of 4124 patients from four EORTC Melanoma Group centres who underwent sentinel lymph node biopsy. Prognostic factors for recurrence and MSM were studied with Cox regression analysis. Significant factors were incorporated in the models. Performance was assessed by discrimination (c-index) and calibration in cross-validation across the four centres. A nomogram was developed for graphical presentation. RESULTS: There were 3180 eligible patients. The final prediction model for recurrence and the calibrated model for MSM included three independent prognostic factors: ulceration, anatomical location and Breslow thickness. The c-index was 0·74 for recurrence and 0·76 for the calibrated MSM model. Cross-validation across the four centres showed reasonable model performance. A nomogram was developed based on these models. One-third of the patients had a 5-year recurrence probability of 8·2 per cent or less, and one-third had a recurrence probability of 23·0 per cent or more. CONCLUSION: A nomogram for predicting recurrence and MSM in patients with melanoma and negative SNs was constructed and validated. It could provide personalized estimates useful for tailoring surveillance strategies (reduce or increase intensity), and selection of patients for adjuvant therapy or clinical trials.

Prediction of survival in patients with oesophageal or junctional cancer receiving neoadjuvant chemoradiotherapy and surgery.

BACKGROUND: The value of conventional prognostic factors is unclear in the era of multimodal treatment for oesophageal cancer. This study aimed to quantify the impact of neoadjuvant chemoradiotherapy (nCRT) and surgery on well established prognostic factors, and to develop and validate a prognostic model. METHODS: Patients treated with nCRT plus surgery were included. Multivariable Cox modelling was used to identify prognostic factors for overall survival. A prediction model for individual survival was developed using stepwise backward selection. The model was internally validated leading to a nomogram for use in clinical practice. RESULTS: Some 626 patients who underwent nCRT plus surgery were included. In the multivariable model, only pretreatment cN category and ypN category were independent prognostic factors. The final prognostic model included cN, ypT and ypN categories, and had moderate discrimination (c-index at internal validation 0·63). CONCLUSION: In patients with oesophageal or oesophagogastric cancer treated with nCRT plus surgery, overall survival can best be estimated using a prediction model based on cN, ypT and ypN categories. Predicted survival according to this model showed only moderate correlation with observed survival, emphasizing the need for new prognostic factors to improve survival prediction.

Daily practice in guideline adherence to adjuvant chemotherapy in stage III colon cancer and predictors of outcome.

INTRODUCTION: Although guidelines recommend adjuvant chemotherapy for stage III colon cancer patients, many patients do not receive adjuvant chemotherapy. The aim of this study was to identify reasons for guideline non-adherence and assess the effect on patient outcomes in a multicenter cohort of stage III colon cancer patients who received surgery plus adjuvant chemotherapy or surgery alone. METHODS: Patients who underwent surgery between 2007 and 2017 were included. Reasons for non-adherence were determined. Propensity score analyses with inverse probability weighting were performed to adjust for confounding factors. Cox proportional hazards regression and risk stratified analyses were performed to assess the association of guideline adherence and other potential predictors with recurrence free survival (RFS). RESULTS: Data of 575 patients were included of whom 61% received adjuvant chemotherapy. In 87 of 222 patients (39%) who did not receive adjuvant chemotherapy, no reason was documented. Only age was predictive for receiving chemotherapy. Patients who received adjuvant chemotherapy had longer RFS (HR 0.42, 95%CI 0.29-0.62, p < 0.001). High T- and N-stage were associated with poorer RFS HR 2.0 (95%CI 1.58-2.71, p < 0.001) and HR 2.19 (95%CI 1.60-2.99, p < 0.001) respectively. Risk groups were identified with distinct prognosis and treatment effect and a nomogram is presented to visualize individualized RFS differences. CONCLUSION: This study shows considerable variation in guideline adherence to adjuvant chemotherapy and poor documentation on reasons for non-adherence. Optimizing adherence and gaining insight in reasons for non-adherence is advocated as this can lead to significant RFS benefit, especially in patients with high T-and N-stage tumors.

Reduced increase of calcium scores using breath-hold in left-sided whole breast irradiation.

PURPOSE: In this prospective longitudinal study, Coronary Artery Calcium (CAC) scores determined before the start of whole breast irradiation were compared with those determined 7 years afterwards. The aim was to examine whether the use of a breath-hold (BH) technique is associated with less increase of CAC scores. METHODS AND MATERIALS: Changes in CAC scores were analysed in 87 breast cancer patients. The results of the following groups were compared: patients receiving right (R) or left-sided radiotherapy using free breathing (L-FB) with those receiving left-sided radiotherapy with BH (L-BH). We compared the changes of CAC scores between these groups over time, testing the hypothesis that a significantly reduced increase of calcium scores is observed when using BH. RESULTS: For L-BH cases, when compared with L-FB cases, for overall as well as for Left Anterior Descending coronary artery (LAD) CAC scores, we noted significantly less increased CAC scores (p < 0.01). This effect of BH was even more striking in the group with CAC scores >0 at baseline. The attenuated increase over time of CAC scores in the L-BH group was robust to correction for age and statin use (p < 0.05). CONCLUSION: After a median follow-up of 7.4 years, we found significantly less increased CAC scores when using BH. This is a relevant finding since higher levels of CAC scores are associated with higher probabilities of coronary artery events. Moreover, it underlines the rationale for the use of BH in left-sided whole breast irradiation.

The value of tumor-stroma ratio as predictor of pathologic response after neoadjuvant chemoradiotherapy in esophageal cancer.

BACKGROUND AND PURPOSE: With currently available techniques, the prediction of pathologic complete response after neoadjuvant chemoradiotherapy is insufficient. The tumor-stroma ratio (TSR) has proven to be a predictor of survival for several types of cancer, including esophageal. The aim of this study was to investigate the value of TSR in predicting pathologic response after neoadjuvant chemoradiotherapy in esophageal cancer patients. MATERIALS AND METHODS: Patients with esophageal adenocarcinoma or squamous cell carcinoma who received neoadjuvant chemoradiotherapy followed by a resection were selected. Haematoxylin and eosin (H&E) stained sections of diagnostic biopsies were collected and TSR was independently assessed by two investigators. Patients were categorized in stroma-low (≤50% stroma) and stroma-high (>50% stroma) groups for further analyses. The tumor regression grade (TRG) was assessed on H&E stained sections of the resected primary tumor to determine pathologic response. RESULTS: A total of 94 patients were included in this study, of which 76 patients were categorized as stroma-low and 18 as stroma-high. Forty-two (45%) patients had a major pathologic response (TRG 1-2), whereas 52 (55%) were considered non-responders. After adjustment for gender, tumor type, cT-status and differentiation grade, patients with a stroma-high tumor showed a higher chance of no response compared to patients with a stroma-low tumor (OR 3.57, 95%CI 1.03-12.31, P = 0.04). CONCLUSION: TSR showed to have the potential to aid in the prediction of pathologic response in esophageal cancer patients receiving neoadjuvant chemoradiotherapy. Larger validation studies are necessary before implementing this method in daily practice.