Weighted metrics are required when evaluating the performance of prediction models in nested case-control studies.

BACKGROUND: Nested case-control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data. METHODS: We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design. RESULTS: Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58-0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62-0.69) and 0.65 (0.61-0.69), respectively. The unweighted O/E ratio was 18.38 (17.67-19.06) in the NCC datasets, while it was 1.69 (1.42-1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53-1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment. CONCLUSIONS: Nested case-control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure.

From decision to reflection: understanding the experiences and unmet care needs of patients treated with immunotherapy for melanoma in the adjuvant or metastatic setting.

BACKGROUND: Despite increased use of immune checkpoint inhibitors (ICIs) in patients with advanced melanoma, little is known about patient experiences during this treatment. This study aimed to gain an in-depth understanding of experiences and unmet care needs of patients treated in the adjuvant or metastatic setting for advanced melanoma regarding their ICI treatment trajectory. METHODS: Interviews and focus groups were conducted among 35 patients treated with ICIs in the adjuvant setting for completely resected stage III (n = 14), or in the metastatic setting for irresectable stage IV (n = 21) melanoma. A thorough thematic content analysis was conducted. RESULTS: Three main themes were identified. When (1) dealing with uncertainty in the decision-making process, adjuvant patients explored the pros and cons, whereas metastatic patients considered immunotherapy their only viable option. Both groups expressed the need for additional guidance. In (2) navigating the immunotherapy course, both perceived the trajectory as intense, experienced a major impact on their and their (close) relatives' lives, and felt the need to (re)gain control. When (3) looking back on the immunotherapy experience, metastatic patients generally felt relieved, while among adjuvant patients, feelings of doubt regarding their choice for ICIs were also reported. CONCLUSIONS: ICI treatment is perceived as intensive for both patient groups, facing both comparable and distinct challenges throughout the treatment trajectory, underscoring the need for stage-specific, individualised guidance. Options regarding flexible follow-ups, low-threshold contact and psychosocial support throughout the treatment trajectory should be explored.

Predicting skin cancer risk from facial images with an explainable artificial intelligence (XAI) based approach: a proof-of-concept study.

Background: Efficient identification of individuals at high risk of skin cancer is crucial for implementing personalized screening strategies and subsequent care. While Artificial Intelligence holds promising potential for predictive analysis using image data, its application for skin cancer risk prediction utilizing facial images remains unexplored. We present a neural network-based explainable artificial intelligence (XAI) approach for skin cancer risk prediction based on 2D facial images and compare its efficacy to 18 established skin cancer risk factors using data from the Rotterdam Study. Methods: The study employed data from the Rotterdam population-based study in which both skin cancer risk factors and 2D facial images and the occurrence of skin cancer were collected from 2010 to 2018. We conducted a deep-learning survival analysis based on 2D facial images using our developed XAI approach. We subsequently compared these results with survival analysis based on skin cancer risk factors using cox proportional hazard regression. Findings: Among the 2810 participants (mean Age = 68.5 ± 9.3 years, average Follow-up = 5.0 years), 228 participants were diagnosed with skin cancer after photo acquisition. Our XAI approach achieved superior predictive accuracy based on 2D facial images (c-index = 0.72, 95% CI: 0.70-0.74), outperforming that of the known risk factors (c-index = 0.59, 95% CI 0.57-0.61). Interpretation: This proof-of-concept study underscores the high potential of harnessing facial images and a tailored XAI approach as an easily accessible alternative over known risk factors for identifying individuals at high risk of skin cancer. Funding: The Rotterdam Study is funded through unrestricted research grants from Erasmus Medical Center and Erasmus University, Rotterdam, Netherlands Organization for the Health Research and Development (ZonMw), the Research Institute for Diseases in the Elderly (RIDE), the Ministry of Education, Culture and Science, the Ministry for Health, Welfare and Sports, the European Commission (DG XII), and the Municipality of Rotterdam. G.V. Roshchupkin is supported by the ZonMw Veni grant (Veni, 549 1936320).