Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Evaluation of Less Invasive Sampling Tools for the Diagnosis of Cutaneous Leishmaniasis.

Background: Diagnosis of cutaneous leishmaniasis (CL) usually relies on invasive samples, but it is unclear whether more patient-friendly tools are good alternatives for diverse lesions when used with polymerase chain reaction (PCR). Methods: Patients with suspected CL were enrolled consecutively in a prospective diagnostic accuracy study. We compared dental broach, tape disc, and microbiopsy samples with PCR as index tests, using PCR with skin slit samples as reference test. Subsequently, we constructed a composite reference test including microscopy, the 3 index tests and skin slit PCR, and we compared these same tests with the composite reference test. We assessed diagnostic accuracy parameters with 95% confidence intervals for all comparisons. Results: Among 344 included patients, 282 (82.0%) had CL diagnosed, and 62 (18.0%) CL absence, by skin slit PCR. The sensitivity and specificity by PCR were 89.0% (95% confidence interval, 84.8%-92.1%) and 58.1% (45.7%-69.5%), respectively, for dental broach, 96.1% (93.2%-97.8%) and 27.4% (17.9%-39.6%) for tape disc, and 74.8% (66.3%-81.7%) and 72.7% (51.8%-86.8%) for microbiopsy. Several reference test-negative patients were consistently positive with the index tests. Using the composite reference test, dental broach, and skin slit had similar diagnostic performance. Discussion: Dental broach seems a less invasive but similarly accurate alternative to skin slit for diagnosing CL when using PCR. Tape discs lack specificity and seem unsuitable for CL diagnosis without cutoff. Reference tests for CL are problematic, since using a single reference test is likely to miss true cases, while composite reference tests are often biased and impractical as they require multiple tests.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique. STUDY DESIGN AND SETTING: We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting.

OBJECTIVES: To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes. STUDY DESIGN AND SETTING: This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation. MAIN OUTCOME MEASURE: In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting. RESULTS: All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71. CONCLUSION: Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.