Addressing bias in prediction models by improving subpopulation calibration.

OBJECTIVE: To illustrate the problem of subpopulation miscalibration, to adapt an algorithm for recalibration of the predictions, and to validate its performance. MATERIALS AND METHODS: In this retrospective cohort study, we evaluated the calibration of predictions based on the Pooled Cohort Equations (PCE) and the fracture risk assessment tool (FRAX) in the overall population and in subpopulations defined by the intersection of age, sex, ethnicity, socioeconomic status, and immigration history. We next applied the recalibration algorithm and assessed the change in calibration metrics, including calibration-in-the-large. RESULTS: 1 021 041 patients were included in the PCE population, and 1 116 324 patients were included in the FRAX population. Baseline overall model calibration of the 2 tested models was good, but calibration in a substantial portion of the subpopulations was poor. After applying the algorithm, subpopulation calibration statistics were greatly improved, with the variance of the calibration-in-the-large values across all subpopulations reduced by 98.8% and 94.3% in the PCE and FRAX models, respectively. DISCUSSION: Prediction models in medicine are increasingly common. Calibration, the agreement between predicted and observed risks, is commonly poor for subpopulations that were underrepresented in the development set of the models, resulting in bias and reduced performance for these subpopulations. In this work, we empirically evaluated an adapted version of the fairness algorithm designed by Hebert-Johnson et al. (2017) and demonstrated its use in improving subpopulation miscalibration. CONCLUSION: A postprocessing and model-independent fairness algorithm for recalibration of predictive models greatly decreases the bias of subpopulation miscalibration and thus increases fairness and equality.

Automated opportunistic osteoporotic fracture risk assessment using computed tomography scans to aid in FRAX underutilization.

Methods for identifying patients at high risk for osteoporotic fractures, including dual-energy X-ray absorptiometry (DXA)1,2 and risk predictors like the Fracture Risk Assessment Tool (FRAX)3-6, are underutilized. We assessed the feasibility of automatic, opportunistic fracture risk evaluation based on routine abdomen or chest computed tomography (CT) scans. A CT-based predictor was created using three automatically generated bone imaging biomarkers (vertebral compression fractures (VCFs), simulated DXA T-scores and lumbar trabecular density) and CT metadata of age and sex. A cohort of 48,227 individuals (51.8% women) aged 50-90 with available CTs before 2012 (index date) were assessed for 5-year fracture risk using FRAX with no bone mineral density (BMD) input (FRAXnb) and the CT-based predictor. Predictions were compared to outcomes of major osteoporotic fractures and hip fractures during 2012-2017 (follow-up period). Compared with FRAXnb, the major osteoporotic fracture CT-based predictor presented better receiver operating characteristic area under curve (AUC), sensitivity and positive predictive value (PPV) (+1.9%, +2.4% and +0.7%, respectively). The AUC, sensitivity and PPV measures of the hip fracture CT-based predictor were noninferior to FRAXnb at a noninferiority margin of 1%. When FRAXnb inputs are not available, the initial evaluation of fracture risk can be done completely automatically based on a single abdomen or chest CT, which is often available for screening candidates7,8.