Calibration plots for multistate risk predictions models.

INTRODUCTION: There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. METHODS: We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. RESULTS: The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability. CONCLUSIONS: We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

Prognostic factors associated with failure of total elbow arthroplasty.

Aims: The aims of this study were to identify and evaluate the current literature examining the prognostic factors which are associated with failure of total elbow arthroplasty (TEA). Methods: Electronic literature searches were conducted using MEDLINE, Embase, PubMed, and Cochrane. All studies reporting prognostic estimates for factors associated with the revision of a primary TEA were included. The risk of bias was assessed using the Quality In Prognosis Studies (QUIPS) tool, and the quality of evidence was assessed using the modified Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) framework. Due to low quality of the evidence and the heterogeneous nature of the studies, a narrative synthesis was used. Results: A total of 19 studies met the inclusion criteria, investigating 28 possible prognostic factors. Most QUIPS domains (84%) were rated as moderate to high risk of bias. The quality of the evidence was low or very low for all prognostic factors. In low-quality evidence, prognostic factors with consistent associations with failure of TEA in more than one study were: the sequelae of trauma leading to TEA, either independently or combined with acute trauma, and male sex. Several other studies investigating sex reported no association. The evidence for other factors was of very low quality and mostly involved exploratory studies. Conclusion: The current evidence investigating the prognostic factors associated with failure of TEA is of low or very low quality, and studies generally have a moderate to high risk of bias. Prognostic factors are subject to uncertainty, should be interpreted with caution, and are of little clinical value. Higher-quality evidence is required to determine robust prognostic factors for failure of TEA.

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration.

OBJECTIVES: In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models. STUDY DESIGN AND SETTING: A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index). RESULTS: The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence. CONCLUSION: We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review.

BACKGROUND: Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome. METHODS: We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size. RESULTS: A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84). CONCLUSIONS: Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Prognostic factors associated with failure of total elbow replacement: a protocol for a systematic review.

INTRODUCTION: Total elbow replacement (TER) has higher failure rates requiring revision surgery compared with the replacement of other joints. Understanding the factors associated with failure is essential for informed decision-making between patients and clinicians, and for reducing the failure rate. This review aims to identify, describe and appraise the literature examining prognostic factors for failure of TER. METHODS AND ANALYSIS: This systematic review will be conducted and reported in line with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines. Electronic literature searches will be conducted using Medline, EMBASE, PubMed and Cochrane. The search strategy will be broad, including a combination of subject headings (MESH) and free text search. This search will be supplemented with a screening of reference lists of the included studies and relevant reviews. Two independent reviewers will screen all search results in two stages (title and abstract, and full text) based on the Population, Index prognostic factor, Comparator prognostic factor, Outcome, Time and Setting criteria. The types of evidence included will be randomised trials, non-randomised trials, prospective and retrospective cohort studies, registry studies and case-control studies. If the literature lacks enough studies, then case series with 50 or more TERs will be considered for inclusion. Data extraction and risk of bias assessment for included studies will be performed by two independent reviewers using the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies for Prognostic Factors and Quality In Prognostic Studies tools.Meta-analyses of prognostic estimates for each factor will be undertaken for studies that are deemed to be sufficiently robust and comparable. Several challenges are likely to arise due to heterogeneity between studies, therefore, subgroup and sensitivity analyses will be performed to account for the differences between studies. Heterogeneity will be assessed using Q and I2 statistics. If I2>40% then pooled estimates will not be reported. When quantitative synthesis is not possible, a narrative synthesis will be undertaken. The quality of the evidence for each prognostic factor will be assessed using the Grades of Recommendation Assessment, Development and Evaluation tool. PROSPERO REGISTRATION NUMBER: CRD42023384756.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques.

INTRODUCTION: This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis. METHODS: We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring. RESULTS: Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors. DISCUSSION: We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

Day-to-day variability of knee pain and the relationship with physical activity in people with knee osteoarthritis: an observational, feasibility study using consumer smartwatches.

OBJECTIVE: To assess the feasibility of using smartwatches in people with knee osteoarthritis (OA) to determine the day-to-day variability of pain and the relationship between daily pain and step count. DESIGN: Observational, feasibility study. SETTING: In July 2017, the study was advertised in newspapers, magazines and, on social media. Participants had to be living/willing to travel to Manchester. Recruitment was in September 2017 and data collection was completed in January 2018. PARTICIPANTS: 26 participants aged>50 years with self-diagnosed symptomatic knee OA were recruited. OUTCOME MEASURES: Participants were provided with a consumer cellular smartwatch with a bespoke app that triggered a series of daily questions including two times per day questions about level of knee pain and one time per month question from the pain subscale of the Knee Injury and Osteoarthritis Outcome Score (KOOS) questionnaire. The smartwatch also recorded daily step counts. RESULTS: Of the 25 participants, 13 were men and their mean age was 65 years (standard deviation (SD) 8 years). The smartwatch app was successful in simultaneously assessing and recording data on knee pain and step count in real time. Knee pain was categorised into sustained high/low or fluctuating levels, but there was considerable day-to-day variation within these categories. Levels of knee pain in general correlated with pain assessed by KOOS. Those with sustained high/low levels of pain had a similar daily step count average (mean 3754 (SD 2524)/4307 (SD 2992)), but those with fluctuating pain had much lower step count levels (mean 2064 (SD 1716)). CONCLUSIONS: Smartwatches can be used to assess pain and physical activity in knee OA. Larger studies may help inform a better understanding of causal links between physical activity patterns and pain. In time, this could inform development of personalised physical activity recommendations for people with knee OA.

Serum HE4 predicts progestin treatment response in endometrial cancer and atypical hyperplasia: A prognostic study.

OBJECTIVE: To investigate serum human epididymis-4 (HE4) as a predictive biomarker of intrauterine progestin response in endometrial cancer and atypical endometrial hyperplasia (AEH). DESIGN: Prospective prognostic factor study. SETTING: Consecutive sample of women attending a tertiary gynaecological oncology centre in northwest England. POPULATION: Women with AEH or early-stage, low-grade endometrial cancer who were unfit for or declined primary surgical management. METHODS: A total of 76 women, 32 with AEH and 44 with endometrial cancer, were treated with a levonorgestrel intrauterine system (LNG-IUS) for 12 months. Endometrial biopsies and imaging were performed to assess treatment response. Pretreatment serum HE4 was analysed by chemiluminescence immunoassay and diagnostic accuracy and logistic regression analyses were performed. MAIN OUTCOME MEASURES: Progestin response at 12 months defined by histology and imaging. RESULTS: The median age and body mass index (BMI) of the final cohort were 52 years (interquartile range [IQR] 33-62 years) and 46 kg/m2 (IQR 38-54 kg/m2 ), respectively. Baseline serum HE4 was significantly higher in non-responders than responders (119.2 pmol/L, IQR 94.0-208.4 pmol/L versus 71.8 pmol/L, IQR 56.1-84.2 pmol/L, p < 0.001). Older age (odds ratio [OR] 0.96, 95% CI 0.93-0.99, p = 0.02), baseline serum HE4 (OR 0.97, 95% CI 0.96-0.99, p = 0.001) and endometrial cancer histology (OR 0.22, 95% CI 0.72-0.68, p = 0.009) were associated with a lower likelihood of progestin treatment response. Serum HE4 remained independently associated with progestin treatment failure when adjusted for age and histology (adjusted hazard ratio 0.97, 95% CI 0.96-0.99, p = 0.008). CONCLUSION: Serum HE4 shows promise as a predictive biomarker of progestin treatment response in endometrial cancer and AEH.

Can prognostic factors for indirect muscle injuries in elite football (soccer) players be identified using data from preseason screening? An exploratory analysis using routinely collected periodic health examination records.

BACKGROUND: In elite football, periodic health examination (PHE) may be useful for injury risk prediction. OBJECTIVE: To explore whether PHE-derived variables are prognostic factors for indirect muscle injuries (IMIs) in elite players. DESIGN: Retrospective cohort study. SETTING: An English Premier League football club. PARTICIPANTS: 134 outfield elite male players, over 5 seasons (1 July 2013-19 May 2018). OUTCOME AND ANALYSIS: The outcome was any time-loss, lower extremity index IMI (I-IMI). Prognostic associations were estimated using odds ratios (ORs) and corresponding statistical significance for 36 variables, derived from univariable and multivariable logistic regression models. Missing data were handled using multiple imputation. Non-linear associations were explored using fractional polynomials. RESULTS: During 317 participant-seasons, 138 I-IMIs were recorded. Univariable associations were determined for previous calf IMI frequency (OR 1.80, 95% CI 1.09 to 2.97), hamstring IMI frequency (OR 1.56, 95% CI 1.17 to 2.09), if the most recent hamstring IMI occurred >12 months but <3 years prior to PHE (OR 2.95, 95% CI 1.51 to 5.73) and age (OR 1.12 per 1-year increase, 95% CI 1.06 to 1.18). Multivariable analyses showed that if a player's most recent previous hamstring IMI was >12 months but <3 years prior to PHE (OR 2.24, 95% CI 1.11 to 4.53), this was the only variable with added prognostic value over and above age, which was a confirmed prognostic factor (OR 1.12 per 1-year increase, 95% CI 1.05 to 1.18). Allowing non-linear associations conferred no advantage over linear associations. CONCLUSION: PHE has limited use for injury risk prediction. Most variables did not add prognostic value over and above age, other than if a player experienced a hamstring IMI >12 months but <3 years prior to PHE. However, the precision of this prognostic association should be confirmed in future. TRIAL REGISTRATION NUMBER: NCT03782389.

Association between psoriatic disease and lifestyle factors and comorbidities: cross-sectional analysis and Mendelian randomization.

OBJECTIVES: To examine associations between PsA and psoriasis vs lifestyle factors and comorbidities by triangulating observational and genetic evidence. METHODS: We analysed cross-sectional data from the UK Biobank (1836 PsA, 8995 psoriasis, 36 000 controls) to describe the association between psoriatic disease and lifestyle factors (including BMI and smoking) and 15 comorbidities [including diabetes and coronary artery disease (CAD)] using logistic models adjusted for age, sex and lifestyle factors. We applied bidirectional Mendelian randomization (MR) to genome-wide association data (3609 PsA and 7804 psoriasis cases, up to 1.2 million individuals for lifestyle factors and 757 601 for comorbidities) to examine causal direction, using the inverse-variance weighted method. RESULTS: BMI was cross-sectionally associated with risk of PsA (OR 1.31 per 5 kg/m2 increase; 95% CI 1.26, 1.37) and psoriasis (OR 1.23; 1.20, 1.26), with consistent MR estimates (PsA OR 1.38; 1.14, 1.67; psoriasis OR 1.36; 1.18, 1.58). In both designs, smoking was more strongly associated with psoriasis than PsA. PsA and psoriasis were cross-sectionally associated with diabetes (OR 1.35 and 1.39, respectively) and CAD (OR 1.56 and 1.38, respective). Genetically predicted glycated haemoglobin (surrogate for diabetes) increased PsA risk (OR 1.18 per 6.7 mmol/mol increase; 1.02, 1.36) but not psoriasis. Genetic liability to PsA (OR 1.05; 1.003, 1.09) and psoriasis (OR 1.03; 1.001, 1.06) were associated with increased risk of CAD. CONCLUSION: Observational and genetic evidence converge to suggest that BMI and glycaemic control are associated with increased psoriatic disease risk, while psoriatic disease is associated with increased CAD risk. Further research is needed to understand the mechanism of these associations.

Clinical prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: A systematic review and meta-analysis.

BACKGROUND: In the management of rheumatoid arthritis (RA), there is a clinical need to identify which patients are at high-risk of not responding to methotrexate (MTX), or experiencing adverse events (AEs), to enable earlier alternative treatments. Many clinical prediction models (CPMs) have previously been developed, but a summary of such models and their methodological quality is lacking. This systematic review aimed to (i) identify and summarize previously published CPMs of MTX outcomes in biologic-naïve RA patients, and (ii) critically appraise their methodological properties. METHODS: Medline and Embase were searched to identify studies developing or validating CPMs of MTX outcomes in RA patients. The risk of bias (ROB) was assessed using PROBAST (prediction model risk of bias assessment tool). A fixed effects meta-analysis summarised discrimination for models with multiple external validations. RESULTS: The systematic review identified 20 CPMs across 13 studies, and 4 validation studies. Three outcome types were used: a state of disease activity (n = 14, 70%); EULAR response criteria (n = 4, 20%); or discontinuation due to AEs (n = 2, 10%). Only one model accounted for potential competing risks, and nine (45%) were internally validated. Eight (40%) models used multiple imputation for missing data, others were often limited to complete case analysis. There was overall high ROB. The meta-analysis summarised c-statistics for two models with multiple external validations was 0.77 (95% CI: 0.69, 0.84) and 0.68 (0.64, 0.71). CONCLUSION: This review highlights several methodological shortcomings that should be addressed in future model development to increase potential for implementation into practice.

Heterogeneity in the association between weather and pain severity among patients with chronic pain: a Bayesian multilevel regression analysis.

INTRODUCTION: Previous studies on the association between weather and pain severity among patients with chronic pain have produced mixed results. In part, this inconsistency may be due to differences in individual pain responses to the weather. METHODS: To test the hypothesis that there might be subgroups of participants with different pain responses to different weather conditions, we examined data from a longitudinal smartphone-based study, Cloudy with a Chance of Pain, conducted between January 2016 and April 2017. The study recruited more than 13,000 participants and recorded daily pain severity on a 5-point scale (range: no pain to very severe pain) along with hourly local weather data for up to 15 months. We used a Bayesian multilevel model to examine the weather-pain association. RESULTS: We found 1 in 10 patients with chronic pain were sensitive to the temperature, 1 in 25 to relative humidity, 1 in 50 to pressure, and 3 in 100 to wind speed, after adjusting for age, sex, belief in the weather-pain association, mood, and activity level. The direction of the weather-pain association differed between people. Although participants seem to be differentially sensitive to weather conditions, there is no definite indication that participants' underlying pain conditions play a role in weather sensitivity. CONCLUSION: This study demonstrated that weather sensitivity among patients with chronic pain is more apparent in some subgroups of participants. In addition, among those sensitive to the weather, the direction of the weather-pain association can differ.

Understanding the Predictors of Missing Location Data to Inform Smartphone Study Design: Observational Study.

BACKGROUND: Smartphone location data can be used for observational health studies (to determine participant exposure or behavior) or to deliver a location-based health intervention. However, missing location data are more common when using smartphones compared to when using research-grade location trackers. Missing location data can affect study validity and intervention safety. OBJECTIVE: The objective of this study was to investigate the distribution of missing location data and its predictors to inform design, analysis, and interpretation of future smartphone (observational and interventional) studies. METHODS: We analyzed hourly smartphone location data collected from 9665 research participants on 488,400 participant days in a national smartphone study investigating the association between weather conditions and chronic pain in the United Kingdom. We used a generalized mixed-effects linear model with logistic regression to identify whether a successfully recorded geolocation was associated with the time of day, participants' time in study, operating system, time since previous survey completion, participant age, sex, and weather sensitivity. RESULTS: For most participants, the app collected a median of 2 out of a maximum of 24 locations (1760/9665, 18.2% of participants), no location data (1664/9665, 17.2%), or complete location data (1575/9665, 16.3%). The median locations per day differed by the operating system: participants with an Android phone most often had complete data (a median of 24/24 locations) whereas iPhone users most often had a median of 2 out of 24 locations. The odds of a successfully recorded location for Android phones were 22.91 times higher than those for iPhones (95% CI 19.53-26.87). The odds of a successfully recorded location were lower during weekends (odds ratio [OR] 0.94, 95% CI 0.94-0.95) and nights (OR 0.37, 95% CI 0.37-0.38), if time in study was longer (OR 0.99 per additional day in study, 95% CI 0.99-1.00), and if a participant had not used the app recently (OR 0.96 per additional day since last survey entry, 95% CI 0.96-0.96). Participant age and sex did not predict missing location data. CONCLUSIONS: The predictors of missing location data reported in our study could inform app settings and user instructions for future smartphone (observational and interventional) studies. These predictors have implications for analysis methods to deal with missing location data, such as imputation of missing values or case-only analysis. Health studies using smartphones for data collection should assess context-specific consequences of high missing data, especially among iPhone users, during the night and for disengaged participants.

Clinical Prediction Models in Sports Medicine: A Guide for Clinicians and Researchers.

SYNOPSIS: Participating in sport carries inherent risk of injury. Clinicians execute high-level clinical reasoning and decision making to support athletes to achieve the best outcomes. Accurately diagnosing a problem, estimating prognosis, or selecting the most suitable intervention for each athlete is challenging. Clinical prediction models are tools to assist clinicians in estimating the risk or probability of a health outcome for an individual by using data from multiple predictors. Although common in general medical literature, clinical prediction models are rare in sports medicine. The purpose of this article was to (1) describe the steps required to develop and validate (ie, evaluate) a clinical prediction model for clinical researchers, and (2) help sports medicine clinicians understand and interpret clinical prediction model studies. Using a case study to illustrate how to implement clinical prediction models in practice, we address the following issues in developing and validating a clinical prediction model: study design and data, sample size, missing data, selecting predictors, handling continuous predictors, model fitting, internal and external validation, performance measures, reporting, and model presentation. Our work builds on initiatives to improve diagnostic and prognostic clinical research, including the PROGnosis RESearch Strategy (PROGRESS) series of papers and textbook and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. J Orthop Sports Phys Ther 2021;51(10):517-525. doi:10.2519/jospt.2021.10697.

Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods.

BACKGROUND: Clinical prediction models (CPMs) predict the risk of health outcomes for individual patients. The majority of existing CPMs only harness cross-sectional patient information. Incorporating repeated measurements, such as those stored in electronic health records, into CPMs may provide an opportunity to enhance their performance. However, the number and complexity of methodological approaches available could make it difficult for researchers to explore this opportunity. Our objective was to review the literature and summarise existing approaches for harnessing repeated measurements of predictor variables in CPMs, primarily to make this field more accessible for applied researchers. METHODS: MEDLINE, Embase and Web of Science were searched for articles reporting the development of a multivariable CPM for individual-level prediction of future binary or time-to-event outcomes and modelling repeated measurements of at least one predictor. Information was extracted on the following: the methodology used, its specific aim, reported advantages and limitations, and software available to apply the method. RESULTS: The search revealed 217 relevant articles. Seven methodological frameworks were identified: time-dependent covariate modelling, generalised estimating equations, landmark analysis, two-stage modelling, joint-modelling, trajectory classification and machine learning. Each of these frameworks satisfies at least one of three aims: to better represent the predictor-outcome relationship over time, to infer a covariate value at a pre-specified time and to account for the effect of covariate change. CONCLUSIONS: The applicability of identified methods depends on the motivation for including longitudinal information and the method's compatibility with the clinical context and available patient data, for both model development and risk estimation in practice.

A review of the use of propensity score diagnostics in papers published in high-ranking medical journals.

BACKGROUND: Propensity scores are widely used to deal with confounding bias in medical research. An incorrectly specified propensity score model may lead to residual confounding bias; therefore it is essential to use diagnostics to assess propensity scores in a propensity score analysis. The current use of propensity score diagnostics in the medical literature is unknown. The objectives of this study are to (1) assess the use of propensity score diagnostics in medical studies published in high-ranking journals, and (2) assess whether the use of propensity score diagnostics differs between studies (a) in different research areas and (b) using different propensity score methods. METHODS: A PubMed search identified studies published in high-impact journals between Jan 1st 2014 and Dec 31st 2016 using propensity scores to answer an applied medical question. From each study we extracted information regarding how propensity scores were assessed and which propensity score method was used. Research area was defined using the journal categories from the Journal Citations Report. RESULTS: A total of 894 papers were included in the review. Of these, 187 (20.9%) failed to report whether the propensity score had been assessed. Commonly reported diagnostics were p-values from hypothesis tests (36.6%) and the standardised mean difference (34.6%). Statistical tests provided marginally stronger evidence for a difference in diagnostic use between studies in different research areas (p = 0.033) than studies using different propensity score methods (p = 0.061). CONCLUSIONS: The use of diagnostics in the propensity score medical literature is far from optimal, with different diagnostics preferred in different areas of medicine. The propensity score literature may improve with focused efforts to change practice in areas where suboptimal practice is most common.

The Value of Preseason Screening for Injury Prediction: The Development and Internal Validation of a Multivariable Prognostic Model to Predict Indirect Muscle Injury Risk in Elite Football (Soccer) Players.

BACKGROUND: In elite football (soccer), periodic health examination (PHE) could provide prognostic factors to predict injury risk. OBJECTIVE: To develop and internally validate a prognostic model to predict individualised indirect (non-contact) muscle injury (IMI) risk during a season in elite footballers, only using PHE-derived candidate prognostic factors. METHODS: Routinely collected preseason PHE and injury data were used from 152 players over 5 seasons (1st July 2013 to 19th May 2018). Ten candidate prognostic factors (12 parameters) were included in model development. Multiple imputation was used to handle missing values. The outcome was any time-loss, index indirect muscle injury (I-IMI) affecting the lower extremity. A full logistic regression model was fitted, and a parsimonious model developed using backward-selection to remove factors that exceeded a threshold that was equivalent to Akaike's Information Criterion (alpha 0.157). Predictive performance was assessed through calibration, discrimination and decision-curve analysis, averaged across all imputed datasets. The model was internally validated using bootstrapping and adjusted for overfitting. RESULTS: During 317 participant-seasons, 138 I-IMIs were recorded. The parsimonious model included only age and frequency of previous IMIs; apparent calibration was perfect, but discrimination was modest (C-index = 0.641, 95% confidence interval (CI) = 0.580 to 0.703), with clinical utility evident between risk thresholds of 37-71%. After validation and overfitting adjustment, performance deteriorated (C-index = 0.589 (95% CI = 0.528 to 0.651); calibration-in-the-large = - 0.009 (95% CI = - 0.239 to 0.239); calibration slope = 0.718 (95% CI = 0.275 to 1.161)). CONCLUSION: The selected PHE data were insufficient prognostic factors from which to develop a useful model for predicting IMI risk in elite footballers. Further research should prioritise identifying novel prognostic factors to improve future risk prediction models in this field. TRIAL REGISTRATION: NCT03782389.

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

OBJECTIVE: To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease. DESIGN: Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group. DATA SOURCES: PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020. STUDY SELECTION: Studies that developed or validated a multivariable covid-19 related prediction model. DATA EXTRACTION: At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool). RESULTS: 37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models. CONCLUSION: Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline. SYSTEMATIC REVIEW REGISTRATION: Protocol https://osf.io/ehc47/, registration https://osf.io/wy245. READERS' NOTE: This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.