Clinical prediction models and the multiverse of madness.

BACKGROUND: Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice. MAIN BODY: We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice. CONCLUSIONS: Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Prediction Models for Bronchopulmonary Dysplasia in Preterm Infants: A Systematic Review and Meta-Analysis.

OBJECTIVE: To review systematically and assess the accuracy of prediction models for bronchopulmonary dysplasia (BPD) at 36 weeks of postmenstrual age. STUDY DESIGN: Searches were conducted in MEDLINE and EMBASE. Studies published between 1990 and 2022 were included if they developed or validated a prediction model for BPD or the combined outcome death/BPD at 36 weeks in the first 14 days of life in infants born preterm. Data were extracted independently by 2 authors following the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (ie, CHARMS) and PRISMA guidelines. Risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (ie, PROBAST). RESULTS: Sixty-five studies were reviewed, including 158 development and 108 externally validated models. Median c-statistic of 0.84 (range 0.43-1.00) was reported at model development, and 0.77 (range 0.41-0.97) at external validation. All models were rated at high risk of bias, due to limitations in the analysis part. Meta-analysis of the validated models revealed increased c-statistics after the first week of life for both the BPD and death/BPD outcome. CONCLUSIONS: Although BPD prediction models perform satisfactorily, they were all at high risk of bias. Methodologic improvement and complete reporting are needed before they can be considered for use in clinical practice. Future research should aim to validate and update existing models.

Multiple cereblon genetic changes are associated with acquired resistance to lenalidomide or pomalidomide in multiple myeloma.

Emergence of drug resistance to all available therapies is the major challenge to improving survival in myeloma. Cereblon (CRBN) is the essential binding protein of the widely used immunomodulatory drugs (IMiDs) and novel CRBN E3 ligase modulator drugs (CELMoDs) in myeloma, as well as certain proteolysis targeting chimeras (PROTACs), in development for a range of diseases. Using whole-genome sequencing (WGS) data from 455 patients and RNA sequencing (RNASeq) data from 655 patients, including newly diagnosed (WGS, n = 198; RNASeq, n = 437), lenalidomide (LEN)-refractory (WGS, n = 203; RNASeq, n = 176), and pomalidomide (POM)-refractory cohorts (WGS, n = 54; RNASeq, n = 42), we found incremental increases in the frequency of 3 CRBN aberrations, namely point mutations, copy losses/structural variations, and a specific variant transcript (exon 10 spliced), with progressive IMiD exposure, until almost one-third of patients had CBRN alterations by the time they were POM refractory. We found all 3 CRBN aberrations were associated with inferior outcomes to POM in those already refractory to LEN, including those with gene copy losses and structural variations, a finding not previously described. This represents the first comprehensive analysis and largest data set of CBRN alterations in myeloma patients as they progress through therapy. It will help inform patient selection for sequential therapies with CRBN-targeting drugs.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review.

BACKGROUND: Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome. METHODS: We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size. RESULTS: A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84). CONCLUSIONS: Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review.

BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review.

BACKGROUND: While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. METHODS: We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item. RESULTS: Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). CONCLUSION: Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. SYSTEMATIC REVIEW REGISTRATION: PROSPERO, CRD42019161764.

Utility of pre-operative haemoglobin concentration to guide peri-operative blood tests for hip and knee arthroplasty: A decision curve analysis.

OBJECTIVE: Assess the prognostic value of pre-operative haemoglobin concentration (Hb) for identifying patients who develop severe post-operative anaemia or require blood transfusion following primary total hip or knee, or unicompartmental knee arthroplasty (THA, TKA, UKA). BACKGROUND: Pre-operative group and save (G&S), and post-operative Hb measurement may be unnecessary for many patients undergoing hip and knee arthroplasty provided individuals at greatest risk of severe post-operative anaemia can be identified. METHODS AND MATERIALS: Patients undergoing THA, TKA, or UKA between 2011 and 2018 were included. Outcomes were post-operative Hb below 70 and 80 g/L, and peri-operative blood transfusion. Logistic regression assessed the association between pre-operative Hb and each outcome. Decision curve analysis compared strategies for selecting patients for G&S and post-operative Hb measurement. RESULTS: 10 015 THA, TKA and UKA procedures were performed in 8582 patients. The incidence of blood transfusion (4.5%) decreased during the study. Using procedure specific Hb thresholds to select patients for pre-operative G&S and post-operative Hb testing had a greater net benefit than selecting all patients, no patients, or patients with pre-operative anaemia. CONCLUSIONS: Pre-operative G&S and post-operative Hb measurement may not be indicated for UKA or TKA when adopting restrictive transfusion thresholds, provided clinicians accept a 0.1% risk of patients developing severe undiagnosed post-operative anaemia (Hb < 70 g/L). The decision to perform these blood tests for THA patients should be based on local institutional data and selection of acceptable risk thresholds.

Mental health and other factors associated with work productivity after injury in the UK: multicentre cohort study.

INTRODUCTION: Mental health conditions are a major contributor to productivity loss and are common after injury. This study quantifies postinjury productivity loss and its association with preinjury and postinjury mental health, injury, demographic, health, social and other factors. METHODS: Multicentre, longitudinal study recruiting hospitalised employed individuals aged 16-69 years with unintentional injuries, followed up at 1, 2, 4 and 12 months. Participants completed questionnaires on injury, demographic factors, health (including mental health), social factors, other factors and on-the-job productivity upon return to work (RTW). ORs were estimated for above median productivity loss using random effects logistic regression. RESULTS: 217 adults had made an RTW at 2, 4 or 12 months after injury: 29% at 2 months, 66% at 4 months and 83% at 12 months. Productivity loss reduced over time: 3.3% of working time at 2 months, 1.7% at 4 months, 1% at 12 months. Significantly higher productivity loss was associated with preinjury psychiatric conditions (OR 21.40, 95% CI 3.50 to 130.78) and post-traumatic stress avoidance symptoms at 1 month (OR for 1-unit increase in score 1.15, 95% CI 1.07 to 1.22). Significantly lower productivity loss was associated with male gender (OR 0.32, 95% CI 0.14 to 0.74), upper and lower limb injuries (vs other body regions, OR 0.15, 95% CI 0.03 to 0.81) and sports injuries (vs home, OR 0.18, 95% CI 0.04 to 0.78). Preinjury psychiatric conditions and gender remained significant in analysis of multiply imputed data. CONCLUSIONS: Unintentional injury results in substantial productivity loss. Females, those with preinjury psychiatric conditions and those with post-traumatic stress avoidance symptoms experience greater productivity loss and may require additional support to enable successful RTW.

GoodReports: developing a website to help health researchers find and use reporting guidelines.

BACKGROUND: Th EQUATOR Network improves the quality and transparency in health research, primarily by promoting awareness and use of reporting guidelines. In 2018, the UK EQUATOR Centre launched GoodReports.org , a website that helps authors find and use reporting guidelines. This paper describes the tool's development so far. We describe user experience and behaviour of using GoodReports.org both inside and outside a journal manuscript submission process. We intend to use our findings to inform future development and testing of the tool. METHODS: We conducted a survey to collect data on user experience of the GoodReports website. We cross-checked a random sample of 100 manuscripts submitted to a partner journal to describe the level of agreement between the tool's checklist recommendation and what we would have recommended. We compared the proportion of authors submitting a completed reporting checklist alongside their manuscripts between groups exposed or not exposed to the GoodReports tool. We also conducted a study comparing completeness of reporting of manuscript text before an author received a reporting guideline recommendation from GoodReports.org with the completeness of the text subsequently submitted to a partner journal. RESULTS: Seventy percent (423/599) of survey respondents rated GoodReports 8 or more out of 10 for usefulness, and 74% (198/267) said they had made changes to their manuscript after using the website. We agreed with the GoodReports reporting guideline recommendation in 84% (72/86) of cases. Of authors who completed the guideline finder questionnaire, 14% (10/69) failed to submit a completed checklist compared to 30% (41/136) who did not use the tool. Of the 69 authors who received a GoodReports reporting guideline recommendation, 20 manuscript pairs could be reviewed before and after use of GoodReports. Five included more information in their methods section after exposure to GoodReports. On average, authors reported 57% of necessary reporting items before completing a checklist on GoodReports.org and 60% after. CONCLUSION: The data suggest that reporting guidance is needed early in the writing process, not at submission stage. We are developing GoodReports by adding more reporting guidelines and by creating editable article templates. We will test whether GoodReports users write more complete study reports in a randomised trial targeting researchers starting to write health research articles.

Electronic identification systems reduce the number of wrong components transfused.

BACKGROUND: Errors in hospital transfusion may cause wrong (blood) components to be transfused. This study assessed the value of electronic identification systems (EISs) in reducing wrong component transfusions (WCTs). METHODS: UK hospitals reporting to Serious Hazards of Transfusion were invited to complete an electronic survey about transfusion including the use of EISs. Further information was requested for WCTs and near-miss WCTs. RESULTS: A response rate of 93 of 222 (42%) hospitals accounted for 38% of UK blood component issues in 2015 and 2016. Thirty-three of 93 (35%) hospitals employ manual procedures and 16 (17%) use EISs throughout the transfusion process; most of the remainder use EISs for blood collection only. Fifty-seven WCTs were identified in approximately two million blood components. The primary error was at blood draw and sample labeling (3), blood collection (15), and administration (2); the remainder were mostly blood bank errors. No WCTs occurred with blood draw and sample labeling or administration with use of EISs. Three WCTs occurred with EISs for blood collection due to incorrect processes for emergency transfusions of group O blood without any adverse effects. Seventeen WCTs occurred with manual processes; one was an ABO-incompatible red blood cell transfusion resulting in renal impairment. Near-miss WCTs were also more frequent with manual procedures than EISs at blood draw and sample labeling and blood collection. CONCLUSIONS: This is the first multicenter study to demonstrate a lower incidence of WCTs and near-miss WCTs with EISs compared to manual processes, and highlights some limitations of both manual and EIS procedures.

Prediction model protocols indicate better adherence to recommended guidelines for study conduct and reporting.

BACKGROUND AND OBJECTIVE: Protocols are invaluable documents for any research study, especially for prediction model studies. However, the mere existence of a protocol is insufficient if key details are omitted. We reviewed the reporting content and details of the proposed design and methods reported in published protocols for prediction model research. METHODS: We searched MEDLINE, Embase, and the Web of Science Core Collection for protocols for studies developing or validating a diagnostic or prognostic model using any modeling approach in any clinical area. We screened protocols published between Jan 1, 2022 and June 30, 2022. We used the abstract, introduction, methods, and discussion sections of The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement to inform data extraction. RESULTS: We identified 30 protocols, of which 28 were describing plans for model development and six for model validation. All protocols were open access, including a preprint. 15 protocols reported prospectively collecting data. 21 protocols planned to use clustered data, of which one-third planned methods to account for it. A planned sample size was reported for 93% development and 67% validation analyses. 16 protocols reported details of study registration, but all protocols reported a statement on ethics approval. Plans for data sharing were reported in 13 protocols. CONCLUSION: Protocols for prediction model studies are uncommon, and few are made publicly available. Those that are available were reasonably well-reported and often described their methods following current prediction model research recommendations, likely leading to better reporting and methods in the actual study.

Open science practices need substantial improvement in prognostic model studies in oncology using machine learning.

OBJECTIVE: To describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology. STUDY DESIGN AND SETTING: We conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices. RESULTS: We identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing statements, with 21 (46%) indicating data were available on request to the authors and seven declaring data sharing was not applicable. Two studies (4%) shared data. Only 12 studies (26%) provided code sharing statements, including 2 (4%) that indicated the code was available on request to the authors. Only 11 studies (24%) provided sufficient information to allow their model to be used in practice. The use of reporting guidelines was rare: eight studies (18%) mentioning using a reporting guideline, with 4 (10%) using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement, 1 (2%) using Minimum Information About Clinical Artificial Intelligence Modeling and Consolidated Standards Of Reporting Trials-Artificial Intelligence, 1 (2%) using Strengthening The Reporting Of Observational Studies In Epidemiology, 1 (2%) using Standards for Reporting Diagnostic Accuracy Studies, and 1 (2%) using Transparent Reporting of Evaluations with Nonrandomized Designs. CONCLUSION: The adoption of open science principles in oncology studies developing prognostic models using machine learning methods is poor. Guidance and an increased awareness of benefits and best practices of open science are needed for prediction research in oncology.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies.

Objectives: To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance. Design: Systematic review and meta-analysis of external validation studies. Data sources: Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023. Eligibility criteria for selecting studies: All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed. Results: 47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125). Conclusions: The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed. Systematic review registration: PROSPERO CRD42022373182.

The rate of nonunion in the MRI-detected occult scaphoid fracture.

Aims: There is a lack of published evidence relating to the rate of nonunion seen in occult scaphoid fractures, diagnosed only after MRI. This study reports the rate of delayed union and nonunion in a cohort of patients with MRI-detected acute scaphoid fractures. Methods: This multicentre cohort study at eight centres in the UK included all patients with an acute scaphoid fracture diagnosed on MRI having presented acutely following wrist trauma with normal radiographs. Data were gathered retrospectively for a minimum of 12 months at each centre. The primary outcome measures were the rate of acute surgery, delayed union, and nonunion. Results: A total of 1,989 patients underwent acute MRI for a suspected scaphoid fracture during the study period, of which 256 patients (12.9%) were diagnosed with a previously occult scaphoid fracture. Of the patients with scaphoid fractures, six underwent early surgical fixation (2.3%) and there was a total of 16 cases of delayed or nonunion (6.3%) in the remaining 250 patients treated with cast immobilization. Of the nine nonunions (3.5%), seven underwent surgery (2.7%), one opted for non-surgical treatment, and one failed to attend follow-up. Of the seven delayed unions (2.7%), one (0.4%) was treated with surgery at two months, one (0.4%) did not attend further follow-up, and the remaining five fractures (1.9%) healed after further cast immobilization. All fractures treated with surgery had united at follow-up. There was one complication of surgery (prominent screw requiring removal). Conclusion: MRI-detected scaphoid fractures are not universally benign, with delayed or nonunion of scaphoid fractures diagnosed only after MRI seen in over 6% despite appropriate initial immobilization, with most of these patients with nonunion requiring surgery to achieve union. This study adds weight to the evidence base supporting the use of early MRI for these patients.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique. STUDY DESIGN AND SETTING: We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Effect of adding systematic family history enquiry to cardiovascular disease risk assessment in primary care: a matched-pair, cluster randomized trial.

BACKGROUND: Evidence of the value of systematically collecting family history in primary care is limited. OBJECTIVE: To evaluate the feasibility of systematically collecting family history of coronary heart disease in primary care and the effect of incorporating these data into cardiovascular risk assessment. DESIGN: Pragmatic, matched-pair, cluster randomized, controlled trial. (International Standardized Randomized Controlled Trial Number Register: ISRCTN 17943542). SETTING: 24 family practices in the United Kingdom. PARTICIPANTS: 748 persons aged 30 to 65 years with no previously diagnosed cardiovascular risk, seen between July 2007 and March 2009. INTERVENTION: Participants in control practices had the usual Framingham-based cardiovascular risk assessment with and without use of existing family history information in their medical records. Participants in intervention practices also completed a questionnaire to systematically collect their family history. All participants were informed of their risk status. Participants with high cardiovascular risk were invited for a consultation. MEASUREMENTS: The primary outcome was the proportion of participants with high cardiovascular risk (10-year risk ≥ 20%). Other measures included questionnaire completion rate and anxiety score. RESULTS: 98% of participants completed the family history questionnaire. The mean increase in proportion of participants classified as having high cardiovascular risk was 4.8 percentage points in the intervention practices, compared with 0.3 percentage point in control practices when family history from patient records was incorporated. The 4.5-percentage point difference between groups (95% CI, 1.7 to 7.2 percentage points) remained significant after adjustment for participant and practice characteristics (P = 0.007). Anxiety scores were similar between groups. LIMITATIONS: Relatively few participants were from ethnic minority or less-educated groups. The potential to explore behavioral change and clinical outcomes was limited. Many data were missing for anxiety scores. CONCLUSION: Systematically collecting family history increases the proportion of persons identified as having high cardiovascular risk for further targeted prevention and seems to have little or no effect on anxiety. PRIMARY FUNDING SOURCE: Genetics Health Services Research program of the United Kingdom Department of Health.

Artificial intelligence in lung cancer diagnostic imaging: a review of the reporting and conduct of research published 2018-2019.

Objective: This study aimed to describe the methodologies used to develop and evaluate models that use artificial intelligence (AI) to analyse lung images in order to detect, segment (outline borders of), or classify pulmonary nodules as benign or malignant. Methods: In October 2019, we systematically searched the literature for original studies published between 2018 and 2019 that described prediction models using AI to evaluate human pulmonary nodules on diagnostic chest images. Two evaluators independently extracted information from studies, such as study aims, sample size, AI type, patient characteristics, and performance. We summarised data descriptively. Results: The review included 153 studies: 136 (89%) development-only studies, 12 (8%) development and validation, and 5 (3%) validation-only. CT scans were the most common type of image type used (83%), often acquired from public databases (58%). Eight studies (5%) compared model outputs with biopsy results. 41 studies (26.8%) reported patient characteristics. The models were based on different units of analysis, such as patients, images, nodules, or image slices or patches. Conclusion: The methods used to develop and evaluate prediction models using AI to detect, segment, or classify pulmonary nodules in medical imaging vary, are poorly reported, and therefore difficult to evaluate. Transparent and complete reporting of methods, results and code would fill the gaps in information we observed in the study publications. Advances in knowledge: We reviewed the methodology of AI models detecting nodules on lung images and found that the models were poorly reported and had no description of patient characteristics, with just a few comparing models' outputs with biopsies results. When lung biopsy is not available, lung-RADS could help standardise the comparisons between the human radiologist and the machine. The field of radiology should not give up principles from the diagnostic accuracy studies, such as the choice for the correct ground truth, just because AI is used. Clear and complete reporting of the reference standard used would help radiologists trust in the performance that AI models claim to have. This review presents clear recommendations about the essential methodological aspects of diagnostic models that should be incorporated in studies using AI to help detect or segmentate lung nodules. The manuscript also reinforces the need for more complete and transparent reporting, which can be helped using the recommended reporting guidelines.