Prognostic Factors and Models to Predict Mortality Outcomes in Patients with Peripheral Arterial Disease: A Systematic Review.

OBJECTIVE: Predicting adverse outcomes in patients with peripheral arterial disease (PAD) is a complex task owing to the heterogeneity in patient and disease characteristics. This systematic review aimed to identify prognostic factors and prognostic models to predict mortality outcomes in patients with PAD Fontaine stage I - III or Rutherford category 0 - 4. DATA SOURCES: PubMed, Embase, and Cochrane Database of Systematic Reviews were searched to identify studies examining individual prognostic factors or studies aiming to develop or validate a prognostic model for mortality outcomes in patients with PAD. REVIEW METHODS: Information on study design, patient population, prognostic factors, and prognostic model characteristics was extracted, and risk of bias was evaluated. RESULTS: Sixty nine studies investigated prognostic factors for mortality outcomes in PAD. Over 80 single prognostic factors were identified, with age as a predictor of death in most of the studies. Other common factors included sex, diabetes, and smoking status. Six studies had low risk of bias in all domains, and the remainder had an unclear or high risk of bias in at least one domain. Eight studies developed or validated a prognostic model. All models included age in their primary model, but not sex. All studies had similar discrimination levels of > 70%. Five of the studies on prognostic models had an overall high risk of bias, whereas two studies had an overall unclear risk of bias. CONCLUSION: This systematic review shows that a large number of prognostic studies have been published, with heterogeneity in patient populations, outcomes, and risk of bias. Factors such as sex, age, diabetes, hypertension, and smoking are significant in predicting mortality risk among patients with PAD Fontaine stage I - III or Rutherford category 0 - 4.

External Validation of Risk Prediction Models to Improve Selection of Patients for Carotid Endarterectomy.

BACKGROUND AND PURPOSE: The net benefit of carotid endarterectomy (CEA) is determined partly by the risk of procedural stroke or death. Current guidelines recommend CEA if 30-day risks are <6% for symptomatic stenosis and <3% for asymptomatic stenosis. We aimed to identify prediction models for procedural stroke or death after CEA and to externally validate these models in a large registry of patients from the United States. METHODS: We conducted a systematic search in MEDLINE and EMBASE for prediction models of procedural outcomes after CEA. We validated these models with data from patients who underwent CEA in the American College of Surgeons National Surgical Quality Improvement Program (2011-2017). We assessed discrimination using C statistics and calibration graphically. We determined the number of patients with predicted risks that exceeded recommended thresholds of procedural risks to perform CEA. RESULTS: After screening 788 reports, 15 studies describing 17 prediction models were included. Nine were developed in populations including both asymptomatic and symptomatic patients, 2 in symptomatic and 5 in asymptomatic populations. In the external validation cohort of 26 293 patients who underwent CEA, 702 (2.7%) developed a stroke or died within 30-days. C statistics varied between 0.52 and 0.64 using all patients, between 0.51 and 0.59 using symptomatic patients, and between 0.49 to 0.58 using asymptomatic patients. The Ontario Carotid Endarterectomy Registry model that included symptomatic status, diabetes, heart failure, and contralateral occlusion as predictors, had C statistic of 0.64 and the best concordance between predicted and observed risks. This model identified 4.5% of symptomatic and 2.1% of asymptomatic patients with procedural risks that exceeded recommended thresholds. CONCLUSIONS: Of the 17 externally validated prediction models, the Ontario Carotid Endarterectomy Registry risk model had most reliable predictions of procedural stroke or death after CEA and can inform patients about procedural hazards and help focus CEA toward patients who would benefit most from it.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review.

BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review.

BACKGROUND: While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. METHODS: We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item. RESULTS: Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). CONCLUSION: Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. SYSTEMATIC REVIEW REGISTRATION: PROSPERO, CRD42019161764.

Transparent Reporting of Multivariable Prediction Models in Journal and Conference Abstracts: TRIPOD for Abstracts.

Clear and informative reporting in titles and abstracts is essential to help readers and reviewers identify potentially relevant studies and decide whether to read the full text. Although the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement provides general recommendations for reporting titles and abstracts, more detailed guidance seems to be desirable. The authors present TRIPOD for Abstracts, a checklist and corresponding guidance for reporting prediction model studies in abstracts. A list of 32 potentially relevant items was the starting point for a modified Delphi procedure involving 110 experts, of whom 71 (65%) participated in the web-based survey. After 2 Delphi rounds, the experts agreed on 21 items as being essential to report in abstracts of prediction model studies. This number was reduced by merging some of the items. In a third round, participants provided feedback on a draft version of TRIPOD for Abstracts. The final checklist contains 12 items and applies to journal and conference abstracts that describe the development or external validation of a diagnostic or prognostic prediction model, or the added value of predictors to an existing model, regardless of the clinical domain or statistical approach used.

Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement.

BACKGROUND: As complete reporting is essential to judge the validity and applicability of multivariable prediction models, a guideline for the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) was introduced. We assessed the completeness of reporting of prediction model studies published just before the introduction of the TRIPOD statement, to refine and tailor its implementation strategy. METHODS: Within each of 37 clinical domains, 10 journals with the highest journal impact factor were selected. A PubMed search was performed to identify prediction model studies published before the launch of TRIPOD in these journals (May 2014). Eligible publications reported on the development or external validation of a multivariable prediction model (either diagnostic or prognostic) or on the incremental value of adding a predictor to an existing model. RESULTS: We included 146 publications (84% prognostic), from which we assessed 170 models: 73 (43%) on model development, 43 (25%) on external validation, 33 (19%) on incremental value, and 21 (12%) on combined development and external validation of the same model. Overall, publications adhered to a median of 44% (25th-75th percentile 35-52%) of TRIPOD items, with 44% (35-53%) for prognostic and 41% (34-48%) for diagnostic models. TRIPOD items that were completely reported for less than 25% of the models concerned abstract (2%), title (5%), blinding of predictor assessment (6%), comparison of development and validation data (11%), model updating (14%), model performance (14%), model specification (17%), characteristics of participants (21%), model performance measures (methods) (21%), and model-building procedures (24%). Most often reported were TRIPOD items regarding overall interpretation (96%), source of data (95%), and risk groups (90%). CONCLUSIONS: More than half of the items considered essential for transparent reporting were not fully addressed in publications of multivariable prediction model studies. Essential information for using a model in individual risk prediction, i.e. model specifications and model performance, was incomplete for more than 80% of the models. Items that require improved reporting are title, abstract, and model-building procedures, as they are crucial for identification and external validation of prediction models.

Increased endorsement of TRIPOD and other reporting guidelines by high impact factor journals: survey of instructions to authors.

OBJECTIVES: To assess the endorsement of reporting guidelines by high impact factor journals over the period 2017-2022, with a specific focus on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. STUDY DESIGN AND SETTING: We searched the online 'instructions to authors' of high impact factor medical journals in February 2017 and in January 2022 for any reference to reporting guidelines and TRIPOD in particular. RESULTS: In 2017, 205 out of 337 (61%) journals mentioned any reporting guideline in their instructions to authors and in 2022 this increased to 245 (73%) journals. A reference to TRIPOD was provided by 27 (8%) journals in 2017 and 67 (20%) in 2022. Of those journals mentioning TRIPOD in 2022, 22% provided a link to the TRIPOD website and 60% linked to TRIPOD information on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website. Twenty-five percent of the journals required adherence to TRIPOD. CONCLUSION: About three-quarters of high-impact medical journals endorse the use of reporting guidelines and 20% endorse TRIPOD. Transparent reporting is important in enhancing the usefulness of health research and endorsement by journals plays a critical role in this.

Both clinical trial register and electronic bibliographic database searches were needed to identify randomized clinical trials for systematic reviews: an evaluation study.

OBJECTIVES: To determine whether clinical trial register (CTR) searches can accurately identify a greater number of completed randomized clinical trials (RCTs) than electronic bibliographic database (EBD) searches for systematic reviews of interventions, and to quantify the number of eligible ongoing trials. STUDY DESIGN AND SETTING: We performed an evaluation study and based our search for RCTs on the eligibility criteria of a systematic review that focused on the underrepresentation of people with chronic kidney disease in cardiovascular RCTs. We conducted a combined search of ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform through the Cochrane Central Register of Controlled Trials to identify eligible RCTs registered up to June 1, 2023. We searched Cochrane Central Register of Controlled Trials, EMBASE, and MEDLINE for publications of eligible RCTs published up to June 5, 2023. Finally, we compared the search results to determine the extent to which the two sources identified the same RCTs. RESULTS: We included 92 completed RCTs. Of these, 81 had results available. Sixty-six completed RCTs with available results were identified by both sources (81% agreement [95% CI: 71-88]). We identified seven completed RCTs with results exclusively by CTR search (9% [95% CI: 4-17]) and eight exclusively by EBD search (10% [95% CI: 5-18]). Eleven RCTs were completed but lacked results (four identified by both sources (36% [95% CI: 15-65]), one exclusively by EBD search (9% [95% CI: 1-38]), and six exclusively by CTR search (55% [95% CI: 28-79])). Also, we identified 42 eligible ongoing RCTs: 16 by both sources (38% [95% CI: 25-53]) and 26 exclusively by CTR search (62% [95% CI: 47-75]). Lastly, we identified four RCTs of unknown status by both sources. CONCLUSION: CTR searches identify a greater number of completed RCTs than EBD searches. Both searches missed some included RCTs. Based on our case study, researchers (eg, information specialists, systematic reviewers) aiming to identify all available RCTs should continue to search both sources. Once the barriers to performing CTR searches alone are targeted, CTR searches may be a suitable alternative.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique. STUDY DESIGN AND SETTING: We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

How to conduct a systematic review and meta-analysis of prognostic model studies.

BACKGROUND: Prognostic models are typically developed to estimate the risk that an individual in a particular health state will develop a particular health outcome, to support (shared) decision making. Systematic reviews of prognostic model studies can help identify prognostic models that need to further be validated or are ready to be implemented in healthcare. OBJECTIVES: To provide a step-by-step guidance on how to conduct and read a systematic review of prognostic model studies and to provide an overview of methodology and guidance available for every step of the review progress. SOURCES: Published, peer-reviewed guidance articles. CONTENT: We describe the following steps for conducting a systematic review of prognosis studies: 1) Developing the review question using the Population, Index model, Comparator model, Outcome(s), Timing, Setting format, 2) Searching and selection of articles, 3) Data extraction using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist, 4) Quality and risk of bias assessment using the Prediction model Risk Of Bias ASsessment (PROBAST) tool, 5) Analysing data and undertaking quantitative meta-analysis, and 6) Presenting summary of findings, interpreting results, and drawing conclusions. Guidance for each step is described and illustrated using a case study on prognostic models for patients with COVID-19. IMPLICATIONS: Guidance for conducting a systematic review of prognosis studies is available, but the implications of these reviews for clinical practice and further research highly depend on complete reporting of primary studies.

One Heartbeat Away from a Prediction Model for Cardiovascular Diseases in Patients with Chronic Kidney Disease: A Systematic Review.

INTRODUCTION: Patients with chronic kidney disease (CKD) have a high risk of cardiovascular disease (CVD). Prediction models, combining clinical and laboratory characteristics, are commonly used to estimate an individual's CVD risk. However, these models are not specifically developed for patients with CKD and may therefore be less accurate. In this review, we aim to give an overview of CVD prognostic studies available, and their methodological quality, specifically for patients with CKD. METHODS: MEDLINE was searched for papers reporting CVD prognostic studies in patients with CKD published between 2012 and 2021. Characteristics regarding patients, study design, outcome measurement, and prediction models were compared between included studies. The risk of bias of studies reporting on prognostic factors or the development/validation of a prediction model was assessed with, respectively, the QUIPS and PROBAST tool. RESULTS: In total, 134 studies were included, of which 123 studies tested the incremental value of one or more predictors to existing models or common risk factors, while only 11 studies reported on the development or validation of a prediction model. Substantial heterogeneity in cohort and study characteristics, such as sample size, event rate, and definition of outcome measurements, was observed across studies. The most common predictors were age (87%), sex (75%), diabetes (70%), and estimated glomerular filtration rate (69%). Most of the studies on prognostic factors have methodological shortcomings, mostly due to a lack of reporting on clinical and methodological information. Of the 11 studies on prediction models, six developed and internally validated a model and four externally validated existing or developed models. Only one study on prognostic models showed a low risk of bias and high applicability. CONCLUSION: A large quantity of prognostic studies has been published, yet their usefulness remains unclear due to incomplete presentation, and lack of external validation of prognostic models. Our review can be used to select the most appropriate prognostic model depending on the patient population, outcome, and risk of bias. Future collaborative efforts should aim at improving existing models by externally validating them, evaluating the addition of new predictors, and assessment of the clinical impact. REGISTRATION: We have registered the protocol of our systematic review on PROSPERO (CRD42021228043).

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models.

BACKGROUND AND OBJECTIVES: We sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques. METHODS: We search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes. RESULTS: We included 152 studies, 58 (38.2% [95% CI 30.8-46.1]) were diagnostic and 94 (61.8% [95% CI 53.9-69.2]) prognostic studies. Most studies reported only the development of prediction models (n = 133, 87.5% [95% CI 81.3-91.8]), focused on binary outcomes (n = 131, 86.2% [95% CI 79.8-90.8), and did not report a sample size calculation (n = 125, 82.2% [95% CI 75.4-87.5]). The most common algorithms used were support vector machine (n = 86/522, 16.5% [95% CI 13.5-19.9]) and random forest (n = 73/522, 14% [95% CI 11.3-17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n = 494/522, 94.6% [95% CI 92.4-96.3]). CONCLUSION: Our review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning-based prediction models. SYSTEMATIC REVIEW REGISTRATION: PROSPERO, CRD42019161764.

Diagnostic yield and safety of navigation bronchoscopy: A systematic review and meta-analysis.

BACKGROUND: Navigation bronchoscopy has seen rapid development in the past decade in terms of new navigation techniques and multi-modality approaches utilizing different techniques and tools. This systematic review analyses the diagnostic yield and safety of navigation bronchoscopy for the diagnosis of peripheral pulmonary nodules suspected of lung cancer. METHODS: An extensive search was performed in Embase, Medline and Cochrane CENTRAL in May 2022. Eligible studies used cone-beam CT-guided navigation (CBCT), electromagnetic navigation (EMN), robotic navigation (RB) or virtual bronchoscopy (VB) as the primary navigation technique. Primary outcomes were diagnostic yield and adverse events. Quality of studies was assessed using QUADAS-2. Random effects meta-analysis was performed, with subgroup analyses for different navigation techniques, newer versus older techniques, nodule size, publication year, and strictness of diagnostic yield definition. Explorative analyses of subgroups reported by studies was performed for nodule size and bronchus sign. RESULTS: A total of 95 studies (n = 10,381 patients; n = 10,682 nodules) were included. The majority (n = 63; 66.3%) had high risk of bias or applicability concerns in at least one QUADAS-2 domain. Summary diagnostic yield was 70.9% (95%-CI 68.4%-73.2%). Overall pneumothorax rate was 2.5%. Newer navigation techniques using advanced imaging and/or robotics(CBCT, RB, tomosynthesis guided EMN; n = 24 studies) had a statistically significant higher diagnostic yield compared to longer established techniques (EMN, VB; n = 82 studies): 77.5% (95%-CI 74.7%-80.1%) vs 68.8% (95%-CI 65.9%-71.6%) (p < 0.001).Explorative subgroup analyses showed that larger nodule size and bronchus sign presence were associated with a statistically significant higher diagnostic yield. Other subgroup analyses showed no significant differences. CONCLUSION: Navigation bronchoscopy is a safe procedure, with the potential for high diagnostic yield, in particular using newer techniques such as RB, CBCT and tomosynthesis-guided EMN. Studies showed a large amount of heterogeneity, making comparisons difficult. Standardized definitions for outcomes with relevant clinical context will improve future comparability.

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review.

OBJECTIVES: In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction. STUDY DESIGN AND SETTING: We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices. RESULTS: We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion. CONCLUSION: The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models.

OBJECTIVES: We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques. STUDY DESIGN AND SETTING: We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty. RESULTS: We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies. CONCLUSION: Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.

What did we learn in 35 years of research on nutrition and supplements for age-related macular degeneration: a systematic review.

The aim of this paper is to summarize all available evidence from systematic reviews, randomized controlled trials (RCTs) and comparative nonrandomized studies (NRS) on the association between nutrition and antioxidant, vitamin, and mineral supplements and the development or progression of age-related macular degeneration (AMD). The Cochrane Database of Systematic Reviews, Cochrane register CENTRAL, MEDLINE and Embase were searched and studies published between January 2015 and May 2021 were included. The certainty of evidence was assessed according to the GRADE methodology. The main outcome measures were development of AMD, progression of AMD, and side effects. We included 7 systematic reviews, 7 RCTs, and 13 NRS. A high consumption of specific nutrients, i.e. ß-carotene, lutein and zeaxanthin, copper, folate, magnesium, vitamin A, niacin, vitamin B6, vitamin C, docosahexaenoic acid, and eicosapentaenoic acid, was associated with a lower risk of progression of early to late AMD (high certainty of evidence). Use of antioxidant supplements and adherence to a Mediterranean diet, characterized by a high consumption of vegetables, whole grains, and nuts and a low consumption of red meat, were associated with a decreased risk of progression of early to late AMD (moderate certainty of evidence). A high consumption of alcohol was associated with a higher risk of developing AMD (moderate certainty of evidence). Supplementary vitamin C, vitamin E, or ß-carotene were not associated with the development of AMD, and supplementary omega-3 fatty acids were not associated with progression to late AMD (high certainty of evidence). Research in the last 35 years included in our overview supports that a high intake of specific nutrients, the use of antioxidant supplements and adherence to a Mediterranean diet decrease the risk of progression of early to late AMD.

The score after 10 years of registration of systematic review protocols.

BACKGROUND: With the exponential growth of published systematic reviews (SR), there is a high potential for overlapping and redundant duplication of work. Prospective protocol registration gives the opportunity to assess the added value of a new study or review, thereby potentially reducing research waste and simultaneously increasing transparency and research quality. The PROSPERO database for SR protocol registration was launched 10 years ago. This study aims to assess the proportion SRs of intervention studies with a protocol registration (or publication) and explore associations of SR characteristics with protocol registration status. METHODS: PubMed was searched for SRs of human intervention studies published in January 2020 and January 2021. After random-stratified sampling and eligibility screening, data extraction on publication and journal characteristics, and protocol registration status, was performed. Both descriptive and multivariable comparative statistical analyses were performed. RESULTS: A total of 357 SRs (2020: n = 163; 2021: n = 194) were included from a random sample of 1267 publications. Of the published SRs, 38% had a protocol. SRs that reported using PRISMA as a reporting guideline had higher odds of having a protocol than publications that did not report PRISMA (OR 2.71; 95% CI: 1.21 to 6.09). SRs with a higher journal impact factor had higher odds of having a protocol (OR 1.12; 95% CI 1.04 to 1.25). Publications from Asia had a lower odds of having a protocol (OR 0.43; 95% CI 0.23 to 0.80, reference category = Europe). Of the 33 SRs published in journals that endorse PROSPERO, 45% did not have a protocol. Most SR protocols were registered in PROSPERO (n = 129; 96%). CONCLUSIONS: We found that 38% of recently published SRs of interventions reported a registered or published protocol. Protocol registration was significantly associated with a higher impact factor of the journal publishing the SR and a more frequent self-reported use of the PRISMA guidelines. In some parts of the world, SR protocols are more often registered or published than others. To guide strategies to increase the uptake of SR protocol registration, further research is needed to gain understanding of the benefits and informativeness of SRs protocols among different stakeholders. SYSTEMATIC REVIEW REGISTRATION: osf.io/9kj7r/.

Risk of bias of prognostic models developed using machine learning: a systematic review in oncology.

BACKGROUND: Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain. METHODS: We conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately. RESULTS: We included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation. CONCLUSIONS: The quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models.