SPIN-PM: A consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model, regardless of the modelling technique. STUDY DESIGN: We followed a three-phase consensus process: (1) pre-meeting literature review to generate items to be included; (2) a series of structured meetings to provide comments, discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) post-meeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-PM which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

The burden of schizophrenia in the Middle East and North Africa region, 1990-2019.

Schizophrenia ranks as the third-most common cause of disability among mental disorders globally. This study presents findings on the prevalence, incidence and years lived with disability (YLDs) as a result of schizophrenia in the Middle East and North Africa (MENA), stratified by age, sex and sociodemographic index (SDI). We collected publicly accessible data from the Global Burden of Disease (GBD) study 2019. This study reports the burden of schizophrenia, from 1990 to 2019, for the 21 countries that comprise MENA. In 2019, MENA exhibited an age-standardised point prevalence of 248.2, an incidence rate of 14.7 and an YLD rate of 158.7 per 100,000, which have not changed substantially between 1990 and 2019. In 2019, the age-standardised YLD rate was highest in Qatar and lowest in Afghanistan. No MENA countries demonstrated noteworthy changes in the burden of schizophrenia from 1990 to 2019. Furthermore, in 2019, the highest number of prevalent cases and the point prevalence were observed among those aged 35-39, with a higher prevalence among males in almost all age categories. Additionally, in 2019, the age-standardised YLD rates in MENA were below the worldwide average. Finally, there was a positive correlation between the burden of schizophrenia and the SDI from 1990 to 2019. The disease burden of schizophrenia has remained relatively stable over the past thirty years. Nevertheless, as the regional life-expectancy continues to increase, the burden of schizophrenia is also expected to rise. Therefore, early planning for the increase in the burden of the disease is urgently needed in the region.

Anterior Cruciate Ligament Return to Sport after Injury Scale (ACL-RSI) Scores over Time After Anterior Cruciate Ligament Reconstruction: A Systematic Review with Meta-analysis.

BACKGROUND: Psychological readiness is an important consideration for athletes and clinicians when making return to sport decisions following anterior cruciate ligament reconstruction (ACLR). To improve our understanding of the extent of deficits in psychological readiness, a systematic review is necessary. OBJECTIVE: To investigate psychological readiness (measured via the Anterior Cruciate Ligament-Return to Sport after Injury scale (ACL-RSI)) over time after ACL tear and understand if time between injury and surgery, age, and sex are associated with ACL-RSI scores. METHODS: Seven databases were searched from the earliest date available to March 22, 2022. Articles reporting ACL-RSI scores after ACL tear were included. Risk of bias was assessed using the ROBINS-I, RoB-2, and RoBANS tools based on the study design. Evidence certainty was assessed for each analysis. Random-effects meta-analyses pooled ACL-RSI scores, stratified by time post-injury and based on treatment approach (i.e., early ACLR, delayed ACLR, and unclear approach). RESULTS: A total of 83 studies were included in this review (78% high risk of bias). Evidence certainty was 'weak' or 'limited' for all analyses. Overall, ACL-RSI scores were higher at 3 to 6 months post-ACLR (mean = 61.5 [95% confidence interval (CI) 58.6, 64.4], I2 = 94%) compared to pre-ACLR (mean = 44.4 [95% CI 38.2, 50.7], I2 = 98%), remained relatively stable, until they reached the highest point 2 to 5 years after ACLR (mean = 70.7 [95% CI 63.0, 78.5], I2 = 98%). Meta-regression suggests shorter time from injury to surgery, male sex, and older age were associated with higher ACL-RSI scores only 3 to 6 months post-ACLR (heterogeneity explained R2 = 47.6%), and this reduced 1-2 years after ACLR (heterogeneity explained R2 = 27.0%). CONCLUSION: Psychological readiness to return to sport appears to improve early after ACL injury, with little subsequent improvement until ≥ 2-years after ACLR. Longer time from injury to surgery, female sex and older age might be negatively related to ACL-RSI scores 12-24 months after ACLR. Due to the weak evidence quality rating and the considerable importance of psychological readiness for long-term outcomes after ACL injury, there is an urgent need for well-designed studies that maximize internal validity and identify additional prognostic factors for psychological readiness at times critical for return to sport decisions. REGISTRATION: Open Science Framework (OSF), https://osf.io/2tezs/ .

Reverse total shoulder replacement versus anatomical total shoulder replacement for osteoarthritis: population based cohort study using data from the National Joint Registry and Hospital Episode Statistics for England.

OBJECTIVES: To answer a national research priority by comparing the risk-benefit and costs associated with reverse total shoulder replacement (RTSR) and anatomical total shoulder replacement (TSR) in patients having elective primary shoulder replacement for osteoarthritis. DESIGN: Population based cohort study using data from the National Joint Registry and Hospital Episode Statistics for England. SETTING: Public hospitals and publicly funded procedures at private hospitals in England, 2012-20. PARTICIPANTS: Adults aged 60 years or older who underwent RTSR or TSR for osteoarthritis with intact rotator cuff tendons. Patients were identified from the National Joint Registry and linked to NHS Hospital Episode Statistics and civil registration mortality data. Propensity score matching and inverse probability of treatment weighting were used to balance the study groups. MAIN OUTCOME MEASURES: The main outcome measure was revision surgery. Secondary outcome measures included serious adverse events within 90 days, reoperations within 12 months, prolonged hospital stay (more than three nights), change in Oxford Shoulder Score (preoperative to six month postoperative), and lifetime costs to the healthcare service. RESULTS: The propensity score matched population comprised 7124 RTSR or TSR procedures (126 were revised), and the inverse probability of treatment weighted population comprised 12 968 procedures (294 were revised) with a maximum follow-up of 8.75 years. RTSR had a reduced hazard ratio of revision in the first three years (hazard ratio local minimum 0.33, 95% confidence interval 0.18 to 0.59) with no clinically important difference in revision-free restricted mean survival time, and a reduced relative risk of reoperations at 12 months (odds ratio 0.45, 95% confidence interval 0.25 to 0.83) with an absolute risk difference of -0.51% (95% confidence interval -0.89 to -0.13). Serious adverse events and prolonged hospital stay risks, change in Oxford Shoulder Score, and modelled mean lifetime costs were similar. Outcomes remained consistent after weighting. CONCLUSIONS: This study's findings provide reassurance that RTSR is an acceptable alternative to TSR for patients aged 60 years or older with osteoarthritis and intact rotator cuff tendons. Despite a significant difference in the risk profiles of revision surgery over time, no statistically significant and clinically important differences between RTSR and TSR were found in terms of long term revision surgery, serious adverse events, reoperations, prolonged hospital stay, or lifetime healthcare costs.

Methods used to develop the SPIRIT 2024 and CONSORT 2024 Statements.

OBJECTIVES: To describe, and explain the rationale for, the methods used and decisions made during development of the updated SPIRIT 2024 and CONSORT 2024 reporting guidelines. METHODS: We developed SPIRIT 2024 and CONSORT 2024 together to facilitate harmonization of the two guidelines, and incorporated content from key extensions. We conducted a scoping review of comments suggesting changes to SPIRIT 2013 and CONSORT 2010, and compiled a list of other possible revisions based on existing SPIRIT and CONSORT extensions, other reporting guidelines, and personal communications. From this, we generated a list of potential modifications or additions to SPIRIT and CONSORT, which we presented to stakeholders for feedback in an international online Delphi survey. The Delphi survey results were discussed at an online expert consensus meeting attended by 30 invited international participants. We then drafted the updated SPIRIT and CONSORT checklists and revised them based on further feedback from meeting attendees. RESULTS: We compiled 83 suggestions for revisions or additions to SPIRIT and/or CONSORT from the scoping review and 85 from other sources, from which we generated 33 potential changes to SPIRIT (n = 5) or CONSORT (n = 28). Of 463 participants invited to take part in the Delphi survey, 317 (68%) responded to Round 1, 303 (65%) to Round 2 and 290 (63%) to Round 3. Two additional potential checklist changes were added to the Delphi survey based on Round 1 comments. Overall, 14/35 (SPIRIT n = 0; CONSORT n = 14) proposed changes reached the predefined consensus threshold (≥80% agreement), and participants provided 3580 free-text comments. The consensus meeting participants agreed with implementing 11/14 of the proposed changes that reached consensus in the Delphi and supported implementing a further 4/21 changes (SPIRIT n = 2; CONSORT n = 2) that had not reached the Delphi threshold. They also recommended further changes to refine key concepts and for clarity. CONCLUSION: The forthcoming SPIRIT 2024 and CONSORT 2024 Statements will provide updated, harmonized guidance for reporting randomized controlled trial protocols and results, respectively. The simultaneous development of the SPIRIT and CONSORT checklists has been informed by current empirical evidence and extensive input from stakeholders. We hope that this report of the methods used will be helpful for developers of future reporting guidelines.

Stakeholder perspectives towards diagnostic artificial intelligence: a co-produced qualitative evidence synthesis.

Background: Diagnosis is a cornerstone of medical practice. Worldwide, there is increased demand for diagnostic services, exacerbating workforce shortages. Artificial intelligence (AI) technologies may improve diagnostic efficiency, accuracy, and access. Understanding stakeholder perspectives is key to informing implementation of complex interventions. We systematically reviewed the literature on stakeholder perspectives on diagnostic AI, including all English-language peer-reviewed primary qualitative or mixed-methods research. Methods: We searched PubMed, Ovid MEDLINE/Embase, Scopus, CINAHL and Web of Science (22/2/2023 and updated 8/2/2024). The Critical Appraisal Skills Programme Checklist informed critical appraisal. We used a 'best-fit' framework approach for analysis, using the Non-adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) framework. This study was pre-registered (PROSPERO CRD42022313782). Findings: We screened 16,577 articles and included 44. 689 participants were interviewed, and 402 participated in focus groups. Four stakeholder groups were described: patients, clinicians, researchers and healthcare leaders. We found an under-representation of patients, researchers and leaders across articles. We summarise the differences and relationships between each group in a conceptual model, hinging on the establishment of trust, engagement and collaboration. We present a modification of the NASSS framework, tailored to diagnostic AI. Interpretation: We provide guidance for future research and implementation of diagnostic AI, highlighting the importance of representing all stakeholder groups. We suggest that implementation strategies consider how any proposed software fits within the extended NASSS-AI framework, and how stakeholder priorities and concerns have been addressed. Funding: RK is supported by an NIHR Doctoral Research Fellowship grant (NIHR302562), which funded patient and public involvement activities, and access to Covidence.

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines.

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

Ethnicity data resource in population-wide health records: completeness, coverage and granularity of diversity.

Intersectional social determinants including ethnicity are vital in health research. We curated a population-wide data resource of self-identified ethnicity data from over 60 million individuals in England primary care, linking it to hospital records. We assessed ethnicity data in terms of completeness, consistency, and granularity and found one in ten individuals do not have ethnicity information recorded in primary care. By linking to hospital records, ethnicity data were completed for 94% of individuals. By reconciling SNOMED-CT concepts and census-level categories into a consistent hierarchy, we organised more than 250 ethnicity sub-groups including and beyond "White", "Black", "Asian", "Mixed" and "Other, and found them to be distributed in proportions similar to the general population. This large observational dataset presents an algorithmic hierarchy to represent self-identified ethnicity data collected across heterogeneous healthcare settings. Accurate and easily accessible ethnicity data can lead to a better understanding of population diversity, which is important to address disparities and influence policy recommendations that can translate into better, fairer health for all.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Prediction model protocols indicate better adherence to recommended guidelines for study conduct and reporting.

BACKGROUND AND OBJECTIVE: Protocols are invaluable documents for any research study, especially for prediction model studies. However, the mere existence of a protocol is insufficient if key details are omitted. We reviewed the reporting content and details of the proposed design and methods reported in published protocols for prediction model research. METHODS: We searched MEDLINE, Embase, and the Web of Science Core Collection for protocols for studies developing or validating a diagnostic or prognostic model using any modeling approach in any clinical area. We screened protocols published between Jan 1, 2022 and June 30, 2022. We used the abstract, introduction, methods, and discussion sections of The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement to inform data extraction. RESULTS: We identified 30 protocols, of which 28 were describing plans for model development and six for model validation. All protocols were open access, including a preprint. 15 protocols reported prospectively collecting data. 21 protocols planned to use clustered data, of which one-third planned methods to account for it. A planned sample size was reported for 93% development and 67% validation analyses. 16 protocols reported details of study registration, but all protocols reported a statement on ethics approval. Plans for data sharing were reported in 13 protocols. CONCLUSION: Protocols for prediction model studies are uncommon, and few are made publicly available. Those that are available were reasonably well-reported and often described their methods following current prediction model research recommendations, likely leading to better reporting and methods in the actual study.

Publishers' and journals' instructions to authors on use of generative artificial intelligence in academic and scientific publishing: bibliometric analysis.

OBJECTIVES: To determine the extent and content of academic publishers' and scientific journals' guidance for authors on the use of generative artificial intelligence (GAI). DESIGN: Cross sectional, bibliometric study. SETTING: Websites of academic publishers and scientific journals, screened on 19-20 May 2023, with the search updated on 8-9 October 2023. PARTICIPANTS: Top 100 largest academic publishers and top 100 highly ranked scientific journals, regardless of subject, language, or country of origin. Publishers were identified by the total number of journals in their portfolio, and journals were identified through the Scimago journal rank using the Hirsch index (H index) as an indicator of journal productivity and impact. MAIN OUTCOME MEASURES: The primary outcomes were the content of GAI guidelines listed on the websites of the top 100 academic publishers and scientific journals, and the consistency of guidance between the publishers and their affiliated journals. RESULTS: Among the top 100 largest publishers, 24% provided guidance on the use of GAI, of which 15 (63%) were among the top 25 publishers. Among the top 100 highly ranked journals, 87% provided guidance on GAI. Of the publishers and journals with guidelines, the inclusion of GAI as an author was prohibited in 96% and 98%, respectively. Only one journal (1%) explicitly prohibited the use of GAI in the generation of a manuscript, and two (8%) publishers and 19 (22%) journals indicated that their guidelines exclusively applied to the writing process. When disclosing the use of GAI, 75% of publishers and 43% of journals included specific disclosure criteria. Where to disclose the use of GAI varied, including in the methods or acknowledgments, in the cover letter, or in a new section. Variability was also found in how to access GAI guidelines shared between journals and publishers. GAI guidelines in 12 journals directly conflicted with those developed by the publishers. The guidelines developed by top medical journals were broadly similar to those of academic journals. CONCLUSIONS: Guidelines by some top publishers and journals on the use of GAI by authors are lacking. Among those that provided guidelines, the allowable uses of GAI and how it should be disclosed varied substantially, with this heterogeneity persisting in some instances among affiliated publishers and journals. Lack of standardization places a burden on authors and could limit the effectiveness of the regulations. As GAI continues to grow in popularity, standardized guidelines to protect the integrity of scientific output are needed.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Confounder Adjustment Using the Disease Risk Score: A Proposal for Weighting Methods.

Propensity score analysis is a common approach to addressing confounding in nonrandomized studies. Its implementation, however, requires important assumptions (e.g., positivity). The disease risk score (DRS) is an alternative confounding score that can relax some of these assumptions. Like the propensity score, the DRS summarizes multiple confounders into a single score, on which conditioning by matching allows the estimation of causal effects. However, matching relies on arbitrary choices for pruning out data (e.g., matching ratio, algorithm, and caliper width) and may be computationally demanding. Alternatively, weighting methods, common in propensity score analysis, are easy to implement and may entail fewer choices, yet none have been developed for the DRS. Here we present 2 weighting approaches: One derives directly from inverse probability weighting; the other, named target distribution weighting, relates to importance sampling. We empirically show that inverse probability weighting and target distribution weighting display performance comparable to matching techniques in terms of bias but outperform them in terms of efficiency (mean squared error) and computational speed (up to >870 times faster in an illustrative study). We illustrate implementation of the methods in 2 case studies where we investigate placebo treatments for multiple sclerosis and administration of aspirin in stroke patients.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Open science practices need substantial improvement in prognostic model studies in oncology using machine learning.

OBJECTIVE: To describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology. STUDY DESIGN AND SETTING: We conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices. RESULTS: We identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing statements, with 21 (46%) indicating data were available on request to the authors and seven declaring data sharing was not applicable. Two studies (4%) shared data. Only 12 studies (26%) provided code sharing statements, including 2 (4%) that indicated the code was available on request to the authors. Only 11 studies (24%) provided sufficient information to allow their model to be used in practice. The use of reporting guidelines was rare: eight studies (18%) mentioning using a reporting guideline, with 4 (10%) using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement, 1 (2%) using Minimum Information About Clinical Artificial Intelligence Modeling and Consolidated Standards Of Reporting Trials-Artificial Intelligence, 1 (2%) using Strengthening The Reporting Of Observational Studies In Epidemiology, 1 (2%) using Standards for Reporting Diagnostic Accuracy Studies, and 1 (2%) using Transparent Reporting of Evaluations with Nonrandomized Designs. CONCLUSION: The adoption of open science principles in oncology studies developing prognostic models using machine learning methods is poor. Guidance and an increased awareness of benefits and best practices of open science are needed for prediction research in oncology.