Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models.

Importance: Large language models (LLMs) may facilitate the labor-intensive process of systematic reviews. However, the exact methods and reliability remain uncertain. Objective: To explore the feasibility and reliability of using LLMs to assess risk of bias (ROB) in randomized clinical trials (RCTs). Design, Setting, and Participants: A survey study was conducted between August 10, 2023, and October 30, 2023. Thirty RCTs were selected from published systematic reviews. Main Outcomes and Measures: A structured prompt was developed to guide ChatGPT (LLM 1) and Claude (LLM 2) in assessing the ROB in these RCTs using a modified version of the Cochrane ROB tool developed by the CLARITY group at McMaster University. Each RCT was assessed twice by both models, and the results were documented. The results were compared with an assessment by 3 experts, which was considered a criterion standard. Correct assessment rates, sensitivity, specificity, and F1 scores were calculated to reflect accuracy, both overall and for each domain of the Cochrane ROB tool; consistent assessment rates and Cohen κ were calculated to gauge consistency; and assessment time was calculated to measure efficiency. Performance between the 2 models was compared using risk differences. Results: Both models demonstrated high correct assessment rates. LLM 1 reached a mean correct assessment rate of 84.5% (95% CI, 81.5%-87.3%), and LLM 2 reached a significantly higher rate of 89.5% (95% CI, 87.0%-91.8%). The risk difference between the 2 models was 0.05 (95% CI, 0.01-0.09). In most domains, domain-specific correct rates were around 80% to 90%; however, sensitivity below 0.80 was observed in domains 1 (random sequence generation), 2 (allocation concealment), and 6 (other concerns). Domains 4 (missing outcome data), 5 (selective outcome reporting), and 6 had F1 scores below 0.50. The consistent rates between the 2 assessments were 84.0% for LLM 1 and 87.3% for LLM 2. LLM 1's κ exceeded 0.80 in 7 and LLM 2's in 8 domains. The mean (SD) time needed for assessment was 77 (16) seconds for LLM 1 and 53 (12) seconds for LLM 2. Conclusions: In this survey study of applying LLMs for ROB assessment, LLM 1 and LLM 2 demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes.

The effects of inulin-type fructans on cardiovascular disease risk factors: systematic review and meta-analysis of randomized controlled trials.

BACKGROUND: Inulin-type fructans (ITF) are the leading prebiotics in the market. Available evidence provides conflicting results regarding the beneficial effects of ITF on cardiovascular disease risk factors. OBJECTIVES: This study aimed to evaluate the effects of ITF supplementation on cardiovascular disease risk factors in adults. METHODS: We searched MEDLINE, EMBASE, Emcare, AMED, CINAHL, and the Cochrane Library databases from inception through May 15, 2022. Eligible randomized controlled trials (RCTs) administered ITF or placebo (for example, control, foods, diets) to adults for ≥2 weeks and reported one or more of the following: low, very-low, or high-density lipoprotein cholesterol (LDL-C, VLDL-C, HDL-C); total cholesterol; apolipoprotein A1 or B; triglycerides; fasting blood glucose; body mass index; body weight; waist circumference; waist-to-hip ratio; systolic or diastolic blood pressure; or hemoglobin A1c. Two reviewers independently and in duplicate screened studies, extracted data, and assessed risk of bias. We pooled data using random-effects model, and assessed the certainty of evidence (CoE) using the Grading of Recommendations, Assessment, Development and Evaluation approach. RESULTS: We identified 1767 studies and included 55 RCTs with 2518 participants in meta-analyses. The pooled estimate showed that ITF supplementation reduced LDL-C [mean difference (MD) -0.14 mmol/L, 95% confidence interval (95% CI: -0.24, -0.05), 38 RCTs, 1879 participants, very low CoE], triglycerides (MD -0.06 mmol/L, 95% CI: -0.12, -0.01, 40 RCTs, 1732 participants, low CoE), and body weight (MD -0.97 kg, 95% CI: -1.28, -0.66, 36 RCTs, 1672 participants, low CoE) but little to no significant effect on other cardiovascular disease risk factors. The effects were larger when study duration was ≥6 weeks and in pre-obese and obese participants. CONCLUSION: ITF may reduce low-density lipoprotein, triglycerides, and body weight. However, due to low to very low CoE, further well-designed and executed trials are needed to confirm these effects. PROSPERO REGISTRATION NUMBER: CRD42019136745.

Assessment of reporting quality of randomized controlled trials investigating the effects of inulin-type fructans supplementation on cardiovascular disease risk factors: A systematic survey.

BACKGROUND: Transparent and detailed reporting of randomized controlled trials (RCTs) is essential to judge its validity and generalizability. We assessed the reporting quality of RCTs examining the effects of inulin-type fructans supplementation on cardiovascular risk factors, before and after the publication of the Consolidated Standards of Reporting Trials (CONSORT) in 2010. METHODS: We searched MEDLINE, EMBASE, Emcare, AMED, the Cochrane Library, and CINAHL from inception to May 15, 2022, including the reference lists of selected RCTs. We screened titles and abstracts and extracted the data independently and in duplicate. We included RCTs that investigated the effects of inulin-type fructans on cardiovascular disease risk factors (e.g., low-density lipoprotein cholesterol, triglycerides, fasting blood glucose) in adults (18 years or older). The primary outcomes of this study were: the overall reporting quality of RCTs (defined as the total number of items [0 to 36] present from the CONSORT checklist) published before and after CONSORT; and the study characteristics (e.g., sample size, significance of primary outcome) predictive of the CONSORT score. The secondary outcome was the reporting of each specific item of the CONSORT checklist during pre- and post-CONSORT periods. The mean difference in the total number of reported items in studies published before and after CONSORT were compared using a t-test and Poisson regression to explore the factors associated with overall reporting quality of RCTs. We used Fisher's exact test to compare the adherence to each of the 36 items during pre- and post-CONSORT periods. RESULTS: We identified 1,767 citations from our systematic search, of which 55 were eligible. There was a significant increase in the reporting of CONSORT items (mean difference 8.5, 95% confidence interval [CI] 5.24 to 11.71) between studies published before and after publication of CONSORT. The sole variable that was predictive of better reporting quality of RCTs was whether the study was published before or after CONSORT (incidence rate ratio 1.67, 95% CI 1.40 to 2.02). Completeness of reporting of RCTs only improved in 15 out of 36 items (41.6%) after the publication of CONSORT. CONCLUSION: The completeness of reporting in RCTs investigating inulin-type fructans supplementation on cardiovascular disease risk factors remains inadequate after the publication of CONSORT. Greater adherence to CONSORT by authors and enforcement of CONSORT by journals may improve the quality of reporting among RCTs.

Saturated fat, the estimated absolute risk and certainty of risk for mortality and major cancer and cardiometabolic outcomes: an overview of systematic reviews.

OBJECTIVE: To assess the impact of reducing saturated fat or fatty foods, or replacing saturated fat with unsaturated fat, carbohydrate or protein, on the risk of mortality and major cancer and cardiometabolic outcomes in adults. METHODS: We searched MEDLINE, EMBASE, CINAHL, and references of included studies for systematic reviews and meta-analyses (SRMAs) of randomized controlled trials (RCTs) and observational studies in adults published in the past 10 years. Eligible reviews investigated reducing saturated fat or fatty foods or replacing saturated fat with unsaturated fat, carbohydrate or protein, on the risk of cancer and cardiometabolic outcomes and assessed the certainty of evidence for each outcome using, for example, the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) approach. We assessed the quality of SRMAs using a modified version of AMSTAR-2. Results were summarized as absolute estimates of effect together with the certainty of effects using a narrative synthesis approach. RESULTS: We included 17 SRMAs (13 reviews of observational studies with follow-up 1 to 34 years; 4 reviews of RCTs with follow-up 1 to 17 years). The quality of two-thirds of the SRMAs was critically low to moderate; the main limitations included deficient reporting of study selection, absolute effect estimates, sources of funding, and a priori subgroups to explore heterogeneity. Our included reviews reported > 100 estimates of effect across 11 critically important cancer and cardiometabolic outcomes. High quality SRMAs consistently and predominantly reported low to very low certainty evidence that reducing or replacing saturated fat was associated with a very small risk reduction in cancer and cardiometabolic endpoints. The risk reductions where approximately divided, some being statistically significant and some being not statistically significant. However, based on 2 moderate to high quality reviews, we found moderate certainty evidence for a small but important effect that was statistically significant for two outcomes (total mortality events [20 fewer events per 1000 followed] and combined cardiovascular events [16 fewer per 1000 followed]). Conversely, 4 moderate to high quality reviews showed very small effects on total mortality, with 3 of these reviews showing non-statistically significant mortality effects. CONCLUSION: Systematic reviews investigating the impact of SFA on mortality and major cancer and cardiometabolic outcomes almost universally suggest very small absolute changes in risk, and the data is based primarily on low and very low certainty evidence. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42020172141.

Effects of inulin-type fructans supplementation on cardiovascular disease risk factors: a protocol for a systematic review and meta-analysis of randomised controlled trials.

INTRODUCTION: This review aims to assess the effects of dietary supplementation with inulin-type fructans (ITF) compared with no supplementation on cardiovascular disease risk factors in adults and assess the quality of trial reporting using the Consolidated Standards of Reporting Trials (CONSORT) and CONSORT for abstract (CONSORT-A) checklists. METHODS AND ANALYSIS: We will search randomised controlled trials (RCTs) in MEDLINE, EMBASE, CINAHL, Emcare, AMED and the Cochrane Database of Systematic Reviews from inception to 31 March 2022, without any language restrictions. The RCTs need to administer ITF in adults for at least 2 weeks and assess effects on at least one cardiovascular risk factor. We will exclude RCTs that (1) assessed the postprandial effects of ITF; (2) included pregnant or lactating participants; (3) enrolled participants undergoing treatment that might affect the response to ITF. We will assess the study risk of bias (RoB) using V.2 of the Cochrane RoB tool for RCTs (RoB 2) and the certainty of the evidence using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach. We will pool data using a random-effects model. We will use the χ2 test to compare compliance of CONSORT and CONSORT-A checklists and Poisson regression to identify factors associated with better reporting. ETHICS AND DISSEMINATION: Ethics approval is not required for secondary analysis of already published data. We will publish the reviews in a peer-review journal. PROSPERO REGISTRATION NUMBER: CRD42019136745.

Causal language use in systematic reviews of observational studies is often inconsistent with intent: a systematic survey.

OBJECTIVES: We systematically evaluated causal language use in systematic reviews of observational studies and explored the relation between language use and the intent of the investigation. STUDY DESIGN AND SETTING: We searched EMBASE, MEDLINE, and Epistemonikos. We randomly selected 199 reviews published in 2019, stratified in a 1:1 ratio by use and nonuse of the Grading of Recommendations Assessment, Development and Evaluation approach to rating quality of evidence. RESULTS: Of 199 reviews of observational studies 56.8% had causal intent. Reviews with causal intent were more likely to investigate therapeutic clinical intervention (33.6% vs. 12.8%). Although 78.8% of those with causal intent used causal language in one or more sections of the title, abstract, or main text, only 4.4% consistently used causal language throughout the manuscript, and 21.2% did not use causal language at all. Of reviews without causal intent, 51.2% used causal language somewhere in the manuscript. CONCLUSION: Systematic reviews of observational studies sometimes do and sometimes do not have causal intent. Both those are inconsistent in causal language use and often use language inconsistent with the intent. Journal policies would better serve clarity of thinking and appropriateness of inferences by demanding authors clearly specify their intent and consistently use language consistent with that intent.

Primary health care seeking behaviour of people with physical disabilities in Bangladesh: a cross-sectional study.

BACKGROUND: People with disabilities constitute about 10% of the total population of Bangladesh. They are more likely to experience poor health than those without disabilities. However, there is a lack of evidence on their primary health care (PHC) seeking behaviour for their general illness. The aim of this study was to understand the PHC seeking behaviour of people with physical disabilities (PWPDs), and to investigate the determinants of such behaviours. METHODS: We surveyed 282 PWPDs, aged ≥18 years, using a structured questionnaire. Participants were recruited from the out-patient department of a rehabilitation centre in Dhaka between November and December 2014. We explored PHC seeking behaviour using frequency distribution. We performed logistic regression to investigate the factors that determined their PHC seeking behaviours for general illness. In our logistic regression model, the outcome variable was whether PWPDs received treatment from a formal health care provider. The predictors were socio-demographic characteristics and clinical characteristics such as type of impairment and type of illness experienced. RESULTS: Among 282 participants, 85% suffered from general illness in the past 6 months. The participants in the higher age group, for example, age group 31-45 years (OR = 3.9, [95% CI 1.2 to 13.4]), 46-59 years (OR = 13.6, [95% CI 2.9 to 63.7) and 60+ years (OR = 12.5, [95% CI 1.7 to 93.0]) were more likely to seek treatment from formal health care providers than the age group 18-30 years. The educational attainment of the primary income earning family member (OR = 3.2, [CI 1.1 to 9.6]), religion (OR = 0.3, [95% CI 0.1 to 0.98]) and mobility aid used (OR = 4.0, [95% CI 1.2 to 13]) were determinants for seeking health care from a formal health care provider. Moreover, the type of illness suffered by participant was a strong predictor of their decision to seek treatment from a formal health care provider. The participants who suffered from urinary tract infections (OR = 10.3, [95% CI 2.3 to 46.6]), ulcers (OR = 13.1, [95% CI 2.11 to 79.3]) and pain (OR = 3.6, [95% CI 1.4 to 9.4]) were more likely to seek treatment from formal health care provider than who suffered from fever. CONCLUSIONS: Age, religion, earning member's education, type of mobility aids used and type of illness suffered were explicative determinants of PHC seeking behaviour of PWPDs. The results suggest that these factors should be considered when devising interventions for this population. Moreover, accessibility, quality of care and expertise of the providers in treating disabled people were among the other factors reported by PWPDs which influence their decision to seek health care. In order to provide inclusive health services, primary health centres need to consider these determinants.