Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system.

BACKGROUND: Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers' needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. METHODS: We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer's assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. RESULTS: Reviewers were equally likely to accept judgment by RobotReviewer as each other's judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. CONCLUSION: Despite being presented with evidence of RobotReviewer's equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.

Personal protective effect of wearing surgical face masks in public spaces on self-reported respiratory symptoms in adults: pragmatic randomised superiority trial.

OBJECTIVE: To evaluate the personal protective effects of wearing versus not wearing surgical face masks in public spaces on self-reported respiratory symptoms over a 14 day period. DESIGN: Pragmatic randomised superiority trial. SETTING: Norway. PARTICIPANTS: 4647 adults aged ≥18 years: 2371 were assigned to the intervention arm and 2276 to the control arm. INTERVENTIONS: Participants in the intervention arm were assigned to wear a surgical face mask in public spaces (eg, shopping centres, streets, public transport) over a 14 day period (mask wearing at home or work was not mentioned). Participants in the control arm were assigned to not wear a surgical face mask in public places. MAIN OUTCOME MEASURES: The primary outcome was self-reported respiratory symptoms consistent with a respiratory infection. Secondary outcomes included self-reported and registered covid-19 infection. RESULTS: Between 10 February 2023 and 27 April 2023, 4647 participants were randomised of whom 4575 (2788 women (60.9%); mean age 51.0 (standard deviation 15.0) years) were included in the intention-to-treat analysis: 2313 (50.6%) in the intervention arm and 2262 (49.4%) in the control arm. 163 events (8.9%) of self-reported symptoms consistent with respiratory infection were reported in the intervention arm and 239 (12.2%) in the control arm. The marginal odds ratio was 0.71 (95% confidence interval (CI) 0.58 to 0.87; P=0.001) favouring the face mask intervention. The absolute risk difference was -3.2% (95% CI -5.2% to -1.3%; P<0.001). No statistically significant effect was found on self- reported (marginal odds ratio 1.07, 95% CI 0.58 to 1.98; P=0.82) or registered covid-19 infection (effect estimate and 95% CI not estimable owing to lack of events in the intervention arm). CONCLUSION: Wearing a surgical face mask in public spaces over 14 days reduces the risk of self-reported symptoms consistent with a respiratory infection, compared with not wearing a surgical face mask. TRIAL REGISTRATION: ClinicalTrials.gov NCT05690516.

Effect modification in network meta-analyses for relapsed/refractory multiple myeloma: systematic review and meta-analysis.

OBJECTIVES: To systematically review and meta-analyse the evidence for effect modification by refractory status and number of treatment lines in relapsed/refractory multiple myeloma (RRMM); and to assess whether effect modification is likely to invalidate network meta-analyses (NMA) that assume negligible modification. DESIGN: Systematic review, meta-analysis and simulation. DATA SOURCES: We systematically searched the literature (e.g., OVID Medline) to identify eligible publications in February 2020 and regularly updated the search until January 2022. We also contacted project stakeholders (including industry) ELIGIBILITY CRITERIA: Phase 2 and 3 randomised controlled trials reporting stratified estimates for comparisons with at least one of a prespecified set of treatments relevant for use in Norwegian RRMM patients. OUTCOMES: We used meta-analysis to estimate relative HRs (RHRs) for overall survival (OS) and progression-free survival (PFS) with respect to refractory status and number of treatment lines. We used the estimated RHRs in simulations to estimate the percentage of NMA results expected to differ significantly in the presence versus absence of effect modification. RESULTS: Among the 42 included publications, stratified estimates were published by and extracted from up to 18 (43%) publications and on as many as 8364 patients. Within-study evidence for effect modification is very weak (p>0.05 for 47 of 49 sets of stratified estimates). The largest RHR estimated was 1.32 (95% CI 1.18 to 1.49) for the modifying effect of refractory status on HR for PFS. Simulations suggest that, in the worst case, this would result in only 4.48% (95% CI 4.42% to 4.54%) of NMA estimates differing statistically significantly in the presence versus absence of effect modification. CONCLUSIONS: Based on the available evidence, effect modification appears to be sufficiently small that it can be neglected in adequately performed NMAs. NMAs can probably be relied on to provide estimates of HRs for OS and PFS in RRMM, subject to caveats discussed herein.

Effects of the Informed Health Choices secondary school intervention on the ability of students in Rwanda to think critically about health choices: A cluster-randomized trial.

AIM: The aim of this trial was to evaluate the effects of the Informed Health Choices intervention on the ability of students in Rwandan to think critically and make Informed Health Choices. METHODS: We conducted a two-arm cluster-randomized trial in 84 lower secondary schools from 10 districts representing five provinces of Rwanda. We used stratified randomization to allocate schools to the intervention or control. One class in each intervention school had ten 40-min lessons taught by a trained teacher in addition to the usual curriculum. Control schools followed the usual curriculum. The primary outcome was a passing score (≥ 9 out of 18 questions answered correctly) for students on the Critical Thinking about Health Test completed within 2 weeks after the intervention. We conducted an intention-to-treat analysis using generalized linear mixed models, accounting for the cluster design using random intercepts. RESULTS: Between February 25 and March 29, 2022, we recruited 3,212 participants. We assigned 1,572 students and 42 teachers to the intervention arm and 1,556 students and 42 teachers to the control arm. The proportion of students who passed the test in the intervention arm was 915/1,572 (58.2%) compared to 302/1,556 (19.4%) in the control arm, adjusted odds ratio 10.6 (95% CI: 6.3-17.8), p < 0.0001, adjusted difference 37.2% (95% CI: 29.5%-45.0%). CONCLUSIONS: The intervention is effective in helping students think critically about health choices. It was possible to improve students' ability to think critically about health in the context of a competence-based curriculum in Rwanda, despite challenging postpandemic conditions.

Effects of the informed health choices secondary school intervention on the ability of students in Kenya to think critically about health choices: A cluster-randomized trial.

AIM: There is an overabundance of claims about the advantages and disadvantages of health interventions. People need to be able to appraise the reliability of these claims. The aim of this two-arm cluster-randomized trial was to evaluate the Informed Health Choices secondary school intervention designed to teach students to assess claims about the effects of health actions and make informed decisions. METHODS: We conducted the trial among students from 80 secondary schools in five subcounties in Kenya. We used stratified randomization to allocate schools to the intervention or control arm. The intervention included a 2-day teacher training workshop and 10 lessons that addressed nine prioritized key concepts for assessing claims about treatment effects. We did not intervene in the control schools. The primary outcome was the proportion of students with a passing score (≥ 9/18 correct answers) on the Critical Thinking about Health test, which included two multiple-choice questions for each concept. RESULTS: Between May 11, 2022, and July 8, 2022, we recruited 3362 students and 80 teachers. We allocated 1863 students and 40 teachers to the intervention and 1499 students and 40 teachers to the control arm. In the intervention schools, 1149/1863 (61.7%) of students achieved a passing score compared to 511/1499 (34.1%) in the control schools (odds ratio 3.6 (95% CI 2.5-5.2), p < 0.0001). CONCLUSIONS: The intervention had a large effect on students' ability to think critically about health interventions. It is possible to integrate the learning of critical thinking about health within Kenya secondary school curriculum.

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study.

BACKGROUND: Machine learning (ML) tools exist that can reduce or replace human activities in repetitive or complex tasks. Yet, ML is underutilized within evidence synthesis, despite the steadily growing rate of primary study publication and the need to periodically update reviews to reflect new evidence. Underutilization may be partially explained by a paucity of evidence on how ML tools can reduce resource use and time-to-completion of reviews. METHODS: This protocol describes how we will answer two research questions using a retrospective study design: Is there a difference in resources used to produce reviews using recommended ML versus not using ML, and is there a difference in time-to-completion? We will also compare recommended ML use to non-recommended ML use that merely adds ML use to existing procedures. We will retrospectively include all reviews conducted at our institute from 1 August 2020, corresponding to the commission of the first review in our institute that used ML. CONCLUSION: The results of this study will allow us to quantitatively estimate the effect of ML adoption on resource use and time-to-completion, providing our organization and others with better information to make high-level organizational decisions about ML.

Use of the informed health choices educational intervention to improve secondary students' ability to think critically about health interventions in Uganda: A cluster-randomized trial.

AIM: The aim was to evaluate the effect of the Informed Health Choices (IHC) educational intervention on secondary students' ability to assess health-related claims and make informed choices. METHODS: In a cluster-randomized trial, we randomized 80 secondary schools (students aged 13-17 years) in Uganda to the intervention or control (usual curriculum). The intervention included a 2-day teacher training workshop, 10 lessons accessed online by teachers and delivered in one school term. The lesson plans were developed for classrooms equipped with a blackboard or a blackboard and projector. The lessons addressed nine prioritized concepts. We used two multiple-choice questions for each concept to evaluate the students' ability to assess claims and make informed choices. The primary outcome was the proportion of students with a passing score (≥9 of 18 questions answered correctly). RESULTS: Eighty schools consented and were randomly allocated. A total of 2477 students in the 40 intervention schools and 2376 students in the 40 control schools participated in this trial. In the intervention schools, 1364 (55%) of students that completed the test had a passing score compared with 586 (25%) of students in the control schools (adjusted difference 33%, 95% CI 26%-39%). CONCLUSIONS: The IHC secondary school intervention improved students' ability to think critically and make informed choices. Well-designed digital resources may improve access to educational material, even in schools without computers or other information and communication technology (ICT). This could facilitate scaling-up use of the resources and help to address inequities associated with limited ICT access.

Machine learning in systematic reviews: Comparing automated text clustering with Lingo3G and human researcher categorization in a rapid review.

Systematic reviews are resource-intensive. The machine learning tools being developed mostly focus on the study identification process, but tools to assist in analysis and categorization are also needed. One possibility is to use unsupervised automatic text clustering, in which each study is automatically assigned to one or more meaningful clusters. Our main aim was to assess the usefulness of an automated clustering method, Lingo3G, in categorizing studies in a simplified rapid review, then compare performance (precision and recall) of this method compared to manual categorization. We randomly assigned all 128 studies in a review to be coded by a human researcher blinded to cluster assignment (mimicking two independent researchers) or by a human researcher non-blinded to cluster assignment (mimicking one researcher checking another's work). We compared time use, precision and recall of manual categorization versus automated clustering. Automated clustering and manual categorization organized studies by population and intervention/context. Automated clustering failed to identify two manually identified categories but identified one additional category not identified by the human researcher. We estimate that automated clustering has similar precision to both blinded and non-blinded researchers (e.g., 88% vs. 89%), but higher recall (e.g., 89% vs. 84%). Manual categorization required 49% more time than automated clustering. Using a specific clustering algorithm, automated clustering can be helpful with categorization of and identifying patterns across studies in simpler systematic reviews. We found that the clustering was sensitive enough to group studies according to linguistic differences that often corresponded to the manual categories.

eRegTime-Time Spent on Health Information Management in Primary Health Care Clinics Using a Digital Health Registry Versus Paper-Based Documentation: Cluster-Randomized Controlled Trial.

BACKGROUND: Digital health interventions have been shown to improve data quality and health services in low- and middle-income countries (LMICs). Nonetheless, in LMICs, systematic assessments of time saved with the use of digital tools are rare. We ran a set of cluster-randomized controlled trials as part of the implementation of a digital maternal and child health registry (eRegistry) in the West Bank, Palestine. OBJECTIVE: In the eRegTime study, we compared time spent on health information management in clinics that use the eRegistry versus the existing paper-based documentation system. METHODS: Intervention (eRegistry) and control (paper documentation) arms were defined by a stratified random subsample of primary health care clinics from the concurrent eRegQual trial. We used time-motion methodology to collect data on antenatal care service provision. Four observers used handheld tablets to record time-use data during one working day per clinic. We estimated relative time spent on health information management for booking and follow-up visits and on client care using mixed-effects linear regression. RESULTS: In total, 22 of the 24 included clinics (12 intervention, 10 control) contributed data; no antenatal care visits occurred in the other two clinics during the study period. A total of 123 and 118 consultations of new pregnancy registrations and follow-up antenatal care visits were observed in the intervention and control groups, respectively. Average time spent on health information management for follow-up antenatal care visits in eRegistry clinics was 5.72 minutes versus 8.10 minutes in control clinics (adjusted relative time 0.69, 95% CI 0.60-0.79; P<.001), and 15.26 minutes versus 18.91 minutes (adjusted relative time 0.96, 95% CI 0.61-1.50; P=.85) for booking visits. The average time spent on documentation, a subcategory of health information management, was 5.50 minutes in eRegistry clinics versus 8.48 minutes in control clinics (adjusted relative time 0.68, 95% CI 0.56-0.83; P<.001). While the average time spent on client care was 5.01 minutes in eRegistry clinics versus 4.91 minutes in control clinics, some uncertainty remains, and the CI was consistent with eRegistry clinics using less, the same, or more time on client care compared to those that use paper (adjusted relative time 0.85, 95% CI 0.64-1.13; P=.27). CONCLUSIONS: The eRegistry captures digital data at point of care during client consultations and generates automated routine reports based on the clinical data entered. Markedly less time (plausibly a saving of at least 18%) was spent on health information management in eRegistry clinics compared to those that use paper-based documentation. This is likely explained by the fact that the eRegistry requires lesser repetitive documentation work than paper-based systems. Adoption of eRegistry-like systems in comparable settings may save valuable and scarce health care resources. TRIAL REGISTRATION: ISRCTN registry ISRCTN18008445; https://doi.org/10.1186/ISRCTN18008445. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/13653.

Quality of information in news media reports about the effects of health interventions: Systematic review and meta-analyses.

Background Many studies have assessed the quality of news reports about the effects of health interventions, but there has been no systematic review of such studies or meta-analysis of their results. We aimed to fill this gap (PROSPERO ID: CRD42018095032). Methods We included studies that used at least one explicit, prespecified and generic criterion to assess the quality of news reports in print, broadcast, or online news media, and specified the sampling frame, and the selection criteria and technique. We assessed criteria individually for inclusion in the meta-analyses, excluding ineligible criteria and criteria with inadequately reported results. We mapped and grouped criteria to facilitate evidence synthesis. Where possible, we extracted the proportion of news reports meeting the included criterion. We performed meta-analyses using a random effects model to estimate such proportions for individual criteria and some criteria groups, and to characterise heterogeneity across studies.  Results We included 44 primary studies in the review, and 18 studies and 108 quality criteria in the meta-analyses. Many news reports gave an unbalanced and oversimplified picture of the potential consequences of interventions. A limited number mention or adequately address conflicts of interest (22%; 95% CI 7%-49%) (low certainty), alternative interventions (36%; 95% CI 26%-47%) (moderate certainty), potential harms (40%; 95% CI 23%-61%) (low certainty), or costs (18%; 95% CI 12%-28%) (moderate certainty), or quantify effects (53%; 95% CI 36%-69%) (low certainty) or report absolute effects (17%; 95% CI 4%-49%) (low certainty).  Discussion There is room for improving health news, but it is logically more important to improve the public's ability to critically appraise health information and make judgements for themselves.

The Norwegian public's ability to assess treatment claims: results of a cross-sectional study of critical health literacy.

Background: Few studies have evaluated the ability of the general public to assess the trustworthiness of claims about the effects of healthcare. For the most part, those studies have used self-reported measures of critical health literacy. Methods: We mailed 4500 invitations to Norwegian adults. Respondents were randomly assigned to one of four online questionnaires that included multiple-choice questions that test understanding of Key Concepts people need to understand to assess healthcare claims. They also included questions about intended behaviours and self-efficacy. One of the four questionnaires was identical to one previously used in two randomised trials of educational interventions in Uganda, facilitating comparisons to Ugandan children, parents, and teachers. We adjusted the results using demographic data to reflect the population. Results: A total of 771 people responded. The adjusted proportion of Norwegian adults who answered correctly was < 50% for 17 of the 30 Key Concepts. On the other hand, less than half answered correctly for 13 concepts. The results for Norwegian adults were better than the results for Ugandan children in the intervention arm of the trial and parents, and similar to those of Ugandan teachers in the intervention arm of the trial. Based on self-report, most Norwegians are likely to find out the basis of treatment claims, but few consider it easy to assess whether claims are based on research and to assess the trustworthiness of research. Conclusions: Norwegian adults do not understand many concepts that are essential for assessing healthcare claims and making informed choices. Future interventions should be tailored to address Key Concepts for which there appears to be a lack of understanding.

Predictors of chronic pain and level of physical function in total knee arthroplasty: a protocol for a systematic review and meta-analysis.

INTRODUCTION: One in five patients undergoing total knee arthroplasty (TKA) experience unchanged or worse pain and physical function 1 year after surgery. Identifying risk factors for unfavourable outcomes is necessary to develop tailored interventions to minimise risk. There is a need to review more current literature with updated methodology that addresses the limitations of earlier systematic reviews and meta-analyses. We present a Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols compliant protocol for a systematic review and meta-analysis of predictors of chronic pain and impaired function after TKA. METHODS AND ANALYSIS: This review will include prospective longitudinal observational studies, or randomised trials (including cluster and crossover designs) that report arm-wise predictors of chronic postsurgical pain or impaired physical function at 3 months, 6 months or 12 months. A comprehensive literature search of studies published between 2000 and 2019 will be performed in Medline, Embase, CINAHL, Cochrane Library and PEDro. Blinded assessment with consensus agreement will be applied for inclusion of studies, data extraction and assessment of bias risk (Quality in Prognosis Studies tool). The co-primary outcomes, pain and impaired function, at 12 months after TKA will be analysed separately. Estimates of association between each outcome and any preoperative or intraoperative factor that may predict chronic pain or impaired physical function will be extracted from the included studies, where possible. For randomised studies, results will only be extracted from TKA arms (or the first period of crossover trials). Estimates of association from the primary evidence will be synthesised narratively, and quantitatively using multivariate meta-analysis to provide 'pooled' estimates of association. Subgroup and sensitivity analyses will be performed. Certainty of evidence for each predictor will be derived from the Grading of Recommendations Assessment, Development and Evaluation framework. ETHICS AND DISSEMINATION: No ethical issues are associated with this project. The results from this review will be published in peer-reviewed journals and presented at international conferences. PROSPERO REGISTRATION NUMBER: CRD42018079069.

Informed Health Choices media intervention for improving people's ability to critically appraise the trustworthiness of claims about treatment effects: a mixed-methods process evaluation of a randomised trial in Uganda.

We developed the Informed Health Choices podcast to improve people's ability to assess claims about the effects of treatments. We evaluated the effects of the podcast in a randomised trial. OBJECTIVES: We conducted this process evaluation to assess the fidelity of the intervention, identify factors that affected the implementation and impact of the intervention and could affect scaling up, and identify potential adverse and beneficial effects. SETTING: The study was conducted in central Uganda in rural, periurban and urban settings. PARTICIPANTS: We collected data on parents who were in the intervention arm of the Informed Health Choices study that evaluated an intervention to improve parents' ability to assess treatment effects. PROCEDURES: We conducted 84 semistructured interviews during the intervention, 19 in-depth interviews shortly after, two focus group discussions with parents, one focus group discussion with research assistants and two in-depth interviews with the principal investigators. We used framework analysis to manage qualitative data, assessed the certainty of the findings using the GRADE-CERQual (Grading of Recommendations, Assessment, Development and Evaluations-Confidence in the Evidence from Reviews of Qualitative Research) approach, and organised findings in a logic model. OUTCOMES: Proportion of participants listening to all episodes; factors influencing the implementation of the podcast; ways to scale up and any adverse and beneficial effects. RESULTS: All participants who completed the study listened to the podcast as intended, perhaps because of the explanatory design and recruitment of parents with a positive attitude. This was also likely facilitated by the podcast being delivered by research assistants, and providing the participants with MP3 players. The podcast was reportedly clear, understandable, credible and entertaining, which motivated them to listen and eased implementation. No additional adverse effects were reported. CONCLUSIONS: Participants experienced the podcast positively and were motivated to engage with it. These findings help to explain the short-term effectiveness of the intervention, but not the decrease in effectiveness over the following year.

Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation.

BACKGROUND: We developed the informed health choices (IHC) primary school resources to teach children how to assess the trustworthiness of claims about the effects of treatments. We evaluated these resources in a randomised trial in Uganda. This paper describes the process evaluation that we conducted alongside this trial. OBJECTIVES: To identify factors affecting the implementation, impact and scaling up of the intervention; and potential adverse and beneficial effects of the intervention. METHODS: All 85 teachers in the 60 schools in the intervention arm of the trial completed a questionnaire after each lesson and at the end of the term. We conducted structured classroom observations at all 60 schools. For interviews and focus groups, we purposively selected six schools. We interviewed district education officers, teachers, head teachers, children and their parents. We used a framework analysis approach to analyse the data. RESULTS: Most of the participants liked the IHC resources and felt that the content was important. This motivated the teachers and contributed to positive attitudes. Although some teachers started out lacking confidence, many found that the children's enthusiasm for the lessons made them more confident. Nearly everyone interviewed thought that the children learnt something important and many thought that it improved their decision-making. The main barrier to scaling up use of the IHC resources that participants identified was the need to incorporate the lessons into the national curriculum. CONCLUSION: The mostly positive findings reflect the trial results, which showed large effects on the children's and the teachers' critical appraisal skills. The main limitations of this evaluation are that the investigators were responsible for both developing and evaluating the intervention.