Patient preferences for breast cancer screening: a systematic review update to inform recommendations by the Canadian Task Force on Preventive Health Care.

BACKGROUND: Different guideline panels, and individuals, may make different decisions based in part on their preferences. Preferences for or against an intervention are viewed as a consequence of the relative importance people place on the expected or experienced health outcomes it incurs. These findings can then be considered as patient input when balancing effect estimates on benefits and harms reported by empirical evidence on the clinical effectiveness of screening programs. This systematic review update examined the relative importance placed by patients on the potential benefits and harms of mammography-based breast cancer screening to inform an update to the 2018 Canadian Task Force on Preventive Health Care's guideline on screening. METHODS: We screened all articles from our previous review (search December 2017) and updated our searches to June 19, 2023 in MEDLINE, PsycINFO, and CINAHL. We also screened grey literature, submissions by stakeholders, and reference lists. The target population was cisgender women and other adults assigned female at birth (including transgender men and nonbinary persons) aged ≥ 35 years and at average or moderately increased risk for breast cancer. Studies of patients with breast cancer were eligible for health-state utility data for relevant outcomes. We sought three types of data, directly through (i) disutilities of screening and curative treatment health states (measuring the impact of the outcome on one's health-related quality of life; utilities measured on a scale of 0 [death] to 1 [perfect health]), and (ii) other preference-based data, such as outcome trade-offs, and indirectly through (iii) the relative importance of benefits versus harms inferred from attitudes, intentions, and behaviors towards screening among patients provided with estimates of the magnitudes of benefit(s) and harms(s). For screening, we used machine learning as one of the reviewers after at least 50% of studies had been reviewed in duplicate by humans; full-text selection used independent review by two humans. Data extraction and risk of bias assessments used a single reviewer with verification. Our main analysis for utilities used data from utility-based health-related quality of life tools (e.g., EQ-5D) in patients; a disutility value of about 0.04 can be considered a minimally important value for the Canadian public. When suitable, we pooled utilities and explored heterogeneity. Disutilities were calculated for screening health states and between different treatment states. Non-utility data were grouped into categories, based on outcomes compared (e.g. for trade-off data), participant age, and our judgements of the net benefit of screening portrayed by the studies. Thereafter, we compared and contrasted findings while considering sample sizes, risk of bias, subgroup findings and data on knowledge scores, and created summary statements for each data set. Certainty assessments followed GRADE guidance for patient preferences and used consensus among at least two reviewers. FINDINGS: Eighty-two studies (38 on utilities) were included. The estimated disutilities were 0.07 for a positive screening result (moderate certainty), 0.03-0.04 for a false positive (FP; "additional testing" resolved as negative for cancer) (low certainty), and 0.08 for untreated screen-detected cancer (moderate certainty) or (low certainty) an interval cancer. At ≤12 months, disutilities of mastectomy (vs. breast-conserving therapy), chemotherapy (vs. none) (low certainty), and radiation therapy (vs. none) (moderate certainty) were 0.02-0.03, 0.02-0.04, and little-to-none, respectively, though in each case findings were somewhat limited in their applicability. Over the longer term, there was moderate certainty for little-to-no disutility from mastectomy versus breast-conserving surgery/lumpectomy with radiation and from radiation. There was moderate certainty that a majority (>50%) and possibly a large majority (>75%) of women probably accept up to six cases of overdiagnosis to prevent one breast-cancer death; there was some uncertainty because of an indication that overdiagnosis was not fully understood by participants in some cases. Low certainty evidence suggested that a large majority may accept that screening may reduce breast-cancer but not all-cause mortality, at least when presented with relatively high rates of breast-cancer mortality reductions (n = 2; 2 and 5 fewer per 1000 screened), and at least a majority accept that to prevent one breast-cancer death at least a few hundred patients will receive a FP result and 10-15 will have a FP resolved through biopsy. An upper limit for an acceptable number of FPs was not evaluated. When using data from studies assessing attitudes, intentions, and screening behaviors, across all age groups but most evident for women in their 40s, preferences reduced as the net benefit presented by study authors decreased in magnitude. In a relatively low net-benefit scenario, a majority of patients in their 40s may not weigh the benefits as greater than the harms from screening whereas for women in their 50s a large majority may prefer screening (low certainty evidence for both ages). There was moderate certainty that a large majority of women 50 years of age and 50 to 69 years of age, who have usually experienced screening, weigh the benefits as greater than the harms from screening in a high net-benefit scenario. A large majority of patients aged 70-71 years who have recently screened probably think the benefits outweigh the harms of continuing to screen. A majority of women in their mid-70s to early 80s may prefer to continue screening. CONCLUSIONS: Evidence across a range of data sources on how informed patients value the potential outcomes from breast-cancer screening will be useful during decision-making for recommendations. The evidence suggests that all of the outcomes examined have importance to women of any age, that there is at least some and possibly substantial (among those in their 40s) variability across and within age groups about the acceptable magnitude of effects across outcomes, and that provision of easily understandable information on the likelihood of the outcomes may be necessary to enable informed decision making. Although studies came from a wide range of countries, there were limited data from Canada and about whether findings applied well across an ethnographically and socioeconomically diverse population. SYSTEMATIC REVIEW REGISTRATION: Protocol available at Open Science Framework https://osf.io/xngsu/ .

Screening for lung cancer with computed tomography: protocol for systematic reviews for the Canadian Task Force on Preventive Health Care.

PURPOSE: Lung cancer is the leading cause of cancer deaths in Canada, and because early cancers are often asymptomatic screening aims to prevent mortality by detecting cancer earlier when treatment is more likely to be curative. These reviews will inform updated recommendations by the Canadian Task Force on Preventive Health Care on screening for lung cancer. METHODS: We will update the review on the benefits and harms of screening with CT conducted for the task force in 2015 and perform de novo reviews on the comparative effects between (i) trial-based selection criteria and use of risk prediction models and (ii) trial-based nodule classification and different nodule classification systems and on patients' values and preferences. We will search Medline, Embase, and Cochrane Central (for questions on benefits and harms from 2015; comparative effects from 2012) and Medline, Scopus, and EconLit (for values and preferences from 2012) via peer-reviewed search strategies, clinical trial registries, and the reference lists of included studies and reviews. Two reviewers will screen all citations (including those in the previous review) and base inclusion decisions on consensus or arbitration by another reviewer. For benefits (i.e., all-cause and cancer-specific mortality and health-related quality of life) and harms (i.e., overdiagnosis, false positives, incidental findings, psychosocial harms from screening, and major complications and mortality from invasive procedures as a result of screening), we will include studies of adults in whom lung cancer is not suspected. We will include randomized controlled trials comparing CT screening with no screening or alternative screening modalities (e.g., chest radiography) or strategies (e.g., CT using different screening intervals, classification systems, and/or patient selection via risk models or biomarkers); non-randomized studies, including modeling studies, will be included for the comparative effects between trial-based and other selection criteria or nodule classification methods. For harms (except overdiagnosis) we will also include non-randomized and uncontrolled studies. For values and preferences, the study design may be any quantitative design that either directly or indirectly measures outcome preferences on outcomes pertaining to lung cancer screening. We will only include studies conducted in Very High Human Development Countries and having full texts in English or French. Data will be extracted by one reviewer with verification by another, with the exception of result data on mortality and cancer incidence (for calculating overdiagnosis) where duplicate extraction will occur. If two or more studies report on the same comparison and it is deemed suitable, we will pool continuous data using a mean difference or standardized mean difference, as applicable, and binary data using relative risks and a DerSimonian and Laird model unless events are rare (< 1%) where we will pool odds ratios using Peto's method or (if zero events) the reciprocal of the opposite treatment arm size correction. For pooling proportions, we will apply suitable transformation (logit or arcsine) depending on the proportions of events. If meta-analysis is not undertaken we will synthesize the data descriptively, considering clinical and methodological differences. For each outcome, two reviewers will independently assess within- and across-study risk of bias and rate the certainty of the evidence using GRADE (Grading of Recommendations Assessment, Development, and Evaluation), and reach consensus. DISCUSSION: Since 2015, additional trials and longer follow-ups or additional data (e.g., harms, specific patient populations) from previously published trials have been published that will improve our understanding of the benefits and harms of screening. The systematic review of values and preferences will allow fulsome insights that will inform the balance of benefits and harms. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42022378858.

Associations between SARS-CoV-2 infection and incidence of new chronic condition diagnoses: a systematic review.

Because of the large number of infected individuals, an estimate of the future burdens of the long-term consequences of SARS-CoV-2 infection is needed. This systematic review examined associations between SARS-CoV-2 infection and incidence of categories of and selected chronic conditions, by age and severity of infection (inpatient vs. outpatient/mixed care). MEDLINE and EMBASE were searched (1 January 2020 to 4 October 2022) and reference lists scanned. We included observational studies from high-income OECD countries with a control group adjusting for sex and comorbidities. Identified records underwent a two-stage screening process. Two reviewers screened 50% of titles/abstracts, after which DistillerAI acted as second reviewer. Two reviewers then screened the full texts of stage one selections. One reviewer extracted data and assessed risk of bias; results were verified by another. Random-effects meta-analysis estimated pooled hazard ratios (HR). GRADE assessed certainty of the evidence. Twenty-five studies were included. Among the outpatient/mixed SARS-CoV-2 care group, there is high certainty of a small-to-moderate increase (i.e. HR 1.26-1.99) among adults ≥65 years of any cardiovascular condition, and of little-to-no difference (i.e. HR 0.75-1.25) in anxiety disorders for individuals <18, 18-64, and ≥65 years old. Among 18-64 and ≥65 year-olds receiving outpatient/mixed care there are probably (moderate certainty) large increases (i.e. HR ≥2.0) in encephalopathy, interstitial lung disease, and respiratory failure. After SARS-CoV-2 infection, there is probably an increased risk of diagnoses for some chronic conditions; whether the magnitude of risk will remain stable into the future is uncertain.

Screening for the primary prevention of fragility fractures among adults aged 40 years and older in primary care: systematic reviews of the effects and acceptability of screening and treatment, and the accuracy of risk prediction tools.

BACKGROUND: To inform recommendations by the Canadian Task Force on Preventive Health Care, we reviewed evidence on the benefits, harms, and acceptability of screening and treatment, and on the accuracy of risk prediction tools for the primary prevention of fragility fractures among adults aged 40 years and older in primary care. METHODS: For screening effectiveness, accuracy of risk prediction tools, and treatment benefits, our search methods involved integrating studies published up to 2016 from an existing systematic review. Then, to locate more recent studies and any evidence relating to acceptability and treatment harms, we searched online databases (2016 to April 4, 2022 [screening] or to June 1, 2021 [predictive accuracy]; 1995 to June 1, 2021, for acceptability; 2016 to March 2, 2020, for treatment benefits; 2015 to June 24, 2020, for treatment harms), trial registries and gray literature, and hand-searched reviews, guidelines, and the included studies. Two reviewers selected studies, extracted results, and appraised risk of bias, with disagreements resolved by consensus or a third reviewer. The overview of reviews on treatment harms relied on one reviewer, with verification of data by another reviewer to correct errors and omissions. When appropriate, study results were pooled using random effects meta-analysis; otherwise, findings were described narratively. Evidence certainty was rated according to the GRADE approach. RESULTS: We included 4 randomized controlled trials (RCTs) and 1 controlled clinical trial (CCT) for the benefits and harms of screening, 1 RCT for comparative benefits and harms of different screening strategies, 32 validation cohort studies for the calibration of risk prediction tools (26 of these reporting on the Fracture Risk Assessment Tool without [i.e., clinical FRAX], or with the inclusion of bone mineral density (BMD) results [i.e., FRAX + BMD]), 27 RCTs for the benefits of treatment, 10 systematic reviews for the harms of treatment, and 12 studies for the acceptability of screening or initiating treatment. In females aged 65 years and older who are willing to independently complete a mailed fracture risk questionnaire (referred to as "selected population"), 2-step screening using a risk assessment tool with or without measurement of BMD probably (moderate certainty) reduces the risk of hip fractures (3 RCTs and 1 CCT, n = 43,736, absolute risk reduction [ARD] = 6.2 fewer in 1000, 95% CI 9.0-2.8 fewer, number needed to screen [NNS] = 161) and clinical fragility fractures (3 RCTs, n = 42,009, ARD = 5.9 fewer in 1000, 95% CI 10.9-0.8 fewer, NNS = 169). It probably does not reduce all-cause mortality (2 RCTs and 1 CCT, n = 26,511, ARD = no difference in 1000, 95% CI 7.1 fewer to 5.3 more) and may (low certainty) not affect health-related quality of life. Benefits for fracture outcomes were not replicated in an offer-to-screen population where the rate of response to mailed screening questionnaires was low. For females aged 68-80 years, population screening may not reduce the risk of hip fractures (1 RCT, n = 34,229, ARD = 0.3 fewer in 1000, 95% CI 4.2 fewer to 3.9 more) or clinical fragility fractures (1 RCT, n = 34,229, ARD = 1.0 fewer in 1000, 95% CI 8.0 fewer to 6.0 more) over 5 years of follow-up. The evidence for serious adverse events among all patients and for all outcomes among males and younger females (<65 years) is very uncertain. We defined overdiagnosis as the identification of high risk in individuals who, if not screened, would never have known that they were at risk and would never have experienced a fragility fracture. This was not directly reported in any of the trials. Estimates using data available in the trials suggest that among "selected" females offered screening, 12% of those meeting age-specific treatment thresholds based on clinical FRAX 10-year hip fracture risk, and 19% of those meeting thresholds based on clinical FRAX 10-year major osteoporotic fracture risk, may be overdiagnosed as being at high risk of fracture. Of those identified as being at high clinical FRAX 10-year hip fracture risk and who were referred for BMD assessment, 24% may be overdiagnosed. One RCT (n = 9268) provided evidence comparing 1-step to 2-step screening among postmenopausal females, but the evidence from this trial was very uncertain. For the calibration of risk prediction tools, evidence from three Canadian studies (n = 67,611) without serious risk of bias concerns indicates that clinical FRAX-Canada may be well calibrated for the 10-year prediction of hip fractures (observed-to-expected fracture ratio [O:E] = 1.13, 95% CI 0.74-1.72, I2 = 89.2%), and is probably well calibrated for the 10-year prediction of clinical fragility fractures (O:E = 1.10, 95% CI 1.01-1.20, I2 = 50.4%), both leading to some underestimation of the observed risk. Data from these same studies (n = 61,156) showed that FRAX-Canada with BMD may perform poorly to estimate 10-year hip fracture risk (O:E = 1.31, 95% CI 0.91-2.13, I2 = 92.7%), but is probably well calibrated for the 10-year prediction of clinical fragility fractures, with some underestimation of the observed risk (O:E 1.16, 95% CI 1.12-1.20, I2 = 0%). The Canadian Association of Radiologists and Osteoporosis Canada Risk Assessment (CAROC) tool may be well calibrated to predict a category of risk for 10-year clinical fractures (low, moderate, or high risk; 1 study, n = 34,060). The evidence for most other tools was limited, or in the case of FRAX tools calibrated for countries other than Canada, very uncertain due to serious risk of bias concerns and large inconsistency in findings across studies. Postmenopausal females in a primary prevention population defined as <50% prevalence of prior fragility fracture (median 16.9%, range 0 to 48% when reported in the trials) and at risk of fragility fracture, treatment with bisphosphonates as a class (median 2 years, range 1-6 years) probably reduces the risk of clinical fragility fractures (19 RCTs, n = 22,482, ARD = 11.1 fewer in 1000, 95% CI 15.0-6.6 fewer, [number needed to treat for an additional beneficial outcome] NNT = 90), and may reduce the risk of hip fractures (14 RCTs, n = 21,038, ARD = 2.9 fewer in 1000, 95% CI 4.6-0.9 fewer, NNT = 345) and clinical vertebral fractures (11 RCTs, n = 8921, ARD = 10.0 fewer in 1000, 95% CI 14.0-3.9 fewer, NNT = 100); it may not reduce all-cause mortality. There is low certainty evidence of little-to-no reduction in hip fractures with any individual bisphosphonate, but all provided evidence of decreased risk of clinical fragility fractures (moderate certainty for alendronate [NNT=68] and zoledronic acid [NNT=50], low certainty for risedronate [NNT=128]) among postmenopausal females. Evidence for an impact on risk of clinical vertebral fractures is very uncertain for alendronate and risedronate; zoledronic acid may reduce the risk of this outcome (4 RCTs, n = 2367, ARD = 18.7 fewer in 1000, 95% CI 25.6-6.6 fewer, NNT = 54) for postmenopausal females. Denosumab probably reduces the risk of clinical fragility fractures (6 RCTs, n = 9473, ARD = 9.1 fewer in 1000, 95% CI 12.1-5.6 fewer, NNT = 110) and clinical vertebral fractures (4 RCTs, n = 8639, ARD = 16.0 fewer in 1000, 95% CI 18.6-12.1 fewer, NNT=62), but may make little-to-no difference in the risk of hip fractures among postmenopausal females. Denosumab probably makes little-to-no difference in the risk of all-cause mortality or health-related quality of life among postmenopausal females. Evidence in males is limited to two trials (1 zoledronic acid, 1 denosumab); in this population, zoledronic acid may make little-to-no difference in the risk of hip or clinical fragility fractures, and evidence for all-cause mortality is very uncertain. The evidence for treatment with denosumab in males is very uncertain for all fracture outcomes (hip, clinical fragility, clinical vertebral) and all-cause mortality. There is moderate certainty evidence that treatment causes a small number of patients to experience a non-serious adverse event, notably non-serious gastrointestinal events (e.g., abdominal pain, reflux) with alendronate (50 RCTs, n = 22,549, ARD = 16.3 more in 1000, 95% CI 2.4-31.3 more, [number needed to treat for an additional harmful outcome] NNH = 61) but not with risedronate; influenza-like symptoms with zoledronic acid (5 RCTs, n = 10,695, ARD = 142.5 more in 1000, 95% CI 105.5-188.5 more, NNH = 7); and non-serious gastrointestinal adverse events (3 RCTs, n = 8454, ARD = 64.5 more in 1000, 95% CI 26.4-13.3 more, NNH = 16), dermatologic adverse events (3 RCTs, n = 8454, ARD = 15.6 more in 1000, 95% CI 7.6-27.0 more, NNH = 64), and infections (any severity; 4 RCTs, n = 8691, ARD = 1.8 more in 1000, 95% CI 0.1-4.0 more, NNH = 556) with denosumab. For serious adverse events overall and specific to stroke and myocardial infarction, treatment with bisphosphonates probably makes little-to-no difference; evidence for other specific serious harms was less certain or not available. There was low certainty evidence for an increased risk for the rare occurrence of atypical femoral fractures (0.06 to 0.08 more in 1000) and osteonecrosis of the jaw (0.22 more in 1000) with bisphosphonates (most evidence for alendronate). The evidence for these rare outcomes and for rebound fractures with denosumab was very uncertain. Younger (lower risk) females have high willingness to be screened. A minority of postmenopausal females at increased risk for fracture may accept treatment. Further, there is large heterogeneity in the level of risk at which patients may be accepting of initiating treatment, and treatment effects appear to be overestimated. CONCLUSION: An offer of 2-step screening with risk assessment and BMD measurement to selected postmenopausal females with low prevalence of prior fracture probably results in a small reduction in the risk of clinical fragility fracture and hip fracture compared to no screening. These findings were most applicable to the use of clinical FRAX for risk assessment and were not replicated in the offer-to-screen population where the rate of response to mailed screening questionnaires was low. Limited direct evidence on harms of screening were available; using study data to provide estimates, there may be a moderate degree of overdiagnosis of high risk for fracture to consider. The evidence for younger females and males is very limited. The benefits of screening and treatment need to be weighed against the potential for harm; patient views on the acceptability of treatment are highly variable. SYSTEMATIC REVIEW REGISTRATION: International Prospective Register of Systematic Reviews (PROSPERO): CRD42019123767.

Risk factors and preventive interventions for post Covid-19 condition: systematic review.

This systematic review examined pre-existing and clinical risk factors for post Covid-19 condition (≥12 weeks after onset), and interventions during acute and post-acute phases of illness that could potentially prevent post Covid-19 condition. The review focuses on studies collecting data during the early phases of the pandemic and prior to the emergence of variants of concern and widespread vaccination. We searched bibliographic databases and grey literature. Two investigators independently reviewed abstracts and full-text articles, and data extraction and risk of bias assessments were verified. Meta-analysis was performed when suitable and we assessed the certainty of evidence using GRADE. We included 31 studies. We found small-to-moderate associations (e.g. adjusted odds ratios 1.5 to <2.0) between female sex and higher non-recovery, fatigue, and dyspnea (moderate certainty). Severe or critical acute-phase Covid-19 severity (versus not) has probably (moderate certainty) a large association (adjusted ratio ≥2.0) with increased cognitive impairment, a small-to-moderate association with more non-recovery, and a little-to-no association with dyspnea. There may be (low certainty) large associations between hospitalization and increased non-recovery, increased dyspnea, and reduced return to work. Other outcomes had low certainty of small-to-moderate or little-to-no association or very low certainty. Several potential preventive interventions were examined, but effects are very uncertain. Guidelines in relation to surveillance, screening, and other services such as access to sickness and disability benefits, might need to focus on females and those with previously severe Covid-19 illness. Continuous assessment of emerging evidence, especially on whether different variants and vaccination impact outcomes, will be important. PROSPERO registration: CRD42021270354.

Incidence, risk factors, natural history, and hypothesised mechanisms of myocarditis and pericarditis following covid-19 vaccination: living evidence syntheses and review.

OBJECTIVES: To synthesise evidence on incidence rates and risk factors for myocarditis and pericarditis after use of mRNA vaccination against covid-19, clinical presentation, short term and longer term outcomes of cases, and proposed mechanisms. DESIGN: Living evidence syntheses and review. DATA SOURCES: Medline, Embase, and the Cochrane Library were searched from 6 October 2020 to 10 January 2022; reference lists and grey literature (to 13 January 2021). One reviewer completed screening and another verified 50% of exclusions, using a machine learning program to prioritise records. A second reviewer verified all exclusions at full text, extracted data, and (for incidence and risk factors) risk of bias assessments using modified Joanna Briggs Institute tools. Team consensus determined certainty of evidence ratings for incidence and risk factors using GRADE (Grading of Recommendations, Assessment, Development and Evaluation). ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Large (>10 000 participants) or population based or multisite observational studies and surveillance data (incidence and risk factors) reporting on confirmed myocarditis or pericarditis after covid-19 mRNA vaccination; case series (n≥5, presentation, short term clinical course and longer term outcomes); opinions, letters, reviews, and primary studies focused on describing or supporting hypothesised mechanisms. RESULTS: 46 studies were included (14 on incidence, seven on risk factors, 11 on characteristics and short term course, three on longer term outcomes, and 21 on mechanisms). Incidence of myocarditis after mRNA vaccines was highest in male adolescents and male young adults (age 12-17 years, range 50-139 cases per million (low certainty); 18-29 years, 28-147 per million (moderate certainty)). For girls and boys aged 5-11 years and women aged 18-29 years, incidence of myocarditis after vaccination with BNT162b2 (Pfizer/BioNTech) could be fewer than 20 cases per million (low certainty). Incidence after a third dose of an mRNA vaccine had very low certainty evidence. For individuals of 18-29 years, incidence of myocarditis is probably higher after vaccination with mRNA-1273 (Moderna) compared with Pfizer (moderate certainty). Among individuals aged 12-17, 18-29, or 18-39 years, incidence of myocarditis or pericarditis after dose two of an mRNA vaccine for covid-19 might be lower when administered ≥31 days compared with ≤30 days after dose one (low certainty). Data specific to men aged 18-29 years indicated that the dosing interval might need to increase to ≥56 days to substantially drop myocarditis or pericarditis incidence. For clinical course and short term outcomes, only one small case series (n=8) was found for 5-11 year olds. In adolescents and adults, most (>90%) myocarditis cases involved men of a median 20-30 years of age and with symptom onset two to four days after a second dose (71-100%). Most people were admitted to hospital (≥84%) for a short duration (two to four days). For pericarditis, data were limited but more variation than myocarditis has been reported in patient age, sex, onset timing, and rate of admission to hospital. Three case series with longer term (3 months; n=38) follow-up suggested persistent echocardiogram abnormalities, as well as ongoing symptoms or a need for drug treatments or restriction from activities in >50% of patients. Sixteen hypothesised mechanisms were described, with little direct supporting or refuting evidence. CONCLUSIONS: These findings indicate that adolescent and young adult men are at the highest risk of myocarditis after mRNA vaccination. Use of a Pfizer vaccine over a Moderna vaccine and waiting for more than 30 days between doses might be preferred for this population. Incidence of myocarditis in children aged 5-11 years is very rare but certainty was low. Data for clinical risk factors were very limited. A clinical course of mRNA related myocarditis appeared to be benign, although longer term follow-up data were limited. Prospective studies with appropriate testing (eg, biopsy and tissue morphology) will enhance understanding of mechanism.

Burden of illness in infants and young children hospitalized for respiratory syncytial virus: A rapid review.

Respiratory syncytial virus (RSV) infections are common among young children and represent a significant burden to patients, their families and the Canadian health system. Here we conduct a rapid review of the burden of RSV illness in children 24 months of age or younger. Four databases (Medline, Embase, Cochrane Database of Clinical Trials, ClinicalTrials.gov from 2014 to 2018), grey literature and reference lists were reviewed for studies on the following: children with or without a risk factor, without prophylaxis and with lab-confirmed RSV infection. Of 29 studies identified, 10 provided within-study comparisons and few examined clinical conditions besides prematurity. For infants of 33-36 weeks gestation (wGA) versus term infants, there was low-to-moderate certainty evidence for an increase in RSV-hospitalizations (n=599,535 infants; RR 2.05 [95% CI 1.89-2.22]; 1.3 more per 100 [1.1-1.5 more]) and hospital length of stay (n=7,597 infants; mean difference 1.00 day [95% CI 0.88-1.12]). There was low-to-moderate certainty evidence of little-to-no difference for infants born at 29-32 versus 33-36 wGA for hospitalization (n=12,812 infants; RR 1.20 [95% CI 0.92-1.56]). There was low certainty evidence of increased mechanical ventilation for hospitalized infants born at 29-32 versus 33-35 wGA (n=212 infants; RR 1.58, 95% CI 0.94-2.65). Among infants born at 32-35 wGA, hospitalization for RSV in infancy may be associated with increased wheeze and asthma-medication use across six-year follow-up (RR range 1.3-1.7). Children with versus without Down syndrome may have increased hospital length of stay (n=7,206 children; mean difference 3.00 days, 95% CI 1.95-4.05; low certainty). Evidence for other within-study comparisons was of very low certainty. In summary, prematurity is associated with greater risk for RSV-hospitalization and longer hospital length of stay, and Down syndrome may be associated with longer hospital stay for RSV. Respiratory syncytial virus-hospitalization in infancy may be associated with greater wheeze and asthma-medication use in early childhood. Lack of a comparison group was a major limitation for many studies.

Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool.

BACKGROUND: Machine learning tools that semi-automate data extraction may create efficiencies in systematic review production. We evaluated a machine learning and text mining tool's ability to (a) automatically extract data elements from randomized trials, and (b) save time compared with manual extraction and verification. METHODS: For 75 randomized trials, we manually extracted and verified data for 21 data elements. We uploaded the randomized trials to an online machine learning and text mining tool, and quantified performance by evaluating its ability to identify the reporting of data elements (reported or not reported), and the relevance of the extracted sentences, fragments, and overall solutions. For each randomized trial, we measured the time to complete manual extraction and verification, and to review and amend the data extracted by the tool. We calculated the median (interquartile range [IQR]) time for manual and semi-automated data extraction, and overall time savings. RESULTS: The tool identified the reporting (reported or not reported) of data elements with median (IQR) 91% (75% to 99%) accuracy. Among the top five sentences for each data element at least one sentence was relevant in a median (IQR) 88% (83% to 99%) of cases. Among a median (IQR) 90% (86% to 97%) of relevant sentences, pertinent fragments had been highlighted by the tool; exact matches were unreliable (median (IQR) 52% [33% to 73%]). A median 48% of solutions were fully correct, but performance varied greatly across data elements (IQR 21% to 71%). Using ExaCT to assist the first reviewer resulted in a modest time savings compared with manual extraction by a single reviewer (17.9 vs. 21.6 h total extraction time across 75 randomized trials). CONCLUSIONS: Using ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements. The tool's ability to identify at least one relevant sentence and highlight pertinent fragments was generally good, but changes to sentence selection and/or highlighting were often required. PROTOCOL: https://doi.org/10.7939/DVN/RQPJKS.

Screening for Gestational Diabetes: Updated Evidence Report and Systematic Review for the US Preventive Services Task Force.

Importance: Gestational diabetes is associated with several poor health outcomes. Objective: To update the 2012 review on screening for gestational diabetes to inform the US Preventive Services Task Force. Data Sources: MEDLINE, EMBASE, and CINAHL (2010 to May 2020), ClinicalTrials.gov, reference lists; surveillance through June 2021. Study Selection: English-language intervention studies for screening and treatment; observational studies on screening; prospective studies on screening test accuracy. Data Extraction and Synthesis: Dual review of titles/abstracts, full-text articles, and study quality. Single-reviewer data abstraction with verification. Random-effects meta-analysis or bivariate analysis (accuracy). Main Outcomes and Measures: Pregnancy, fetal/neonatal, and long-term health outcomes; harms of screening; accuracy. Results: A total of 76 studies were included (18 randomized clinical trials [RCTs] [n = 31 241], 2 nonrandomized intervention studies [n = 190], 56 observational studies [n = 261 678]). Direct evidence on benefits of screening vs no screening was limited to 4 observational studies with inconsistent findings and methodological limitations. Screening was not significantly associated with serious or long-term harm. In 5 RCTs (n = 25 772), 1-step (International Association of Diabetes and Pregnancy Study Group) vs 2-step (Carpenter and Coustan) screening was significantly associated with increased likelihood of gestational diabetes (11.5% vs 4.9%) but no improved health outcomes. At or after 24 weeks of gestation, oral glucose challenge tests with 140- and 135-mg/dL cutoffs had sensitivities of 82% and 93%, respectively, and specificities of 82% and 79%, respectively, against Carpenter and Coustan criteria, and a test with a 140-mg/dL cutoff had sensitivity of 85% and specificity of 81% against the National Diabetes Group Data criteria. Fasting plasma glucose tests with cutoffs of 85 and 90 mg/dL had sensitivities of 88% and 81% and specificities of 73% and 82%, respectively, against Carpenter and Coustan criteria. Based on 8 RCTs and 1 nonrandomized study (n = 3982), treatment was significantly associated with decreased risk of primary cesarean deliveries (relative risk [RR], 0.70 [95% CI, 0.54-0.91]; absolute risk difference [ARD], 5.3%), shoulder dystocia (RR, 0.42 [95% CI, 0.23-0.77]; ARD, 1.3%), macrosomia (RR, 0.53 [95% CI, 0.41-0.68]; ARD, 8.9%), large for gestational age (RR, 0.56 [95% CI, 0.47-0.66]; ARD, 8.4%), birth injuries (odds ratio, 0.33 [95% CI, 0.11-0.99]; ARD, 0.2%), and neonatal intensive care unit admissions (RR, 0.73 [95% CI, 0.53-0.99]; ARD, 2.0%). The association with reduction in preterm deliveries was not significant (RR, 0.75 [95% CI, 0.56-1.01]). Conclusions and Relevance: Direct evidence on screening vs no screening remains limited. One- vs 2-step screening was not significantly associated with improved health outcomes. At or after 24 weeks of gestation, treatment of gestational diabetes was significantly associated with improved health outcomes.

Health inequities related to vaccination: An evidence map of potentially influential factors and systematic review of interventions.

INTRODUCTION: The National Advisory Committee on Immunization (NACI) makes recommendations for vaccines in Canada. To inform considerations for equity when making recommendations, the NACI Secretariat developed a matrix of factors that may influence vaccine equity. To inform the matrix we mapped the evidence for P2ROGRESS And Other factors potentially associated with unequal levels of illness or death from vaccine-preventable diseases (VPDs) and systematically reviewed the evidence for interventions aimed at reducing inequities. METHODS: In October 2019 we searched Medline, Embase, and CINAHL. Two reviewers agreed on the included studies. Our primary outcomes were VPD-related hospitalizations and deaths. Secondary outcomes were differential vaccine access, and exposure, susceptibility, severity, and consequences of VPDs. Two reviewers appraised the certainty of evidence. We mapped the evidence for P2ROGRESS And Other factors and summarized the findings descriptively. We summarized the interventions narratively. RESULTS: We identified 413 studies reporting on P2ROGRESS And Other factors. The most commonly investigated factors included age (n = 374, 89%), pre-existing conditions (n = 179, 42%), and gender identity or sex (n = 144, 34%). We identified 2 trials investigating the effects of interventions. One (n = 1249) provided very low certainty evidence that staff vaccination policies may reduce hospitalizations and deaths from influenza among private care home residents. The other (n not reported) provided very low certainty evidence that universal vaccination by nurses in clinics may reduce hospitalizations for rotavirus gastroenteritis compared with vaccination by physicians or no intervention. CONCLUSIONS: There is a large body of studies reporting on hospitalizations and deaths from VPDs stratified by P2ROGRESS And Other factors. We found only two trials examining the effects of interventions on hospitalization for or mortality from VPDs. This review has been helpful to NACI and will be helpful to similar organizations aiming to systematically identify and target health inequities through the development of vaccine program recommendations.

Risk factors for severity of COVID-19: a rapid review to inform vaccine prioritisation in Canada.

OBJECTIVES: Rapid review to determine the magnitude of association between potential risk factors and severity of COVID-19, to inform vaccine prioritisation in Canada. SETTING: Ovid MEDLINE(R) ALL, Epistemonikos COVID-19 in L·OVE Platform, McMaster COVID-19 Evidence Alerts and websites were searched to 15 June 2020. Eligible studies were conducted in high-income countries and used multivariate analyses. PARTICIPANTS: After piloting, screening, data extraction and quality appraisal were performed by a single experienced reviewer. Of 3740 unique records identified, 34 were included that reported on median 596 (range 44-418 794) participants, aged 42-84 years. 19/34 (56%) were good quality. OUTCOMES: Hospitalisation, intensive care unit admission, length of stay in hospital or intensive care unit, mechanical ventilation, severe disease, mortality. RESULTS: Authors synthesised findings narratively and appraised the certainty of the evidence for each risk factor-outcome association. There was low or moderate certainty evidence for a large (≥2-fold) magnitude of association between hospitalisation in people with COVID-19, and: obesity class III, heart failure, diabetes, chronic kidney disease, dementia, age >45 years, male gender, black race/ethnicity (vs non-Hispanic white), homelessness and low income. Age >60 and >70 years may be associated with large increases in mechanical ventilation and severe disease, respectively. For mortality, a large magnitude of association may exist with liver disease, Bangladeshi ethnicity (vs British white), age >45 years, age >80 years (vs 65-69 years) and male gender among 20-64 years (but not older). Associations with hospitalisation and mortality may be very large (≥5-fold) for those aged ≥60 years. CONCLUSIONS: Increasing age (especially >60 years) may be the most important risk factor for severe outcomes. High-quality primary research accounting for multiple confounders is needed to better understand the magnitude of associations for severity of COVID-19 with several other factors. PROSPERO REGISTRATION NUMBER: CRD42020198001.

Screening for chlamydia and/or gonorrhea in primary health care: systematic reviews on effectiveness and patient preferences.

BACKGROUND: We conducted systematic reviews on the benefits and harms of screening compared with no screening or alternative screening approaches for Chlamydia trachomatis (CT) and Neisseria gonorrhoeae (NG) in non-pregnant sexually active individuals, and on the relative importance patients' place on the relevant outcomes. Findings will inform recommendations by the Canadian Task Force on Preventive Health Care. METHODS: We searched five databases (to January 24, 2020), trial registries, conference proceedings, and reference lists for English and French literature published since 1996. Screening, study selection, and risk of bias assessments were independently undertaken by two reviewers, with consensus for final decisions. Data extraction was conducted by one reviewer and checked by another for accuracy and completeness. Meta-analysis was conducted where appropriate. We used the GRADE approach to rate the certainty of the evidence. The Task Force and content experts provided input on determining thresholds for important effect sizes and on interpretation of findings. RESULTS: Of 41 included studies, 17 and 11 reported on benefits and harms of screening, respectively, and 14 reported on patient preferences. Universal screening for CT in general populations 16 to 29 years of age, using population-based or opportunistic approaches achieving low screening rates, may make little-to-no difference for a female's risk of pelvic inflammatory disease (PID) (2 RCTs, n=141,362; 0.3 more in 1000 [7.6 fewer to 11 more]) or ectopic pregnancy (1 RCT, n=15,459; 0.20 more per 1000 [2.2 fewer to 3.9 more]). It may also not make a difference for CT transmission (3 RCTs, n=41,709; 3 fewer per 1000 [11.5 fewer to 6.9 more]). However, benefits may be achieved for reducing PID if screening rates are increased (2 trials, n=30,652; 5.7 fewer per 1000 [10.8 fewer to 1.1 more]), and for reducing CT and NG transmission when intensely screening high-prevalence female populations (2 trials, n=6127; 34.3 fewer per 1000 [4 to 58 fewer]; NNS 29 [17 to 250]). Evidence on infertility in females from CT screening and on transmission of NG in males and both sexes from screening for CT and NG is very uncertain. No evidence was found for cervicitis, chronic pelvic pain, or infertility in males from CT screening, or on any clinical outcomes from NG screening. Undergoing screening, or having a diagnosis of CT, may cause a small-to-moderate number of people to experience some degree of harm, mainly due to feelings of stigmatization and anxiety about future infertility risk. The number of individuals affected in the entire screening-eligible population is likely smaller. Screening may make little-to-no difference for general anxiety, self-esteem, or relationship break-up. Evidence on transmission from studies comparing home versus clinic screening is very uncertain. Four studies on patient preferences found that although utility values for the different consequences of CT and NG infections are probably quite similar, when considering the duration of the health state experiences, infertility and chronic pelvic pain are probably valued much more than PID, ectopic pregnancy, and cervicitis. How patients weigh the potential benefits versus harms of screening is very uncertain (1 survey, 10 qualitative studies); risks to reproductive health and transmission appear to be more important than the (often transient) psychosocial harms. DISCUSSION: Most of the evidence on screening for CT and/or NG offers low or very low certainty about the benefits and harms. Indirectness from use of comparison groups receiving some screening, incomplete outcome ascertainment, and use of outreach settings was a major contributor to uncertainty. Patient preferences indicate that the potential benefits from screening appear to outweigh the possible harms. Direct evidence about which screening strategies and intervals to use, which age to start and stop screening, and whether screening males in addition to females is necessary to prevent clinical outcomes is scarce, and further research in these areas would be informative. Apart from the evidence in this review, information on factors related to equity, acceptability, implementation, cost/resources, and feasibility will support recommendations made by the Task Force. SYSTEMATIC REVIEW REGISTRATION: International Prospective Register of Systematic Reviews (PROSPERO), registration number CRD42018100733 .

LOCATE: a prospective evaluation of the value of Leveraging Ongoing Citation Acquisition Techniques for living Evidence syntheses.

BACKGROUND: Living systematic reviews (LSRs) can expedite evidence synthesis by incorporating new evidence in real time. However, the methods needed to identify new studies in a timely manner are not well established. OBJECTIVES: To explore the value of complementary search approaches in terms of search performance, impact on results and conclusions, screening workload, and feasibility compared to the reference standard. METHODS: We developed three complementary search approaches for a systematic review on treatments for bronchiolitis: Automated Full Search, PubMed Similar Articles, and Scopus Citing References. These were automated to retrieve results monthly; pairs of reviewers screened the records and commented on feasibility. After 1 year, we conducted a full update search (reference standard). For each complementary approach, we compared search performance (proportion missed, number needed to read [NNR]) and reviewer workload (number of records screened, time required) to the reference standard. We investigated the impact of the new trials on the effect estimate and certainty of evidence for the primary outcomes. We summarized comments about feasibility. RESULTS: Via the reference standard, reviewers screened 505 titles/abstracts, 24 full texts, and identified four new trials (NNR 127; 12.4 h). Of the complementary approaches, only the Automated Full Search located all four trials; these were located 6 to 12 months sooner than via the reference standard but did not alter the results nor certainty in the evidence. The Automated Full Search was the most resource-intensive approach (816 records screened; NNR 204; 17.1 h). The PubMed Similar Articles and Scopus Citing References approaches located far fewer records (452 and 244, respectively), thereby requiring less screening time (9.4 and 5.2 h); however, each approach located only one of the four new trials. Reviewers found it feasible and convenient to conduct monthly screening for searches of this yield (median 15-65 records/month). CONCLUSIONS: The Automated Full Search was the most resource-intensive approach, but also the only to locate all of the newly published trials. Although the monthly screening time for the PubMed Similar Articles and Scopus Citing Articles was far less, most relevant records were missed. These approaches were feasible to integrate into reviewer work processes. SYSTEMATIC REVIEW REGISTRATION: Open Science Framework. https://doi.org/10.17605/OSF.IO/6M28H .

Fall prevention interventions for older community-dwelling adults: systematic reviews on benefits, harms, and patient values and preferences.

BACKGROUND: An estimated 20-30% of community-dwelling Canadian adults aged 65 years or older experience one or more falls each year. Fall-related injuries are a leading cause of hospitalization and can lead to functional independence. Many fall prevention interventions, often based on modifiable risk factors, have been studied. Apart from the magnitude of the benefits and harms from different interventions, the preferences of older adults for different interventions as well as the relative importance they place on the different potential outcomes may influence recommendations by guideline panels. These reviews on benefits and harms of interventions, and on patient values and preferences, will inform the Canadian Task Force on Preventive Health Care to develop recommendations on fall prevention for primary care providers. METHODS: To review the benefits and harms of fall prevention interventions, we will update a previous systematic review of randomized controlled trials with adaptations to modify the classification of interventions and narrow the scope to community-dwelling older adults and primary-care relevant interventions. Four databases (MEDLINE, Embase, Cochrane Central Register of Controlled Trials, Ageline), reference lists, trial registries, and relevant websites will be searched, using limits for randomized trials and date (2016 onwards). We will classify interventions according to the Prevention of Falls Network Europe (ProFANE) Group's taxonomy. Outcomes include fallers, falls, injurious falls, fractures, hip fractures, institutionalization, health-related quality of life, functional status, and intervention-related adverse effects. For studies not included in the previous review, screening, study selection, data extraction on outcomes, and risk of bias assessments will be independently undertaken by two reviewers with consensus used for final decisions. Where quantitative analysis is suitable, network or pairwise meta-analysis will be conducted using a frequentist approach in Stata. Assessment of the transitivity and coherence of the network meta-analyses will be undertaken. For the reviews on patient preferences and outcome valuation (relative importance of outcomes), we will perform de novo reviews with searches in three databases (MEDLINE, PsycInfo, and CINAHL) and reference lists for cross-sectional, longitudinal quantitative, or qualitative studies published from 2000. Selection, data extraction, and risk of bias assessments suitable for each study design will be performed in duplicate. The analysis will be guided by a narrative synthesis approach, which may include meta-analysis for health-state utilities. We will use the CINeMa approach to a rate the certainty of the evidence for outcomes on intervention effects analyzed using network meta-analysis and the GRADE approach for all other outcomes. DISCUSSION: We will describe the flow of literature and characteristics of all studies and present results of all analyses and summary of finding tables. We will compare our findings to others and discuss the limitations of the reviews and the available literature. SYSTEMATIC REVIEW REGISTRATION: This protocol has not been registered.

Screening for the prevention and early detection of cervical cancer: protocol for systematic reviews to inform Canadian recommendations.

PURPOSE: To inform recommendations by the Canadian Task Force on Preventive Health Care on screening in primary care for the prevention and early detection of cervical cancer by systematically reviewing evidence of (a) effectiveness; (b) test accuracy; (c) individuals' values and preferences; and (d) strategies aimed at improving screening rates. METHODS: De novo reviews will be conducted to evaluate effectiveness and to assess values and preferences. For test accuracy and strategies to improve screening rates, we will integrate studies from existing systematic reviews with search updates to the present. Two Cochrane reviews will provide evidence of adverse pregnancy outcomes from the conservative management of cervical intraepithelial neoplasia. We will search Medline, Embase, and Cochrane Central (except for individuals' values and preferences, where Medline, Scopus, and EconLit will be searched) via peer-reviewed search strategies and the reference lists of included studies and reviews. We will search ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform for ongoing trials. Two reviewers will screen potentially eligible studies and agree on those to include. Data will be extracted by one reviewer with verification by another. Two reviewers will independently assess risk of bias and reach consensus. Where possible and suitable, we will pool studies via meta-analysis. We will compare accuracy data per outcome and per comparison using the Rutter and Gatsonis hierarchical summary receiver operating characteristic model and report relative sensitivities and specificities. Findings on values and preferences will be synthesized using a narrative synthesis approach and thematic analysis, depending on study designs. Two reviewers will appraise the certainty of evidence for all outcomes using GRADE (Grading of Recommendations Assessment, Development and Evaluation) and come to consensus. DISCUSSION: The publication of guidance on screening in primary care for the prevention and early detection of cervical cancer by the Task Force in 2013 focused on cytology. Since 2013, new studies using human papillomavirus tests for cervical screening have been published that will improve our understanding of screening in primary care settings. This review will inform updated recommendations based on currently available studies and address key evidence gaps noted in our previous review.

Making Narrative Statements to Describe Treatment Effects.

Accurately describing treatment effects using plain language and narrative statements is a critical step in communicating research findings to end users. However, the process of developing these narratives has not been historically guided by a specific framework. The Agency for Healthcare Research and Quality Evidence-based Practice Center Program developed guidance for narrative summaries of treatment effects that identifies five constructs. We explicitly identify these constructs to facilitate developing narrative statements: (1) direction of effect, (2) size of effect, (3) clinical importance, (4) statistical significance, and (5) strength or certainty of evidence. These constructs clearly overlap. It may not always be feasible to address all five constructs. Based on context and intended audience, investigators can determine which constructs will be most important to address in narrative statements.

A systematic review of factors that influence the acceptability of vaccines among Canadians.

BACKGROUND: Canada's National Advisory Committee on Immunization (NACI) provides guidance on the use of vaccines in Canada. To support the expansion of its mandate to include considerations for vaccine acceptability when making recommendations, the NACI Secretariat developed a matrix of factors that influence acceptability. To inform and validate the matrix, we systematically reviewed evidence for factors that influence vaccine acceptability, and for interventions aimed at improving acceptability. METHODS: On 10-11 October 2018 we searched four bibliographic databases, the Theses Canada Portal, and ClinicalTrials.gov. Two reviewers agreed on the included studies. From each study, we extracted information about the participants, intervention or exposure, comparator, and relevant outcomes. Due to heterogeneity in the reported factors and acceptability indicators we synthesized the findings narratively. We appraised the certainty of evidence using GRADE. For each vaccine-preventable disease we populated a matrix of factors for which there was evidence of an influence on acceptability. RESULTS: One hundred studies (>1 million participants) contributed data relevant to the public, 16 (6191 participants) to healthcare providers, and three (84 participants) to policymakers. There were 43 intervention studies (~2 million participants). Across vaccines, we identified low certainty evidence for 70 factors relevant to the general population, 56 to high-risk groups, and 30 to healthcare providers. The perceived safety and importance of the vaccine, vaccination history, and receiving a recommendation from a healthcare provider were common influential factors. We found low certainty evidence that reminders for childhood vaccines and policies or delivery models for rotavirus vaccines could improve uptake and coverage. Evidence for other interventions was of very low certainty. CONCLUSIONS: The NACI vaccine acceptability matrix is useful for categorizing acceptability factors for the general public. Reminder systems may improve the uptake of childhood vaccines. Policies that make the rotavirus vaccine universally available and easily accessible may improve coverage. FUNDING: This systematic review was completed under contract to the Public Health Agency of Canada, Contract #4600001536.

Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews.

BACKGROUND: We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr's predictions varied by review or study-level characteristics. METHODS: For a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining records, as predicted by the tool. We retrospectively simulated the liberal-accelerated screening approach. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool's predictions varied by review and study-level characteristics. RESULTS: Using the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records that were included in the final reports, but saved a median (IQR) 26 (9, 42) h of screening time. One missed study was included in eight pairwise meta-analyses in one systematic review. The pooled effect for just one of those meta-analyses changed considerably (from MD (95% CI) - 1.53 (- 2.92, - 0.15) to - 1.17 (- 2.70, 0.36)). Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P = 0.37) or intervention type (simple or complex, P = 0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions (P = 0.01), or that included only trials (95%) vs. multiple designs (86%) (P = 0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) (P = 0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) (P = 0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P = 0.02) were more often correctly predicted as relevant. CONCLUSION: Our screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. Several of our findings are paradoxical and require further study to fully understand the tasks to which ML-assisted screening is best suited. The findings should be interpreted in light of the fact that the protocol was prepared for the funder, but not published a priori. Because we used a convenience sample, the findings may be prone to selection bias. The results may not be generalizable to other samples of reviews, ML tools, or screening approaches. The small number of missed studies across reviews with pairwise meta-analyses hindered strong conclusions about the effect of missed studies on the results and conclusions of systematic reviews.

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools.

BACKGROUND: We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool. METHODS: We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey. RESULTS: Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33-90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s). CONCLUSIONS: The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer's decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers' self-identified preferences may improve their compatibility with present workflows. SYSTEMATIC REVIEW REGISTRATION: Not applicable.