Deep learning to refine the identification of high-quality clinical research articles from the biomedical literature: Performance evaluation.

BACKGROUND: Identifying practice-ready evidence-based journal articles in medicine is a challenge due to the sheer volume of biomedical research publications. Newer approaches to support evidence discovery apply deep learning techniques to improve the efficiency and accuracy of classifying sound evidence. OBJECTIVE: To determine how well deep learning models using variants of Bidirectional Encoder Representations from Transformers (BERT) identify high-quality evidence with high clinical relevance from the biomedical literature for consideration in clinical practice. METHODS: We fine-tuned variations of BERT models (BERTBASE, BioBERT, BlueBERT, and PubMedBERT) and compared their performance in classifying articles based on methodological quality criteria. The dataset used for fine-tuning models included titles and abstracts of >160,000 PubMed records from 2012 to 2020 that were of interest to human health which had been manually labeled based on meeting established critical appraisal criteria for methodological rigor. The data was randomly divided into 80:10:10 sets for training, validating, and testing. In addition to using the full unbalanced set, the training data was randomly undersampled into four balanced datasets to assess performance and select the best performing model. For each of the four sets, one model that maintained sensitivity (recall) at ≥99% was selected and were ensembled. The best performing model was evaluated in a prospective, blinded test and applied to an established reference standard, the Clinical Hedges dataset. RESULTS: In training, three of the four selected best performing models were trained using BioBERTBASE. The ensembled model did not boost performance compared with the best individual model. Hence a solo BioBERT-based model (named DL-PLUS) was selected for further testing as it was computationally more efficient. The model had high recall (>99%) and 60% to 77% specificity in a prospective evaluation conducted with blinded research associates and saved >60% of the work required to identify high quality articles. CONCLUSIONS: Deep learning using pretrained language models and a large dataset of classified articles produced models with improved specificity while maintaining >99% recall. The resulting DL-PLUS model identifies high-quality, clinically relevant articles from PubMed at the time of publication. The model improves the efficiency of a literature surveillance program, which allows for faster dissemination of appraised research.

Preliminary comparison of the performance of the National Library of Medicine's systematic review publication type and the sensitive clinical queries filter for systematic reviews in PubMed.

OBJECTIVE: The National Library of Medicine (NLM) inaugurated a "publication type" concept to facilitate searches for systematic reviews (SRs). On the other hand, clinical queries (CQs) are validated search strategies designed to retrieve scientifically sound, clinically relevant original and review articles from biomedical literature databases. We compared the retrieval performance of the SR publication type (SR[pt]) against the most sensitive CQ for systematic review articles (CQrs) in PubMed. METHODS: We ran date-limited searches of SR[pt] and CQrs to compare the relative yield of articles and SRs, focusing on the differences in retrieval of SRs by SR[pt] but not CQrs (SR[pt] NOT CQrs) and CQrs NOT SR[pt]. Random samples of articles retrieved in each of these comparisons were examined for SRs until a consistent pattern became evident. RESULTS: For SR[pt] NOT CQrs, the yield was relatively low in quantity but rich in quality, with 79% of the articles being SRs. For CQrs NOT SR[pt], the yield was high in quantity but low in quality, with only 8% being SRs. For CQrs AND SR[pt], the quality was highest, with 92% being SRs. CONCLUSIONS: We found that SR[pt] had high precision and specificity for SRs but low recall (sensitivity), whereas CQrs had much higher recall. SR[pt] OR CQrs added valid SRs to the CQrs yield at low cost (i.e., added few non-SRs). For searches that are intended to be exhaustive for SRs, SR[pt] can be added to existing sensitive search filters.

Context and Approach in Reporting Evaluations of Electronic Health Record-Based Implementation Projects.

Electronic health records (EHRs) are ubiquitous yet still evolving, resulting in a moving target for determining the effects of context (features of the work environment, such as organization, payment systems, user training, and roles) on EHR implementation projects. Electronic health records have become instrumental in effecting quality improvement innovations and providing data to evaluate them. However, reports of studies typically fail to provide adequate descriptions of contextual details to permit readers to apply the findings. As for any evaluation, the quality of reporting is essential to learning from, and disseminating, the results. Extensive guidelines exist for reporting of virtually all types of applied health research, but they are not tailored to capture some contextual factors that may affect the outcomes of EHR implementations, such as attitudes toward implementation, format and amount of training, post go-live support, amount of local customization, and time diverted from direct interaction with patients to computers. Nevertheless, evaluators of EHR-based innovations can choose reporting guidelines that match the general purpose of their evaluation and the stage of their investigation (planning, protocol, execution, and analysis) and should report relevant contextual details (including, if pertinent, any pressures to help justify the huge investments and many years required for some implementations). Reporting guidelines are based on the scientific principles and practices that underlie sound research and should be consulted from the earliest stages of planning evaluations and onward, serving as guides for how evaluations should be conducted as well as reported.

Effect of Patient-Centered Transitional Care Services on Clinical Outcomes in Patients Hospitalized for Heart Failure: The PACT-HF Randomized Clinical Trial.

Importance: Health care services that support the hospital-to-home transition can improve outcomes in patients with heart failure (HF). Objective: To test the effectiveness of the Patient-Centered Care Transitions in HF transitional care model in patients hospitalized for HF. Design, Setting, and Participants: Stepped-wedge cluster randomized trial of 2494 adults hospitalized for HF across 10 hospitals in Ontario, Canada, from February 2015 to March 2016, with follow-up until November 2016. Interventions: Hospitals were randomized to receive the intervention (n = 1104 patients), in which nurse-led self-care education, a structured hospital discharge summary, a family physician follow-up appointment less than 1 week after discharge, and, for high-risk patients, structured nurse homevisits and heart function clinic care were provided to patients, or usual care (n = 1390 patients), in which transitional care was left to the discretion of clinicians. Main Outcomes and Measures: Primary outcomes were hierarchically ordered as composite all-cause readmission, emergency department (ED) visit, or death at 3 months; and composite all-cause readmission or ED visit at 30 days. Secondary outcomes were B-PREPARED score for discharge preparedness (range: 0 [most prepared] to 22 [least prepared]); the 3-Item Care Transitions Measure (CTM-3) for quality of transition (range: 0 [worst transition] to 100 [best transition]); the 5-level EQ-5D version (EQ-5D-5L) for quality of life (range: 0 [dead] to 1 [full health]); and quality-adjusted life-years (QALY; range: 0 [dead] to 0.5 [full health at 6 months]). Results: Among eligible patients, all 2494 (mean age, 77.7 years; 1258 [50.4%] women) completed the trial. There was no significant difference between the intervention and usual care groups in the first primary composite outcome (545 [49.4%] vs 698 [50.2%] events, respectively; hazard ratio [HR], 0.99 [95% CI, 0.83-1.19]) or in the second primary composite outcome (304 [27.5%] vs 408 [29.3%] events, respectively; HR, 0.93 [95% CI, 0.73-1.18]). There were significant differences between the intervention and usual care groups in the secondary outcomes of mean B-PREPARED score at 6 weeks (16.6 vs 13.9; difference, 2.65 [95% CI, 1.37-3.92]; P < .001); mean CTM-3 score at 6 weeks (76.5 vs 70.3; difference, 6.16 [95% CI, 0.90-11.43]; P = .02); and mean EQ-5D-5L score at 6 weeks (0.7 vs 0.7; difference, 0.06 [95% CI, 0.01 to 0.11]; P = .02) and 6 months (0.7 vs 0.6; difference, 0.06 [95% CI, 0.01-0.12]; P = .02). There was no significant difference in mean QALY between groups at 6 months (0.3 vs 0.3; difference, 0.00 [95% CI, -0.02 to 0.02]; P = .98). Conclusions and Relevance: Among patients with HF in Ontario, Canada, implementation of a patient-centered transitional care model compared with usual care did not improve a composite of clinical outcomes. Whether this type of intervention could be effective in other health care systems or locations would require further research. Trial Registration: ClinicalTrials.gov Identifier: NCT02112227.

Knowledge to action: Rationale and design of the Patient-Centered Care Transitions in Heart Failure (PACT-HF) stepped wedge cluster randomized trial.

INTRODUCTION: Heart Failure (HF) is a common cause of hospitalization in older adults. The transition from hospital to home is high-risk, and gaps in transitional care can increase the risk of re-hospitalization and death. Combining health care services supported by meta-analyses, we designed the PACT-HF transitional care model. METHODS: Adopting an integrated Knowledge Translation (iKT) approach in which decision-makers and clinicians are partners in research, we implement and test the effectiveness of PACT-HF among patients hospitalized for HF. We use a pragmatic stepped wedge cluster randomized trial design to introduce the complex health service intervention to 10 large hospitals in a randomized sequence until all hospitals initiate the intervention. The goal is for all patients hospitalized with HF to receive self-care education, multidisciplinary care, and early follow-up with their health care providers; and in addition, for high-risk patients to receive post-discharge nurse-led home visits and outpatient care in Heart Function clinics. This requires integration of care across hospitals, home care agencies, and outpatient clinics in our publicly funded health care system. While hospitals are the unit of recruitment and analysis, patients (estimated sample size of 3200) are the unit of analysis. Primary outcomes are hierarchically ordered as time to composite all-cause readmissions / emergency department (ED) visits / death at 3 months and time to composite all-cause readmissions / ED visits at 30 days. In a nested study of 8 hospitals, we measure the patient-centered outcomes of Discharge Preparedness, Care Transitions Quality, and Quality Adjusted Life Years (QALY); and the 6-month health care resource use and costs. We obtain all clinical and cost outcomes via linkages to provincial administrative databases. CONCLUSIONS: This protocol describes the implementation and testing of a transitional care model comprising health care services informed by high-level evidence. The study adopts an iKT and pragmatic approach, uses a robust study design, links clinical trial data with outcomes held in administrative databases, and includes patient-reported outcomes. Findings will have implications on clinical practice, health care policy, and Knowledge Translation (KT) research methodology.

A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study.

BACKGROUND: A major barrier to the practice of evidence-based medicine is efficiently finding scientifically sound studies on a given clinical topic. OBJECTIVE: To investigate a deep learning approach to retrieve scientifically sound treatment studies from the biomedical literature. METHODS: We trained a Convolutional Neural Network using a noisy dataset of 403,216 PubMed citations with title and abstract as features. The deep learning model was compared with state-of-the-art search filters, such as PubMed's Clinical Query Broad treatment filter, McMaster's textword search strategy (no Medical Subject Heading, MeSH, terms), and Clinical Query Balanced treatment filter. A previously annotated dataset (Clinical Hedges) was used as the gold standard. RESULTS: The deep learning model obtained significantly lower recall than the Clinical Queries Broad treatment filter (96.9% vs 98.4%; P<.001); and equivalent recall to McMaster's textword search (96.9% vs 97.1%; P=.57) and Clinical Queries Balanced filter (96.9% vs 97.0%; P=.63). Deep learning obtained significantly higher precision than the Clinical Queries Broad filter (34.6% vs 22.4%; P<.001) and McMaster's textword search (34.6% vs 11.8%; P<.001), but was significantly lower than the Clinical Queries Balanced filter (34.6% vs 40.9%; P<.001). CONCLUSIONS: Deep learning performed well compared to state-of-the-art search filters, especially when citations were not indexed. Unlike previous machine learning approaches, the proposed deep learning model does not require feature engineering, or time-sensitive or proprietary features, such as MeSH terms and bibliometrics. Deep learning is a promising approach to identifying reports of scientifically rigorous clinical research. Further work is needed to optimize the deep learning model and to assess generalizability to other areas, such as diagnosis, etiology, and prognosis.

Development and use of a content search strategy for retrieving studies on patients' views and preferences.

BACKGROUND: Identifying scientific literature addressing patients' views and preferences is complex due to the wide range of studies that can be informative and the poor indexing of this evidence. Given the lack of guidance we developed a search strategy to retrieve this type of evidence. METHODS: We assembled an initial list of terms from several sources, including the revision of the terms and indexing of topic-related studies and, methods research literature, and other relevant projects and systematic reviews. We used the relative recall approach, evaluating the capacity of the designed search strategy for retrieving studies included in relevant systematic reviews for the topic. We implemented in practice the final version of the search strategy for conducting systematic reviews and guidelines, and calculated search's precision and the number of references needed to read (NNR). RESULTS: We assembled an initial version of the search strategy, which had a relative recall of 87.4% (yield of 132/out of 151 studies). We then added some additional terms from the studies not initially identified, and re-tested this improved version against the studies included in a new set of systematic reviews, reaching a relative recall of 85.8% (151/out of 176 studies, 95% CI 79.9 to 90.2). This final version of the strategy includes two sets of terms related with two domains: "Patient Preferences and Decision Making" and "Health State Utilities Values". When we used the search strategy for the development of systematic reviews and clinical guidelines we obtained low precision values (ranging from 2% to 5%), and the NNR from 20 to 50. CONCLUSIONS: This search strategy fills an important research gap in this field. It will help systematic reviewers, clinical guideline developers, and policy-makers to retrieve published research on patients' views and preferences. In turn, this will facilitate the inclusion of this critical aspect when formulating heath care decisions, including recommendations.

Limits of search filter development.

OBJECTIVE: The research attempted to develop search filters for biomedical literature databases that improve retrieval of studies of clinical relevance for the nursing and rehabilitation professions. METHODS: Diagnostic testing framework compared machine-culled and practitioner-nominated search terms with a hand-tagged clinical literature database. RESULTS: We were unable to: (1) develop filters for nursing, likely because of the overlapping and expanding scope of practice for nurses in comparison with medical professionals, or (2) develop filters for rehabilitation, because of its broad scope and the profession's multifaceted understanding of "health and ability." CONCLUSIONS: We found limitations on search filter development for these health professions: nursing and rehabilitation.

High-performance information search filters for CKD content in PubMed, Ovid MEDLINE, and EMBASE.

BACKGROUND: Finding relevant articles in large bibliographic databases such as PubMed, Ovid MEDLINE, and EMBASE to inform care and future research is challenging. Articles relevant to chronic kidney disease (CKD) are particularly difficult to find because they are often published under different terminology and are found across a wide range of journal types. STUDY DESIGN: We used computer automation within a diagnostic test assessment framework to develop and validate information search filters to identify CKD articles in large bibliographic databases. SETTING & PARTICIPANTS: 22,992 full-text articles in PubMed, Ovid MEDLINE, or EMBASE. INDEX TEST: 1,374,148 unique search filters. REFERENCE TEST: We established the reference standard of article relevance to CKD by manual review of all full-text articles using prespecified criteria to determine whether each article contained CKD content or not. We then assessed filter performance by calculating sensitivity, specificity, and positive predictive value for the retrieval of CKD articles. Filters with high sensitivity and specificity for the identification of CKD articles in the development phase (two-thirds of the sample) were then retested in the validation phase (remaining one-third of the sample). RESULTS: We developed and validated high-performance CKD search filters for each bibliographic database. Filters optimized for sensitivity reached at least 99% sensitivity, and filters optimized for specificity reached at least 97% specificity. The filters were complex; for example, one PubMed filter included more than 89 terms used in combination, including "chronic kidney disease," "renal insufficiency," and "renal fibrosis." In proof-of-concept searches, physicians found more articles relevant to the topic of CKD with the use of these filters. LIMITATIONS: As knowledge of the pathogenesis of CKD grows and definitions change, these filters will need to be updated to incorporate new terminology used to index relevant articles. CONCLUSIONS: PubMed, Ovid MEDLINE, and EMBASE can be filtered reliably for articles relevant to CKD. These high-performance information filters are now available online and can be used to better identify CKD content in large bibliographic databases.

Feasibility of a randomized controlled trial to evaluate the impact of decision boxes on shared decision-making processes.

BACKGROUND: Decision boxes (DBoxes) are two-page evidence summaries to prepare clinicians for shared decision making (SDM). We sought to assess the feasibility of a clustered Randomized Controlled Trial (RCT) to evaluate their impact. METHODS: A convenience sample of clinicians (nurses, physicians and residents) from six primary healthcare clinics who received eight DBoxes and rated their interest in the topic and satisfaction. After consultations, their patients rated their involvement in decision-making processes (SDM-Q-9 instrument). We measured clinic and clinician recruitment rates, questionnaire completion rates, patient eligibility rates, and estimated the RCT needed sample size. RESULTS: Among the 20 family medicine clinics invited to participate in this study, four agreed to participate, giving an overall recruitment rate of 20%. Of 148 clinicians invited to the study, 93 participated (63%). Clinicians rated an interest in the topics ranging 6.4-8.2 out of 10 (with 10 highest) and a satisfaction with DBoxes of 4 or 5 out of 5 (with 5 highest) for 81% DBoxes. For the future RCT, we estimated that a sample size of 320 patients would allow detecting a 9% mean difference in the SDM-Q-9 ratings between our two arms (0.02 ICC; 0.05 significance level; 80% power). CONCLUSIONS: Clinicians' recruitment and questionnaire completion rates support the feasibility of the planned RCT. The level of interest of participants for the DBox topics, and their level of satisfaction with the Dboxes demonstrate the acceptability of the intervention. Processes to recruit clinics and patients should be optimized.

A process evaluation accompanying an attempted randomized controlled trial of an evidence service for health system policymakers.

BACKGROUND: We developed an evidence service that draws inputs from Health Systems Evidence (HSE), which is a comprehensive database of research evidence about governance, financial and delivery arrangements within health systems and about implementation strategies relevant to health systems. Our goal was to evaluate whether, how and why a 'full-serve' evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care as compared to a 'self-serve' evidence service. METHODS: We attempted to conduct a two-arm, 10-month randomized controlled trial (RCT), along with a follow-up qualitative process evaluation, but we terminated the RCT when we failed to reach our recruitment target. For the qualitative process evaluation we modified the original interview guide to allow us to explore the (1) factors influencing participation in the trial; (2) usage of HSE, factors explaining usage patterns, and strategies to increase usage; (3) participation in training workshops and use of other supports; and (4) views about and experiences with key HSE features. RESULTS: We terminated the RCT given our 15% recruitment rate. Six factors were identified by those who had agreed to participate in the trial as encouraging their participation: relevance of the study to participants' own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors; and participation of colleagues. Most reported that they never, infrequently or inconsistently used HSE and suggested strategies to increase its use, including regular email reminders and employee training. However, only two participants indicated that employee training, in the form of a workshop about finding and using research evidence, had influenced their use of HSE. Most participants found HSE features to be intuitive and helpful, although registration/sign-in and some page formats (particularly the advanced search page and detailed search results page) discouraged their use or did not optimize the user experience. CONCLUSIONS: The qualitative findings informed a re-design of HSE, which allows users to more efficiently find and use research evidence about how to strengthen or reform health systems or in how to get cost-effective programs, services and drugs to those who need them. Our experience with RCT recruitment suggests the need to consider changing the unit of allocation to divisions instead of individuals within divisions, among other lessons. TRIAL REGISTRATION: This protocol for this study is published in Implementation Science and registered with ClinicalTrials.gov ( HHS/FHS REB 10-267 ).

Users' guide to the surgical literature: how to perform a high-quality literature search.

The article "Users" guide to the surgical literature: how to perform a "literature search" was published in 2003, but the continuing technological developments in databases and search filters have rendered that guide out of date. The present guide fills an existing gap in this area; it provides the reader with strategies for developing a searchable clinical question, creating an efficient search strategy,accessing appropriate databases, and skillfully retrieving the best evidence to address the research question.

High-performance information search filters for acute kidney injury content in PubMed, Ovid Medline and Embase.

BACKGROUND: We frequently fail to identify articles relevant to the subject of acute kidney injury (AKI) when searching the large bibliographic databases such as PubMed, Ovid Medline or Embase. To address this issue, we used computer automation to create information search filters to better identify articles relevant to AKI in these databases. METHODS: We first manually reviewed a sample of 22 992 full-text articles and used prespecified criteria to determine whether each article contained AKI content or not. In the development phase (two-thirds of the sample), we developed and tested the performance of >1.3-million unique filters. Filters with high sensitivity and high specificity for the identification of AKI articles were then retested in the validation phase (remaining third of the sample). RESULTS: We succeeded in developing and validating high-performance AKI search filters for each bibliographic database with sensitivities and specificities in excess of 90%. Filters optimized for sensitivity reached at least 97.2% sensitivity, and filters optimized for specificity reached at least 99.5% specificity. The filters were complex; for example one PubMed filter included >140 terms used in combination, including 'acute kidney injury', 'tubular necrosis', 'azotemia' and 'ischemic injury'. In proof-of-concept searches, physicians found more articles relevant to topics in AKI with the use of the filters. CONCLUSIONS: PubMed, Ovid Medline and Embase can be filtered for articles relevant to AKI in a reliable manner. These high-performance information filters are now available online and can be used to better identify AKI content in large bibliographic databases.

The validity of recommendations from clinical guidelines: a survival analysis.

BACKGROUND: Clinical guidelines should be updated to maintain their validity. Our aim was to estimate the length of time before recommendations become outdated. METHODS: We used a retrospective cohort design and included recommendations from clinical guidelines developed in the Spanish National Health System clinical guideline program since 2008. We performed a descriptive analysis of references, recommendations and resources used, and a survival analysis of recommendations using the Kaplan-Meier method. RESULTS: We included 113 recommendations from 4 clinical guidelines with a median of 4 years since the most recent search (range 3.9-4.4 yr). We retrieved 39 136 references (range 3343-14 787) using an exhaustive literature search, 668 of which were related to the recommendations in our sample. We identified 69 (10.3%) key references, corresponding to 25 (22.1%) recommendations that required updating. Ninety-two percent (95% confidence interval 86.9-97.0) of the recommendations were valid 1 year after their development. This probability decreased at 2 (85.7%), 3 (81.3%) and 4 years (77.8%). INTERPRETATION: Recommendations quickly become outdated, with 1 out of 5 recommendations being out of date after 3 years. Waiting more than 3 years to review a guideline is potentially too long.

Interventions for enhancing medication adherence.

BACKGROUND: People who are prescribed self administered medications typically take only about half their prescribed doses. Efforts to assist patients with adherence to medications might improve the benefits of prescribed medications. OBJECTIVES: The primary objective of this review is to assess the effects of interventions intended to enhance patient adherence to prescribed medications for medical conditions, on both medication adherence and clinical outcomes. SEARCH METHODS: We updated searches of The Cochrane Library, including CENTRAL (via http://onlinelibrary.wiley.com/cochranelibrary/search/), MEDLINE, EMBASE, PsycINFO (all via Ovid), CINAHL (via EBSCO), and Sociological Abstracts (via ProQuest) on 11 January 2013 with no language restriction. We also reviewed bibliographies in articles on patient adherence, and contacted authors of relevant original and review articles. SELECTION CRITERIA: We included unconfounded RCTs of interventions to improve adherence with prescribed medications, measuring both medication adherence and clinical outcome, with at least 80% follow-up of each group studied and, for long-term treatments, at least six months follow-up for studies with positive findings at earlier time points. DATA COLLECTION AND ANALYSIS: Two review authors independently extracted all data and a third author resolved disagreements. The studies differed widely according to medical condition, patient population, intervention, measures of adherence, and clinical outcomes. Pooling results according to one of these characteristics still leaves highly heterogeneous groups, and we could not justify meta-analysis. Instead, we conducted a qualitative analysis with a focus on the RCTs with the lowest risk of bias for study design and the primary clinical outcome. MAIN RESULTS: The present update included 109 new RCTs published since the previous update in January 2007, bringing the total number of RCTs to 182; we found five RCTs from the previous update to be ineligible and excluded them. Studies were heterogeneous for patients, medical problems, treatment regimens, adherence interventions, and adherence and clinical outcome measurements, and most had high risk of bias. The main changes in comparison with the previous update include that we now: 1) report a lack of convincing evidence also specifically among the studies with the lowest risk of bias; 2) do not try to classify studies according to intervention type any more, due to the large heterogeneity; 3) make our database available for collaboration on sub-analyses, in acknowledgement of the need to make collective advancement in this difficult field of research. Of all 182 RCTs, 17 had the lowest risk of bias for study design features and their primary clinical outcome, 11 from the present update and six from the previous update. The RCTs at lowest risk of bias generally involved complex interventions with multiple components, trying to overcome barriers to adherence by means of tailored ongoing support from allied health professionals such as pharmacists, who often delivered intense education, counseling (including motivational interviewing or cognitive behavioral therapy by professionals) or daily treatment support (or both), and sometimes additional support from family or peers. Only five of these RCTs reported improvements in both adherence and clinical outcomes, and no common intervention characteristics were apparent. Even the most effective interventions did not lead to large improvements in adherence or clinical outcomes. AUTHORS' CONCLUSIONS: Across the body of evidence, effects were inconsistent from study to study, and only a minority of lowest risk of bias RCTs improved both adherence and clinical outcomes. Current methods of improving medication adherence for chronic health problems are mostly complex and not very effective, so that the full benefits of treatment cannot be realized. The research in this field needs advances, including improved design of feasible long-term interventions, objective adherence measures, and sufficient study power to detect improvements in patient-important clinical outcomes. By making our comprehensive database available for sharing we hope to contribute to achieving these advances.

A modified evidence-based practice- knowledge, attitudes, behaviour and decisions/outcomes questionnaire is valid across multiple professions involved in pain management.

BACKGROUND: A validated and reliable instrument was developed to knowledge, attitudes and behaviours with respect to evidence-based practice (EBB-KABQ) in medical trainees but requires further adaptation and validation to be applied across different health professionals. METHODS: A modified 33-item evidence-based practice scale (EBP-KABQ) was developed to evaluate EBP perceptions and behaviors in clinicians. An international sample of 673 clinicians interested in treatment of pain (mean age = 45 years, 48% occupational therapists/physical therapists, 25% had more than 5 years of clinical training) completed an online English version of the questionnaire and demographics. Scaling properties (internal consistency, floor/ceiling effects) and construct validity (association with EBP activities, comparator constructs) were examined. A confirmatory factor analysis was used to assess the 4-domain structure EBP knowledge, attitudes, behavior, outcomes/decisions). RESULTS: The EBP-KABQ scale demonstrated high internal consistency (Cronbach's alpha = 0.85), no evident floor/ceiling effects, and support for a priori construct validation hypotheses. A 4-factor structure provided the best fit statistics (CFI =0.89, TLI =0.86, and RMSEA = 0.06). CONCLUSIONS: The EBP-KABQ scale demonstrates promising psychometric properties in this sample. Areas for improvement are described.

Net improvement of correct answers to therapy questions after pubmed searches: pre/post comparison.

BACKGROUND: Clinicians search PubMed for answers to clinical questions although it is time consuming and not always successful. OBJECTIVE: To determine if PubMed used with its Clinical Queries feature to filter results based on study quality would improve search success (more correct answers to clinical questions related to therapy). METHODS: We invited 528 primary care physicians to participate, 143 (27.1%) consented, and 111 (21.0% of the total and 77.6% of those who consented) completed the study. Participants answered 14 yes/no therapy questions and were given 4 of these (2 originally answered correctly and 2 originally answered incorrectly) to search using either the PubMed main screen or PubMed Clinical Queries narrow therapy filter via a purpose-built system with identical search screens. Participants also picked 3 of the first 20 retrieved citations that best addressed each question. They were then asked to re-answer the original 14 questions. RESULTS: We found no statistically significant differences in the rates of correct or incorrect answers using the PubMed main screen or PubMed Clinical Queries. The rate of correct answers increased from 50.0% to 61.4% (95% CI 55.0%-67.8%) for the PubMed main screen searches and from 50.0% to 59.1% (95% CI 52.6%-65.6%) for Clinical Queries searches. These net absolute increases of 11.4% and 9.1%, respectively, included previously correct answers changing to incorrect at a rate of 9.5% (95% CI 5.6%-13.4%) for PubMed main screen searches and 9.1% (95% CI 5.3%-12.9%) for Clinical Queries searches, combined with increases in the rate of being correct of 20.5% (95% CI 15.2%-25.8%) for PubMed main screen searches and 17.7% (95% CI 12.7%-22.7%) for Clinical Queries searches. CONCLUSIONS: PubMed can assist clinicians answering clinical questions with an approximately 10% absolute rate of improvement in correct answers. This small increase includes more correct answers partially offset by a decrease in previously correct answers.

Retrieving clinical evidence: a comparison of PubMed and Google Scholar for quick clinical searches.

BACKGROUND: Physicians frequently search PubMed for information to guide patient care. More recently, Google Scholar has gained popularity as another freely accessible bibliographic database. OBJECTIVE: To compare the performance of searches in PubMed and Google Scholar. METHODS: We surveyed nephrologists (kidney specialists) and provided each with a unique clinical question derived from 100 renal therapy systematic reviews. Each physician provided the search terms they would type into a bibliographic database to locate evidence to answer the clinical question. We executed each of these searches in PubMed and Google Scholar and compared results for the first 40 records retrieved (equivalent to 2 default search pages in PubMed). We evaluated the recall (proportion of relevant articles found) and precision (ratio of relevant to nonrelevant articles) of the searches performed in PubMed and Google Scholar. Primary studies included in the systematic reviews served as the reference standard for relevant articles. We further documented whether relevant articles were available as free full-texts. RESULTS: Compared with PubMed, the average search in Google Scholar retrieved twice as many relevant articles (PubMed: 11%; Google Scholar: 22%; P<.001). Precision was similar in both databases (PubMed: 6%; Google Scholar: 8%; P=.07). Google Scholar provided significantly greater access to free full-text publications (PubMed: 5%; Google Scholar: 14%; P<.001). CONCLUSIONS: For quick clinical searches, Google Scholar returns twice as many relevant articles as PubMed and provides greater access to free full-text articles.

Impact of PubMed search filters on the retrieval of evidence by physicians.

BACKGROUND: Physicians face challenges when searching PubMed for research evidence, and they may miss relevant articles while retrieving too many nonrelevant articles. We investigated whether the use of search filters in PubMed improves searching by physicians. METHODS: We asked a random sample of Canadian nephrologists to answer unique clinical questions derived from 100 systematic reviews of renal therapy. Physicians provided the search terms that they would type into PubMed to locate articles to answer these questions. We entered the physician-provided search terms into PubMed and applied two types of search filters alone or in combination: a methods-based filter designed to identify high-quality studies about treatment (clinical queries "therapy") and a topic-based filter designed to identify studies with renal content. We evaluated the comprehensiveness (proportion of relevant articles found) and efficiency (ratio of relevant to nonrelevant articles) of the filtered and nonfiltered searches. Primary studies included in the systematic reviews served as the reference standard for relevant articles. RESULTS: The average physician-provided search terms retrieved 46% of the relevant articles, while 6% of the retrieved articles were relevant (corrected) (the ratio of relevant to nonrelevant articles was 1:16). The use of both filters together produced a marked improvement in efficiency, resulting in a ratio of relevant to nonrelevant articles of 1:5 (16 percentage point improvement; 99% confidence interval 9% to 22%; p < 0.003) with no substantive change in comprehensiveness (44% of relevant articles found; p = 0.55). INTERPRETATION: The use of PubMed search filters improves the efficiency of physician searches. Improved search performance may enhance the transfer of research into practice and improve patient care.

How current are leading evidence-based medical textbooks? An analytic survey of four online textbooks.

BACKGROUND: The consistency of treatment recommendations of evidence-based medical textbooks with more recently published evidence has not been investigated to date. Inconsistencies could affect the quality of medical care. OBJECTIVE: To determine the frequency with which topics in leading online evidence-based medical textbooks report treatment recommendations consistent with more recently published research evidence. METHODS: Summarized treatment recommendations in 200 clinical topics (ie, disease states) covered in four evidence-based textbooks--UpToDate, Physicians' Information Education Resource (PIER), DynaMed, and Best Practice--were compared with articles identified in an evidence rating service (McMaster Premium Literature Service, PLUS) since the date of the most recent topic updates in each textbook. Textbook treatment recommendations were compared with article results to determine if the articles provided different, new conclusions. From these findings, the proportion of topics which potentially require updating in each textbook was calculated. RESULTS: 478 clinical topics were assessed for inclusion to find 200 topics that were addressed by all four textbooks. The proportion of topics for which there was 1 or more recently published articles found in PLUS with evidence that differed from the textbooks' treatment recommendations was 23% (95% CI 17-29%) for DynaMed, 52% (95% CI 45-59%) for UpToDate, 55% (95% CI 48-61%) for PIER, and 60% (95% CI 53-66%) for Best Practice (χ(2) (3)=65.3, P<.001). The time since the last update for each textbook averaged from 170 days (range 131-209) for DynaMed, to 488 days (range 423-554) for PIER (P<.001 across all textbooks). CONCLUSIONS: In online evidence-based textbooks, the proportion of topics with potentially outdated treatment recommendations varies substantially.