Increasing the efficiency of study selection for systematic reviews using prioritization tools and a single-screening approach.

BACKGROUND: Systematic literature screening is a key component in systematic reviews. However, this approach is resource intensive as generally two persons independently of each other (double screening) screen a vast number of search results. To develop approaches for increasing efficiency, we tested the use of text mining to prioritize search results as well as the involvement of only one person (single screening) in the study selection process. METHOD: Our study is based on health technology assessments (HTAs) of drug and non-drug interventions. Using a sample size calculation, we consecutively included 11 searches resulting in 33 study selection processes. Of the three screeners for each search, two used screening tools with prioritization (Rayyan, EPPI Reviewer) and one a tool without prioritization. For each prioritization tool, we investigated the proportion of citations classified as relevant at three cut-offs or STOP criteria (after screening 25%, 50% and 75% of the citation set). For each STOP criterion, we measured sensitivity (number of correctly identified relevant studies divided by the total number of relevant studies in the study pool). In addition, we determined the number of relevant studies identified per single screening round and investigated whether missed studies were relevant to the HTA conclusion. RESULTS: Overall, EPPI Reviewer performed better than Rayyan and identified the vast majority (88%, Rayyan 66%) of relevant citations after screening half of the citation set. As long as additional information sources were screened, it was sufficient to apply a single-screening approach to identify all studies relevant to the HTA conclusion. Although many relevant publications (n = 63) and studies (n = 29) were incorrectly excluded, ultimately only 5 studies could not be identified at all in 2 of the 11 searches (1x 1 study, 1x 4 studies). However, their omission did not change the overall conclusion in any HTA. CONCLUSIONS: EPPI Reviewer helped to identify relevant citations earlier in the screening process than Rayyan. Single screening would have been sufficient to identify all studies relevant to the HTA conclusion. However, this requires screening of further information sources. It also needs to be considered that the credibility of an HTA may be questioned if studies are missing, even if they are not relevant to the HTA conclusion.

Information specialist collaboration in Europe: collaborative methods, processes, and infrastructure through EUnetHTA.

The history of European health technology assessment (HTA) goes back more than 30 years. Almost as old as HTA agencies themselves is the desire to achieve European collaboration. This gained further impetus with the establishment of the European Network of Health Technology Assessment (EUnetHTA) in 2006. In this context, the field of information management faced specific challenges. Although these services are an integral part of HTA and information specialists play a key role here, this field is often not adequately represented in the HTA agencies within EUnetHTA. Furthermore, the organization of HTA production, including the types of HTAs produced, as well as funding, varies considerably. In order to meet these different conditions, information specialists have created various products and defined processes. With the EUnetHTA guideline, a common methodological understanding for the production of rapid Relative Effectiveness Assessments now exists. Furthermore, the Standard Operating Procedures map the complex information retrieval processes within EUnetHTA in a hands-on manner. The newly established Information Specialist Network (ISN) will in future ensure that information specialists are involved in all EUnetHTA assessments and that the methods are applied consistently in all assessments. In addition, the steering committee of the ISN manages enquiries and can be contacted to discuss methodological issues. Major barriers such as heterogeneity in the daily work of the EUnetHTA members can only be overcome through more collaboration and training.

Development and validation of study filters for identifying controlled non-randomized studies in PubMed and Ovid MEDLINE.

A retrospective analysis published by the German Institute for Quality and Efficiency in Health Care (IQWiG) in 2018 concluded that no filter for non-randomized studies (NRS) achieved sufficient sensitivity (≥92%), a precondition for comprehensive information retrieval. New NRS filters are therefore required, taking into account the challenges related to this study type. Our evaluation focused on the development of study filters for NRS with a control group ("controlled NRS"), as this study type allows the calculation of an effect size. In addition, we assumed that due to the more explicit search syntax, controlled NRS are easier to identify than non-controlled ones, potentially resulting in better performance measures of study filters for controlled NRS. Our aim was to develop study filters for identifying controlled NRS in PubMed and Ovid MEDLINE. We developed two new search filters that can assist clinicians and researchers in identifying controlled NRS in PubMed and Ovid MEDLINE. The reference set was based on 2110 publications in Medline extracted from 271 Cochrane reviews and on 4333 irrelevant references. The first filter maximizes sensitivity (92.42%; specificity 79.67%, precision 68.49%) and should be used when a comprehensive search is needed. The second filter maximizes specificity (92.06%; precision 82.98%, sensitivity 80.94%) and should be used when a more focused search is sufficient.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation.

BACKGROUND: Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations. METHODS: Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets. RESULTS: A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted. CONCLUSIONS: The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

Effective study selection using text mining or a single-screening approach: a study protocol.

BACKGROUND: Systematic information retrieval generally requires a two-step selection process for studies, which is conducted by two persons independently of one another (double-screening approach). To increase efficiency, two methods seem promising, which will be tested in the planned study: the use of text mining to prioritize search results as well as the involvement of only one person in the study selection process (single-screening approach). The aim of the present study is to examine the following questions related to the process of study selection: Can the use of the Rayyan or EPPI Reviewer tools to prioritize the results of study selection increase efficiency? How accurately does a single-screening approach identify relevant studies? Which advantages or disadvantages (e.g., shortened screening time or increase in the number of full texts ordered) does a single-screening versus a double-screening approach have? METHODS: Our study is a prospective analysis of study selection processes based on benefit assessments of drug and non-drug interventions. It consists of two parts: firstly, the evaluation of a single-screening approach based on a sample size calculation (11 study selection processes, including 33 single screenings) and involving different screening tools and, secondly, the evaluation of the conventional double-screening approach based on five conventional study selection processes. In addition, the advantages and disadvantages of the single-screening versus the double-screening approach with regard to the outcomes "number of full texts ordered" and "time required for study selection" are analyzed. The previous work experience of the screeners is considered as a potential effect modifier. DISCUSSION: No study comparing the features of prioritization tools is currently available. Our study can thus contribute to filling this evidence gap. This study is also the first to investigate a range of questions surrounding the screening process and to include an a priori sample size calculation, thus enabling statistical conclusions. In addition, the impact of missing studies on the conclusion of a benefit assessment is calculated. SYSTEMATIC REVIEW REGISTRATION: Not applicable.

Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity.

OBJECTIVES: To analyze the availability of randomized controlled trials (RCTs) of new drugs in trial registries and to develop and test different search strategies in ClinicalTrials.gov (CT.gov), the EU Clinical Trials Register (EU-CTR), and the International Clinical Trials Registry Platform (ICTRP). STUDY DESIGN AND SETTING: Information from dossiers submitted by pharmaceutical companies was analyzed regarding the registration of the included RCTs in CT.gov, EU-CTR and ICTRP; different search strategies were developed and tested to determine performance. RESULTS: A total of 192 (95%) of 203 RCTs on newly approved drugs were registered in CT.gov; the 11 nonregistered trials were completed before 2005 or represented non-RCTs. Simple searches for RCTs on 18 new drugs using the generic drug name yielded a sensitivity of 94% in CT.gov (EU-CTR: 71%; ICTRP: 60%). The main reason for study nondetection was the sole use of the drug code in the registry entries. Simple searches for RCTs on 13 conditions using reasonably inferred search terms yielded a sensitivity of 100% in CT.gov. CONCLUSION: Almost all relevant RCTs on newly approved drugs will probably be identified in CT.gov alone. A sensitive search in CT.gov can be conducted using single search terms. The searches in ICTRP and EU-CTR should include several search terms (e.g., derived via text analysis).

Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one.

BACKGROUND: In the development of search strategies for systematic reviews, "conceptual approaches" are generally recommended to identify appropriate search terms for those parts of the strategies for which no validated search filters exist. However, "objective approaches" based on search terms identified by text analysis are increasingly being applied. OBJECTIVES: To prospectively compare an objective with a conceptual approach for the development of search strategies. METHODS: Two different MEDLINE search strategies were developed in parallel for five systematic reviews covering a range of topics and study designs. The Institute for Quality and Efficiency in Health Care (IQWiG) applied an objective approach, and external experts applied a conceptual approach for the same research questions. For each systematic review, the citations retrieved were combined and the overall pool of citations screened to determine sensitivity and precision. RESULTS: The objective approach yielded a weighted mean sensitivity and precision of 97% and 5%. The corresponding values for the conceptual approach were 75% and 4%. CONCLUSION: Our findings indicate that the objective approach applied by IQWiG for search strategy development yields higher sensitivity than and similar precision to a conceptual approach. The main advantage of the objective approach is that it produces consistent results across searches.

The Role of End-of-Life Issues in the Design and Reporting of Cancer Clinical Trials: A Structured Literature Review.

BACKGROUND: Randomized controlled trials (RCTs) are important sources of information on the benefits and harms patients may expect from treatment options. The aim of this structured literature review by the German Institute for Quality and Efficiency in Health Care was to explore whether and how the end-of-life (EoL) situation of patients with advanced cancer is considered in RCTs investigating anti-cancer treatments. METHODS: Our journal pool comprised 19 medical journals, namely five preselected key general medical journals as well as 14 specialist journals (mainly cancer) identified via a scoping search. We systematically searched these journals in MEDLINE to identify RCTs investigating anti-cancer treatments for the following four cancer types: glioblastoma, lung cancer (stage IIIb-IV), malignant melanoma (stage IV), and pancreatic cancer (search via OVID; November 2012). We selected a representative sample of 100 publications, that is, the 25 most recent publications for each cancer type. EoL was defined as a life expectancy of ≤ two years. We assessed the information provided on (1) the descriptions of the terminal stage of the disease, (2) the therapeutic goal (i.e. the intended therapeutic benefit of the intervention studied), (3) the study endpoints assessed, (4) the authors' concluding appraisal of the intervention's effects, and (5) the terminology referring to the patients' EoL situation. RESULTS: Median survival was ≤ one year for each of the four cancer types. Descriptions of the terminal stage of the disease were ambiguous or lacking in 29/100 publications. One or more therapeutic goals were mentioned in 51/100 publications; these goals were patient-relevant in 38 publications (survival alone: 30/38; health-related quality of life (HRQoL) or HRQoL and survival: 6/38; symptom control or symptom control and survival: 2/38). Primary endpoints included survival (50%), surrogates (44%), and safety (3%). Patient-reported outcomes (PROs) were assessed in 36/100 RCTs. The implications of treatment-related harms for the patients were discussed in 22/100 appraisals. Terminology referring to the patients' EoL situation (e.g. "terminal") was scarce, whereas terms suggesting control of the disease (e.g. "cancer control") were common. CONCLUSIONS: The EoL situation of patients with advanced cancer should be more carefully considered in clinical trials. Although the investigation and robust reporting of PROs is a prerequisite for informed decision-making in healthcare, they are rarely defined as endpoints and HRQoL is rarely mentioned as a therapeutic goal. Suggestions for improving standards for study design and reporting are presented.

Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.

BACKGROUND: Different approaches can be adopted for the development of search strategies of systematic reviews. The objective approach draws on already established text analysis methods for developing search filters. Our aim was to determine whether the objective approach for the development of search strategies was noninferior to the conceptual approach commonly used in Cochrane reviews (CRs). METHODS: We conducted a search for CRs published in the Cochrane Library. The studies included in the CRs were searched for in MEDLINE and represented the total set. We then tested whether references previously removed could be identified via the objective approach. We also reconstructed the original search strategies from the CRs to determine why references could not be identified by the objective approach. As we performed the validation of the search strategies without study filters, we used only sensitivity as a quality measure and did not calculate precision. RESULTS: The objective approach yielded a mean sensitivity of 96% based on 13 searches. The noninferiority test showed that this approach was noninferior to the conceptual approach used in the CRs (P < 0.002). An additional descriptive analysis showed that the original MEDLINE strategies could identify only 86% of all references; however, this lower sensitivity was largely due to one CR. CONCLUSION: To the best of our knowledge, our findings indicate for the first time that the objective approach for the development of search strategies is noninferior to the conceptual approach.

Appraisal tools for clinical practice guidelines: a systematic review.

INTRODUCTION: Clinical practice guidelines can improve healthcare processes and patient outcomes, but are often of low quality. Guideline appraisal tools aim to help potential guideline users in assessing guideline quality. We conducted a systematic review of publications describing guideline appraisal tools in order to identify and compare existing tools. METHODS: Among others we searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews from 1995 to May 2011 for relevant primary and secondary publications. We also handsearched the reference lists of relevant publications. On the basis of the available literature we firstly generated 34 items to be used in the comparison of appraisal tools and grouped them into thirteen quality dimensions. We then extracted formal characteristics as well as questions and statements of the appraisal tools and assigned them to the items. RESULTS: We identified 40 different appraisal tools. They covered between three and thirteen of the thirteen possible quality dimensions and between three and 29 of the possible 34 items. The main focus of the appraisal tools were the quality dimensions "evaluation of evidence" (mentioned in 35 tools; 88%), "presentation of guideline content" (34 tools; 85%), "transferability" (33 tools; 83%), "independence" (32 tools; 80%), "scope" (30 tools; 75%), and "information retrieval" (29 tools; 73%). The quality dimensions "consideration of different perspectives" and "dissemination, implementation and evaluation of the guideline" were covered by only twenty (50%) and eighteen tools (45%) respectively. CONCLUSIONS: Most guideline appraisal tools assess whether the literature search and the evaluation, synthesis and presentation of the evidence in guidelines follow the principles of evidence-based medicine. Although conflicts of interest and norms and values of guideline developers, as well as patient involvement, affect the trustworthiness of guidelines, they are currently insufficiently considered. Greater focus should be placed on these issues in the further development of guideline appraisal tools.

Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews.

OBJECTIVE: The Institute for Quality and Efficiency in Health Care (IQWiG) assesses the added benefit of new drugs by means of company dossiers. The pharmaceutical company performs the information retrieval, which is then assessed by IQWiG. Our aim was to determine whether PubMed's Related Citations (RelCits) and/or a simple-structured Boolean search (SSBS) are efficient and reliable search techniques to assess the completeness of an evidence base consisting of published randomized controlled trials (RCTs). STUDY DESIGN AND SETTING: Retrospective analysis of citations included as relevant in systematic reviews (SRs) of drugs. The proportion of relevant citations identified by the above-mentioned search techniques was determined. Relative sensitivity, precision, and the number needed to read (NNR) were then calculated. RESULTS: A total of 19 SRs included 330 relevant PubMed citations. The single techniques yielded either insufficient completeness, reliability, or efficiency. The first 20 RelCits plus SSBS achieved high completeness and reliability (sensitivity: 98.1%, range: 80-100%) and sufficient efficiency (precision: 5.0%, NNR: 25). The first 50 RelCit plus SSBS achieved slightly better completeness and reliability, but slightly worse efficiency. CONCLUSION: Combining the first 20 RelCits and an SSBS in PubMed is a suitable method to assess the completeness of an evidence base of published RCTs.

Randomized controlled trials on PET: a systematic review of topics, design, and quality.

UNLABELLED: Randomized controlled trials (RCTs) add important information to diagnostic accuracy studies in the evaluation of PET and PET/CT. We evaluated how many RCTs on PET existed, which clinical topics they addressed, and what their design and quality were. METHODS: We searched MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials (Clinical Trials) up to August 2010. We also searched in ClinicalTrials.gov and the International Clinical Trials Registry Platform for ongoing RCTs up to March 2011. Titles and abstracts and full texts were screened independently by 2 reviewers. Study characteristics were extracted with standard extraction sheets for ongoing and published RCTs, and risk of bias was assessed for published ones. RESULTS: We identified 54 RCTs, 12 of which were published. The main topics in published studies were non-small cell lung cancer and colorectal cancer; only 3 were conducted in nononcologic fields (this trend was similar in ongoing studies, in which the most common topic was Hodgkin disease). The main indications in the oncologic PET studies were staging in published studies and restaging (mostly including an early assessment of treatment response) in ongoing ones. All except 1 of the published studies applied a marker-based strategy design, whereas about 43% (18/42) of ongoing studies use a more efficient design (Enrichment Design or Marker by Treatment Interaction Design). CONCLUSION: A relatively high number of ongoing RCTs of PET in several oncologic fields are expected to produce robust results over the next few years. For nononcologic topics, further high-quality studies are still needed to ascertain the benefit of this technique for patients. As funding is usually difficult in nondrug topics, alternative concepts of funding, which should also involve the manufacturers of diagnostic devices, but also more efficient study designs, should be applied to bridge the evidence gap on PET in the near future.

Routine development of objectively derived search strategies.

BACKGROUND: Over the past few years, information retrieval has become more and more professionalized, and information specialists are considered full members of a research team conducting systematic reviews. Research groups preparing systematic reviews and clinical practice guidelines have been the driving force in the development of search strategies, but open questions remain regarding the transparency of the development process and the available resources. An empirically guided approach to the development of a search strategy provides a way to increase transparency and efficiency. METHODS: Our aim in this paper is to describe the empirically guided development process for search strategies as applied by the German Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, or "IQWiG"). This strategy consists of the following steps: generation of a test set, as well as the development, validation and standardized documentation of the search strategy. RESULTS: We illustrate our approach by means of an example, that is, a search for literature on brachytherapy in patients with prostate cancer. For this purpose, a test set was generated, including a total of 38 references from 3 systematic reviews. The development set for the generation of the strategy included 25 references. After application of textual analytic procedures, a strategy was developed that included all references in the development set. To test the search strategy on an independent set of references, the remaining 13 references in the test set (the validation set) were used. The validation set was also completely identified. DISCUSSION: Our conclusion is that an objectively derived approach similar to that used in search filter development is a feasible way to develop and validate reliable search strategies. Besides creating high-quality strategies, the widespread application of this approach will result in a substantial increase in the transparency of the development process of search strategies.

An evaluation of searching the German CCMed database for the production of systematic reviews.

OBJECTIVES: National databases may be useful sources in the production of a systematic review (SR). The aim of this study was to assess the potential benefit of a systematic search in the German database 'Current Contents Medizin' (CCMed). METHODS: The study was conducted on the basis of published SRs that included CCMed as a literature source. Eligible SRs were identified through a systematic search in medline, embase and The Cochrane Library. The websites of German Health Technology Assessment agencies were also screened. Citations of primary studies included as relevant in the SRs were extracted and then categorised. RESULTS: The search yielded 52 eligible SRs. A total of 1505 relevant citations were extracted. Seventy-seven of these articles were published in journals indexed in CCMed. Thirty-two of the 77 citations were indexed in CCMed, but only eight of the 32 were unique. Of these eight citations, seven were not identified by a systematic search, but by handsearching. Only one unique citation, an observational study, was identified in CCMed by a systematic search. CONCLUSIONS: In the production of SRs, a systematic search in CCMed identifies relevant studies only in exceptional cases. Therefore, the routine inclusion of this database in systematic searches does not appear meaningful.

[Developments of nursing research within German-speaking countries - publications from 1988 until 2007 in the journal "Pflege"]. / Entwicklungen der deutschsprachigen Pflegewissenschaft im Spiegel der PFLEGE - Veröffentlichung von 1988 - 2007.

"Pflege" is the nursing research journal with the largest circulation in the German-speaking area and has been nursing research experts' only communication platform for a considerable time. Analysing the structure of articles aims to focus on development and alteration of the German-speaking region's nursing research. The study consists of a retrospective analysis of publications in the nursing research journal "Pflege". 589 articles from 1988 until 2007 could be included into the analysis. Research questions refer to the amount of empirical studies and the study designs in quantitative projects. Almost 50 % of all publications of the "Pflege" represent results of empirical research; the remaining publications come from "other publications" and increasingly literature reviews. Research designs are mainly simple cross-sectional surveys; only 20 % are intervention studies (including five randomised controlled trials). The importance of intervention studies will increase in future. This development cannot be seen in the "Pflege". There is a need for further bibliometric analysis to be conducted to find out whether German-speaking nurse researchers actually seldom conduct intervention studies, or whether they prefer to publish in journals with a high impact factor.

Identifying nurse staffing research in Medline: development and testing of empirically derived search strategies with the PubMed interface.

BACKGROUND: The identification of health services research in databases such as PubMed/Medline is a cumbersome task. This task becomes even more difficult if the field of interest involves the use of diverse methods and data sources, as is the case with nurse staffing research. This type of research investigates the association between nurse staffing parameters and nursing and patient outcomes. A comprehensively developed search strategy may help identify nurse staffing research in PubMed/Medline. METHODS: A set of relevant references in PubMed/Medline was identified by means of three systematic reviews. This development set was used to detect candidate free-text and MeSH terms. The frequency of these terms was compared to a random sample from PubMed/Medline in order to identify terms specific to nurse staffing research, which were then used to develop a sensitive, precise and balanced search strategy. To determine their precision, the newly developed search strategies were tested against a) the pool of relevant references extracted from the systematic reviews, b) a reference set identified from an electronic journal screening, and c) a sample from PubMed/Medline. Finally, all newly developed strategies were compared to PubMed's Health Services Research Queries (PubMed's HSR Queries). RESULTS: The sensitivities of the newly developed search strategies were almost 100% in all of the three test sets applied; precision ranged from 6.1% to 32.0%. PubMed's HSR queries were less sensitive (83.3% to 88.2%) than the new search strategies. Only minor differences in precision were found (5.0% to 32.0%). CONCLUSIONS: As with other literature on health services research, nurse staffing studies are difficult to identify in PubMed/Medline. Depending on the purpose of the search, researchers can choose between high sensitivity and retrieval of a large number of references or high precision, i.e. and an increased risk of missing relevant references, respectively. More standardized terminology (e.g. by consistent use of the term "nurse staffing") could improve the precision of future searches in this field. Empirically selected search terms can help to develop effective search strategies. The high consistency between all test sets confirmed the validity of our approach.