Peer review of searches for studies for health technology assessments, systematic reviews, and other evidence syntheses.

INTRODUCTION: Peer review of searches is a process whereby both the search strategies and the search process description are reviewed, ideally using an evidence-based checklist. RATIONALE: As the search strategy underpins any well-conducted evidence synthesis, its quality could affect the final result. Evidence shows, however, that search strategies are prone to error. FINDINGS: There is increasing awareness and use of the PRESS Evidence-Based Checklist and peer review of search strategies, at the outset of evidence syntheses, prior to the searches being run, and this is now recommended by a number of evidence synthesis organizations. RECOMMENDATIONS AND CONCLUSIONS: Searches for evidence syntheses should be peer reviewed by a suitably qualified and experienced librarian or information specialist after being designed, ideally, by another suitably qualified and experienced librarian or information specialist. Peer review of searches should take place at two important stages in the evidence synthesis process; at the outset of the project prior to the searches being run and at the prepublication stage. There is little empirical evidence, however, to support the effectiveness of peer review of searches. Further research is required to assess this. Those wishing to stay up to date with the latest developments in information retrieval, including peer review of searches, should consult the SuRe Info resource (http://www.sure-info.org), which seeks to help information specialists and others by providing easy access to the findings from current information retrieval methods research and thus support more research-based information retrieval practice.

In vitro mechanical analysis of complete-arch mandibular implant-supported fixed prostheses abutment screws after cyclic loading.

STATEMENT OF PROBLEM: Clinicians question when to evaluate for worn or loose implant-supported retainer screws to prevent possible clinical complications. PURPOSE: The purpose of this study was to compare differences among initial and postdynamically loaded detorque values and identify physical structural changes of prosthetic retaining screws in a simulated implant-supported mandibular complete fixed prosthesis. MATERIAL AND METHODS: Nine groups and nonloaded controls comprising a 5-implant-supported, milled titanium framework were fabricated and assembled (screw torque 35 Ncm). Dynamic loading (20 to 220 N) was applied to simulate 2 years of oral function. After testing, screw detorque values were measured (ΔT, initial-detorque value). A scanning electron microscopic analysis of screw threads was used to assess physical changes. Data were analyzed by 2-way ANOVA to determine the influence of loading and implant position on ΔT (α=.05). RESULTS: ΔT values of loaded and nonloaded groups were compared separately at each implant position and showed a significant difference only for the implant in the central position (P=.002). All positions were compared in terms of ΔT values separately for loaded and nonloaded conditions. A significantly higher ΔT was found in 1 cantilever area of the loaded group, whereas a significantly lower ΔT value was found in the central position in the nonloaded group. No statistically significant differences were found in physical changes between loading and nonloading or among implant positions. CONCLUSIONS: When delivering a multiimplant supported prosthesis, the application of dynamic loading and the sequence in which implant screws are tightened could influence the subsequent detorque value of a screw; they have no effect on the physical appearance of screws after extended function.

Reporting methodological search filter performance comparisons: a literature review.

BACKGROUND: Methodological search filters are tools for retrieving database records reporting studies which use a specific research method. Choosing a filter is likely to be based on filter performance data. This review examines which measures are reported, and the way that filter performance is presented, in filter comparisons. METHODS: Studies were identified from the current content and pending update (2010) of a filter website. Eligible studies compared two or more methodological search filters designed to identify randomised controlled trials, diagnostic test accuracy studies, systematic reviews or economic evaluations. RESULTS: Eighteen studies met the inclusion criteria. The number of filters compared in a single study ranged from 2 to 38. The most commonly reported measures were sensitivity/recall and precision. All studies displayed results in tables and gave results as percentages or proportions. Two studies supplemented results tables with graphical displays of data: a bar graph of the proportion of retrieved and missed gold standard references per filter; a forest plot of the overall sensitivity and specificity of each filter. CONCLUSIONS: Sensitivity/recall and precision are the most frequently reported performance measures. This review highlights the potential for presenting results in novel and innovative ways to aid filter selection.

Choosing and using methodological search filters: searchers' views.

BACKGROUND: Search filters or hedges are search strategies developed to assist information specialists and librarians to retrieve different types of evidence from bibliographic databases. The objectives of this project were to learn about searchers' filter use, how searchers choose search filters and what information they would like to receive to inform their choices. METHODS: Interviews with information specialists working in, or for, the National Institute for Health and Care Excellence (NICE) were conducted. An online questionnaire survey was also conducted and advertised via a range of email lists. RESULTS: Sixteen interviews were undertaken and 90 completed questionnaires were received. The use of search filters tends to be linked to reducing a large amount of literature, introducing focus and assisting with searches that are based on a single study type. Respondents use numerous ways to identify search filters and can find choosing between different filters problematic because of knowledge gaps and lack of time. CONCLUSIONS: Search filters are used mainly for reducing large result sets (introducing focus) and assisting with searches focused on a single study type. Features that would help with choosing filters include making information about filters less technical, offering ratings and providing more detail about filter validation strategies and filter provenance.

Searching for medical devices - Practical guidance.

Systematic reviews of medical devices have generally adopted the same methodology for the conduct of their reviews as reviews of other clinical interventions, in particular, medicines. It has been suggested that these methods may need to be developed to account for the challenges of reviewing the evidence for medical devices when compared to reviews of medicines. The purpose of this research note is to focus on the methods of searching for medical devices. Our aim is to set out guidance on 'how to search' for medical devices. This includes: Defining what you are searching for; How to design a search strategy; Searching bibliographic databases; Searching beyond bibliographic databases; and Search reporting. The research note is written by three experienced searchers/researchers with experience of critically appraising Medical Technologies Evaluation Programme (MTEP) submissions, undertaking systematic searches and/or reviews of medical devices, or developing guidance for searching for studies for systematic reviews of the effectiveness of interventions, including medical devices. The research note includes a worked example of a search for a fictitious medical device.

Relationship of denture cast measurements to width of maxillary anterior teeth.

STATEMENT OF PROBLEM: When making complete dentures, clinicians may have difficulty with selection of properly sized denture teeth. PURPOSE: The purpose of this study was to determine if there are specific measurements made on an edentulous cast that could be useful to clinicians for selection of proper maxillary anterior denture tooth width. MATERIAL AND METHODS: Following Institutional Review Board approval, measurements were made on 50 maxillary and mandibular sets of complete denture casts and their marked and contoured occlusion rims and record bases consecutively submitted to a dental laboratory. The following 6 measurements were recorded for each set of casts: left mid-maxillary to right mid-maxillary, representing the distance between the respective residual ridge crest points adjacent to each canine eminence; left retromolar pad to right retromolar pad; incisive papilla to left hamular notch; incisive papilla to right hamular notch; left hamular notch to right hamular notch; and incisive papilla to vibrating line. These measurements were statistically compared with the control, the canine-to-canine distance as marked on the contoured maxillary occlusion rim to indicate the relaxed commissures. The casts were also classified as small, medium or large based upon the tooth width specified by the commercial mold guide for their respective canine-to-canine control measurement. The estimation bias was determined for each of the 6 measurements, and for those measurements for which the bias was significantly different from zero, a bias-correction was applied. St. Laurent's coefficient was used to describe the agreement between each (bias-corrected) measurement and the canine-to-canine control. A commercial guide was used to determine the accuracy of each of the 6 bias-corrected measurements in classifying denture teeth width as small, medium or large as compared to the same classification using the canine-to-canine control. RESULTS: The bias-corrected measurement from the left hamular notch mark to the right hamular notch mark, obtained by adding 10 mm, exhibited the greatest degree of agreement with the control. This measurement was also the only one of the 6 that correctly classified more than 50% of both the medium and large casts. None of the 6 measurements classified more than 30% of the small casts correctly. CONCLUSIONS: Of the 6 measurements considered here, the bias-corrected measurement from left hamular notch mark to right hamular notch mark was preferred for selecting denture teeth width.

Enhancing access to reports of randomized trials published world-wide--the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library.

BACKGROUND: Randomized trials are essential in assessing the effects of healthcare interventions and are a key component in systematic reviews of effectiveness. Searching for reports of randomized trials in databases is problematic due to the absence of appropriate indexing terms until the 1990s and inconsistent application of these indexing terms thereafter. OBJECTIVES: The objectives of this study are to devise a search strategy for identifying reports of randomized trials in EMBASE which are not already indexed as trials in MEDLINE and to make these reports easily accessible by including them in the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library, with the permission of Elsevier, the publishers of EMBASE. METHODS: A highly sensitive search strategy was designed for EMBASE based on free-text and thesaurus terms which occurred frequently in the titles, abstracts, EMTREE terms (or some combination of these) of reports of trials indexed in EMBASE. This search strategy was run against EMBASE from 1980 to 2005 (1974 to 2005 for four of the terms) and records retrieved by the search, which were not already indexed as randomized trials in MEDLINE, were downloaded from EMBASE, printed and read. An analysis of the language of publication was conducted for the reports of trials published in 2005 (the most recent year completed at the time of this study). RESULTS: Twenty-two search terms were used (including nine which were later rejected due to poor cumulative precision). More than a third of a million records were downloaded and scanned and approximately 80,000 reports of trials were identified which were not already indexed as randomized trials in MEDLINE. These are now easily identifiable in CENTRAL, in The Cochrane Library. Cumulative sensitivity ranged from 0.1% to 60% and cumulative precision ranged from 8% to 61%. The truncated term 'random$' identified 60% of the total number of reports of trials but only 35% of the more than 130,000 records retrieved by this term were reports of trials. The language analysis for the sample year 2005 indicated that of the 18,427 reports indexed as randomized trials in MEDLINE, 959 (5%) were in languages other than English. The EMBASE search identified an additional 658 reports in languages other than English, of which the highest number were in Chinese (320). CONCLUSION: The results of the search to date have greatly increased access to reports of trials in EMBASE, especially in some languages other than English. The search strategy used was subjectively derived from a small 'gold standard' set of test records and was not validated in an independent test set. We intend to design an objectively-derived validated search strategy using logistic regression based on the frequency of occurrence of terms in the approximately 80,000 reports of randomized trials identified compared with the frequency of these terms across the entire EMBASE database.

So many filters, so little time: the development of a search filter appraisal checklist.

OBJECTIVES: The authors developed a tool to assess the quality of search filters designed to retrieve records for studies with specific research designs (e.g., diagnostic studies). METHODS: The UK InterTASC Information Specialists' Sub-Group (ISSG), a group of experienced health care information specialists, reviewed the literature to evaluate existing search filter appraisal tools and determined that existing tools were inadequate for their needs. The group held consensus meetings to develop a new filter appraisal tool consisting of a search filter appraisal checklist and a structured abstract. ISSG members tested the final checklist using three published search filters. RESULTS: The detailed ISSG Search Filter Appraisal Checklist captures relevance criteria and methods used to develop and test search filters. The checklist includes categorical and descriptive responses and is accompanied by a structured abstract that provides a summary of key quality features of a filter. DISCUSSION: The checklist is a comprehensive appraisal tool that can assist health sciences librarians and others in choosing search filters. The checklist reports filter design methods and search performance measures, such as sensitivity and precision. The checklist can also aid filter developers by indicating information on core methods that should be reported to help assess filter suitability. The generalizability of the checklist for non-methods filters remains to be explored.

Challenges of identifying unpublished data from clinical trials: Getting the best out of clinical trials registers and other novel sources.

Clinical trial data are essential for assessments of the effectiveness of health care interventions. Information about ongoing or completed, but not yet formally published, trials has been more difficult to identify until the development of clinical trials registers and portals. This paper summarises research evidence on identifying sources of trial data, how and when to search those sources, and which future developments may enhance access to and retrieval of unpublished trial evidence. We conducted a literature search for relevant studies and provide a narrative review of the evidence from these studies. Clinical trial data can be found in resources including clinical trials registers, regulatory agency sources, health technology assessment websites and manufacturers' websites, and submissions for regulatory approval. The challenges of searching these resources are described. Trials registers are relatively unsophisticated in terms of their search interfaces, and searchers need to adapt to each individual register. There is overlap across registers, but little research on the degree and nature of overlap and how best to search. Despite these challenges, trials registers and other resources can be rich sources of additional unique trial data, which may not be available from journal reports. New initiatives, such as OpenTrials, aim to consolidate and link all structured data and documentation related to clinical trials. No single resource gives access to all trials, and multiple registers should be searched as sensitively as possible. Searching is challenging and should be adequately resourced. Information specialists should monitor new developments which may reduce the challenges over the coming years.

When to include clinical study reports and regulatory documents in systematic reviews.

Reporting bias is a major threat to the validity and credibility of systematic reviews. This article outlines the rationale for accessing clinical study reports and other regulatory documents (regulatory data) as a means of addressing reporting bias and identifies factors that may help decide whether (or not) to include regulatory data in systematic reviews. The article also describes the origins and current state of regulatory data access and summarises a survey of current systematic reviewers' practices in considering regulatory data for inclusion in systematic reviews. How to access and extract regulatory data is not addressed. Organisations and other stakeholders such as Cochrane should encourage the use of data from clinical study reports as an important source of data in reviews of pharmaceutical interventions particularly when the intervention in question is of high importance and the risk of reporting bias is great.

The use of clinical study reports to enhance the quality of systematic reviews: a survey of systematic review authors.

BACKGROUND: Clinical study reports (CSRs) are produced for marketing authorisation applications. They often contain considerably more information about, and data from, clinical trials than corresponding journal publications. Use of data from CSRs might help circumvent reporting bias, but many researchers appear to be unaware of their existence or potential value. Our survey aimed to gain insight into the level of familiarity, understanding and use of CSRs, and to raise awareness of their potential within the systematic review community. We also aimed to explore the potential barriers faced when obtaining and using CSRs in systematic reviews. METHODS: Online survey of systematic reviewers who (i) had requested or used CSRs, (ii) had considered but not used CSRs and (iii) had not considered using CSRs was conducted. Cochrane reviewers were contacted twice via the Cochrane monthly digest. Non-Cochrane reviewers were reached via journal and other website postings. RESULTS: One hundred sixty respondents answered an open invitation and completed the questionnaire; 20/160 (13%) had previously requested or used CSRs and other regulatory documents, 7/160 (4%) had considered but not used CSRs and 133/160 (83%) had never considered this data source. Survey respondents mainly sought data from the European Medicines Agency (EMA) and/or the Food and Drug Administration (FDA). Motivation for using CSRs stemmed mainly from concerns about reporting bias 11/20 (55%), specifically outcome reporting bias 11/20 (55%) and publication bias 5/20 (25%). The barriers to using CSRs noted by all types of respondents included current limited access to these documents (43 respondents), the time and resources needed to obtain and include these data in evidence syntheses (n = 25) and lack of guidance about how to use these sources in systematic reviews (n = 26). CONCLUSIONS: Most respondents (irrespective of whether they had previously used them) agreed that access to CSRs is important, and suggest that further guidance on how to use and include these data would help to promote their use in future systematic reviews. Most respondents who received CSRs considered them to be valuable in their systematic review and/or meta-analysis.

Sleep disordered breathing in infants and children: a review of the literature.

The objective of this report was to review the etiology, diagnosis, and treatment of sleep disordered breathing (SDB) in children and infants. English peer-reviewed SDB literature identified by MEDLINE and a manual search conducted between 1999 and 2006 was selected. The keywords used for the search included: (1) children; (2) sleep disorder; (3) snoring; and (4) obstructive sleep apnea. A total of 153 manuscripts was identified. A delay in treatment of SDB children may be caused by several factors and may result in serious but generally reversible problems, including: (1) impaired growth; (2) neurocognitive and behavioral dysfunction; and (3) cardiorespiratory failure. Adenotonsillectomy is the treatment of choice, and continuous positive airway pressure may be an option for patients who are not candidates for surgery or who do not respond to surgery. Minimal information is available concerning the dental treatment of these disorders. With the devastating effects sleep disorders can have on children and their families, dentists must recognize obvious symptoms and refer these patients for management by physicians.

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.

BACKGROUND: Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this. OBJECTIVES: This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided. METHODS: Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. RESULTS: The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists' Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important. LIMITATIONS: The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs. CONCLUSIONS: Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results. FUNDING: The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council-NIHR Methodology Research Programme (grant number G0901496).

How to identify randomized controlled trials in MEDLINE: ten years on.

OBJECTIVE: The researchers sought to assess whether the widely used 1994 Cochrane Highly Sensitive Search Strategy (HSSS) for randomized controlled trials (RCTs) in MEDLINE could be improved in terms of sensitivity, precision, or parsimony. METHODS: A gold standard of 1,347 RCT records and a comparison group of 2,400 non-trials were randomly selected from MEDLINE. Terms occurring in at least 1% of RCT records were identified. Fifty percent of the RCT and comparison group records were randomly selected, and the ability of the terms to discriminate RCTs from non-trial records was determined using logistic regression. The best performing combinations of terms were tested on the remaining records and in MEDLINE. RESULTS: The best discriminating term was "Clinical Trial" (Publication Type). In years where the Cochrane assessment of MEDLINE records had taken place, the strategies identified few additional unindexed records of trials. In years where Cochrane assessment has yet to take place, "Randomized Controlled Trial" (Publication Type) proved highly sensitive and precise. Adding six more search terms identified further, unindexed trials at reasonable levels of precision and with sensitivity almost equal to the Cochrane HSSS. CONCLUSIONS: Most reports of RCTs in MEDLINE can now be identified easily using "Randomized Controlled Trial" (Publication Type). More sensitive searches can be achieved by a brief strategy, the Centre for Reviews and Dissemination/Cochrane Highly Sensitive Search Strategy (2005 revision).

PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement.

OBJECTIVE: To develop an evidence-based guideline for Peer Review of Electronic Search Strategies (PRESS) for systematic reviews (SRs), health technology assessments, and other evidence syntheses. STUDY DESIGN AND SETTING: An SR, Web-based survey of experts, and consensus development forum were undertaken to identify checklists that evaluated or validated electronic literature search strategies and to determine which of their elements related to search quality or errors. RESULTS: Systematic review: No new search elements were identified for addition to the existing (2008-2010) PRESS 2015 Evidence-Based Checklist, and there was no evidence refuting any of its elements. Results suggested that structured PRESS could identify search errors and improve the selection of search terms. Web-based survey of experts: Most respondents felt that peer review should be undertaken after the MEDLINE search had been prepared but before it had been translated to other databases. Consensus development forum: Of the seven original PRESS elements, six were retained: translation of the research question; Boolean and proximity operators; subject headings; text word search; spelling, syntax and line numbers; and limits and filters. The seventh (skilled translation of the search strategy to additional databases) was removed, as there was consensus that this should be left to the discretion of searchers. An updated PRESS 2015 Guideline Statement was developed, which includes the following four documents: PRESS 2015 Evidence-Based Checklist, PRESS 2015 Recommendations for Librarian Practice, PRESS 2015 Implementation Strategies, and PRESS 2015 Guideline Assessment Form. CONCLUSION: The PRESS 2015 Guideline Statement should help to guide and improve the peer review of electronic literature search strategies.

Number and size of randomized trials reported in general health care journals from 1948 to 1997.

BACKGROUND: Randomized trials are important for controlling selection biases, and where sufficient numbers of participants are involved, have the potential to yield reliable estimates of treatment effects. METHODS: We investigated trends in the number and size of randomized trials reported in general health care journals from 1948 to 1997. From the handsearching of 18 general health care journals we collected data on the number of reports of randomized trials in each journal per year, and the number of participants in each trial. RESULTS: A total of 5503 reports of trials were identified in 18 general health care journals. More than a third appeared in the British Medical Journal. The peak period for trial reports was the mid 1980s, with more in 1986 than any other year (242). By the mid 1990s the number per year had declined by a third. Trials with fewer than 100 participants accounted for most of the reports (69%). In spite of the overall decline in the number of trial reports, those involving 100 participants or more continued to increase throughout the period studied. CONCLUSIONS: The continued increase in the number of larger trials reported is encouraging, especially if it represents an increase in the size of trials more generally. Further research is needed to determine whether the trends over time identified here are reflective more of trends in the actual conduct of, rather than simply the reporting, of randomized trials.

Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials.

The Cochrane Collaboration has established a centralized database of controlled trials and other studies of health care interventions (called CENTRAL) that serves as the best available resource for all those preparing and maintaining systematic reviews or otherwise searching for trials. CENTRAL is available on The Cochrane Library. This article describes the history and methods of CENTRAL's development and the results of an analysis of the current composition of CENTRAL. As of September 2000, CENTRAL contained almost 300,000 citations to reports of trials, contributed mainly by Cochrane Groups and Centers around the world. Development of CENTRAL has been an ambitious, scholarly undertaking and has resulted in a valuable resource: CENTRAL includes citations to controlled trials that may not be indexed in MEDLINE, EMBASE, or other bibliographic databases; citations published internationally in many languages; and citations that are available only in conference proceedings or other hard-to-access sources.