Increasing the efficiency of study selection for systematic reviews using prioritization tools and a single-screening approach.

BACKGROUND: Systematic literature screening is a key component in systematic reviews. However, this approach is resource intensive as generally two persons independently of each other (double screening) screen a vast number of search results. To develop approaches for increasing efficiency, we tested the use of text mining to prioritize search results as well as the involvement of only one person (single screening) in the study selection process. METHOD: Our study is based on health technology assessments (HTAs) of drug and non-drug interventions. Using a sample size calculation, we consecutively included 11 searches resulting in 33 study selection processes. Of the three screeners for each search, two used screening tools with prioritization (Rayyan, EPPI Reviewer) and one a tool without prioritization. For each prioritization tool, we investigated the proportion of citations classified as relevant at three cut-offs or STOP criteria (after screening 25%, 50% and 75% of the citation set). For each STOP criterion, we measured sensitivity (number of correctly identified relevant studies divided by the total number of relevant studies in the study pool). In addition, we determined the number of relevant studies identified per single screening round and investigated whether missed studies were relevant to the HTA conclusion. RESULTS: Overall, EPPI Reviewer performed better than Rayyan and identified the vast majority (88%, Rayyan 66%) of relevant citations after screening half of the citation set. As long as additional information sources were screened, it was sufficient to apply a single-screening approach to identify all studies relevant to the HTA conclusion. Although many relevant publications (n = 63) and studies (n = 29) were incorrectly excluded, ultimately only 5 studies could not be identified at all in 2 of the 11 searches (1x 1 study, 1x 4 studies). However, their omission did not change the overall conclusion in any HTA. CONCLUSIONS: EPPI Reviewer helped to identify relevant citations earlier in the screening process than Rayyan. Single screening would have been sufficient to identify all studies relevant to the HTA conclusion. However, this requires screening of further information sources. It also needs to be considered that the credibility of an HTA may be questioned if studies are missing, even if they are not relevant to the HTA conclusion.

Choosing health technology assessment and systematic review topics: the development of priority-setting criteria for patients' and consumers' interests.

BACKGROUND: The Institute for Quality and Efficiency in Health Care (IQWiG) was established in 2003 by the German parliament. Its legislative responsibilities are health technology assessment, mostly to support policy making and reimbursement decisions. It also has a mandate to serve patients' interests directly, by assessing and communicating evidence for the general public. OBJECTIVES: To develop a priority-setting framework based on the interests of patients and the general public. METHODS: A theoretical framework for priority setting from a patient/consumer perspective was developed. The process of development began with a poll to determine level of lay and health professional interest in the conclusions of 124 systematic reviews (194 responses). Data sources to identify patients' and consumers' information needs and interests were identified. RESULTS: IQWiG's theoretical framework encompasses criteria for quality of evidence and interest, as well as being explicit about editorial considerations, including potential for harm. Dimensions of "patient interest" were identified, such as patients' concerns, information seeking, and use. Rather than being a single item capable of measurement by one means, the concept of "patients' interests" requires consideration of data and opinions from various sources. CONCLUSIONS: The best evidence to communicate to patients/consumers is right, relevant and likely to be considered interesting and/or important to the people affected. What is likely to be interesting for the community generally is sufficient evidence for a concrete conclusion, in a common condition. More research is needed on characteristics of information that interest patients and consumers, methods of evaluating the effectiveness of priority setting, and methods to determine priorities for disinvestment.