The NICE OECD countries' geographic search filters: Part 1-methodology for developing the draft MEDLINE and Embase (Ovid) filters.

OBJECTIVE: There are no existing validated search filters for the group of 37 Organisation for Economic Co-operation and Development (OECD) countries. This study describes how information specialists from the United Kingdom's National Institute for Health and Care Excellence (NICE) developed and evaluated novel OECD countries' geographic search filters for MEDLINE and Embase (Ovid) to improve literature search effectiveness for evidence about OECD countries. METHODS: We created the draft filters using an alternative approach to standard filter construction. They are composed entirely of geographic subject headings and are designed to retain OECD country evidence by excluding non-OECD country evidence using the NOT Boolean operator. To evaluate the draft filters' effectiveness, we used MEDLINE and Embase literature searches for three NICE guidelines that retrieved >5,000 search results. A 10% sample of the excluded references was screened to check that OECD country evidence was not inadvertently excluded. RESULTS: The draft MEDLINE filter reduced results for each NICE guideline by 9.5% to 12.9%. In Embase, search results were reduced by 10.7% to 14%. Of the sample references, 7 of 910 (0.8%) were excluded inadvertently. These references were from a guideline about looked-after minors that concerns both OECD and non-OECD countries. CONCLUSION: The draft filters look promising-they reduced search result volumes while retaining most OECD country evidence from MEDLINE and Embase. However, we advise caution when using them in topics about both non-OECD and OECD countries. We have created final versions of the search filters and will validate them in a future study.

The NICE OECD countries' geographic search filters: Part 2-validation of the MEDLINE and Embase (Ovid) filters.

OBJECTIVE: We previously developed draft MEDLINE and Embase (Ovid) geographic search filters for Organisation for Economic Co-operation and Development (OECD) countries to assess their feasibility for finding evidence about the countries. Here, we describe the validation of these search filters. METHODS: We identified OECD country references from thirty National Institute for Health and Care Excellence (NICE) guidelines to generate gold standard sets for MEDLINE (n=2,065) and Embase (n=2,023). We validated the filters by calculating their recall against these sets. We then applied the filters to existing search strategies for three OECD-focused NICE guideline reviews (NG103 on flu vaccination, NG140 on abortion care, and NG146 on workplace health) to calculate the filters' impact on the number needed to read (NNR) of the searches. RESULTS: The filters both achieved 99.95% recall against the gold standard sets. Both filters achieved 100% recall for the three NICE guideline reviews. The MEDLINE filter reduced NNR from 256 to 232 for the NG103 review, from 38 to 27 for the NG140 review, and from 631 to 591 for the NG146 review. The Embase filter reduced NNR from 373 to 341 for the NG103 review, from 101 to 76 for the NG140 review, and from 989 to 925 for the NG146 review. CONCLUSION: The NICE OECD countries' search filters are the first validated filters for the countries. They can save time for research topics about OECD countries by finding the majority of evidence about OECD countries while reducing search result volumes in comparison to no filter use.

The NICE MEDLINE and Embase (Ovid) health apps search filters: development of validated filters to retrieve evidence about health apps.

OBJECTIVES: Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid). METHODS: A selection of medical informatics journals was hand searched to identify a "gold standard" (GS) set of references about apps. The GS set was divided into a development and validation set. The filters' search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters' overall recall, specificity, and precision were calculated using all the references identified from the hand search. RESULTS: Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision. CONCLUSIONS: The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

The Embase UK filter: validation of a geographic search filter to retrieve research about the UK from OVID Embase.

BACKGROUND: The authors developed a validated geographic search filter to retrieve research about the United Kingdom (UK) from OVID Embase. It was created to be used alongside their previously published OVID MEDLINE UK filter in systematic literature searches for context-sensitive topics. OBJECTIVES: To develop a validated geographic search filter to retrieve research about the UK from OVID Embase. METHODS: The Embase UK filter was translated from the MEDLINE UK filter. A gold standard set of references was generated using the relative recall method. The set contained references to publications about the UK that had informed National Institute for Health and Care Excellence (NICE) guidance and it was used to validate the filter. Recall, precision and number-needed-to-read (NNR) were calculated using a case study. RESULTS: The validated Embase UK filter demonstrated 99.8% recall against the references with UK identifiers in the gold standard set. In the case study, the Embase UK filter demonstrated 98.5% recall, 7.6% precision and a NNR of 13. CONCLUSION: The Embase UK filter can be used alongside the MEDLINE UK filter. The filters have the potential to save time and associated resource costs when they are used for context-sensitive topics that require research about UK settings.

The medline UK filter: development and validation of a geographic search filter to retrieve research about the UK from OVID medline.

BACKGROUND: A validated geographic search filter for the retrieval of research about the United Kingdom (UK) from bibliographic databases had not previously been published. OBJECTIVES: To develop and validate a geographic search filter to retrieve research about the UK from OVID medline with high recall and precision. METHODS: Three gold standard sets of references were generated using the relative recall method. The sets contained references to studies about the UK which had informed National Institute for Health and Care Excellence (NICE) guidance. The first and second sets were used to develop and refine the medline UK filter. The third set was used to validate the filter. Recall, precision and number-needed-to-read (NNR) were calculated using a case study. RESULTS: The validated medline UK filter demonstrated 87.6% relative recall against the third gold standard set. In the case study, the medline UK filter demonstrated 100% recall, 11.4% precision and a NNR of nine. CONCLUSION: A validated geographic search filter to retrieve research about the UK with high recall and precision has been developed. The medline UK filter can be applied to systematic literature searches in OVID medline for topics with a UK focus.

Validated geographic search filters for bibliographic databases: a scoping review protocol.

OBJECTIVE: The purpose of this scoping review is to identify validated geographic search filters and report their methodology and performance measures. INTRODUCTION: Data on specific geographic areas can be required for evidence syntheses topics, such as the investigation of regional inequalities in health care or to answer context-specific epidemiological questions. Search filters are useful tools for reviewers aiming to identify publications with common characteristics in bibliographic databases. Geographic search filters limit the literature search results to a specific geographic feature (eg, a country or region). INCLUSION CRITERIA: We will include reports on validated geographic search filters that aim to identify research evidence about a defined geographic area (eg, a country/region or a group of countries/regions). METHODS: This review will be conducted in accordance with JBI methodology for scoping reviews. The literature search will be conducted in PubMed and Embase. The InterTASC Information Specialists' Sub-Group Search Filter resource and Google Scholar will also be searched. Reports published in any language, from database inception to the present, will be considered for inclusion. Two researchers will independently screen the title, abstract, and full text of the search results. A third reviewer will be consulted in the event of any disagreements. The data extraction will include study characteristics, basic characteristics of the geographical search filter (eg, country/region), and the methods used to develop and validate the search filter. The extracted data will be summarized narratively and presented in a table. REVIEW REGISTRATION: Open Science Framework https://osf.io/5czhs.

Economic evaluations of artificial intelligence-based healthcare interventions: a systematic literature review of best practices in their conduct and reporting.

Objectives: Health economic evaluations (HEEs) help healthcare decision makers understand the value of new technologies. Artificial intelligence (AI) is increasingly being used in healthcare interventions. We sought to review the conduct and reporting of published HEEs for AI-based health interventions. Methods: We conducted a systematic literature review with a 15-month search window (April 2021 to June 2022) on 17th June 2022 to identify HEEs of AI health interventions and update a previous review. Records were identified from 3 databases (Medline, Embase, and Cochrane Central). Two reviewers screened papers against predefined study selection criteria. Data were extracted from included studies using prespecified data extraction tables. Included studies were quality assessed using the National Institute for Health and Care Excellence (NICE) checklist. Results were synthesized narratively. Results: A total of 21 studies were included. The most common type of AI intervention was automated image analysis (9/21, 43%) mainly used for screening or diagnosis in general medicine and oncology. Nearly all were cost-utility (10/21, 48%) or cost-effectiveness analyses (8/21, 38%) that took a healthcare system or payer perspective. Decision-analytic models were used in 16/21 (76%) studies, mostly Markov models and decision trees. Three (3/16, 19%) used a short-term decision tree followed by a longer-term Markov component. Thirteen studies (13/21, 62%) reported the AI intervention to be cost effective or dominant. Limitations tended to result from the input data, authorship conflicts of interest, and a lack of transparent reporting, especially regarding the AI nature of the intervention. Conclusion: Published HEEs of AI-based health interventions are rapidly increasing in number. Despite the potentially innovative nature of AI, most have used traditional methods like Markov models or decision trees. Most attempted to assess the impact on quality of life to present the cost per QALY gained. However, studies have not been comprehensively reported. Specific reporting standards for the economic evaluation of AI interventions would help improve transparency and promote their usefulness for decision making. This is fundamental for reimbursement decisions, which in turn will generate the necessary data to develop flexible models better suited to capturing the potentially dynamic nature of AI interventions.

The NICE Evidence Standards Framework for digital health and care technologies - Developing and maintaining an innovative evidence framework with global impact.

OBJECTIVE: In 2018, the UK National Institute for Health and Care Excellence (NICE), in partnership with Public Health England, NHS England, NHS Improvement and others, developed an evidence standards framework (ESF) for digital health and care technologies (DHTs). The ESF was designed to provide a standardised approach to guide developers and commissioners on the levels of evidence needed for the clinical and economic evaluation of DHTs by health and care systems. METHODS: The framework was developed using an agile and iterative methodology that included a literature review of existing initiatives and comparison of these against the requirements set by NHS England; iterative consultation with stakeholders through an expert working group and workshops; and questionnaire-based stakeholder input on a publicly available draft document. RESULTS: The evidence standards framework has been well-received and to date the ESF has been viewed online over 55,000 times and downloaded over 19,000 times. CONCLUSIONS: In April 2021 we published an update to the ESF. Here, we summarise the process through which the ESF was developed, reflect on its global impact to date, and describe NICE's ongoing work to maintain and improve the framework in the context for a fast moving, innovative field.

The NICE UK geographic search filters for MEDLINE and Embase (Ovid): Post-development study to further evaluate precision and number-needed-to-read when retrieving UK evidence.

BACKGROUND: The National Institute for Health and Care Excellence's (NICE) United Kingdom (UK) geographic search filters for MEDLINE and Embase (OVID) retrieve evidence in literature searches for UK-focused research topics with high recall. Their precision and number-needed-to-read (NNR) was examined previously in case studies using a single review. This paper details a larger post-development study that was conducted to test the NICE UK filters' precision and NNR more extensively. METHODS: The filters' recall of included UK references from 100 reviews was calculated. As reproducible search strategies were not available for every review, the MEDLINE filter's precision and NNR were calculated using strategies from 25 reviews. Strategies from nine reviews were used for the Embase filter. RESULTS: The MEDLINE filter achieved an average of 96.4% recall for the included UK references from the 100 reviews and the Embase filter achieved an average of 97.4% recall. Compared to not using a filter, the MEDLINE filter achieved an average of 98.9% recall for the 25 reviews. Precision was increased by an average of 7.8 times, reducing the NNR from 357 to 46. The Embase filter achieved an average of 97.1% recall for the nine reviews. Precision was increased by an average of 5.1 times, reducing the NNR from 746 to 146. CONCLUSION: There is more evidence to demonstrate that the NICE UK filters retrieve the majority of UK evidence from MEDLINE and Embase while increasing precision and reducing NNR. The filters can save time spent on selecting evidence for UK-focused research topics.

Introducing GRADE across the NICE clinical guideline program.

OBJECTIVES: Grading of Recommendations Assessment, Development and Evaluation (GRADE) is a system for rating the confidence in estimates of effect and grading guideline recommendations. It promotes evaluation of the quality of the evidence for each outcome and an assessment of balance between desirable and undesirable outcomes leading to a judgment about the strength of the recommendation. In 2007, the National Institute for Health and Clinical Excellence began introducing GRADE across its clinical guideline program to enable separation of judgments about the evidence quality from judgments about the strength of the recommendation. STUDY DESIGN AND SETTING: We describe the process of implementing GRADE across guidelines. RESULTS: Use of GRADE has been positively received by both technical staff and guideline development group members. CONCLUSION: A shift in thinking about confidence in the evidence was required leading to a more structured and transparent approach to decision making. Practical problems were also encountered; these have largely been resolved, but some areas require further work, including the application of imprecision and presenting results from analyses considering more than two alternative interventions. The use of GRADE for nonrandomized and diagnostic accuracy studies needs to be refined.