Longitudinal clinical decision support for assessing decisions over time: State-of-the-art and future directions.

Objective: Patients and clinicians rarely experience healthcare decisions as snapshots in time, but clinical decision support (CDS) systems often represent decisions as snapshots. This scoping review systematically maps challenges and facilitators to longitudinal CDS that are applied at two or more timepoints for the same decision made by the same patient or clinician. Methods: We searched Embase, PubMed, and Medline databases for articles describing development, validation, or implementation of patient- or clinician-facing longitudinal CDS. Validated quality assessment tools were used for article selection. Challenges and facilitators to longitudinal CDS are reported according to PRISMA-ScR guidelines. Results: Eight articles met inclusion criteria; each article described a unique CDS. None used entirely automated data entry, none used living guidelines for updating the evidence base or knowledge engine as new evidence emerged during the longitudinal study, and one included formal readiness for change assessments. Seven of eight CDS were implemented and evaluated prospectively. Challenges were primarily related to suboptimal study design (with unique challenges for each study) or user interface. Facilitators included use of randomized trial designs for prospective enrollment, increased CDS uptake during longitudinal exposure, and machine-learning applications that are tailored to the CDS use case. Conclusions: Despite the intuitive advantages of representing healthcare decisions longitudinally, peer-reviewed literature on longitudinal CDS is sparse. Existing reports suggest opportunities to incorporate longitudinal CDS frameworks, automated data entry, living guidelines, and user readiness assessments. Generating best practice guidelines for longitudinal CDS would require a greater depth and breadth of published work and expert opinion.

Striking a match between FHIR-based patient data and FHIR-based eligibility criteria.

Inputs and Outputs: The Strike-a-Match Function, written in JavaScript version ES6+, accepts the input of two datasets (one dataset defining eligibility criteria for research studies or clinical decision support, and one dataset defining characteristics for an individual patient). It returns an output signaling whether the patient characteristics are a match for the eligibility criteria. Purpose: Ultimately, such a system will play a "matchmaker" role in facilitating point-of-care recognition of patient-specific clinical decision support. Specifications: The eligibility criteria are defined in HL7 FHIR (version R5) EvidenceVariable Resource JSON structure. The patient characteristics are provided in an FHIR Bundle Resource JSON including one Patient Resource and one or more Observation and Condition Resources which could be obtained from the patient's electronic health record. Application: The Strike-a-Match Function determines whether or not the patient is a match to the eligibility criteria and an Eligibility Criteria Matching Software Demonstration interface provides a human-readable display of matching results by criteria for the clinician or patient to consider. This is the first software application, serving as proof of principle, that compares patient characteristics and eligibility criteria with all data exchanged using HL7 FHIR JSON. An Eligibility Criteria Matching Software Library at https://fevir.net/110192 provides a method for sharing functions using the same information model.

Representation of evidence-based clinical practice guideline recommendations on FHIR.

BACKGROUND: Various formalisms have been developed to represent clinical practice guideline recommendations in a computer-interpretable way. However, none of the existing formalisms leverage the structured and computable information that emerge from the evidence-based guideline development process. Thus, we here propose a FHIR-based format that uses computer-interpretable representations of the knowledge artifacts that emerge during the process of evidence-based guideline development to directly serve as the basis of evidence-based recommendations. METHODS: We identified the information required to represent evidence-based clinical practice guideline recommendations and reviewed the knowledge artifacts emerging during the evidence-based guideline development process. We then conducted a consensus-based design process with domain experts to develop an information model for guideline recommendation representation that is structurally aligned to the evidence-based guideline recommendation development process and a corresponding representation based on FHIR resources developed for evidence-based medicine (EBMonFHIR). The resulting recommendations were modelled and represented in conformance with the FHIR Clinical Guidelines (CPG-on-FHIR) implementation guide. RESULTS: The information model of evidence-based clinical guideline recommendations and its EBMonFHIR-/CPG-on-FHIR-based representation contain the clinical contents of individual guideline recommendations, a set of metadata for the recommendations, the ratings for the recommendations (e.g., strength of recommendation, certainty of overall evidence), the ratings of certainty of evidence for individual outcomes (e.g., risk of bias) and links to the underlying evidence (systematic reviews based on primary studies). We created profiles and an implementation guide for all FHIR resources required to represent the knowledge artifacts generated during evidence-based guideline development and their re-use as the basis for recommendations and used the profiles to implement an exemplary clinical guideline recommendation. CONCLUSIONS: The FHIR implementation guide presented here can be used to directly link the evidence assessment process of evidence-based guideline recommendation development, i.e. systematic reviews and evidence grading, and the underlying evidence from primary studies to the resulting guideline recommendations. This not only allows the evidence on which recommendations are based on to be evaluated transparently and critically, but also enables guideline developers to leverage computable evidence in a more direct way to facilitate the generation of computer-interpretable guideline recommendations.

Reflections on defining a standard for computable expression of scientific knowledge: What teach us Yoda can.

Science advances at a slow pace but can be accelerated with a standard for computable expression of scientific knowledge, more precisely a technical standard for electronic data exchange of machine-interpretable data expressing scientific knowledge. Efforts to achieve this vision include Evidence-Based Medicine on Fast Healthcare Interoperability Resources (EBMonFHIR), COVID-19 Knowledge Accelerator (COKA), Computable Publishing LLC, Scientific Knowledge Accelerator Foundation, and the Fast Evidence Interoperability Resources (FEvIR) Platform. The vision for communicating scientific research results to be instantly found, viewed, sent, received, and incorporated into thousands of system is a Just-in-time Evidence Dissemination and Integration (JEDI) vision. Reflections on JEDI teachings in a fun way helps explain the focus on the "Resource" to make science computable, how a precise specification of evidence changes perspectives, the need to reset standard terminologies, the inclusive nature of efforts to achieve the vision, and the critical demand to realize the vision.

A Framework for the Development of Living Practice Guidelines in Health Care.

BACKGROUND: Living practice guidelines are increasingly being used to ensure that recommendations are responsive to rapidly emerging evidence. OBJECTIVE: To develop a framework that characterizes the processes of development of living practice guidelines in health care. DESIGN: First, 3 background reviews were conducted: a scoping review of methods papers, a review of handbooks of guideline-producing organizations, and an analytic review of selected living practice guidelines. Second, the core team drafted the first version of the framework. Finally, the core team refined the framework through an online survey and online discussions with a multidisciplinary international group of stakeholders. SETTING: International. PARTICIPANTS: Multidisciplinary group of 51 persons who have experience with guidelines. MEASUREMENTS: Not applicable. RESULTS: A major principle of the framework is that the unit of update in a living guideline is the individual recommendation. In addition to providing definitions, the framework addresses several processes. The planning process should address the organization's adoption of the living methodology as well as each specific guideline project. The production process consists of initiation, maintenance, and retirement phases. The reporting should cover the evidence surveillance time stamp, the outcome of reassessment of the body of evidence (when applicable), and the outcome of revisiting a recommendation (when applicable). The dissemination process may necessitate the use of different venues, including one for formal publication. LIMITATION: This study does not provide detailed or practical guidance for how the described concepts would be best implemented. CONCLUSION: The framework will help guideline developers in planning, producing, reporting, and disseminating living guideline projects. It will also help research methodologists study the processes of living guidelines. PRIMARY FUNDING SOURCE: None.

Clinical Quality Measures: A Challenge for-and to-Family Physicians.

INTRODUCTION: Improving design, selection and implementation of appropriate clinical quality measures can reduce harms and costs of health care and improve the quality and experience of care delivery. These measures have not been evaluated for appropriateness for use in performance measurement in a systematic, reproducible, and widely accepted manner. METHODS: We defined 10 criteria for evaluation of measure appropriateness in 4 domains: Patient-centeredness of outcomes, specification of population measured and measure detail, reliable evidence that benefits likely outweigh harms, and independence from significant confounders. We applied these criteria to 24 measures under consideration for statewide use in Massachusetts in public and private incentive-based programs. We appraised each measure as Appropriate or Not Appropriate for such use. RESULTS: We rated 15 measures as Appropriate (62.5%). Three measures (12.5%) were considered Appropriate only if applied at a system level but not for patient-provider assessment and 6 measures (25%) were rated Not Appropriate. Reasons for designation as "Not Appropriate" included benefits not clearly outweighing harms, lack of preservation of patient autonomy, inappropriate specification of population and measure detail, confounding by locus of control, and confounding by social determinants of health. CONCLUSIONS: Using this consensus-driven, 10-criteria methodology we were able to evaluate appropriateness of clinical quality measures. This methodology may improve measure design and inform selection of the most appropriate measures for use in quality measurement, financial incentives, and reporting.

Categorizing metadata to help mobilize computable biomedical knowledge.

INTRODUCTION: Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine-interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T. METHODS: We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject-predicate-object triples to more clearly differentiate metadata categories. RESULTS: We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK-to-CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories. CONCLUSION: A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

Processes for evidence summarization for patient decision aids: A Delphi consensus study.

BACKGROUND: Patient decision aids (PDAs) should provide evidence-based information so patients can make informed decisions. Yet, PDA developers do not have an agreed-upon process to select, synthesize and present evidence in PDAs. OBJECTIVE: To reach the consensus on an evidence summarization process for PDAs. DESIGN: A two-round modified Delphi survey. SETTING AND PARTICIPANTS: A group of international experts in PDA development invited developers, scientific networks, patient groups and listservs to complete Delphi surveys. DATA COLLECTION: We emailed participants the study description and a link to the online survey. Participants were asked to rate each potential criterion (omit, possible, desirable, essential) and provide qualitative feedback. ANALYSIS: Criteria in each round were retained if rated by >80% of participants as desirable or essential. If two or more participants suggested rewording, reordering or merging, the steering group considered the suggestion. RESULTS: Following two Delphi survey rounds, the evidence summarization process included defining the decision, reporting the processes and policies of the evidence summarization process, assembling the editorial team and managing (collect, manage, report) their conflicts of interest, conducting a systematic search, selecting and appraising the evidence, presenting the harms and benefits in plain language, and describing the method of seeking external review and the plan for updating the evidence (search, selection and appraisal of new evidence). CONCLUSION: A multidisciplinary stakeholder group reached consensus on an evidence summarization process to guide the creation of high-quality PDAs. PATIENT CONTRIBUTION: A patient partner was part of the steering group and involved in the development of the Delphi survey.

Making science computable: Developing code systems for statistics, study design, and risk of bias.

The COVID-19 crisis led a group of scientific and informatics experts to accelerate development of an infrastructure for electronic data exchange for the identification, processing, and reporting of scientific findings. The Fast Healthcare Interoperability Resources (FHIR®) standard which is overcoming the interoperability problems in health information exchange was extended to evidence-based medicine (EBM) knowledge with the EBMonFHIR project. A 13-step Code System Development Protocol was created in September 2020 to support global development of terminologies for exchange of scientific evidence. For Step 1, we assembled expert working groups with 55 people from 26 countries by October 2020. For Step 2, we identified 23 commonly used tools and systems for which the first version of code systems will be developed. For Step 3, a total of 368 non-redundant concepts were drafted to become display terms for four code systems (Statistic Type, Statistic Model, Study Design, Risk of Bias). Steps 4 through 13 will guide ongoing development and maintenance of these terminologies for scientific exchange. When completed, the code systems will facilitate identifying, processing, and reporting research results and the reliability of those results. More efficient and detailed scientific communication will reduce cost and burden and improve health outcomes, quality of life, and patient, caregiver, and healthcare professional satisfaction. We hope the achievements reached thus far will outlive COVID-19 and provide an infrastructure to make science computable for future generations. Anyone may join the effort at https://www.gps.health/covid19_knowledge_accelerator.html.

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper.

OBJECTIVES: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields. STUDY DESIGN AND SETTING: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group. RESULTS: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan. CONCLUSION: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

Consistency of Recommendations for Evaluation and Management of Hypertension.

Importance: Hypertension is very common, but guideline recommendations for hypertension have been controversial, are of increasing interest, and have profound implications. Objective: To systematically assess the consistency of recommendations regarding hypertension management across clinical practice guidelines (CPGs). Design, Setting, and Participants: This cross-sectional study of hypertension management recommendations included CPGs that had been published as of April 2018. Two point-of-care resources that provided graded recommendations were included for secondary analyses. Discrete and unambiguous specifications of the population, intervention, and comparison states were used to define a series of reference recommendations. Three raters reached consensus on coding the direction and strength of each recommendation made by each CPG. Three independent raters reached consensus on the importance of each reference recommendation. Main Outcomes and Measures: The main outcomes were rates of consistency for direction and strength among CPGs. Sensitivity analyses testing the robustness were conducted by excluding recommendation statements that were described as insufficient evidence, excluding single recommendation sources, and stratifying by importance of recommendations. Results: The analysis included 8 CPGs with a total of 71 reference recommendations, 68 of which had clear recommendations from 2 or more CPGs. Across CPGs, 22 recommendations (32%) were consistent in direction and strength, 18 recommendations (27%) were consistent in direction but inconsistent in strength, and 28 recommendations (41%) were inconsistent in direction. The rate of consistency was lower in secondary analyses. When insufficient evidence ratings were excluded, there was still substantial inconsistency, and a leave-one-out sensitivity analysis suggested the inconsistency could not be attributed to any single recommendation source. Inconsistency in direction was more common for recommendations deemed to be of lower importance (11 of 20 recommendations [55%]), but 17 of 48 high-importance recommendations (35%) had inconsistency in direction. Conclusions and Relevance: Hypertension is a common chronic condition with widespread expectations surrounding guideline-based care, yet CPGs have a high rate of inconsistency. Further investigations should determine the reasons for inconsistency, the implications for recommendation development, and the role of synthesis across recommendations for optimal guidance of clinical care.

Defining certainty of net benefit: a GRADE concept paper.

Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology is used to assess and report certainty of evidence and strength of recommendations. This GRADE concept article is not GRADE guidance but introduces certainty of net benefit, defined as the certainty that the balance between desirable and undesirable health effects is favourable. Determining certainty of net benefit requires considering certainty of effect estimates, the expected importance of outcomes and variability in importance, and the interaction of these concepts. Certainty of net harm is the certainty that the net effect is unfavourable. Guideline panels using or testing this approach might limit strong recommendations to actions with a high certainty of net benefit or against actions with a moderate or high certainty of net harm. Recommendations may differ in direction or strength from that suggested by the certainty of net benefit or harm when influenced by cost, equity, acceptability or feasibility.

Modified Delphi survey for the evidence summarisation of patient decision aids: Study protocol.

INTRODUCTION: Information included in a patient decision aid (PDA) can significantly influence patients' decisions and is, therefore, expected to be evidence-based and rigorously selected and summarised. PDA developers have not yet agreed on a standardised process for the selection and summarisation of the supporting evidence. We intend to generate consensus on a process (and related steps and criteria) for selecting and summarising evidence for PDAs using a modified Delphi survey. METHODS AND ANALYSIS: We will develop an evidence summarisation process specific to PDA development by using a consensus-based Delphi approach, surveying international experts and stakeholders with two to three rounds. To increase generalisability and acceptability, we will distribute the survey to the following stakeholder groups: PDA developers, researchers with expertise in shared decision making, PDA development and evidence summarisation, members of the International Patient Decision Aids Standards (IPDAS) collaboration, policy makers with expertise in PDA certification and patient stakeholder groups. For each criterion, if at least 80% of survey participants rank the criterion as most important/least important, we will consider that consensus has been achieved. ETHICS AND DISSEMINATION: It is critical for PDAs to have accurate and trustworthy evidence-based information about the risks and benefits of health treatments and tests, as these decision aids help patients make important choices. We want to generate consensus on an approach for selecting and summarising the evidence included in PDAs, which can be widely implemented by PDA developers. Dartmouth College's Committee for the Protection of Human Subjects approved this protocol. We will publish our results in a peer reviewed journal.

The GRADE Working Group clarifies the construct of certainty of evidence.

OBJECTIVE: To clarify the grading of recommendations assessment, development and evaluation (GRADE) definition of certainty of evidence and suggest possible approaches to rating certainty of the evidence for systematic reviews, health technology assessments, and guidelines. STUDY DESIGN AND SETTING: This work was carried out by a project group within the GRADE Working Group, through brainstorming and iterative refinement of ideas, using input from workshops, presentations, and discussions at GRADE Working Group meetings to produce this document, which constitutes official GRADE guidance. RESULTS: Certainty of evidence is best considered as the certainty that a true effect lies on one side of a specified threshold or within a chosen range. We define possible approaches for choosing threshold or range. For guidelines, what we call a fully contextualized approach requires simultaneously considering all critical outcomes and their relative value. Less-contextualized approaches, more appropriate for systematic reviews and health technology assessments, include using specified ranges of magnitude of effect, for example, ranges of what we might consider no effect, trivial, small, moderate, or large effects. CONCLUSION: It is desirable for systematic review authors, guideline panelists, and health technology assessors to specify the threshold or ranges they are using when rating the certainty in evidence.