Enhanced family history-based algorithms increase the identification of individuals meeting criteria for genetic testing of hereditary cancer syndromes but would not reduce disparities on their own.

OBJECTIVE: This study aimed to 1) investigate algorithm enhancements for identifying patients eligible for genetic testing of hereditary cancer syndromes using family history data from electronic health records (EHRs); and 2) assess their impact on relative differences across sex, race, ethnicity, and language preference. MATERIALS AND METHODS: The study used EHR data from a tertiary academic medical center. A baseline rule-base algorithm, relying on structured family history data (structured data; SD), was enhanced using a natural language processing (NLP) component and a relaxed criteria algorithm (partial match [PM]). The identification rates and differences were analyzed considering sex, race, ethnicity, and language preference. RESULTS: Among 120,007 patients aged 25-60, detection rate differences were found across all groups using the SD (all P < 0.001). Both enhancements increased identification rates; NLP led to a 1.9 % increase and the relaxed criteria algorithm (PM) led to an 18.5 % increase (both P < 0.001). Combining SD with NLP and PM yielded a 20.4 % increase (P < 0.001). Similar increases were observed within subgroups. Relative differences persisted across most categories for the enhanced algorithms, with disproportionately higher identification of patients who are White, Female, non-Hispanic, and whose preferred language is English. CONCLUSION: Algorithm enhancements increased identification rates for patients eligible for genetic testing of hereditary cancer syndromes, regardless of sex, race, ethnicity, and language preference. However, differences in identification rates persisted, emphasizing the need for additional strategies to reduce disparities such as addressing underlying biases in EHR family health information and selectively applying algorithm enhancements for disadvantaged populations. Systematic assessment of differences in algorithm performance across population subgroups should be incorporated into algorithm development processes.

Artificial Intelligence and Cancer Control: Toward Prioritizing Justice, Equity, Diversity, and Inclusion (JEDI) in Emerging Decision Support Technologies.

PURPOSE FOR REVIEW: This perspective piece has two goals: first, to describe issues related to artificial intelligence-based applications for cancer control as they may impact health inequities or disparities; and second, to report on a review of systematic reviews and meta-analyses of artificial intelligence-based tools for cancer control to ascertain the extent to which discussions of justice, equity, diversity, inclusion, or health disparities manifest in syntheses of the field's best evidence. RECENT FINDINGS: We found that, while a significant proportion of existing syntheses of research on AI-based tools in cancer control use formal bias assessment tools, the fairness or equitability of models is not yet systematically analyzable across studies. Issues related to real-world use of AI-based tools for cancer control, such as workflow considerations, measures of usability and acceptance, or tool architecture, are more visible in the literature, but still addressed only in a minority of reviews. Artificial intelligence is poised to bring significant benefits to a wide range of applications in cancer control, but more thorough and standardized evaluations and reporting of model fairness are required to build the evidence base for AI-based tool design for cancer and to ensure that these emerging technologies promote equitable healthcare.

A method for structuring complex clinical knowledge and its representational formalisms to support composite knowledge interoperability in healthcare.

INTRODUCTION: The use and interoperability of clinical knowledge starts with the quality of the formalism utilized to express medical expertise. However, a crucial challenge is that existing formalisms are often suboptimal, lacking the fidelity to represent complex knowledge thoroughly and concisely. Often this leads to difficulties when seeking to unambiguously capture, share, and implement the knowledge for care improvement in clinical information systems used by providers and patients. OBJECTIVES: To provide a systematic method to address some of the complexities of knowledge composition and interoperability related to standards-based representational formalisms of medical knowledge. METHODS: Several cross-industry (Healthcare, Linguistics, System Engineering, Standards Development, and Knowledge Engineering) frameworks were synthesized into a proposed reference knowledge framework. The framework utilizes IEEE 42010, the MetaObject Facility, the Semantic Triangle, an Ontology Framework, and the Domain and Comprehensibility Appropriateness criteria. The steps taken were: 1) identify foundational cross-industry frameworks, 2) select architecture description method, 3) define life cycle viewpoints, 4) define representation and knowledge viewpoints, 5) define relationships between neighboring viewpoints, and 6) establish characteristic definitions of the relationships between components. System engineering principles applied included separation of concerns, cohesion, and loose coupling. RESULTS: A "Multilayer Metamodel for Representation and Knowledge" (M*R/K) reference framework was defined. It provides a standard vocabulary for organizing and articulating medical knowledge curation perspectives, concepts, and relationships across the artifacts created during the life cycle of language creation, authoring medical knowledge, and knowledge implementation in clinical information systems such as electronic health records (EHR). CONCLUSION: M*R/K provides a systematic means to address some of the complexities of knowledge composition and interoperability related to medical knowledge representations used in diverse standards. The framework may be used to guide the development, assessment, and coordinated use of knowledge representation formalisms. M*R/K could promote the alignment and aggregated use of distinct domain-specific languages in composite knowledge artifacts such as clinical practice guidelines (CPGs).

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

A qualitative study of prevalent laboratory information systems and data communication patterns for genetic test reporting.

PURPOSE: The availability of genetic test data within the electronic health record (EHR) is a pillar of the US vision for an interoperable health IT infrastructure and a learning health system. Although EHRs have been highly investigated, evaluation of the information systems used by the genetic labs has received less attention-but is necessary for achieving optimal interoperability. This study aimed to characterize how US genetic testing labs handle their information processing tasks. METHODS: We followed a qualitative research method that included interviewing lab representatives and a panel discussion to characterize the information flow models. RESULTS: Ten labs participated in the study. We identified three generic lab system models and their relevant characteristics: a backbone system with additional specialized systems for interpreting genetic results, a brokering system that handles housekeeping and communication, and a single primary system for results interpretation and report generation. CONCLUSION: Labs have heterogeneous workflows and generally have a low adoption of standards when sending genetic test reports back to EHRs. Core interpretations are often delivered as free text, limiting their computational availability for clinical decision support tools. Increased provision of genetic test data in discrete and standard-based formats by labs will benefit individual and public health.

A qualitative investigation of biomedical informatics interoperability standards for genetic test reporting: benefits, challenges, and motivations from the testing laboratory's perspective.

PURPOSE: Genetic laboratory test reports can often be of limited computational utility to the receiving clinical information systems, such as clinical decision support systems. Many health-care interoperability (HC) standards aim to tackle this problem, but the perceived benefits, challenges, and motivations for implementing HC interoperability standards from the labs' perspective has not been systematically assessed. METHODS: We surveyed genetic testing labs across the United States and conducted a semistructured interview with responding lab representatives. We conducted a thematic analysis of the interview transcripts to identify relevant themes. A panel of experts discussed and validated the identified themes. RESULTS: Nine labs participated in the interview, and 24 relevant themes were identified within five domains. These themes included the challenge of complex and changing genetic knowledge, the motivation of competitive advantage, provided financial incentives, and the benefit of supporting the learning health system. CONCLUSION: Our study identified the labs' perspective on various aspects of implementing HC interoperability standards in producing and communicating genetic test reports. Interviewees frequently reported that increased adoption of HC standards may be motivated by competition and programs incentivizing and regulating the incorporation of interoperability standards for genetic test data, which could benefit quality control, research, and other areas.

Predictive analytics for step-up therapy: Supervised or semi-supervised learning?

BACKGROUND: Step-up therapy is a patient management approach that aims to balance the efficacy, costs and risks posed by different lines of medications. While the initiation of first line medications is a straightforward decision, stepping-up a patient to the next treatment line is often more challenging and difficult to predict. By identifying patients who are likely to move to the next line of therapy, prediction models could be used to help healthcare organizations with resource planning and chronic disease management. OBJECTIVE: To compared supervised learning versus semi-supervised learning to predict which rheumatoid arthritis patients will move from the first line of therapy (i.e., conventional synthetic disease-modifying antirheumatic drugs) to the next line of therapy (i.e., disease-modifying antirheumatic drugs or targeted synthetic disease-modifying antirheumatic drugs) within one year. MATERIALS AND METHODS: Five groups of features were extracted from an administrative claims database: demographics, medications, diagnoses, provider characteristics, and procedures. Then, a variety of supervised and semi-supervised learning methods were implemented to identify the most optimal method of each approach and assess the contribution of each feature group. Finally, error analysis was conducted to understand the behavior of misclassified patients. RESULTS: XGBoost yielded the highest F-measure (42%) among the supervised approaches and one-class support vector machine achieved the highest F-measure (65%) among the semi-supervised approaches. The semi-supervised approach had significantly higher F-measure (65% vs. 42%; p < 0.01), precision (51% vs. 33%; p < 0.01), and recall (89% vs. 59%; p < 0.01) than the supervised approach. Excluding demographic, drug, diagnosis, provider, and procedure features reduced theF-measure from 65% to 61%, 57%, 54%, 51% and 49% respectively (p < 0.01). The error analysis showed that a substantial portion of false positive patients will change their line of therapy shortly after the prediction period. CONCLUSION: This study showed that supervised learning approaches are not an optimal option for a difficult clinical decision regarding step-up therapy. More specifically, negative class labels in step-up therapy data are not a robust ground truth, because the costs and risks associated with higher line of therapy impact objective decision making of patients and providers. The proposed semi-supervised learning approach can be applied to other step-up therapy applications.

CASIDE: A data model for interoperable cancer survivorship information based on FHIR.

Cancer survivorship has traditionally received little research attention although it is associated with a variety of long-term consequences and also many other comorbidities. There is an urgent need to increase research on this area, and the secondary use of healthcare data has the potential to provide valuable insights on survivors' health trajectories. However, cancer survivors' data is often stored in silos and collected inconsistently. In this study we present CASIDE, an interoperable data model for cancer survivorship information that aims to accelerate the secondary use of healthcare data and data sharing across institutions. It is designed to provide a holistic view of the cancer survivor, taking into account not just the clinical data but also the patient's own perspective, and is built upon the emerging Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR) standard. Advantages of adopting FHIR and challenges in information modelling using this standard are discussed. CASIDE is a generalizable approach that is already being used as a support tool for the development of downstream applications to support clinical decision making and can contribute to translational collaborative research on cancer survivorship.

Developing a sampling method and preliminary taxonomy for classifying COVID-19 public health guidance for healthcare organizations and the general public.

BACKGROUND: Development and dissemination of public health (PH) guidance to healthcare organizations and the general public (e.g., businesses, schools, individuals) during emergencies like the COVID-19 pandemic is vital for policy, clinical, and public decision-making. Yet, the rapidly evolving nature of these events poses significant challenges for guidance development and dissemination strategies predicated on well-understood concepts and clearly defined access and distribution pathways. Taxonomies are an important but underutilized tool for guidance authoring, dissemination and updating in such dynamic scenarios. OBJECTIVE: To design a rapid, semi-automated method for sampling and developing a PH guidance taxonomy using widely available Web crawling tools and streamlined manual content analysis. METHODS: Iterative samples of guidance documents were taken from four state PH agency websites, the US Center for Disease Control and Prevention, and the World Health Organization. Documents were used to derive and refine a preliminary taxonomy of COVID-19 PH guidance via content analysis. RESULTS: Eight iterations of guidance document sampling and taxonomy revisions were performed, with a final corpus of 226 documents. The preliminary taxonomy contains 110 branches distributed between three major domains: stakeholders (24 branches), settings (25 branches) and topics (61 branches). Thematic saturation measures indicated rapid saturation (≤5% change) for the domains of "stakeholders" and "settings", and "topic"-related branches for clinical decision-making. Branches related to business reopening and economic consequences remained dynamic throughout sampling iterations. CONCLUSION: The PH guidance taxonomy can support public health agencies by aligning guidance development with curation and indexing strategies; supporting targeted dissemination; increasing the speed of updates; and enhancing public-facing guidance repositories and information retrieval tools. Taxonomies are essential to support knowledge management activities during rapidly evolving scenarios such as disease outbreaks and natural disasters.

Comparing models of delivery for cancer genetics services among patients receiving primary care who meet criteria for genetic evaluation in two healthcare systems: BRIDGE randomized controlled trial.

BACKGROUND: Advances in genetics and sequencing technologies are enabling the identification of more individuals with inherited cancer susceptibility who could benefit from tailored screening and prevention recommendations. While cancer family history information is used in primary care settings to identify unaffected patients who could benefit from a cancer genetics evaluation, this information is underutilized. System-level population health management strategies are needed to assist health care systems in identifying patients who may benefit from genetic services. In addition, because of the limited number of trained genetics specialists and increasing patient volume, the development of innovative and sustainable approaches to delivering cancer genetic services is essential. METHODS: We are conducting a randomized controlled trial, entitled Broadening the Reach, Impact, and Delivery of Genetic Services (BRIDGE), to address these needs. The trial is comparing uptake of genetic counseling, uptake of genetic testing, and patient adherence to management recommendations for automated, patient-directed versus enhanced standard of care cancer genetics services delivery models. An algorithm-based system that utilizes structured cancer family history data available in the electronic health record (EHR) is used to identify unaffected patients who receive primary care at the study sites and meet current guidelines for cancer genetic testing. We are enrolling eligible patients at two healthcare systems (University of Utah Health and New York University Langone Health) through outreach to a randomly selected sample of 2780 eligible patients in the two sites, with 1:1 randomization to the genetic services delivery arms within sites. Study outcomes are assessed through genetics clinic records, EHR, and two follow-up questionnaires at 4 weeks and 12 months after last genetic counseling contactpre-test genetic counseling. DISCUSSION: BRIDGE is being conducted in two healthcare systems with different clinical structures and patient populations. Innovative aspects of the trial include a randomized comparison of a chatbot-based genetic services delivery model to standard of care, as well as identification of at-risk individuals through a sustainable EHR-based system. The findings from the BRIDGE trial will advance the state of the science in identification of unaffected patients with inherited cancer susceptibility and delivery of genetic services to those patients. TRIAL REGISTRATION: BRIDGE is registered as NCT03985852 . The trial was registered on June 6, 2019 at clinicaltrials.gov .

Context and Approach in Reporting Evaluations of Electronic Health Record-Based Implementation Projects.

Electronic health records (EHRs) are ubiquitous yet still evolving, resulting in a moving target for determining the effects of context (features of the work environment, such as organization, payment systems, user training, and roles) on EHR implementation projects. Electronic health records have become instrumental in effecting quality improvement innovations and providing data to evaluate them. However, reports of studies typically fail to provide adequate descriptions of contextual details to permit readers to apply the findings. As for any evaluation, the quality of reporting is essential to learning from, and disseminating, the results. Extensive guidelines exist for reporting of virtually all types of applied health research, but they are not tailored to capture some contextual factors that may affect the outcomes of EHR implementations, such as attitudes toward implementation, format and amount of training, post go-live support, amount of local customization, and time diverted from direct interaction with patients to computers. Nevertheless, evaluators of EHR-based innovations can choose reporting guidelines that match the general purpose of their evaluation and the stage of their investigation (planning, protocol, execution, and analysis) and should report relevant contextual details (including, if pertinent, any pressures to help justify the huge investments and many years required for some implementations). Reporting guidelines are based on the scientific principles and practices that underlie sound research and should be consulted from the earliest stages of planning evaluations and onward, serving as guides for how evaluations should be conducted as well as reported.

Patient Interactions With an Automated Conversational Agent Delivering Pretest Genetics Education: Descriptive Study.

BACKGROUND: Cancer genetic testing to assess an individual's cancer risk and to enable genomics-informed cancer treatment has grown exponentially in the past decade. Because of this continued growth and a shortage of health care workers, there is a need for automated strategies that provide high-quality genetics services to patients to reduce the clinical demand for genetics providers. Conversational agents have shown promise in managing mental health, pain, and other chronic conditions and are increasingly being used in cancer genetic services. However, research on how patients interact with these agents to satisfy their information needs is limited. OBJECTIVE: Our primary aim is to assess user interactions with a conversational agent for pretest genetics education. METHODS: We conducted a feasibility study of user interactions with a conversational agent who delivers pretest genetics education to primary care patients without cancer who are eligible for cancer genetic evaluation. The conversational agent provided scripted content similar to that delivered in a pretest genetic counseling visit for cancer genetic testing. Outside of a core set of information delivered to all patients, users were able to navigate within the chat to request additional content in their areas of interest. An artificial intelligence-based preprogrammed library was also established to allow users to ask open-ended questions to the conversational agent. Transcripts of the interactions were recorded. Here, we describe the information selected, time spent to complete the chat, and use of the open-ended question feature. Descriptive statistics were used for quantitative measures, and thematic analyses were used for qualitative responses. RESULTS: We invited 103 patients to participate, of which 88.3% (91/103) were offered access to the conversational agent, 39% (36/91) started the chat, and 32% (30/91) completed the chat. Most users who completed the chat indicated that they wanted to continue with genetic testing (21/30, 70%), few were unsure (9/30, 30%), and no patient declined to move forward with testing. Those who decided to test spent an average of 10 (SD 2.57) minutes on the chat, selected an average of 1.87 (SD 1.2) additional pieces of information, and generally did not ask open-ended questions. Those who were unsure spent 4 more minutes on average (mean 14.1, SD 7.41; P=.03) on the chat, selected an average of 3.67 (SD 2.9) additional pieces of information, and asked at least one open-ended question. CONCLUSIONS: The pretest chat provided enough information for most patients to decide on cancer genetic testing, as indicated by the small number of open-ended questions. A subset of participants were still unsure about receiving genetic testing and may require additional education or interpersonal support before making a testing decision. Conversational agents have the potential to become a scalable alternative for pretest genetics education, reducing the clinical demand on genetics providers.

A systematic review of theoretical constructs in CDS literature.

BACKGROUND: Studies that examine the adoption of clinical decision support (CDS) by healthcare providers have generally lacked a theoretical underpinning. The Unified Theory of Acceptance and Use of Technology (UTAUT) model may provide such a theory-based explanation; however, it is unknown if the model can be applied to the CDS literature. OBJECTIVE: Our overall goal was to develop a taxonomy based on UTAUT constructs that could reliably characterize CDS interventions. METHODS: We used a two-step process: (1) identified randomized controlled trials meeting comparative effectiveness criteria, e.g., evaluating the impact of CDS interventions with and without specific features or implementation strategies; (2) iteratively developed and validated a taxonomy for characterizing differential CDS features or implementation strategies using three raters. RESULTS: Twenty-five studies with 48 comparison arms were identified. We applied three constructs from the UTAUT model and added motivational control to characterize CDS interventions. Inter-rater reliability was as follows for model constructs: performance expectancy (κ = 0.79), effort expectancy (κ = 0.85), social influence (κ = 0.71), and motivational control (κ = 0.87). CONCLUSION: We found that constructs from the UTAUT model and motivational control can reliably characterize features and associated implementation strategies. Our next step is to examine the quantitative relationships between constructs and CDS adoption.

Iterative heuristic design of temporal graphic displays with clinical domain experts.

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Discordance in Information Exchange Between Providers During Care Transitions for Surgical Patients.

BACKGROUND: The exchange of health information between primary care providers (PCPs) and surgeons is critical during transitions of care for older patients with multiple comorbidities; however, it is unknown to what extent this process occurs. This study was designed to characterize the extent to which factors associated with older patient's recovery, such as functional status, cognitive status, social status, and emotional factors, are shared among PCPs and surgical providers during care transitions. MATERIALS AND METHODS: We prospectively identified 15 patients aged over 60 y with ≥3 comorbidities referred for general and vascular surgery procedures at a Veterans Administrative and academic medical center. Semistructured Critical Decision Method interviews were conducted with patients along with their surgical providers and referring PCPs. Thematic content analysis was performed independently by five reviewers on the cognitive processes associated with functional status, cognitive status, social status, and emotional factors. Interrater reliability between providers and patients was assessed using Cohen's kappa. RESULTS: Forty-seven Critical Decision Method interviews were conducted, which included 20 paired interviews between a PCP and a surgeon and 16 paired interviews that involved a patient and a provider. The majority of patients reported experiencing poor information exchange between their PCP and surgeon (58%) and feeling they were primarily responsible for communicating their own health information during care transitions (67%). In paired interviews between PCPs and surgeons, there was nearly perfect agreement for the shared knowledge of cognitive (kappa: 0.83) and emotional (kappa 1) factors. In contrast, there was only minimal agreement for shared knowledge of functional status (kappa 0.38) and social status (kappa: 0.34). CONCLUSIONS: Information exchange between PCPs and surgical providers is often discordant during transitions of surgical care for medically complex older patients, particularly when it pertains to communicating their functional or social status.

Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features.

OBJECTIVES: Finding recent clinical studies that warrant changes in clinical practice ("high impact" clinical studies) in a timely manner is very challenging. We investigated a machine learning approach to find recent studies with high clinical impact to support clinical decision making and literature surveillance. METHODS: To identify recent studies, we developed our classification model using time-agnostic features that are available as soon as an article is indexed in PubMed®, such as journal impact factor, author count, and study sample size. Using a gold standard of 541 high impact treatment studies referenced in 11 disease management guidelines, we tested the following null hypotheses: (1) the high impact classifier with time-agnostic features (HI-TA) performs equivalently to PubMed's Best Match sort and a MeSH-based Naïve Bayes classifier; and (2) HI-TA performs equivalently to the high impact classifier with both time-agnostic and time-sensitive features (HI-TS) enabled in a previous study. The primary outcome for both hypotheses was mean top 20 precision. RESULTS: The differences in mean top 20 precision between HI-TA and three baselines (PubMed's Best Match, a MeSH-based Naïve Bayes classifier, and HI-TS) were not statistically significant (12% vs. 3%, p = 0.101; 12% vs. 11%, p = 0.720; 12% vs. 25%, p = 0.094, respectively). Recall of HI-TA was low (7%). CONCLUSION: HI-TA had equivalent performance to state-of-the-art approaches that depend on time-sensitive features. With the advantage of relying only on time-agnostic features, the proposed approach can be used as an adjunct to help clinicians identify recent high impact clinical studies to support clinical decision-making. However, low recall limits the use of HI-TA for literature surveillance.

Critical care information display approaches and design frameworks: A systematic review and meta-analysis.

OBJECTIVE: To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians. METHODS: We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author. RESULTS: Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible. CONCLUSION: Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.

Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era.

The US health system has recently achieved widespread adoption of electronic health record (EHR) systems, primarily driven by financial incentives provided by the Meaningful Use (MU) program. Although successful in promoting EHR adoption and use, the program, and other contributing factors, also produced important unintended consequences (UCs) with far-reaching implications for the US health system. Based on our own experiences from large health information technology (HIT) adoption projects and a collection of key studies in HIT evaluation, we discuss the most prominent UCs of MU: failed expectations, EHR market saturation, innovation vacuum, physician burnout, and data obfuscation. We identify challenges resulting from these UCs and provide recommendations for future research to empower the broader medical and informatics communities to realize the full potential of a now digitized health system. We believe that fixing these unanticipated effects will demand efforts from diverse players such as health care providers, administrators, HIT vendors, policy makers, informatics researchers, funding agencies, and outside developers; promotion of new business models; collaboration between academic medical centers and informatics research departments; and improved methods for evaluations of HIT.

Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis.

BACKGROUND: Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use. OBJECTIVE: The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes. METHODS: We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies. RESULTS: Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I2>50%) across studies. When compared with other resources-7 studies, not amenable to meta-analytic pooling-the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource. CONCLUSIONS: Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.

Data standards for interoperability of care team information to support care coordination of complex pediatric patients.

OBJECTIVE: Seamless access to information about the individuals and organizations involved in the care of a specific patient ("care teams") is crucial to effective and efficient care coordination. This is especially true for vulnerable and complex patient populations such as pediatric patients with special needs. Despite wide adoption of electronic health records (EHR), current EHR systems do not adequately support the visualization and management of care teams within and across health care organizations. Electronic health information exchange has the potential to address this issue. In the present study, we assessed the adequacy of available health information exchange data standards to support the information needs related to care coordination of complex pediatric patients. METHODS: We derived data elements from the information needs of clinicians and parents to support patient care teams; and mapped them to data elements in the Health Level Seven (HL7) Consolidated Clinical Document Architecture (C-CDA) standard and in the HL7 Fast Healthcare Interoperability Resources (FHIR) standard. We also identified additional C-CDA data elements and FHIR resources that include patients' care team members. RESULTS: Information about care team members involved in patient care is generally well-represented in the C-CDA and FHIR specifications. However, there are gaps related to patients' non-clinical events and care team actions. In addition, there is no single place to find information about care team members; rather, information about practitioners and organizations may be available in several different types of C-CDA data elements and FHIR resources. CONCLUSION: Through standards-based electronic health information exchange, it appears to be feasible to build patient care team representations irrespective of the location of patient care. In order to gather care team information across disparate systems, exchange of multiple C-CDA documents and/or execution of multiple FHIR queries will be necessary. This approach has the potential to enable comprehensive patient care team views that may help improve care coordination.