Pediatric Septic Shock Care Pathways in General Emergency Departments: A Qualitative Study Targeting How to Really Make it Work.

OBJECTIVES: Many academic pediatric emergency departments (PEDs) have successfully implemented pediatric septic shock care pathways. However, many general emergency departments (GEDs), who see the majority of pediatric ED visits, have not. This study aims to compare the workflow, resources, communication, and decision making across these 2 settings to inform the future implementation of a standardized care pathway for children with septic shock in the GED. METHODS: We used the critical incident technique to conduct semistructured interviews with 24 ED physicians, nurses, and technicians at one PED and 2 GEDs regarding pediatric septic shock care. We performed a thematic analysis using the Framework Method to develop our coding schema through inductive and deductive analyses. We continued an iterative process of revising the schema until we reached consensus agreement and thematic saturation. RESULTS: We identified the following 6 themes: (1) functioning like a "well-oiled machine" may be key to high performance; (2) experiencing the sequence of care for children with sepsis as invariant and predictable may be essential to high-quality performance; (3) resilience and flexibility are characteristic of high levels of performance; (4) believing that "the buck stops here" may contribute to more accountability; (5) continuous system learning is essential; and (6) computerized clinical decision support may not be optimized to drive decision-making at the point of care. Commentary from GED and PED participants differed across the 6 themes, providing insight into the approach for standardized care pathway implementation in GEDs. CONCLUSIONS: Pediatric septic shock workflow, decision making, and system performance differ between the PED and GEDs. Implementation of a standardized care pathway in GEDs will require a tailored approach. Specific recommendations include (1) improving shared situation awareness; (2) simulation for knowledge, skill, and team-based training; and (3) promoting a culture of continuous learning.

Shedding Light on the Black Box: Explaining Deep Neural Network Prediction of Clinical Outcomes.

Deep neural network models are emerging as an important method in healthcare delivery, following the recent success in other domains such as image recognition. Due to the multiple non-linear inner transformations, deep neural networks are viewed by many as black boxes. For practical use, deep learning models require explanations that are intuitive to clinicians. In this study, we developed a deep neural network model to predict outcomes following major cardiovascular procedures, using temporal image representation of past medical history as input. We created a novel explanation for the prediction of the model by defining impact scores that associate clinical observations with the outcome. For comparison, a logistic regression model was fitted to the same dataset. We compared the impact scores and log odds ratios by calculating three types of correlations, which provided a partial validation of the impact scores. The deep neural network model achieved an area under the receiver operating characteristics curve (AUC) of 0.787, compared to 0.746 for the logistic regression model. Moderate correlations were found between the impact scores and the log odds ratios. Impact scores generated by the explanation algorithm has the potential to shed light on the "black box" deep neural network model and could facilitate its adoption by clinicians.

Discordance in Information Exchange Between Providers During Care Transitions for Surgical Patients.

BACKGROUND: The exchange of health information between primary care providers (PCPs) and surgeons is critical during transitions of care for older patients with multiple comorbidities; however, it is unknown to what extent this process occurs. This study was designed to characterize the extent to which factors associated with older patient's recovery, such as functional status, cognitive status, social status, and emotional factors, are shared among PCPs and surgical providers during care transitions. MATERIALS AND METHODS: We prospectively identified 15 patients aged over 60 y with ≥3 comorbidities referred for general and vascular surgery procedures at a Veterans Administrative and academic medical center. Semistructured Critical Decision Method interviews were conducted with patients along with their surgical providers and referring PCPs. Thematic content analysis was performed independently by five reviewers on the cognitive processes associated with functional status, cognitive status, social status, and emotional factors. Interrater reliability between providers and patients was assessed using Cohen's kappa. RESULTS: Forty-seven Critical Decision Method interviews were conducted, which included 20 paired interviews between a PCP and a surgeon and 16 paired interviews that involved a patient and a provider. The majority of patients reported experiencing poor information exchange between their PCP and surgeon (58%) and feeling they were primarily responsible for communicating their own health information during care transitions (67%). In paired interviews between PCPs and surgeons, there was nearly perfect agreement for the shared knowledge of cognitive (kappa: 0.83) and emotional (kappa 1) factors. In contrast, there was only minimal agreement for shared knowledge of functional status (kappa 0.38) and social status (kappa: 0.34). CONCLUSIONS: Information exchange between PCPs and surgical providers is often discordant during transitions of surgical care for medically complex older patients, particularly when it pertains to communicating their functional or social status.

Physicians' perception of alternative displays of clinical research evidence for clinical decision support - A study with case vignettes.

OBJECTIVE: To design alternate information displays that present summaries of clinical trial results to clinicians to support decision-making; and to compare the displays according to efficacy and acceptability. METHODS: A 6-between (information display presentation order) by 3-within (display type) factorial design. Two alternate displays were designed based on Information Foraging theory: a narrative summary that reduces the content to a few sentences; and a table format that structures the display according to the PICO (Population, Intervention, Comparison, Outcome) framework. The designs were compared with the summary display format available in PubMed. Physicians were asked to review five clinical studies retrieved for a case vignette; and were presented with the three display formats. Participants were asked to rate their experience with each of the information displays according to a Likert scale questionnaire. RESULTS: Twenty physicians completed the study. Overall, participants rated the table display more highly than either the text summary or PubMed's summary format (5.9vs. 5.4vs. 3.9 on a scale between 1 [strongly disagree] and 7 [strongly agree]). Usefulness ratings of seven pieces of information, i.e. patient population, patient age range, sample size, study arm, primary outcome, results of primary outcome, and conclusion, were high (average across all items=4.71 on a 1 to 5 scale, with 1=not at all useful and 5=very useful). Study arm, primary outcome, and conclusion scored the highest (4.9, 4.85, and 4.85 respectively). Participants suggested additional details such as rate of adverse effects. CONCLUSION: The table format reduced physicians' perceived cognitive effort when quickly reviewing clinical trial information and was more favorably received by physicians than the narrative summary or PubMed's summary format display.

Translation of Contextual Control Model to chronic disease management: A paradigm to guide design of cognitive support systems.

BACKGROUND: Electronic health records (EHRs) continue to be criticized for providing poor cognitive support. Defining cognitive support has lacked theoretical foundation. We developed a measurement model of cognitive support based on the Contextual Control Model (COCOM), which describes control characteristics of an "orderly" joint system and proposes 4 levels of control: scrambled, opportunistic, tactical, and strategic. METHODS: 35 clinicians (5 centers) were interviewed pre and post outpatient clinical visits and audiotaped during the visit. Behaviors pertaining to hypertension management were systematically mapped to the COCOM control characteristics of: (1) time horizon, (2) uncertainty assessment, (3) consideration of multiple goals, (4) causal model described, and (5) explicitness of plan. Each encounter was classified for overall mode of control. Visits with deviation versus no deviation from hypertension goals were compared. RESULTS: Reviewer agreement was high. Control characteristics differed significantly between deviation groups (Wilcox rank sum p<.01). K-means cluster analysis of control characteristics, stratified by deviation were distinct, with higher goal deviations associated with more control characteristics. CONCLUSION: The COCOM control characteristics appear to be areas of potential yield for improved user-experience design.

Understanding complex clinical reasoning in infectious diseases for improving clinical decision support design.

BACKGROUND: Clinical experts' cognitive mechanisms for managing complexity have implications for the design of future innovative healthcare systems. The purpose of the study is to examine the constituents of decision complexity and explore the cognitive strategies clinicians use to control and adapt to their information environment. METHODS: We used Cognitive Task Analysis (CTA) methods to interview 10 Infectious Disease (ID) experts at the University of Utah and Salt Lake City Veterans Administration Medical Center. Participants were asked to recall a complex, critical and vivid antibiotic-prescribing incident using the Critical Decision Method (CDM), a type of Cognitive Task Analysis (CTA). Using the four iterations of the Critical Decision Method, questions were posed to fully explore the incident, focusing in depth on the clinical components underlying the complexity. Probes were included to assess cognitive and decision strategies used by participants. RESULTS: The following three themes emerged as the constituents of decision complexity experienced by the Infectious Diseases experts: 1) the overall clinical picture does not match the pattern, 2) a lack of comprehension of the situation and 3) dealing with social and emotional pressures such as fear and anxiety. All these factors contribute to decision complexity. These factors almost always occurred together, creating unexpected events and uncertainty in clinical reasoning. Five themes emerged in the analyses of how experts deal with the complexity. Expert clinicians frequently used 1) watchful waiting instead of over- prescribing antibiotics, engaged in 2) theory of mind to project and simulate other practitioners' perspectives, reduced very complex cases into simple 3) heuristics, employed 4) anticipatory thinking to plan and re-plan events and consulted with peers to share knowledge, solicit opinions and 5) seek help on patient cases. CONCLUSION: The cognitive strategies to deal with decision complexity found in this study have important implications for design future decision support systems for the management of complex patients.

A qualitative evaluation of the crucial attributes of contextual information necessary in EHR design to support patient-centered medical home care.

BACKGROUND: Effective implementation of a Primary Care Medical Home model of care (PCMH) requires integration of patients' contextual information (physical, mental, social and financial status) into an easily retrievable information source for the healthcare team and clinical decision-making. This project explored clinicians' perceptions about important attributes of contextual information for clinical decision-making, how contextual information is expressed in CPRS clinical documentation as well as how clinicians in a highly computerized environment manage information flow related to these areas. METHODS: A qualitative design using Cognitive Task Analyses and a modified Critical Incident Technique were used. The study was conducted in a large VA with a fully implemented EHR located in the western United States. Seventeen providers working in a PCMH model of care in Primary Care, Home Based Care and Geriatrics reported on a recent difficult transition requiring contextual information for decision-making. The transcribed interviews were qualitatively analyzed for thematic development related to contextual information using an iterative process and multiple reviewers with ATLAS@ti software. RESULTS: Six overarching themes emerged as attributes of contextual information: Informativeness, goal language, temporality, source attribution, retrieval effort, and information quality. CONCLUSIONS: These results indicate that specific attributes are needed to in order for contextual information to fully support clinical decision-making in a Medical Home care delivery environment. Improved EHR designs are needed for ease of contextual information access, displaying linkages across time and settings, and explicit linkages to both clinician and patient goals. Implications relevant to providers' information needs, team functioning and EHR design are discussed.

Text summarization in the biomedical domain: a systematic review of recent research.

OBJECTIVE: The amount of information for clinicians and clinical researchers is growing exponentially. Text summarization reduces information as an attempt to enable users to find and understand relevant source texts more quickly and effortlessly. In recent years, substantial research has been conducted to develop and evaluate various summarization techniques in the biomedical domain. The goal of this study was to systematically review recent published research on summarization of textual documents in the biomedical domain. MATERIALS AND METHODS: MEDLINE (2000 to October 2013), IEEE Digital Library, and the ACM digital library were searched. Investigators independently screened and abstracted studies that examined text summarization techniques in the biomedical domain. Information is derived from selected articles on five dimensions: input, purpose, output, method and evaluation. RESULTS: Of 10,786 studies retrieved, 34 (0.3%) met the inclusion criteria. Natural language processing (17; 50%) and a hybrid technique comprising of statistical, Natural language processing and machine learning (15; 44%) were the most common summarization approaches. Most studies (28; 82%) conducted an intrinsic evaluation. DISCUSSION: This is the first systematic review of text summarization in the biomedical domain. The study identified research gaps and provides recommendations for guiding future research on biomedical text summarization. CONCLUSION: Recent research has focused on a hybrid technique comprising statistical, language processing and machine learning techniques. Further research is needed on the application and evaluation of text summarization in real research or patient care settings.

Veterans Healthcare Administration providers' attitudes and perceptions regarding pragmatic trials embedded at the point of care.

Background The Veterans Healthcare Administration (VA) is implementing an adaptation of a pragmatic trial program, Point of Care Research (POC-R). The goal of POC-R is to embed research into clinical practice, contributing to a Learning Healthcare System. Provider acceptance and participation in POC-R is essential to its successful implementation. The purpose of this study is to evaluate provider's perceptions and beliefs regarding the POC-R program. Methods Provider focus groups and interviews were conducted at seven VA medical facilities involving 62 providers. A semi-structured script was used that included descriptions of four use cases and targeted questions regarding perceptions, concerns, and attitudes about the POC-R program. Sessions were audio-taped, de-identified, transcribed, and analyzed using systematic qualitative techniques to create response categories and overarching themes. Results The emergent themes were as follows: (1) POC-R is a valuable component of evidence-based practice, providing an opportunity to base clinical practice on more generalizable evidence as well as providing tools to improve local practice; (2) POC-R highlights the tension between the need for autonomy of practice and compliance with protocols; (3) POC-R may create increased time and burden resulting from added research responsibilities; (4) concern about the scientific validity and reliability of results; (5) potential for a negative impact on the provider-patient relationship; and (6) uncertainty regarding what constitutes equipoise, given differences in provider knowledge and preferences. Despite substantive concerns, barriers were generally felt to be solvable. Implementation should include provider education, careful attention to workflow for all arms of the study, inclusion of the entire team, and adequate oversight. Limitations The study design is qualitative with limited implications for causal inference. Participants are from the VA and may not be representative of other clinicians. Conclusion VA providers are supportive of the importance and value of pragmatic trials in general and of POC-R in particular. However, providers have significant concerns regarding the burden, ethics, and evidence regarding equipoise. Results are discussed in terms of implementation recommendations.

Design and implementation of electronic health record-based tools to support a weight management program in primary care.

Objectives: This paper reports on a mixed methods formative evaluation to support the design and implementation of information technology (IT) tools for a primary care weight management intervention delivered through the patient portal using primary care staff as coaches. Methods: We performed a qualitative needs assessment, designed the IT tools to support the weight management program, and developed implementation tracking metrics. Implementation tracking metrics were designed to use real world electronic health record (EHR) data. Results: The needs assessment revealed IT requirements as well as barriers and facilitators to implementation of EHR-based weight management interventions in primary care. We developed implementation metrics for the IT tools. These metrics were used in weekly project team calls to make sure that project resources were allocated to areas of need. Conclusion: This study identifies the important role of IT in supporting weight management through patient identification, weight and activity tracking in the patient portal, and the use of the EHR as a population management tool. An intensive multi-level implementation approach is required for successful primary care-based weight management interventions including well-designed IT tools, comprehensive involvement of clinic leadership, and implementation tracking metrics to guide the process of workflow integration. This study helps to bridge the gap between informatics and implementation by using socio-technical formative evaluation methods early in order to support the implementation of IT tools. Trial registration: clinicaltrials.gov, NCT04420936. Registered June 9, 2020.

Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial.

OBJECTIVES: We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2). MATERIALS AND METHODS: A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected. RESULTS: Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians. DISCUSSION: Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved. CONCLUSION: The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.

"Are we there yet?" Ten persistent hazards and inefficiencies with the use of medication administration technology from the perspective of practicing nurses.

OBJECTIVES: (1) Characterize persistent hazards and inefficiencies in inpatient medication administration; (2) Explore cognitive attributes of medication administration tasks; and (3) Discuss strategies to reduce medication administration technology-related hazards. MATERIALS AND METHODS: Interviews were conducted with 32 nurses practicing at 2 urban, eastern and western US health systems. Qualitative analysis using inductive and deductive coding included consensus discussion, iterative review, and coding structure revision. We abstracted hazards and inefficiencies through the lens of risks to patient safety and the cognitive perception-action cycle (PAC). RESULTS: Persistent safety hazards and inefficiencies related to MAT organized around the PAC cycle included: (1) Compatibility constraints create information silos; (2) Missing action cues; (3) Intermittent communication flow between safety monitoring systems and nurses; (4) Occlusion of important alerts by other, less helpful alerts; (5) Dispersed information: Information required for tasks is not collocated; (6) Inconsistent data organization: Mismatch of the display and the user's mental model; (7) Hidden medication administration technologies (MAT) limitations: Inaccurate beliefs about MAT functionality contribute to overreliance on the technology; (8) Software rigidity causes workarounds; (9) Cumbersome dependencies between technology and the physical environment; and (10) Technology breakdowns require adaptive actions. DISCUSSION: Errors might persist in medication administration despite successful Bar Code Medication Administration and Electronic Medication Administration Record deployment for reducing errors. Opportunities to improve MAT require a deeper understanding of high-level reasoning in medication administration, including control over the information space, collaboration tools, and decision support. CONCLUSION: Future medication administration technology should consider a deeper understanding of nursing knowledge work for medication administration.

Clinical trial recruitment in primary care: exploratory factor analysis of a questionnaire to measure barriers and facilitators to primary care providers' involvement.

BACKGROUND: Recruitment of sufficient participants for clinical trials remains challenging. Primary care is an important avenue for patient recruitment but is underutilized. We developed and pilot tested a questionnaire to measure relevant barriers and facilitators to primary care providers' involvement in recruiting patients for clinical trials. METHODS: Prior research informed the development of the questionnaire. The initial instrument was revised using feedback obtained from cognitive interviews. We invited all primary care providers practicing within the University of Utah Health system to complete the revised questionnaire. We used a mixed-mode design to collect paper responses via in-person recruitment and email contacts to collect responses online. Descriptive statistics, exploratory factor analysis, Cronbach's alpha, and multivariable regression analyses were conducted. RESULTS: Sixty-seven primary care providers participated in the survey. Exploratory factor analysis suggested retaining five factors, representing the importance of clinical trial recruitment in providers' professional identity, clinic-level interventions to facilitate referral, patient-related barriers, concerns about patient health management, and knowledge gaps. The five factors exhibited good or high internal consistency reliability. Professional identity and clinic-level intervention factors were significant predictors of providers' intention to participate in clinical trial recruitment activities. CONCLUSIONS: Results of this exploratory analysis provide preliminary evidence of the internal structure, internal consistency reliability, and predictive validity of the questionnaire to measure factors relevant to primary care providers' involvement in clinical trial recruitment.

Accuracies of Training Labels and Machine Learning Models: Experiments on Delirium and Simulated Data.

Supervised predictive models require labeled data for training purposes. Complete and accurate labeled data is not always available, and imperfectly labeled data may need to serve as an alternative. An important question is if the accuracy of the labeled data creates a performance ceiling for the trained model. In this study, we trained several models to recognize the presence of delirium in clinical documents using data with annotations that are not completely accurate. In the external evaluation, the support vector machine model with a linear kernel performed best, achieving an area under the curve of 89.3% and accuracy of 88%, surpassing the 80% accuracy of the training sample. We then generated a set of simulated data and carried out a series of experiments which demonstrated that models trained on imperfect data can (but do not always) outperform the accuracy of the training data.

Implementing lung cancer screening in primary care: needs assessment and implementation strategy design.

Lung cancer screening with low-dose computed tomography (CT) could help avert thousands of deaths each year. Since the implementation of screening is complex and underspecified, there is a need for systematic and theory-based strategies. Explore the implementation of lung cancer screening in primary care, in the context of integrating a decision aid into the electronic health record. Design implementation strategies that target hypothesized mechanisms of change and context-specific barriers. The study had two phases. The Qualitative Analysis phase included semi-structured interviews with primary care physicians to elicit key task behaviors (e.g., ordering a low-dose CT) and understand the underlying behavioral determinants (e.g., social influence). The Implementation Strategy Design phase consisted of defining implementation strategies and hypothesizing causal pathways to improve screening with a decision aid. Three key task behaviors and four behavioral determinants emerged from 14 interviews. Implementation strategies were designed to target multiple levels of influence. Strategies included increasing provider self-efficacy toward performing shared decision making and using the decision aid, improving provider performance expectancy toward ordering a low-dose CT, increasing social influence toward performing shared decision making and using the decision aid, and addressing key facilitators to using the decision aid. This study contributes knowledge about theoretical determinants of key task behaviors associated with lung cancer screening. We designed implementation strategies according to causal pathways that can be replicated and tested at other institutions. Future research is needed to evaluate the effectiveness of these strategies and to determine the contexts in which they can be effectively applied.

Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burnout?

The psychology of motivation can help us understand the impact of electronic health records (EHRs) on clinician burnout both directly and indirectly. Informatics approaches to EHR usability tend to focus on the extrinsic motivation associated with successful completion of clearly defined tasks in clinical workflows. Intrinsic motivation, which includes the need for autonomy, sense-making, creativity, connectedness, and mastery is not well supported by current designs and workflows. This piece examines existing research on the importance of 3 psychological drives in relation to healthcare technology: goal-based decision-making, sense-making, and agency/autonomy. Because these motives are ubiquitous, foundational to human functioning, automatic, and unconscious, they may be overlooked in technological interventions. The results are increased cognitive load, emotional distress, and unfulfilling workplace environments. Ultimately, we hope to stimulate new research on EHR design focused on expanding functionality to support intrinsic motivation, which, in turn, would decrease burnout and improve care.

Computerized Mortality Prediction for Community-acquired Pneumonia at 117 Veterans Affairs Medical Centers.

Rationale: Computerized severity assessment for community-acquired pneumonia could improve consistency and reduce clinician burden. Objectives: To develop and compare 30-day mortality-prediction models using electronic health record data, including a computerized score with all variables from the original Pneumonia Severity Index (PSI) except confusion and pleural effusion ("ePSI score") versus models with additional variables. Methods: Among adults with community-acquired pneumonia presenting to emergency departments at 117 Veterans Affairs Medical Centers between January 1, 2006, and December 31, 2016, we compared an ePSI score with 10 novel models employing logistic regression, spline, and machine learning methods using PSI variables, age, sex and 26 physiologic variables as well as all 69 PSI variables. Models were trained using encounters before January 1, 2015; tested on encounters during and after January 1, 2015; and compared using the areas under the receiver operating characteristic curve, confidence intervals, and patient event rates at a threshold PSI score of 970. Results: Among 297,498 encounters, 7% resulted in death within 30 days. When compared using the ePSI score (confidence interval [CI] for the area under the receiver operating characteristic curve, 0.77-0.78), performance increased with model complexity (CI for the logistic regression PSI model, 0.79-0.80; CI for the boosted decision-tree algorithm machine learning PSI model using the Extreme Gradient Boosting algorithm [mlPSI] with the 19 original PSI factors, 0.83-0.85) and the number of variables (CI for the logistic regression PSI model using all 69 variables, 0.84-085; CI for the mlPSI with all 69 variables, 0.86-0.87). Models limited to age, sex, and physiologic variables also demonstrated high performance (CI for the mlPSI with age, sex, and 26 physiologic factors, 0.84-0.85). At an ePSI score of 970 and a mortality-risk cutoff of <2.7%, the ePSI score identified 31% of all patients as being at "low risk"; the mlPSI with age, sex, and 26 physiologic factors identified 53% of all patients as being at low risk; and the mlPSI with all 69 variables identified 56% of all patients as being at low risk, with similar rates of mortality, hospitalization, and 7-day secondary hospitalization being determined. Conclusions: Computerized versions of the PSI accurately identified patients with pneumonia who were at low risk of death. More complex models classified more patients as being at low risk of death and as having similar adverse outcomes.

Standards-Based Clinical Decision Support Platform to Manage Patients Who Meet Guideline-Based Criteria for Genetic Evaluation of Familial Cancer.

PURPOSE: The ubiquitous adoption of electronic health records (EHRs) with family health history (FHH) data provides opportunities for tailoring cancer screening strategies to individuals. We aimed to enable a standards-based clinical decision support (CDS) platform for identifying and managing patients who meet guidelines for genetic evaluation of hereditary cancer. METHODS: The CDS platform (www.opencds.org) was used to implement algorithms based on the 2018 National Comprehensive Cancer Network guidelines for genetic evaluation of hereditary breast/ovarian and colorectal cancer. The platform was designed to be interfaced with different EHR systems via the Health Level Seven International Fast Healthcare Interoperability Resources standard. The platform was integrated with the Epic EHR and evaluated in a pilot study at an academic health care system. RESULTS: The CDS platform was executed against a target population of 143,012 patients; 5,245 (3.7%) met criteria for genetic evaluation based on the FHH recorded in the EHR. In a clinical pilot study, genetic counselors attempted to reach out to 71 of the patients. Of those patients, 25 (35%) scheduled an appointment, 10 (14%) declined, 2 (3%) did not need genetic counseling, 7 (10%) said they would consider it in the future, and 27 (38%) were unreachable. To date, 13 (52%) of the scheduled patients completed visits, and 2 (15%) of those were found to have pathogenic variants in cancer predisposition genes. CONCLUSION: A standards-based CDS platform integrated with EHR systems is a promising population-based approach to identify patients who are appropriate candidates for genetic evaluation of hereditary cancers.

Integrated displays to improve chronic disease management in ambulatory care: A SMART on FHIR application informed by mixed-methods user testing.

OBJECTIVE: The study sought to evaluate a novel electronic health record (EHR) add-on application for chronic disease management that uses an integrated display to decrease user cognitive load, improve efficiency, and support clinical decision making. MATERIALS AND METHODS: We designed a chronic disease management application using the technology framework known as SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources). We used mixed methods to obtain user feedback on a prototype to support ambulatory providers managing chronic obstructive pulmonary disease. Each participant managed 2 patient scenarios using the regular EHR with and without access to our prototype in block-randomized order. The primary outcome was the percentage of expert-recommended ideal care tasks completed. Timing, keyboard and mouse use, and participant surveys were also collected. User experiences were captured using a retrospective think-aloud interview analyzed by concept coding. RESULTS: With our prototype, the 13 participants completed more recommended care (81% vs 48%; P < .001) and recommended tasks per minute (0.8 vs 0.6; P = .03) over longer sessions (7.0 minutes vs 5.4 minutes; P = .006). Keystrokes per task were lower with the prototype (6 vs 18; P < .001). Qualitative themes elicited included the desire for reliable presentation of information which matches participants' mental models of disease and for intuitive navigation in order to decrease cognitive load. DISCUSSION: Participants completed more recommended care by taking more time when using our prototype. Interviews identified a tension between using the inefficient but familiar EHR vs learning to use our novel prototype. Concept coding of user feedback generated actionable insights. CONCLUSIONS: Mixed methods can support the design and evaluation of SMART on FHIR EHR add-on applications by enhancing understanding of the user experience.