Pediatric Septic Shock Care Pathways in General Emergency Departments: A Qualitative Study Targeting How to Really Make it Work.

OBJECTIVES: Many academic pediatric emergency departments (PEDs) have successfully implemented pediatric septic shock care pathways. However, many general emergency departments (GEDs), who see the majority of pediatric ED visits, have not. This study aims to compare the workflow, resources, communication, and decision making across these 2 settings to inform the future implementation of a standardized care pathway for children with septic shock in the GED. METHODS: We used the critical incident technique to conduct semistructured interviews with 24 ED physicians, nurses, and technicians at one PED and 2 GEDs regarding pediatric septic shock care. We performed a thematic analysis using the Framework Method to develop our coding schema through inductive and deductive analyses. We continued an iterative process of revising the schema until we reached consensus agreement and thematic saturation. RESULTS: We identified the following 6 themes: (1) functioning like a "well-oiled machine" may be key to high performance; (2) experiencing the sequence of care for children with sepsis as invariant and predictable may be essential to high-quality performance; (3) resilience and flexibility are characteristic of high levels of performance; (4) believing that "the buck stops here" may contribute to more accountability; (5) continuous system learning is essential; and (6) computerized clinical decision support may not be optimized to drive decision-making at the point of care. Commentary from GED and PED participants differed across the 6 themes, providing insight into the approach for standardized care pathway implementation in GEDs. CONCLUSIONS: Pediatric septic shock workflow, decision making, and system performance differ between the PED and GEDs. Implementation of a standardized care pathway in GEDs will require a tailored approach. Specific recommendations include (1) improving shared situation awareness; (2) simulation for knowledge, skill, and team-based training; and (3) promoting a culture of continuous learning.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Through the Narrative Looking Glass: Commentary on "Impact of Electronic Health Records on Information Practices in Mental Health Contexts: Scoping Review".

The authors of "Impact of Electronic Health Records on Information Practices in Mental Health Contexts: Scoping Review" have effectively brought to our attention the failure of the electronic health record (EHR) to represent the human context. Because mental health or behavioral disorders (and functional status in general) emerge from an interaction between the individual's characteristics and the social context, it is essentially a failure to represent the human context. The assessment and treatment of these disorders must reflect how the person lives, their degree of social connectedness, their personal motivation, and their cultural background. This type of information is best communicated both through narrative and in collaboration with other providers and the patient-largely because human social memory is organized around situation models and natural episodes. Neither functionality is currently available in most EHRs. Narrative communication is effective for several reasons: (1) it supports the communication of goals between providers; (2) it allows the author to express their belief in others' perspectives (theory of mind), for example, those who will be reading these notes; and (3) it supports the incorporation of the patient's personal perspective. The failure of the EHR to support mental health information data and information practices is, therefore, essentially a failure to support the basic communication functions necessary for the narrative. The authors have rightly noted the problems of the EHR in this domain, but perhaps they did not completely link the problems to the lack of functionality to support narrative communication. Suggestions for adding design elements are discussed.

Analysis of the cognitive demands of electronic health record use.

The goal of this study was to elicit the cognitive demands facing clinicians when using an electronic health record (EHR) system and learn the cues and strategies expert clinicians rely on to manage those demands. This study differs from prior research by applying a joint cognitive systems perspective to examining the cognitive aspects of clinical work. We used a cognitive task analysis (CTA) method specifically tailored to elicit the cognitive demands of an EHR system from expert clinicians from different sites in a variety of inpatient and outpatient roles. The analysis of the interviews revealed 145 unique cognitive demands of using an EHR, which were organized into 22 distinct themes across seven broad categories. In addition to confirming previously published themes of cognitive demands, the main emergent themes of this study are: 1) The EHR does not help clinicians develop and maintain awareness of the big picture; 2) The EHR does not support clinicians' need to reason about patients' current and future states, including effects of potential treatments; and 3) The EHR limits agency of clinicians to work individually and collaboratively. Implications for theory and EHR design and evaluation are discussed.

Feasibility and acceptability of a game-based symptom-reporting app for children with cancer: perspectives of children and parents.

BACKGROUND: Children with cancer have difficulty identifying and describing the multiple symptoms they experience during hospitalization and between clinical encounters. Mobile health resources, including apps, are potential solutions to support child-centric symptom reporting. This study evaluated the feasibility and acceptability of a newly developed game-based symptom-reporting app for school-age children with cancer. PROCEDURE: Nineteen school-age children (6-12 years of age) receiving treatment for cancer at a COG institution in the Intermountain West of the United States used a game-based symptom-reporting app between clinical visits. Feasibility was evaluated through a summary of actual days of app use and interaction with each of the app's features. Children and their parents participated in interviews regarding the app's acceptability. RESULTS: Children used the app a median of 4 days (range 1-12) and interacted most frequently with the symptom reporting and the drawing features. Children enjoyed aspects of the app that supported their creativity and provided choices. Parents endorsed the interactive nature of the app and the value of the child providing his/her own report. Both children and parents identified additional opportunities to enhance the child's user experience. CONCLUSION: Study results support the preliminary feasibility and acceptability of the app. Children's and parents' responses supported the developmental relevance of the app and its role in enhancing the child's autonomy and serving as an outlet for creativity. Future directions include optimizing the child user's experience and investigating the app's role as a resource to enhance shared decision-making for symptom management.

Implementing innovative service delivery models in genetic counseling: a qualitative analysis of facilitators and barriers.

Given the increasing demand for genetic counseling, implementation of innovative service delivery models (SDMs) has been proposed to improve access and increase the efficiency of genetic counseling services. This study investigated the key considerations in genetic counseling practices that impact the decision to implement innovative SDMs, as well as barriers and facilitators to implementation. A cross-sectional, qualitative research design was employed, using focus groups. Genetic counselors (GCs) were recruited from respondents to an online survey about SDMs and by posting an invitation to the National Society of Genetic Counselors (NSGC) listserv. A total of 30 participants participated in either in-person (n = 3) or online (n = 5) focus groups, facilitated by members of the Service Delivery Model subcommittee of the NSGC Access and Service Delivery Committee. Transcribed audio and written notes from the focus groups were coded, utilizing an inductive thematic analysis method. Five overarching themes that impact decisions to implement and use innovative SDMs were identified: (a) Patient volume and efficiency are key motivators in trying innovative SDMs; (b) the decision to change is facilitated by available resources; (c) the implementation of innovative SDMs is multidimensional and complex process; (d) there is concern about the impact on patient-provider relationships with the utilization of innovative SDMs; and (e) measuring outcomes of innovative SDM facilitates acquisition of additional resources and support. Approaches to innovative SDM selection and implementation vary by institution needs, resources, and population as demonstrated by the variety of approaches to similar barriers. Outcomes related to the implementation of innovative SDMs in genetic counseling practice should be measured to demonstrate the value of innovative genetic counseling SDMs, improvement of access to care, and to justify need for additional resources to support implementation of these models.

A systematic review of theoretical constructs in CDS literature.

BACKGROUND: Studies that examine the adoption of clinical decision support (CDS) by healthcare providers have generally lacked a theoretical underpinning. The Unified Theory of Acceptance and Use of Technology (UTAUT) model may provide such a theory-based explanation; however, it is unknown if the model can be applied to the CDS literature. OBJECTIVE: Our overall goal was to develop a taxonomy based on UTAUT constructs that could reliably characterize CDS interventions. METHODS: We used a two-step process: (1) identified randomized controlled trials meeting comparative effectiveness criteria, e.g., evaluating the impact of CDS interventions with and without specific features or implementation strategies; (2) iteratively developed and validated a taxonomy for characterizing differential CDS features or implementation strategies using three raters. RESULTS: Twenty-five studies with 48 comparison arms were identified. We applied three constructs from the UTAUT model and added motivational control to characterize CDS interventions. Inter-rater reliability was as follows for model constructs: performance expectancy (κ = 0.79), effort expectancy (κ = 0.85), social influence (κ = 0.71), and motivational control (κ = 0.87). CONCLUSION: We found that constructs from the UTAUT model and motivational control can reliably characterize features and associated implementation strategies. Our next step is to examine the quantitative relationships between constructs and CDS adoption.

Iterative heuristic design of temporal graphic displays with clinical domain experts.

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Shedding Light on the Black Box: Explaining Deep Neural Network Prediction of Clinical Outcomes.

Deep neural network models are emerging as an important method in healthcare delivery, following the recent success in other domains such as image recognition. Due to the multiple non-linear inner transformations, deep neural networks are viewed by many as black boxes. For practical use, deep learning models require explanations that are intuitive to clinicians. In this study, we developed a deep neural network model to predict outcomes following major cardiovascular procedures, using temporal image representation of past medical history as input. We created a novel explanation for the prediction of the model by defining impact scores that associate clinical observations with the outcome. For comparison, a logistic regression model was fitted to the same dataset. We compared the impact scores and log odds ratios by calculating three types of correlations, which provided a partial validation of the impact scores. The deep neural network model achieved an area under the receiver operating characteristics curve (AUC) of 0.787, compared to 0.746 for the logistic regression model. Moderate correlations were found between the impact scores and the log odds ratios. Impact scores generated by the explanation algorithm has the potential to shed light on the "black box" deep neural network model and could facilitate its adoption by clinicians.

Decreases in Antimicrobial Use Associated With Multihospital Implementation of Electronic Antimicrobial Stewardship Tools.

BACKGROUND: Antimicrobial stewards may benefit from comparative data to inform interventions that promote optimal inpatient antimicrobial use. METHODS: Antimicrobial stewards from 8 geographically dispersed Veterans Affairs (VA) inpatient facilities participated in the development of antimicrobial use visualization tools that allowed for comparison to facilities of similar complexity. The visualization tools consisted of an interactive web-based antimicrobial dashboard and, later, a standardized antimicrobial usage report updated at user-selected intervals. Stewards participated in monthly learning collaboratives. The percent change in average monthly antimicrobial use (all antimicrobial agents, anti-methicillin-resistant Staphylococcus aureus [anti-MRSA] agents, and antipseudomonal agents) was analyzed using a pre-post (January 2014-January 2016 vs July 2016-January 2018) design with segmented regression and external comparison with uninvolved control facilities (n = 118). RESULTS: Intervention sites demonstrated a 2.1% decrease (95% confidence interval [CI], -5.7% to 1.6%) in total antimicrobial use pre-post intervention vs a 2.5% increase (95% CI, 0.8% to 4.1%) in nonintervention sites (absolute difference, 4.6%; P = .025). Anti-MRSA antimicrobial use decreased 11.3% (95% CI, -16.0% to -6.3%) at intervention sites vs a 6.6% decrease (95% CI, -9.1% to -3.9%) at nonintervention sites (absolute difference, 4.7%; P = .092). Antipseudomonal antimicrobial use decreased 3.4% (95% CI, -8.2% to 1.7%) at intervention sites vs a 3.6% increase (95% CI, 0.8% to 6.5%) at nonintervention sites (absolute difference, 7.0%; P = .018). CONCLUSIONS: Comparative data visualization tool use by stewards at 8 VA facilities was associated with significant reductions in overall antimicrobial and antipseudomonal use relative to uninvolved facilities.

Family Experience With Pierre Robin Sequence: A Qualitative Study.

OBJECTIVE: To identify concepts and constructs important to parents of children with Pierre Robin Sequence (PRS). DESIGN: Qualitative study. SETTING: All children received some care at a tertiary hospital with additional care at outside facilities. Interviews were conducted in nonclinical locations, including remote locations. PARTICIPANTS: Parents of children <5 years old with a diagnosis of PRS. Prior treatments included observation, positioning, nasal trumpet, mandibular distraction osteogenesis, tracheostomy, and gastrostomy. INTERVENTION: Semi-structured interviews with individuals (4) and with groups (focus groups, 4) were conducted using open-ended questions and non-leading prompts. Transcripts were analyzed with iterative open and axial coding. Concepts and constructs were identified and refined into codes and central themes. Interviews were conducted until thematic saturation was achieved. RESULTS: Sixteen parents were interviewed. Their experiences were coded into 5 main themes, which can be summarized as: (1) child's symptoms/well-being, (2) parents' grief/isolation, (3) family stress, (4) relationships with providers, and (5) psychological and technical growth. Difficulty with feeding, weight gain, and breathing problems were core physical issues described by participants with associated intense fear. Participants described frustration from not only lack of care coordination, slow diagnoses, and poor communication but also gratitude for providers who served as advocates. Participants described gradual development of knowledge/competencies. CONCLUSIONS: Families of children with PRS have experiences that profoundly affect their lives. Child's physical symptoms/well-being and parents' psychosocial well-being provide content for a future PRS-specific quality-of-life instrument. Concepts that emerged also provide a framework to improve parents' experience and enhance their children's quality of care.

Discordance in Information Exchange Between Providers During Care Transitions for Surgical Patients.

BACKGROUND: The exchange of health information between primary care providers (PCPs) and surgeons is critical during transitions of care for older patients with multiple comorbidities; however, it is unknown to what extent this process occurs. This study was designed to characterize the extent to which factors associated with older patient's recovery, such as functional status, cognitive status, social status, and emotional factors, are shared among PCPs and surgical providers during care transitions. MATERIALS AND METHODS: We prospectively identified 15 patients aged over 60 y with ≥3 comorbidities referred for general and vascular surgery procedures at a Veterans Administrative and academic medical center. Semistructured Critical Decision Method interviews were conducted with patients along with their surgical providers and referring PCPs. Thematic content analysis was performed independently by five reviewers on the cognitive processes associated with functional status, cognitive status, social status, and emotional factors. Interrater reliability between providers and patients was assessed using Cohen's kappa. RESULTS: Forty-seven Critical Decision Method interviews were conducted, which included 20 paired interviews between a PCP and a surgeon and 16 paired interviews that involved a patient and a provider. The majority of patients reported experiencing poor information exchange between their PCP and surgeon (58%) and feeling they were primarily responsible for communicating their own health information during care transitions (67%). In paired interviews between PCPs and surgeons, there was nearly perfect agreement for the shared knowledge of cognitive (kappa: 0.83) and emotional (kappa 1) factors. In contrast, there was only minimal agreement for shared knowledge of functional status (kappa 0.38) and social status (kappa: 0.34). CONCLUSIONS: Information exchange between PCPs and surgical providers is often discordant during transitions of surgical care for medically complex older patients, particularly when it pertains to communicating their functional or social status.

Interactive Visual Displays for Interpreting the Results of Clinical Trials: Formative Evaluation With Case Vignettes.

BACKGROUND: At the point of care, evidence from randomized controlled trials (RCTs) is underutilized in helping clinicians meet their information needs. OBJECTIVE: To design interactive visual displays to help clinicians interpret and compare the results of relevant RCTs for the management of a specific patient, and to conduct a formative evaluation with physicians comparing interactive visual versus narrative displays. METHODS: We followed a user-centered and iterative design process succeeded by development of information display prototypes as a Web-based application. We then used a within-subjects design with 20 participants (8 attendings and 12 residents) to evaluate the usability and problem-solving impact of the information displays. We compared subjects' perceptions of the interactive visual displays versus narrative abstracts. RESULTS: The resulting interactive visual displays present RCT results side-by-side according to the Population, Intervention, Comparison, and Outcome (PICO) framework. Study participants completed 19 usability tasks in 3 to 11 seconds with a success rate of 78% to 100%. Participants favored the interactive visual displays over narrative abstracts according to perceived efficiency, effectiveness, effort, user experience and preference (all P values <.001). CONCLUSIONS: When interpreting and applying RCT findings to case vignettes, physicians preferred interactive graphical and PICO-framework-based information displays that enable direct comparison of the results from multiple RCTs compared to the traditional narrative and study-centered format. Future studies should investigate the use of interactive visual displays to support clinical decision making in care settings and their effect on clinician and patient outcomes.

Making cognitive decision support work: Facilitating adoption, knowledge and behavior change through QI.

This paper evaluates the role of facilitation in the successful implementation of Computerized Decision Support (CDS). Facilitation processes include education, specialized computerized decision support, and work process reengineering. These techniques, as well as modeling and feedback enhance self-efficacy, which we propose is one of the factors that mediate the effectiveness of any CDS. In this study, outpatient clinics implemented quality improvement (QI) projects focused on improving geriatric care. Quality Improvement is the systematic process of improving quality through continuous measurement and targeted actions. The program, entitled "Advancing Geriatric Education through Quality Improvement" (AGE QI), consisted of a 6-month, QI based, intervention: (1) 2h didactic session, (2) 1h QI planning session, (3) computerized decision support design and implementation, (4) QI facilitation activities, (5) outcome feedback, and (6) 20h of CME. Specifically, we examined the impact of the QI based program on clinician's perceived self-efficacy in caring for older adults and the relationship of implementation support and facilitation on perceived success. The intervention was implemented at 3 institutions, 27 community healthcare system clinics, and 134 providers. This study reports the results of pre/post surveys for the forty-nine clinicians who completed the full CME program. Self-efficacy ratings for specific clinical behaviors related to care of older adults were assessed using a Likert based instrument. Self-ratings of efficacy improved across the following domains (depression, falls, end-of-life, functional status and medication management) and specifically in QI targeted domains and were associated with overall clinic improvements.

Physicians' perception of alternative displays of clinical research evidence for clinical decision support - A study with case vignettes.

OBJECTIVE: To design alternate information displays that present summaries of clinical trial results to clinicians to support decision-making; and to compare the displays according to efficacy and acceptability. METHODS: A 6-between (information display presentation order) by 3-within (display type) factorial design. Two alternate displays were designed based on Information Foraging theory: a narrative summary that reduces the content to a few sentences; and a table format that structures the display according to the PICO (Population, Intervention, Comparison, Outcome) framework. The designs were compared with the summary display format available in PubMed. Physicians were asked to review five clinical studies retrieved for a case vignette; and were presented with the three display formats. Participants were asked to rate their experience with each of the information displays according to a Likert scale questionnaire. RESULTS: Twenty physicians completed the study. Overall, participants rated the table display more highly than either the text summary or PubMed's summary format (5.9vs. 5.4vs. 3.9 on a scale between 1 [strongly disagree] and 7 [strongly agree]). Usefulness ratings of seven pieces of information, i.e. patient population, patient age range, sample size, study arm, primary outcome, results of primary outcome, and conclusion, were high (average across all items=4.71 on a 1 to 5 scale, with 1=not at all useful and 5=very useful). Study arm, primary outcome, and conclusion scored the highest (4.9, 4.85, and 4.85 respectively). Participants suggested additional details such as rate of adverse effects. CONCLUSION: The table format reduced physicians' perceived cognitive effort when quickly reviewing clinical trial information and was more favorably received by physicians than the narrative summary or PubMed's summary format display.

Identifying complexity in infectious diseases inpatient settings: An observation study.

BACKGROUND: Understanding complexity in healthcare has the potential to reduce decision and treatment uncertainty. Therefore, identifying both patient and task complexity may offer better task allocation and design recommendation for next-generation health information technology system design. OBJECTIVE: To identify specific complexity-contributing factors in the infectious disease domain and the relationship with the complexity perceived by clinicians. METHOD: We observed and audio recorded clinical rounds of three infectious disease teams. Thirty cases were observed for a period of four consecutive days. Transcripts were coded based on clinical complexity-contributing factors from the clinical complexity model. Ratings of complexity on day 1 for each case were collected. We then used statistical methods to identify complexity-contributing factors in relationship to perceived complexity of clinicians. RESULTS: A factor analysis (principal component extraction with varimax rotation) of specific items revealed three factors (eigenvalues>2.0) explaining 47% of total variance, namely task interaction and goals (10 items, 26%, Cronbach's Alpha=0.87), urgency and acuity (6 items, 11%, Cronbach's Alpha=0.67), and psychosocial behavior (4 items, 10%, Cronbach's alpha=0.55). A linear regression analysis showed no statistically significant association between complexity perceived by the physicians and objective complexity, which was measured from coded transcripts by three clinicians (Multiple R-squared=0.13, p=0.61). There were no physician effects on the rating of perceived complexity. CONCLUSION: Task complexity contributes significantly to overall complexity in the infectious diseases domain. The different complexity-contributing factors found in this study can guide health information technology system designers and researchers for intuitive design. Thus, decision support tools can help reduce the specific complexity-contributing factors. Future studies aimed at understanding clinical domain-specific complexity-contributing factors can ultimately improve task allocation and design for intuitive clinical reasoning.

Think twice: A cognitive perspective of an antibiotic timeout intervention to improve antibiotic use.

OBJECTIVES: To understand clinicians' impressions of and decision-making processes regarding an informatics-supported antibiotic timeout program to re-evaluate the appropriateness of continuing vancomycin and piperacillin/tazobactam. METHODS: We implemented a multi-pronged informatics intervention, based on Dual Process Theory, to prompt discontinuation of unwarranted vancomycin and piperacillin/tazobactam on or after day three in a large Veterans Affairs Medical Center. Two workflow changes were introduced to facilitate cognitive deliberation about continuing antibiotics at day three: (1) teams completed an electronic template note, and (2) a paper summary of clinical and antibiotic-related information was provided to clinical teams. Shortly after starting the intervention, six focus groups were conducted with users or potential users. Interviews were recorded and transcribed. Iterative thematic analysis identified recurrent themes from feedback. RESULTS: Themes that emerged are represented by the following quotations: (1) captures and controls attention ("it reminds us to think about it"), (2) enhances informed and deliberative reasoning ("it makes you think twice"), (3) redirects decision direction ("…because [there was no indication] I just [discontinued] it without even trying"), (4) fosters autonomy and improves team empowerment ("the template… forces the team to really discuss it"), and (5) limits use of emotion-based heuristics ("my clinical concern is high enough I think they need more aggressive therapy…"). CONCLUSIONS: Requiring template completion to continue antibiotics nudged clinicians to re-assess the appropriateness of specified antibiotics. Antibiotic timeouts can encourage deliberation on overprescribed antibiotics without substantially curtailing autonomy. An effective nudge should take into account clinician's time, workflow, and thought processes.

Translation of Contextual Control Model to chronic disease management: A paradigm to guide design of cognitive support systems.

BACKGROUND: Electronic health records (EHRs) continue to be criticized for providing poor cognitive support. Defining cognitive support has lacked theoretical foundation. We developed a measurement model of cognitive support based on the Contextual Control Model (COCOM), which describes control characteristics of an "orderly" joint system and proposes 4 levels of control: scrambled, opportunistic, tactical, and strategic. METHODS: 35 clinicians (5 centers) were interviewed pre and post outpatient clinical visits and audiotaped during the visit. Behaviors pertaining to hypertension management were systematically mapped to the COCOM control characteristics of: (1) time horizon, (2) uncertainty assessment, (3) consideration of multiple goals, (4) causal model described, and (5) explicitness of plan. Each encounter was classified for overall mode of control. Visits with deviation versus no deviation from hypertension goals were compared. RESULTS: Reviewer agreement was high. Control characteristics differed significantly between deviation groups (Wilcox rank sum p<.01). K-means cluster analysis of control characteristics, stratified by deviation were distinct, with higher goal deviations associated with more control characteristics. CONCLUSION: The COCOM control characteristics appear to be areas of potential yield for improved user-experience design.

Physician activity during outpatient visits and subjective workload.

We describe methods for capturing and analyzing EHR use and clinical workflow of physicians during outpatient encounters and relating activity to physicians' self-reported workload. We collected temporally-resolved activity data including audio, video, EHR activity, and eye-gaze along with post-visit assessments of workload. These data are then analyzed through a combination of manual content analysis and computational techniques to temporally align streams, providing a range of process measures of EHR usage, clinical workflow, and physician-patient communication. Data was collected from primary care and specialty clinics at the Veterans Administration San Diego Healthcare System and UCSD Health, who use Electronic Health Record (EHR) platforms, CPRS and Epic, respectively. Grouping visit activity by physician, site, specialty, and patient status enables rank-ordering activity factors by their correlation to physicians' subjective work-load as captured by NASA Task Load Index survey. We developed a coding scheme that enabled us to compare timing studies between CPRS and Epic and extract patient and visit complexity profiles. We identified similar patterns of EHR use and navigation at the 2 sites despite differences in functions, user interfaces and consequent coded representations. Both sites displayed similar proportions of EHR function use and navigation, and distribution of visit length, proportion of time physicians attended to EHRs (gaze), and subjective work-load as measured by the task load survey. We found that visit activity was highly variable across individual physicians, and the observed activity metrics ranged widely as correlates to subjective workload. We discuss implications of our study for methodology, clinical workflow and EHR redesign.

A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support.

Mobile health platforms offer significant opportunities for improving diabetic self-care, but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide distinct usability information about systems. The purpose of this study was to complete a usability evaluation of a mobile health system for diabetes patients using a modified heuristic evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients' self-care, and (3) in-depth severity factor ratings. Experts identified 129 usability problems with 274 heuristic violations for the system. The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match Between System and Real World at 22.3% (n = 61). Average severity ratings across system views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The large volume of violations with severe ratings indicated clear priorities for redesign. The modified heuristic approach allowed evaluators to identify unique and important issues, including ones related to self-management and patient safety. This article provides a template for one type of expert evaluation adding to the informaticists' toolbox when needing to conduct a fast, resource-efficient and user-oriented heuristic evaluation.