Research gaps in routine health information system design barriers to data quality and use in low- and middle-income countries: A literature review.

Despite the potential impact of health information system (HIS) design barriers on health data quality and use and, ultimately, health outcomes in low- and middle-income countries (LMICs), no comprehensive literature review has been conducted to study them in this context. We therefore conducted a formal literature review to understand system design barriers to data quality and use in LMICs and to identify any major research gaps related understanding how system design affects data use. We conducted an electronic search across 4 scientific databases-PubMed, Web of Science, Embase, and Global Health-and consulted a data use expert. Following a systematic inclusion and exclusion process, 316 publications (316 abstracts and 18 full papers) were included in the review. We found a paucity of scientific publications that explicitly describe system design factors that hamper data quality or data use for decision making. Although user involvement, work flow, human-computer interactions, and user experience are critical aspects of system design, our findings suggest that these issues are not discussed or conceptualized in the literature. Findings also showed that individual training efforts focus primarily on imparting data analysis skills. The adverse impact of HIS design barriers on data integrity and health system performance may be even bigger in LMICs than elsewhere, leading to errors in population health management and clinical care. We argue for integrating systems thinking into HIS strengthening efforts to reduce the HIS design-user reality gap.

Comparative Visualization of the RNA Suboptimal Conformational Ensemble In Vivo.

When a ribonucleic acid (RNA) molecule folds, it often does not adopt a single, well-defined conformation. The folding energy landscape of an RNA is highly dependent on its nucleotide sequence and molecular environment. Cellular molecules sometimes alter the energy landscape, thereby changing the ensemble of likely low-energy conformations. The effects of these energy landscape changes on the conformational ensemble are particularly challenging to visualize for large RNAs. We have created a robust approach for visualizing the conformational ensemble of RNAs that is well suited for in vitro versus in vivo comparisons. Our method creates a stable map of conformational space for a given RNA sequence. We first identify single point mutations in the RNA that maximally sample suboptimal conformational space based on the ensemble's partition function. Then, we cluster these diverse ensembles to identify the most diverse partition functions for Boltzmann stochastic sampling. By using, to our knowledge, a novel nestedness distance metric, we iteratively add mutant suboptimal ensembles to converge on a stable 2D map of conformational space. We then compute the selective 2' hydroxyl acylation by primer extension (SHAPE)-directed ensemble for the RNA folding under different conditions, and we project these ensembles on the map to visualize. To validate our approach, we established a conformational map of the Vibrio vulnificus add adenine riboswitch that reveals five classes of structures. In the presence of adenine, projection of the SHAPE-directed sampling correctly identified the on-conformation; without the ligand, only off-conformations were visualized. We also collected the whole-transcript in vitro and in vivo SHAPE-MaP for human ß-actin messenger RNA that revealed similar global folds in both conditions. Nonetheless, a comparison of in vitro and in vivo data revealed that specific regions exhibited significantly different SHAPE-MaP profiles indicative of structural rearrangements, including rearrangement consistent with binding of the zipcode protein in a region distal to the stop codon.

A methodology for interactive mining and visual analysis of clinical event patterns using electronic health record data.

Patients' medical conditions often evolve in complex and seemingly unpredictable ways. Even within a relatively narrow and well-defined episode of care, variations between patients in both their progression and eventual outcome can be dramatic. Understanding the patterns of events observed within a population that most correlate with differences in outcome is therefore an important task in many types of studies using retrospective electronic health data. In this paper, we present a method for interactive pattern mining and analysis that supports ad hoc visual exploration of patterns mined from retrospective clinical patient data. Our approach combines (1) visual query capabilities to interactively specify episode definitions, (2) pattern mining techniques to help discover important intermediate events within an episode, and (3) interactive visualization techniques that help uncover event patterns that most impact outcome and how those associations change over time. In addition to presenting our methodology, we describe a prototype implementation and present use cases highlighting the types of insights or hypotheses that our approach can help uncover.

Using Counterfactuals to Improve Causal Inferences From Visualizations.

Traditional approaches to data visualization have often focused on comparing different subsets of data, and this is reflected in the many techniques developed and evaluated over the years for visual comparison. Similarly, common workflows for exploratory visualization are built upon the idea of users interactively applying various filter and grouping mechanisms in search of new insights. This paradigm has proven effective at helping users identify correlations between variables that can inform thinking and decision-making. However, recent studies show that consumers of visualizations often draw causal conclusions even when not supported by the data. Motivated by these observations, this article highlights recent advances from a growing community of researchers exploring methods that aim to directly support visual causal inference. However, many of these approaches have their own limitations, which limit their use in many real-world scenarios. This article, therefore, also outlines a set of key open challenges and corresponding priorities for new research to advance the state of the art in visual causal inference.

Clinical Decision Support for Surgery: A Mixed Methods Study on Design and Implementation Perspectives From Urologists.

OBJECTIVE: To assess urologist attitudes toward clinical decision support (CDS) embedded into the electronic health record (EHR) and define design needs to facilitate implementation and impact. With recent advances in big data and artificial intelligence (AI), enthusiasm for personalized, data-driven tools to improve surgical decision-making has grown, but the impact of current tools remains limited. METHODS: A sequential explanatory mixed methods study from 2019 to 2020 was performed. First, survey responses from the 2019 American Urological Association Annual Census evaluated attitudes toward an automatic CDS tool that would display risk/benefit data. This was followed by the purposeful sampling of 25 urologists and qualitative interviews assessing perspectives on CDS impact and design needs. Bivariable, multivariable, and coding-based thematic analysis were applied and integrated. RESULTS: Among a weighted sample of 12,366 practicing urologists, the majority agreed CDS would help decision-making (70.9%, 95% CI 68.7%-73.2%), aid patient counseling (78.5%, 95% CI 76.5%-80.5%), save time (58.1%, 95% CI 55.7%-60.5%), and improve patient outcomes (42.9%, 95% CI 40.5%-45.4%). More years in practice was negatively associated with agreement (P <.001). Urologists described how CDS could bolster evidence-based care, personalized medicine, resource utilization, and patient experience. They also identified multiple implementation barriers and provided suggestions on form, functionality, and visual design to improve usefulness and ease of use. CONCLUSION: Urologists have favorable attitudes toward the potential for clinical decision support in the EHR. Smart design will be critical to ensure effective implementation and impact.

A Design Space for Surfacing Content Recommendations in Visual Analytic Platforms.

Recommendation algorithms have been leveraged in various ways within visualization systems to assist users as they perform of a range of information tasks. One common focus for these techniques has been the recommendation of content, rather than visual form, as a means to assist users in the identification of information that is relevant to their task context. A wide variety of techniques have been proposed to address this general problem, with a range of design choices in how these solutions surface relevant information to users. This paper reviews the state-of-the-art in how visualization systems surface recommended content to users during users' visual analysis; introduces a four-dimensional design space for visual content recommendation based on a characterization of prior work; and discusses key observations regarding common patterns and future research opportunities.

Electronic Health Record Use and Perceptions among Urologic Surgeons.

OBJECTIVE: Electronic health records (EHRs) have become widely adopted with increasing emphasis on improving care delivery. Improvements in surgery may be limited by specialty-specific issues that impact EHR usability and engagement. Accordingly, we examined EHR use and perceptions in urology, a diverse surgical specialty. METHODS: We conducted a national, sequential explanatory mixed methods study. Through the 2019 American Urological Association Census, we surveyed urologic surgeons on EHR use and perceptions and then identified associated characteristics through bivariable and multivariable analyses. Using purposeful sampling, we interviewed 25 urologists and applied coding-based thematic analysis, which was then integrated with survey findings. RESULTS: Among 2,159 practicing urologic surgeons, 2,081 (96.4%) reported using an EHR. In the weighted sample (n = 12,366), over 90% used the EHR for charting, viewing results, and order entry with most using information exchange functions (59.0-79.6%). In contrast, only 35.8% felt the EHR increases clinical efficiency, whereas 43.1% agreed it improves patient care, which related thematically to information management, administrative burden, patient safety, and patient-surgeon interaction. Quantitatively and qualitatively, use and perceptions differed by years in practice and practice type with more use and better perceptions among more recent entrants into the urologic workforce and those in academic/multispecialty practices, who may have earlier EHR exposure, better infrastructure, and more support. CONCLUSION: Despite wide and substantive usage, EHRs engender mixed feelings, especially among longer-practicing surgeons and those in lower-resourced settings (e.g., smaller and private practices). Beyond reducing administrative burden and simplifying information management, efforts to improve care delivery through the EHR should focus on surgeon engagement, particularly in the community, to boost implementation and user experience.

Improving Visualization Interpretation Using Counterfactuals.

Complex, high-dimensional data is used in a wide range of domains to explore problems and make decisions. Analysis of high-dimensional data, however, is vulnerable to the hidden influence of confounding variables, especially as users apply ad hoc filtering operations to visualize only specific subsets of an entire dataset. Thus, visual data-driven analysis can mislead users and encourage mistaken assumptions about causality or the strength of relationships between features. This work introduces a novel visual approach designed to reveal the presence of confounding variables via counterfactual possibilities during visual data analysis. It is implemented in CoFact, an interactive visualization prototype that determines and visualizes counterfactual subsets to better support user exploration of feature relationships. Using publicly available datasets, we conducted a controlled user study to demonstrate the effectiveness of our approach; the results indicate that users exposed to counterfactual visualizations formed more careful judgments about feature-to-outcome relationships.

Survey on Visual Analysis of Event Sequence Data.

Event sequence data record series of discrete events in the time order of occurrence. They are commonly observed in a variety of applications ranging from electronic health records to network logs, with the characteristics of large-scale, high-dimensional and heterogeneous. This high complexity of event sequence data makes it difficult for analysts to manually explore and find patterns, resulting in ever-increasing needs for computational and perceptual aids from visual analytics techniques to extract and communicate insights from event sequence datasets. In this paper, we review the state-of-the-art visual analytics approaches, characterize them with our proposed design space, and categorize them based on analytical tasks and applications. From our review of relevant literature, we have also identified several remaining research challenges and future research opportunities.

Interpretable Anomaly Detection in Event Sequences via Sequence Matching and Visual Comparison.

Anomaly detection is a common analytical task that aims to identify rare cases that differ from the typical cases that make up the majority of a dataset. When analyzing event sequence data, the task of anomaly detection can be complex because the sequential and temporal nature of such data results in diverse definitions and flexible forms of anomalies. This, in turn, increases the difficulty in interpreting detected anomalies. In this article, we propose a visual analytic approach for detecting anomalous sequences in an event sequence dataset via an unsupervised anomaly detection algorithm based on Variational AutoEncoders. We further compare the anomalous sequences with their reconstructions and with the normal sequences through a sequence matching algorithm to identify event anomalies. A visual analytics system is developed to support interactive exploration and interpretations of anomalies through novel visualization designs that facilitate the comparison between anomalous sequences and normal sequences. Finally, we quantitatively evaluate the performance of our anomaly detection algorithm, demonstrate the effectiveness of our system through case studies, and report feedback collected from study participants.

Selection-Bias-Corrected Visualization via Dynamic Reweighting.

The collection and visual analysis of large-scale data from complex systems, such as electronic health records or clickstream data, has become increasingly common across a wide range of industries. This type of retrospective visual analysis, however, is prone to a variety of selection bias effects, especially for high-dimensional data where only a subset of dimensions is visualized at any given time. The risk of selection bias is even higher when analysts dynamically apply filters or perform grouping operations during ad hoc analyses. These bias effects threaten the validity and generalizability of insights discovered during visual analysis as the basis for decision making. Past work has focused on bias transparency, helping users understand when selection bias may have occurred. However, countering the effects of selection bias via bias mitigation is typically left for the user to accomplish as a separate process. Dynamic reweighting (DR) is a novel computational approach to selection bias mitigation that helps users craft bias-corrected visualizations. This paper describes the DR workflow, introduces key DR visualization designs, and presents statistical methods that support the DR process. Use cases from the medical domain, as well as findings from domain expert user interviews, are also reported.

Analysing EHR navigation patterns and digital workflows among physicians during ICU pre-rounds.

BACKGROUND: Some physicians in intensive care units (ICUs) report that electronic health records (EHRs) can be cumbersome and disruptive to workflow. There are significant gaps in our understanding of the physician-EHR interaction. OBJECTIVE: To better understand how clinicians use the EHR for chart review during ICU pre-rounds through the characterisation and description of screen navigation pathways and workflow patterns. METHOD: We conducted a live, direct observational study of six physician trainees performing electronic chart review during daily pre-rounds in the 30-bed medical ICU at a large academic medical centre in the Southeastern United States. A tailored checklist was used by observers for data collection. RESULTS: We observed 52 distinct live patient chart review encounters, capturing a total of 2.7 hours of pre-rounding chart review activity by six individual physicians. Physicians reviewed an average of 8.7 patients (range = 5-12), spending a mean of 3:05 minutes per patient (range = 1:34-5:18). On average, physicians visited 6.3 (±3.1) total EHR screens per patient (range = 1-16). Four unique screens were viewed most commonly, accounting for over half (52.7%) of all screen visits: results review (17.9%), summary/overview (13.0%), flowsheet (12.7%), and the chart review tab (9.1%). Navigation pathways were highly variable, but several common screen transition patterns emerged across users. Average interrater reliability for the paired EHR observation was 80.0%. CONCLUSION: We observed the physician-EHR interaction during ICU pre-rounds to be brief and highly focused. Although we observed a high degree of "information sprawl" in physicians' digital navigation, we also identified common launch points for electronic chart review, key high-traffic screens and common screen transition patterns. IMPLICATIONS: From the study findings, we suggest recommendations towards improved EHR design.

Visual Causality Analysis of Event Sequence Data.

Causality is crucial to understanding the mechanisms behind complex systems and making decisions that lead to intended outcomes. Event sequence data is widely collected from many real-world processes, such as electronic health records, web clickstreams, and financial transactions, which transmit a great deal of information reflecting the causal relations among event types. Unfortunately, recovering causalities from observational event sequences is challenging, as the heterogeneous and high-dimensional event variables are often connected to rather complex underlying event excitation mechanisms that are hard to infer from limited observations. Many existing automated causal analysis techniques suffer from poor explainability and fail to include an adequate amount of human knowledge. In this paper, we introduce a visual analytics method for recovering causalities in event sequence data. We extend the Granger causality analysis algorithm on Hawkes processes to incorporate user feedback into causal model refinement. The visualization system includes an interactive causal analysis framework that supports bottom-up causal exploration, iterative causal verification and refinement, and causal comparison through a set of novel visualizations and interactions. We report two forms of evaluation: a quantitative evaluation of the model improvements resulting from the user-feedback mechanism, and a qualitative evaluation through case studies in different application domains to demonstrate the usefulness of the system.

Enabling Longitudinal Exploratory Analysis of Clinical COVID Data.

As the COVID-19 pandemic continues to impact the world, data is being gathered and analyzed to better understand the disease. Recognizing the potential for visual analytics technologies to support exploratory analysis and hypothesis generation from longitudinal clinical data, a team of collaborators worked to apply existing event sequence visual analytics technologies to a longitudinal clinical data from a cohort of 998 patients with high rates of COVID-19 infection. This paper describes the initial steps toward this goal, including: (1) the data transformation and processing work required to prepare the data for visual analysis, (2) initial findings and observations, and (3) qualitative feedback and lessons learned which highlight key features as well as limitations to address in future work.

FacetAtlas: multifaceted visualization for rich text corpora.

Documents in rich text corpora usually contain multiple facets of information. For example, an article about a specific disease often consists of different facets such as symptom, treatment, cause, diagnosis, prognosis, and prevention. Thus, documents may have different relations based on different facets. Powerful search tools have been developed to help users locate lists of individual documents that are most related to specific keywords. However, there is a lack of effective analysis tools that reveal the multifaceted relations of documents within or cross the document clusters. In this paper, we present FacetAtlas, a multifaceted visualization technique for visually analyzing rich text corpora. FacetAtlas combines search technology with advanced visual analytical tools to convey both global and local patterns simultaneously. We describe several unique aspects of FacetAtlas, including (1) node cliques and multifaceted edges, (2) an optimized density map, and (3) automated opacity pattern enhancement for highlighting visual patterns, (4) interactive context switch between facets. In addition, we demonstrate the power of FacetAtlas through a case study that targets patient education in the health care domain. Our evaluation shows the benefits of this work, especially in support of complex multifaceted data analysis.

A rapidly deployed, interactive, online visualization system to support fatality management during the coronavirus disease 2019 (COVID-19) pandemic.

OBJECTIVE: To create an online visualization to support fatality management in North Carolina. MATERIALS AND METHODS: A web application aggregates online datasets for coronavirus disease 2019 (COVID-19) infection rates and morgue utilization. The data are visualized through an interactive, online dashboard. RESULTS: The web application was shared with state and local public health officials across North Carolina. Users could adjust interactive maps and other statistical charts to view live reports of metrics at multiple aggregation levels (eg, county or region). The application also provides access to detailed tabular data for individual facilities. DISCUSSION: Stakeholders found this tool helpful for providing situational awareness of capacity, hotspots, and utilization fluctuations. Timely reporting of facility and county data were key, and future work can help streamline the data collection process. There is potential to generalize the technology to other use cases. CONCLUSIONS: This dashboard facilitates fatality management by visualizing county and regional aggregate statistics in North Carolina.

Selection Bias Tracking and Detailed Subset Comparison for High-Dimensional Data.

The collection of large, complex datasets has become common across a wide variety of domains. Visual analytics tools increasingly play a key role in exploring and answering complex questions about these large datasets. However, many visualizations are not designed to concurrently visualize the large number of dimensions present in complex datasets (e.g. tens of thousands of distinct codes in an electronic health record system). This fact, combined with the ability of many visual analytics systems to enable rapid, ad-hoc specification of groups, or cohorts, of individuals based on a small subset of visualized dimensions, leads to the possibility of introducing selection bias-when the user creates a cohort based on a specified set of dimensions, differences across many other unseen dimensions may also be introduced. These unintended side effects may result in the cohort no longer being representative of the larger population intended to be studied, which can negatively affect the validity of subsequent analyses. We present techniques for selection bias tracking and visualization that can be incorporated into high-dimensional exploratory visual analytics systems, with a focus on medical data with existing data hierarchies. These techniques include: (1) tree-based cohort provenance and visualization, including a user-specified baseline cohort that all other cohorts are compared against, and visual encoding of cohort "drift", which indicates where selection bias may have occurred, and (2) a set of visualizations, including a novel icicle-plot based visualization, to compare in detail the per-dimension differences between the baseline and a user-specified focus cohort. These techniques are integrated into a medical temporal event sequence visual analytics tool. We present example use cases and report findings from domain expert user interviews.

Visual Analysis of High-Dimensional Event Sequence Data via Dynamic Hierarchical Aggregation.

Temporal event data are collected across a broad range of domains, and a variety of visual analytics techniques have been developed to empower analysts working with this form of data. These techniques generally display aggregate statistics computed over sets of event sequences that share common patterns. Such techniques are often hindered, however, by the high-dimensionality of many real-world event sequence datasets which can prevent effective aggregation. A common coping strategy for this challenge is to group event types together prior to visualization, as a pre-process, so that each group can be represented within an analysis as a single event type. However, computing these event groupings as a pre-process also places significant constraints on the analysis. This paper presents a new visual analytics approach for dynamic hierarchical dimension aggregation. The approach leverages a predefined hierarchy of dimensions to computationally quantify the informativeness, with respect to a measure of interest, of alternative levels of grouping within the hierarchy at runtime. This information is then interactively visualized, enabling users to dynamically explore the hierarchy to select the most appropriate level of grouping to use at any individual step within an analysis. Key contributions include an algorithm for interactively determining the most informative set of event groupings for a specific analysis context, and a scented scatter-plus-focus visualization design with an optimization-based layout algorithm that supports interactive hierarchical exploration of alternative event type groupings. We apply these techniques to high-dimensional event sequence data from the medical domain and report findings from domain expert interviews.

Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation.

OBJECTIVE: This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. METHODS: A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. RESULTS: After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. DISCUSSION: When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. CONCLUSION: Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Contextual Visualization.

Unseen information can lead to various "threats to validity" when analyzing complex datasets using visual tools, resulting in potentially biased findings. We enumerate sources of unseen information and argue that a new focus on contextual visualization methods is needed to inform users of these threats and to mitigate their effects.