An implementation framework to improve the transparency and reproducibility of computational models of infectious diseases.

Computational models of infectious diseases have become valuable tools for research and the public health response against epidemic threats. The reproducibility of computational models has been limited, undermining the scientific process and possibly trust in modeling results and related response strategies, such as vaccination. We translated published reproducibility guidelines from a wide range of scientific disciplines into an implementation framework for improving reproducibility of infectious disease computational models. The framework comprises 22 elements that should be described, grouped into 6 categories: computational environment, analytical software, model description, model implementation, data, and experimental protocol. The framework can be used by scientific communities to develop actionable tools for sharing computational models in a reproducible way.

Utilization and Outcomes of Clinically Indicated Invasive Cardiac Care in Veterans with Acute Coronary Syndrome and Chronic Kidney Disease.

SIGNIFICANCE STATEMENT: Of studies reporting an association of CKD with lower use of invasive cardiac care to treat acute coronary syndrome (ACS), just one accounted for the appropriateness of such care. However, its findings in patients hospitalized nearly 30 years ago may not apply to current practice. In a more recent cohort of 64,695 veterans hospitalized with ACS, CKD was associated with a 32% lower likelihood of receiving invasive care determined to be clinically indicated. Among patients with CKD, not receiving such care was associated with a 1.39-fold higher risk of 6-month mortality. Efforts to elucidate the reasons for this disparity in invasive care in patients with ACS and CKD and implement tailored interventions to enhance its use in this population may offer the potential to improve clinical outcomes. BACKGROUND: Previous studies have shown that patients with CKD are less likely than those without CKD to receive invasive care to treat acute coronary syndrome (ACS). However, few studies have accounted for whether such care was clinically indicated or assessed whether nonuse of such care was associated with adverse health outcomes. METHODS: We conducted a retrospective cohort study of US veterans who were hospitalized at Veterans Affairs Medical Centers from January 2013 through December 2017 and received a discharge diagnosis of ACS. We used multivariable logistic regression to investigate the association of CKD with use of invasive care (coronary angiography, with or without revascularization; coronary artery bypass graft surgery; or both) deemed clinically indicated based on Global Registry of Acute Coronary Events 2.0 risk scores that denoted a 6-month predicted all-cause mortality ≥5%. Using propensity scoring and inverse probability weighting, we examined the association of nonuse of clinically indicated invasive care with 6-month all-cause mortality. RESULTS: Among 34,430 patients with a clinical indication for invasive care, the 18,780 patients with CKD were less likely than the 15,650 without CKD to receive such care (adjusted odds ratio, 0.68; 95% confidence interval, 0.65 to 0.72). Among patients with CKD, nonuse of invasive care was associated with higher risk of 6-month all-cause mortality (absolute risk, 21.5% versus 15.5%; absolute risk difference 6.0%; adjusted risk ratio, 1.39; 95% confidence interval, 1.29 to 1.49). Findings were consistent across multiple sensitivity analyses. CONCLUSIONS: In contemporary practice, veterans with CKD who experience ACS are less likely than those without CKD to receive clinically indicated invasive cardiac care. Nonuse of such care is associated with increased mortality.

Inter-rater reliability of the infectious disease modeling reproducibility checklist (IDMRC) as applied to COVID-19 computational modeling research.

BACKGROUND: Infectious disease computational modeling studies have been widely published during the coronavirus disease 2019 (COVID-19) pandemic, yet they have limited reproducibility. Developed through an iterative testing process with multiple reviewers, the Infectious Disease Modeling Reproducibility Checklist (IDMRC) enumerates the minimal elements necessary to support reproducible infectious disease computational modeling publications. The primary objective of this study was to assess the reliability of the IDMRC and to identify which reproducibility elements were unreported in a sample of COVID-19 computational modeling publications. METHODS: Four reviewers used the IDMRC to assess 46 preprint and peer reviewed COVID-19 modeling studies published between March 13th, 2020, and July 30th, 2020. The inter-rater reliability was evaluated by mean percent agreement and Fleiss' kappa coefficients (κ). Papers were ranked based on the average number of reported reproducibility elements, and average proportion of papers that reported each checklist item were tabulated. RESULTS: Questions related to the computational environment (mean κ = 0.90, range = 0.90-0.90), analytical software (mean κ = 0.74, range = 0.68-0.82), model description (mean κ = 0.71, range = 0.58-0.84), model implementation (mean κ = 0.68, range = 0.39-0.86), and experimental protocol (mean κ = 0.63, range = 0.58-0.69) had moderate or greater (κ > 0.41) inter-rater reliability. Questions related to data had the lowest values (mean κ = 0.37, range = 0.23-0.59). Reviewers ranked similar papers in the upper and lower quartiles based on the proportion of reproducibility elements each paper reported. While over 70% of the publications provided data used in their models, less than 30% provided the model implementation. CONCLUSIONS: The IDMRC is the first comprehensive, quality-assessed tool for guiding researchers in reporting reproducible infectious disease computational modeling studies. The inter-rater reliability assessment found that most scores were characterized by moderate or greater agreement. These results suggest that the IDMRC might be used to provide reliable assessments of the potential for reproducibility of published infectious disease modeling publications. Results of this evaluation identified opportunities for improvement to the model implementation and data questions that can further improve the reliability of the checklist.

A retrospective textual analysis of sexual and reproductive health counseling for adolescent and young adult people with epilepsy of gestational capacity.

Rationale The American Academy of Neurology (AAN) recommends annual sexual and reproductive health (SRH) counseling for all people with epilepsy of gestational capacity (PWEGC). Child neurologists report discussing SRH concerns infrequently with adolescents. Limited research exists regarding documentation of such counseling. METHODS: We retrospectively studied clinical notes using natural language processing to investigate child neurologists' documentation of SRH counseling for adolescent and young adult PWEGC. We segmented notes into sentences and evaluated for references to menstruation, sexual activity, contraception, folic acid, teratogens, and pregnancy. We developed training sets in a labeling application and used machine learning to identify additional counseling instances. We repeated this iteratively until we identified no new relevant sentences. We validated results using external reviewers; after removing sentences reviewers disagreed on (n = 13/400), we calculated Cohen's kappa values between the model and reviewers (>0.98 for all categories). We evaluated labeled texts for each patient per calendar year with descriptive statistics and logistic regression, adjusting for race/ethnicity, age, and teratogen use. RESULTS: Data comprised 971 PWEGC age 13-21 years with 2277 patient-years and 3663 outpatient child neurology notes. Nearly half of patient-years lacked SRH counseling documentation (49.1%). Among all patients, 38.0% never had SRH counseling documented. Documentation was present regarding menstruation in 26.7% of patient-years, folic acid in 25.0%, contraception in 21.9%, pregnancy in 3.5%, teratogens in 3.0%, and sexual activity in 1.8%. Documentation regarding menstruation and contraception was associated with prescription of antiseizure medications that have a higher risk of teratogenic effects (OR = 1.27, p = 0.020, 95% CI = [1.04,1.54]; OR = 1.27, p = 0.027, 95% CI = [1.03,1.58]). Documentation regarding contraception, folic acid, and sexual activity was increased among older patients (OR = 1.28, p < 0.001, 95% CI = [1.21,1.35]; OR = 1.26, p < 0.001, 95% CI = [1.19,1.32]; OR = 1.26, p = 0.004, 95% CI = [1.08,1.47]). Documentation regarding sexual activity was decreased among patients identifying as White/Non-Hispanic (OR = 0.39, p = 0.007, 95% CI = [0.20,0.78]). CONCLUSION: Child neurologists counsel PWEGC on SRH less frequently than recommended by the AAN based on documentation.

Gene coexpression networks reveal novel molecular endotypes in alpha-1 antitrypsin deficiency.

BACKGROUND: Alpha-1 antitrypsin deficiency (AATD) is a genetic condition that causes early onset pulmonary emphysema and airways obstruction. The complete mechanisms via which AATD causes lung disease are not fully understood. To improve our understanding of the pathogenesis of AATD, we investigated gene expression profiles of bronchoalveolar lavage (BAL) and peripheral blood mononuclear cells (PBMCs) in AATD individuals. METHODS: We performed RNA-Seq on RNA extracted from matched BAL and PBMC samples isolated from 89 subjects enrolled in the Genomic Research in Alpha-1 Antitrypsin Deficiency and Sarcoidosis (GRADS) study. Subjects were stratified by genotype and augmentation therapy. Supervised and unsupervised differential gene expression analyses were performed using Weighted Gene Co-expression Network Analysis (WGCNA) to identify gene profiles associated with subjects' clinical variables. The genes in the most significant WGCNA module were used to cluster AATD individuals. Gene validation was performed by NanoString nCounter Gene Expression Assay. RESULT: We observed modest effects of AATD genotype and augmentation therapy on gene expression. When WGCNA was applied to BAL transcriptome, one gene module, ME31 (2312 genes), correlated with the highest number of clinical variables and was functionally enriched with numerous immune T-lymphocyte related pathways. This gene module identified two distinct clusters of AATD individuals with different disease severity and distinct PBMC gene expression patterns. CONCLUSIONS: We successfully identified novel clusters of AATD individuals where severity correlated with increased immune response independent of individuals' genotype and augmentation therapy. These findings may suggest the presence of previously unrecognised disease endotypes in AATD that associate with T-lymphocyte immunity and disease severity.

Transcriptomics of bronchoalveolar lavage cells identifies new molecular endotypes of sarcoidosis.

BACKGROUND: Sarcoidosis is a multisystem granulomatous disease of unknown origin with a variable and often unpredictable course and pattern of organ involvement. In this study we sought to identify specific bronchoalveolar lavage (BAL) cell gene expression patterns indicative of distinct disease phenotypic traits. METHODS: RNA sequencing by Ion Torrent Proton was performed on BAL cells obtained from 215 well-characterised patients with pulmonary sarcoidosis enrolled in the multicentre Genomic Research in Alpha-1 Antitrypsin Deficiency and Sarcoidosis (GRADS) study. Weighted gene co-expression network analysis and nonparametric statistics were used to analyse genome-wide BAL transcriptome. Validation of results was performed using a microarray expression dataset of an independent sarcoidosis cohort (Freiburg, Germany; n=50). RESULTS: Our supervised analysis found associations between distinct transcriptional programmes and major pulmonary phenotypic manifestations of sarcoidosis including T-helper type 1 (Th1) and Th17 pathways associated with hilar lymphadenopathy, transforming growth factor-ß1 (TGFB1) and mechanistic target of rapamycin (MTOR) signalling with parenchymal involvement, and interleukin (IL)-7 and IL-2 with airway involvement. Our unsupervised analysis revealed gene modules that uncovered four potential sarcoidosis endotypes including hilar lymphadenopathy with increased acute T-cell immune response; extraocular organ involvement with PI3K activation pathways; chronic and multiorgan disease with increased immune response pathways; and multiorgan involvement, with increased IL-1 and IL-18 immune and inflammatory responses. We validated the occurrence of these endotypes using gene expression, pulmonary function tests and cell differentials from Freiburg. CONCLUSION: Taken together, our results identify BAL gene expression programmes that characterise major pulmonary sarcoidosis phenotypes and suggest the presence of distinct disease molecular endotypes.

Systematic discovery of the functional impact of somatic genome alterations in individual tumors through tumor-specific causal inference.

Cancer is mainly caused by somatic genome alterations (SGAs). Precision oncology involves identifying and targeting tumor-specific aberrations resulting from causative SGAs. We developed a novel tumor-specific computational framework that finds the likely causative SGAs in an individual tumor and estimates their impact on oncogenic processes, which suggests the disease mechanisms that are acting in that tumor. This information can be used to guide precision oncology. We report a tumor-specific causal inference (TCI) framework, which estimates causative SGAs by modeling causal relationships between SGAs and molecular phenotypes (e.g., transcriptomic, proteomic, or metabolomic changes) within an individual tumor. We applied the TCI algorithm to tumors from The Cancer Genome Atlas (TCGA) and estimated for each tumor the SGAs that causally regulate the differentially expressed genes (DEGs) in that tumor. Overall, TCI identified 634 SGAs that are predicted to cause cancer-related DEGs in a significant number of tumors, including most of the previously known drivers and many novel candidate cancer drivers. The inferred causal relationships are statistically robust and biologically sensible, and multiple lines of experimental evidence support the predicted functional impact of both the well-known and the novel candidate drivers that are predicted by TCI. TCI provides a unified framework that integrates multiple types of SGAs and molecular phenotypes to estimate which genome perturbations are causally influencing one or more molecular/cellular phenotypes in an individual tumor. By identifying major candidate drivers and revealing their functional impact in an individual tumor, TCI sheds light on the disease mechanisms of that tumor, which can serve to advance our basic knowledge of cancer biology and to support precision oncology that provides tailored treatment of individual tumors.

Testing the face validity and inter-rater agreement of a simple approach to drug-drug interaction evidence assessment.

Low concordance between drug-drug interaction (DDI) knowledge bases is a well-documented concern. One potential cause of inconsistency is variability between drug experts in approach to assessing evidence about potential DDIs. In this study, we examined the face validity and inter-rater reliability of a novel DDI evidence evaluation instrument designed to be simple and easy to use. METHODS: A convenience sample of participants with professional experience evaluating DDI evidence was recruited. Participants independently evaluated pre-selected evidence items for 5 drug pairs using the new instrument. For each drug pair, participants labeled each evidence item as sufficient or insufficient to establish the existence of a DDI based on the evidence categories provided by the instrument. Participants also decided if the overall body of evidence supported a DDI involving the drug pair. Agreement was computed both at the evidence item and drug pair levels. A cut-off of ≥ 70% was chosen as the agreement threshold for percent agreement, while a coefficient > 0.6 was used as the cut-off for chance-corrected agreement. Open ended comments were collected and coded to identify themes related to the participants' experience using the novel approach. RESULTS: The face validity of the new instrument was established by two rounds of evaluation involving a total of 6 experts. Fifteen experts agreed to participate in the reliability assessment, and 14 completed the study. Participant agreement on the sufficiency of 22 of the 34 evidence items (65%) did not exceed the a priori agreement threshold. Similarly, agreement on the sufficiency of evidence for 3 of the 5 drug pairs (60%) was poor. Chance-corrected agreement at the drug pair level further confirmed the poor interrater reliability of the instrument (Gwet's AC1 = 0.24, Conger's Kappa = 0.24). Participant comments suggested several possible reasons for the disagreements including unaddressed subjectivity in assessing an evidence item's type and study design, an infeasible separation of evidence evaluation from the consideration of clinical relevance, and potential issues related to the evaluation of DDI case reports. CONCLUSIONS: Even though the key findings were negative, the study's results shed light on how experts approach DDI evidence assessment, including the importance situating evidence assessment within the context of consideration of clinical relevance. Analysis of participant comments within the context of the negative findings identified several promising future research directions including: novel computer-based support for evidence assessment; formal evaluation of a more comprehensive evidence assessment approach that requires consideration of specific, explicitly stated, clinical consequences; and more formal investigation of DDI case report assessment instruments.

Leveraging Eye Tracking to Prioritize Relevant Medical Record Data: Comparative Machine Learning Study.

BACKGROUND: Electronic medical record (EMR) systems capture large amounts of data per patient and present that data to physicians with little prioritization. Without prioritization, physicians must mentally identify and collate relevant data, an activity that can lead to cognitive overload. To mitigate cognitive overload, a Learning EMR (LEMR) system prioritizes the display of relevant medical record data. Relevant data are those that are pertinent to a context-defined as the combination of the user, clinical task, and patient case. To determine which data are relevant in a specific context, a LEMR system uses supervised machine learning models of physician information-seeking behavior. Since obtaining information-seeking behavior data via manual annotation is slow and expensive, automatic methods for capturing such data are needed. OBJECTIVE: The goal of the research was to propose and evaluate eye tracking as a high-throughput method to automatically acquire physician information-seeking behavior useful for training models for a LEMR system. METHODS: Critical care medicine physicians reviewed intensive care unit patient cases in an EMR interface developed for the study. Participants manually identified patient data that were relevant in the context of a clinical task: preparing a patient summary to present at morning rounds. We used eye tracking to capture each physician's gaze dwell time on each data item (eg, blood glucose measurements). Manual annotations and gaze dwell times were used to define target variables for developing supervised machine learning models of physician information-seeking behavior. We compared the performance of manual selection and gaze-derived models on an independent set of patient cases. RESULTS: A total of 68 pairs of manual selection and gaze-derived machine learning models were developed from training data and evaluated on an independent evaluation data set. A paired Wilcoxon signed-rank test showed similar performance of manual selection and gaze-derived models on area under the receiver operating characteristic curve (P=.40). CONCLUSIONS: We used eye tracking to automatically capture physician information-seeking behavior and used it to train models for a LEMR system. The models that were trained using eye tracking performed like models that were trained using manual annotations. These results support further development of eye tracking as a high-throughput method for training clinical decision support systems that prioritize the display of relevant medical record data.

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.

BACKGROUND: There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework for the design of user-centered displays of explanations. This new framework served as the basis for qualitative inquiries and design review sessions with critical care nurses and physicians that informed the design of a user-centered explanation display for an ML-based prediction tool. METHODS: We used our framework to propose explanation displays for predictions from a pediatric intensive care unit (PICU) in-hospital mortality risk model. Proposed displays were based on a model-agnostic, instance-level explanation approach based on feature influence, as determined by Shapley values. Focus group sessions solicited critical care provider feedback on the proposed displays, which were then revised accordingly. RESULTS: The proposed displays were perceived as useful tools in assessing model predictions. However, specific explanation goals and information needs varied by clinical role and level of predictive modeling knowledge. Providers preferred explanation displays that required less information processing effort and could support the information needs of a variety of users. Providing supporting information to assist in interpretation was seen as critical for fostering provider understanding and acceptance of the predictions and explanations. The user-centered explanation display for the PICU in-hospital mortality risk model incorporated elements from the initial displays along with enhancements suggested by providers. CONCLUSIONS: We proposed a framework for the design of user-centered displays of explanations for ML models. We used the proposed framework to motivate the design of a user-centered display of an explanation for predictions from a PICU in-hospital mortality risk model. Positive feedback from focus group participants provides preliminary support for the use of model-agnostic, instance-level explanations of feature influence as an approach to understand ML model predictions in healthcare and advances the discussion on how to effectively communicate ML model information to healthcare providers.

A Whole-Genome Analysis Framework for Effective Identification of Pathogenic Regulatory Variants in Mendelian Disease.

The interpretation of non-coding variants still constitutes a major challenge in the application of whole-genome sequencing in Mendelian disease, especially for single-nucleotide and other small non-coding variants. Here we present Genomiser, an analysis framework that is able not only to score the relevance of variation in the non-coding genome, but also to associate regulatory variants to specific Mendelian diseases. Genomiser scores variants through either existing methods such as CADD or a bespoke machine learning method and combines these with allele frequency, regulatory sequences, chromosomal topological domains, and phenotypic relevance to discover variants associated to specific Mendelian disorders. Overall, Genomiser is able to identify causal regulatory variants as the top candidate in 77% of simulated whole genomes, allowing effective detection and discovery of regulatory variants in Mendelian disease.

The triple variable index combines information generated over time from common monitoring variables to identify patients expressing distinct patterns of intraoperative physiology.

BACKGROUND: Mean arterial pressure (MAP), bispectral index (BIS), and minimum alveolar concentration (MAC) represent valuable, yet dynamic intraoperative monitoring variables. They provide information related to poor outcomes when considered together, however their collective behavior across time has not been characterized. METHODS: We have developed the Triple Variable Index (TVI), a composite variable representing the sum of z-scores from MAP, BIS, and MAC values that occur together during surgery. We generated a TVI expression profile, defined as the sequential TVI values expressed across time, for each surgery where concurrent MAP, BIS, and MAC monitoring occurred in an adult patient (≥18 years) at the University of Pittsburgh Medical Center between January and July 2014 (n = 5296). Patterns of TVI expression were identified using k-means clustering and compared across numerous patient, procedure, and outcome characteristics. TVI and the triple low state were compared as prediction models for 30-day postoperative mortality. RESULTS: The median frequency MAP, BIS, and MAC were recorded was one measurement every 3, 5, and 5 min. Three expression patterns were identified: elevated, mixed, and depressed. The elevated pattern displayed the highest average MAP, BIS, and MAC values (86.5 mmHg, 45.3, and 0.98, respectively), while the depressed pattern displayed the lowest values (76.6 mmHg, 38.0, 0.66). Patterns (elevated, mixed, depressed) were distinct across the following characteristics: average patient age (52, 53, 54 years), American Society of Anesthesiologists Physical Status 4 (6.7, 16.1, 27.3%) and 5 (0.1, 0.6, 1.6%) categories, cardiac (2.2, 6.5, 16.1%) and emergent (5.8, 10.5, 12.8%) surgery, cardiopulmonary bypass use (0.3, 2.6, 9.8%), intraoperative medication administration including etomidate (3.0, 7.3, 12.6%), hydromorphone (47.6, 26.3, 25.2%), ketamine (11.2, 4.6, 3.0%), dexmedetomidine (18.4, 16.6, 13.6%), phenylephrine (74.0, 74.8, 83.0), epinephrine (2.0, 6.0, 18.0%), norepinephrine (2.4, 7.5, 21.2%), vasopressin (3.4, 7.6, 21.0%), succinylcholine (74.0, 69.0, 61.9%), intraoperative hypotension (28.8, 33.0, 52.3%) and the triple low state (9.4, 30.3, 80.0%) exposure, and 30-day postoperative mortality (0.8, 2.7, 5.6%). TVI was a better predictor of patients that died or survived in the 30 days following surgery compared to cumulative triple low state exposure (AUC 0.68 versus 0.62, p < 0.05). CONCLUSIONS: Surgeries that share similar patterns of TVI expression display distinct patient, procedure, and outcome characteristics.

Using machine learning to selectively highlight patient information.

BACKGROUND: Electronic medical record (EMR) systems need functionality that decreases cognitive overload by drawing the clinician's attention to the right data, at the right time. We developed a Learning EMR (LEMR) system that learns statistical models of clinician information-seeking behavior and applies those models to direct the display of data in future patients. We evaluated the performance of the system in identifying relevant patient data in intensive care unit (ICU) patient cases. METHODS: To capture information-seeking behavior, we enlisted critical care medicine physicians who reviewed a set of patient cases and selected data items relevant to the task of presenting at morning rounds. Using patient EMR data as predictors, we built machine learning models to predict their relevancy. We prospectively evaluated the predictions of a set of high performing models. RESULTS: On an independent evaluation data set, 25 models achieved precision of 0.52, 95% CI [0.49, 0.54] and recall of 0.77, 95% CI [0.75, 0.80] in identifying relevant patient data items. For data items missed by the system, the reviewers rated the effect of not seeing those data from no impact to minor impact on patient care in about 82% of the cases. CONCLUSION: Data-driven approaches for adaptively displaying data in EMR systems, like the LEMR system, show promise in using information-seeking behavior of clinicians to identify and highlight relevant patient data.

HemOnc: A new standard vocabulary for chemotherapy regimen representation in the OMOP common data model.

Systematic application of observational data to the understanding of impacts of cancer treatments requires detailed information models allowing meaningful comparisons between treatment regimens. Unfortunately, details of systemic therapies are scarce in registries and data warehouses, primarily due to the complex nature of the protocols and a lack of standardization. Since 2011, we have been creating a curated and semi-structured website of chemotherapy regimens, HemOnc.org. In coordination with the Observational Health Data Sciences and Informatics (OHDSI) Oncology Subgroup, we have transformed a substantial subset of this content into the OMOP common data model, with bindings to multiple external vocabularies, e.g., RxNorm and the National Cancer Institute Thesaurus. Currently, there are >73,000 concepts and >177,000 relationships in the full vocabulary. Content related to the definition and composition of chemotherapy regimens has been released within the ATHENA tool (athena.ohdsi.org) for widespread utilization by the OHDSI membership. Here, we describe the rationale, data model, and initial contents of the HemOnc vocabulary along with several use cases for which it may be valuable.

The Monarch Initiative: an integrative data and analytic platform connecting phenotypes to genotypes across species.

The correlation of phenotypic outcomes with genetic variation and environmental factors is a core pursuit in biology and biomedicine. Numerous challenges impede our progress: patient phenotypes may not match known diseases, candidate variants may be in genes that have not been characterized, model organisms may not recapitulate human or veterinary diseases, filling evolutionary gaps is difficult, and many resources must be queried to find potentially significant genotype-phenotype associations. Non-human organisms have proven instrumental in revealing biological mechanisms. Advanced informatics tools can identify phenotypically relevant disease models in research and diagnostic contexts. Large-scale integration of model organism and clinical research data can provide a breadth of knowledge not available from individual sources and can provide contextualization of data back to these sources. The Monarch Initiative (monarchinitiative.org) is a collaborative, open science effort that aims to semantically integrate genotype-phenotype data from many species and sources in order to support precision medicine, disease modeling, and mechanistic exploration. Our integrated knowledge graph, analytic tools, and web services enable diverse users to explore relationships between phenotypes and genotypes across species.

Assessing Ghana's eHealth workforce: implications for planning and training.

BACKGROUND: eHealth-the proficient application of information and communication technology to support healthcare delivery-has been touted as one of the best solutions to address quality and accessibility challenges in healthcare. Although eHealth could be of more value to health systems in low- and middle-income countries (LMICs) where resources are limited, identification of a competent workforce which can develop and maintain eHealth systems is a key barrier to adoption. Very little is known about the actual or optimal states of the eHealth workforce needs of LMICs. The objective of this study was to develop a framework to characterize and assess the eHealth workforce of hospitals in LMICs. METHODS: To characterize and assess the sufficiency of the workforce, we designed this study in twofold. First, we developed a general framework to categorize the eHealth workforce at any LMIC setting. Second, we combined qualitative data, using semi-structured interviews and the Workload Indicator of Staffing Needs (WISN) to assess the sufficiency of the eHealth workforce in selected hospitals in a LMIC setting like Ghana. RESULTS: We surveyed 76 (60%) of the eHealth staff from three hospitals in Ghana-La General Hospital, University of Ghana Hospital, and Greater Accra Regional Hospital. We identified two main eHealth cadres, technical support/information technology (IT) and health information management (HIM). While the HIM cadre presented diversity in expertise, the IT group was dominated by training in Science (42%) and Engineering (55%), and the majority (87%) had at least a bachelor's degree. Health information clerk (32%), health information officer (25%), help desk specialist (20%), and network administrator (11%) were the most dominant roles. Based on the WISN assessment, the eHealth workforce at all the surveyed sites was insufficient. La General and University of Ghana were operating at 10% of required IT staff capacity, while Ridge was short by 42%. CONCLUSIONS: We have developed a framework to characterize and assess the eHealth workforce in LMICs. Applying it to a case study in Ghana has given us a better understanding of potential eHealth staffing needs in LMICs, while providing the quantitative basis for building the requisite human capital to drive eHealth initiatives. Educators can also use our results to explore competency gaps and refine curricula for burgeoning training programs. The findings of this study can serve as a springboard for other LMICs to assess the effects of a well-trained eHealth workforce on the return on eHealth investments.

Physician activity during outpatient visits and subjective workload.

We describe methods for capturing and analyzing EHR use and clinical workflow of physicians during outpatient encounters and relating activity to physicians' self-reported workload. We collected temporally-resolved activity data including audio, video, EHR activity, and eye-gaze along with post-visit assessments of workload. These data are then analyzed through a combination of manual content analysis and computational techniques to temporally align streams, providing a range of process measures of EHR usage, clinical workflow, and physician-patient communication. Data was collected from primary care and specialty clinics at the Veterans Administration San Diego Healthcare System and UCSD Health, who use Electronic Health Record (EHR) platforms, CPRS and Epic, respectively. Grouping visit activity by physician, site, specialty, and patient status enables rank-ordering activity factors by their correlation to physicians' subjective work-load as captured by NASA Task Load Index survey. We developed a coding scheme that enabled us to compare timing studies between CPRS and Epic and extract patient and visit complexity profiles. We identified similar patterns of EHR use and navigation at the 2 sites despite differences in functions, user interfaces and consequent coded representations. Both sites displayed similar proportions of EHR function use and navigation, and distribution of visit length, proportion of time physicians attended to EHRs (gaze), and subjective work-load as measured by the task load survey. We found that visit activity was highly variable across individual physicians, and the observed activity metrics ranged widely as correlates to subjective workload. We discuss implications of our study for methodology, clinical workflow and EHR redesign.

Information needs for making clinical recommendations about potential drug-drug interactions: a synthesis of literature review and interviews.

BACKGROUND: Drug information compendia and drug-drug interaction information databases are critical resources for clinicians and pharmacists working to avoid adverse events due to exposure to potential drug-drug interactions (PDDIs). Our goal is to develop information models, annotated data, and search tools that will facilitate the interpretation of PDDI information. To better understand the information needs and work practices of specialists who search and synthesize PDDI evidence for drug information resources, we conducted an inquiry that combined a thematic analysis of published literature with unstructured interviews. METHODS: Starting from an initial set of relevant articles, we developed search terms and conducted a literature search. Two reviewers conducted a thematic analysis of included articles. Unstructured interviews with drug information experts were conducted and similarly coded. Information needs, work processes, and indicators of potential strengths and weaknesses of information systems were identified. RESULTS: Review of 92 papers and 10 interviews identified 56 categories of information needs related to the interpretation of PDDI information including drug and interaction information; study design; evidence including clinical details, quality and content of reports, and consequences; and potential recommendations. We also identified strengths/weaknesses of PDDI information systems. CONCLUSIONS: We identified the kinds of information that might be most effective for summarizing PDDIs. The drug information experts we interviewed had differing goals, suggesting a need for detailed information models and flexible presentations. Several information needs not discussed in previous work were identified, including temporal overlaps in drug administration, biological plausibility of interactions, and assessment of the quality and content of reports. Richly structured depictions of PDDI information may help drug information experts more effectively interpret data and develop recommendations. Effective information models and system designs will be needed to maximize the utility of this information.

Measuring time utilization of pharmacists in the Birmingham Free Clinic dispensary.

BACKGROUND: Free and charitable clinics are a critical part of America's healthcare safety net. Although informatics tools have the potential to mitigate many of the organizational and service-related challenges facing these clinics, little research attention has been paid to the workflows and potential impact of electronic systems in these settings. In previous work, we performed a qualitative investigation at a free clinic dispensary to identify workflow challenges that may be alleviated through introduction of informatics interventions. However, this earlier study did not quantify the magnitude of these challenges. Time-motion studies offer a precise standard in quantifying healthcare workers' time expenditures on clinical activities, and can provide valuable insight into system specifications. These data, informed by a lean healthcare perspective, provide a quality improvement framework intended to maximize value and eliminate waste in inefficient workflow processes. METHODS: We performed a continuous observation time-motion study in the Birmingham Free Clinic dispensary. Two researchers followed pharmacists over the course of three general clinic sessions and recorded the duration of specific tasks. Pharmacists were then asked to identify tasks as value-added or non-value-added to facilitate calculation of the value quotient, a metric used to determine a workflow's level of efficiency. RESULTS: Four high-level workflow categories occupied almost 95 % of pharmacist time: prescription (Rx) preparation (39.8 %), clinician interaction (21.5 %), EMR operations (14.8 %), and patient interaction (18.7 %). Pharmacists invested the largest portion of time in prescription preparation, with 21.8 % of pharmacist time spent handwriting medication labels. Based on value categorizations made by the pharmacists, the average value quotient was found to be 40.3 %, indicating that pharmacists spend more than half of their time completing tasks they consider to be non-value-added. CONCLUSIONS: Our results show that pharmacists spend a large portion of their time preparing prescriptions, primarily the handwritten labeling of medication bottles and documentation tasks, which is not an optimal utilization of pharmacist expertise. The value quotient further supports that there are many wasteful tasks that may benefit from workflow redesign and health information technology, which could result in efficiency improvements for pharmacists.