Assessing Open-Ended Human-Computer Collaboration Systems: Applying a Hallmarks Approach.

There is a growing desire to create computer systems that can collaborate with humans on complex, open-ended activities. These activities typically have no set completion criteria and frequently involve multimodal communication, extensive world knowledge, creativity, and building structures or compositions through multiple steps. Because these systems differ from question and answer (Q&A) systems, chatbots, and simple task-oriented assistants, new methods for evaluating such collaborative computer systems are needed. Here, we present a set of criteria for evaluating these systems, called Hallmarks of Human-Machine Collaboration. The Hallmarks build on the success of heuristic evaluation used by the user interface community and past evaluation techniques used in the spoken language and chatbot communities. They consist of observable characteristics indicative of successful collaborative communication, grouped into eight high-level properties: robustness; habitability; mutual contribution of meaningful content; context-awareness; consistent human engagement; provision of rationale; use of elementary concepts to teach and learn new concepts; and successful collaboration. We present examples of how we used these Hallmarks in the DARPA Communicating with Computers (CwC) program to evaluate diverse activities, including story and music generation, interactive building with blocks, and exploration of molecular mechanisms in cancer. We used the Hallmarks as guides for developers and as diagnostics, assessing systems with the Hallmarks to identify strengths and opportunities for improvement using logs from user studies, surveying the human partner, third-party review of creative products, and direct tests. Informal feedback from CwC technology developers indicates that the use of the Hallmarks for program evaluation helped guide development. The Hallmarks also made it possible to identify areas of progress and major gaps in developing systems where the machine is an equal, creative partner.

Pathogen metadata platform: software for accessing and analyzing pathogen strain information.

BACKGROUND: Pathogen metadata includes information about where and when a pathogen was collected and the type of environment it came from. Along with genomic nucleotide sequence data, this metadata is growing rapidly and becoming a valuable resource not only for research but for biosurveillance and public health. However, current freely available tools for analyzing this data are geared towards bioinformaticians and/or do not provide summaries and visualizations needed to readily interpret results. RESULTS: We designed a platform to easily access and summarize data about pathogen samples. The software includes a PostgreSQL database that captures metadata useful for disease outbreak investigations, and scripts for downloading and parsing data from NCBI BioSample and BioProject into the database. The software provides a user interface to query metadata and obtain standardized results in an exportable, tab-delimited format. To visually summarize results, the user interface provides a 2D histogram for user-selected metadata types and mapping of geolocated entries. The software is built on the LabKey data platform, an open-source data management platform, which enables developers to add functionalities. We demonstrate the use of the software in querying for a pathogen serovar and for genome sequence identifiers. CONCLUSIONS: This software enables users to create a local database for pathogen metadata, populate it with data from NCBI, easily query the data, and obtain visual summaries. Some of the components, such as the database, are modular and can be incorporated into other data platforms. The source code is freely available for download at https://github.com/wchangmitre/bioattribution .

Metabolic modeling predicts metabolite changes in Mycobacterium tuberculosis.

BACKGROUND: Mycobacterium tuberculosis (MTB) is the causal agent of the disease tuberculosis (TB). Metabolic adaptations are thought to be critical to the survival of MTB during pathogenesis. Computational tools that can be used to study MTB metabolism in silico and prioritize resource-intensive experimental work could significantly accelerate research. RESULTS: We have developed E-Flux-MFC, an enhancement of our original E-Flux method that enables the prediction of changes in the production of external and internal metabolites corresponding to gene expression measurements. We have used this method to simulate the changes in the metabolic state of Mycobacterium tuberculosis (MTB). We have validated the accuracy of E-Flux-MFC for predicting changes in lipids and metabolites during a hypoxia time course using previously published metabolomics and transcriptomics data. We have further validated the accuracy of the method for predicting changes in MTB lipids following the deletion and induction of two well-studied transcription factors (TFs). We have applied the method to predict the metabolic impact of the induction of each of the approximately 180 MTB TFs using a previously generated and publically available expression data set. CONCLUSIONS: E-flux-MFC can be used to study global changes in MTB metabolites from gene expression data associated with environmental and genetic perturbations. The application of this method to a data set of MTB TF perturbations provides a resource for studying the large number of TFs whose functions remain unknown. Most TFs impact metabolites indirectly through the propagation of gene expression changes through the regulatory network rather than through their direct regulons. E-Flux-MFC is also applicable to any organism for which accurate metabolic models are available.

The Mycobacterium tuberculosis regulatory network and hypoxia.

We have taken the first steps towards a complete reconstruction of the Mycobacterium tuberculosis regulatory network based on ChIP-Seq and combined this reconstruction with system-wide profiling of messenger RNAs, proteins, metabolites and lipids during hypoxia and re-aeration. Adaptations to hypoxia are thought to have a prominent role in M. tuberculosis pathogenesis. Using ChIP-Seq combined with expression data from the induction of the same factors, we have reconstructed a draft regulatory network based on 50 transcription factors. This network model revealed a direct interconnection between the hypoxic response, lipid catabolism, lipid anabolism and the production of cell wall lipids. As a validation of this model, in response to oxygen availability we observe substantial alterations in lipid content and changes in gene expression and metabolites in corresponding metabolic pathways. The regulatory network reveals transcription factors underlying these changes, allows us to computationally predict expression changes, and indicates that Rv0081 is a regulatory hub.