DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record.

OBJECTIVE: Heatlhcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable, and reliable machine learning models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient, safe and high-quality manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher-created models into a widely used electronic medical record system. MATERIALS AND METHODS: We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within electronic medical record software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact. RESULTS: We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating 12 machine learning models trained using electronic medical record data that predict laboratory diagnostic results, triggered by clinician button-clicks in Stanford Health Care's electronic medical record. DISCUSSION: Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. When possible, we recommend using prospectively estimated performance measures during silent trials to make final go decisions for model deployment. CONCLUSION: Machine learning applications in healthcare are extensively researched, but successful translations to the bedside are rare. By describing DEPLOYR, we aim to inform machine learning deployment best practices and help bridge the model implementation gap.

Expression of PD-1 by T Cells in Malignant Glioma Patients Reflects Exhaustion and Activation.

PURPOSE: Glioblastoma (GBM) is the most common primary malignant tumor in the central nervous system. Our recent preclinical work has suggested that PD-1/PD-L1 plays an important immunoregulatory role to limit effective antitumor T-cell responses induced by active immunotherapy. However, little is known about the functional role that PD-1 plays on human T lymphocytes in patients with malignant glioma.Experimental Design: In this study, we examined the immune landscape and function of PD-1 expression by T cells from tumor and peripheral blood in patients with malignant glioma. RESULTS: We found several differences between PD-1+ tumor-infiltrating lymphocytes (TIL) and patient-matched PD-1+ peripheral blood T lymphocytes. Phenotypically, PD-1+ TILs exhibited higher expression of markers of activation and exhaustion than peripheral blood PD-1+ T cells, which instead had increased markers of memory. A comparison of the T-cell receptor variable chain populations revealed decreased diversity in T cells that expressed PD-1, regardless of the location obtained. Functionally, peripheral blood PD-1+ T cells had a significantly increased proliferative capacity upon activation compared with PD-1- T cells. CONCLUSIONS: Our evidence suggests that PD-1 expression in patients with glioma reflects chronically activated effector T cells that display hallmarks of memory and exhaustion depending on its anatomic location. The decreased diversity in PD-1+ T cells suggests that the PD-1-expressing population has a narrower range of cognate antigen targets compared with the PD-1 nonexpression population. This information can be used to inform how we interpret immune responses to PD-1-blocking therapies or other immunotherapies.

NetworkPainter: dynamic intracellular pathway animation in Cytobank.

BACKGROUND: High-throughput technologies such as flow and mass cytometry have the potential to illuminate cellular networks. However, analyzing the data produced by these technologies is challenging. Visualization is needed to help researchers explore this data. RESULTS: We developed a web-based software program, NetworkPainter, to enable researchers to analyze dynamic cytometry data in the context of pathway diagrams. NetworkPainter provides researchers a graphical interface to draw and "paint" pathway diagrams with experimental data, producing animated diagrams which display the activity of each network node at each time point. CONCLUSION: NetworkPainter enables researchers to more fully explore multi-parameter, dynamical cytometry data.

Cytobank: providing an analytics platform for community cytometry data analysis and collaboration.

Cytometry is used extensively in clinical and laboratory settings to diagnose and track cell subsets in blood and tissue. High-throughput, single-cell approaches leveraging cytometry are developed and applied in the computational and systems biology communities by researchers, who seek to improve the diagnosis of human diseases, map the structures of cell signaling networks, and identify new cell types. Data analysis and management present a bottleneck in the flow of knowledge from bench to clinic. Multi-parameter flow and mass cytometry enable identification of signaling profiles of patient cell samples. Currently, this process is manual, requiring hours of work to summarize multi-dimensional data and translate these data for input into other analysis programs. In addition, the increase in the number and size of collaborative cytometry studies as well as the computational complexity of analytical tools require the ability to assemble sufficient and appropriately configured computing capacity on demand. There is a critical need for platforms that can be used by both clinical and basic researchers who routinely rely on cytometry. Recent advances provide a unique opportunity to facilitate collaboration and analysis and management of cytometry data. Specifically, advances in cloud computing and virtualization are enabling efficient use of large computing resources for analysis and backup. An example is Cytobank, a platform that allows researchers to annotate, analyze, and share results along with the underlying single-cell data.

Web-based analysis and publication of flow cytometry experiments.

Cytobank is a Web-based application for storage, analysis, and sharing of flow cytometry experiments. Researchers use a Web browser to log in and use a wide range of tools developed for basic and advanced flow cytometry. In addition to providing access to standard cytometry tools from any computer, Cytobank creates a platform and community for developing new analysis and publication tools. Figure layouts created on Cytobank are designed to allow transparent access to the underlying experiment annotation and data processing steps. Since all flow cytometry files and analysis data are stored on a central server, experiments and figures can be viewed or edited by anyone with the proper permission, from any computer with Internet access. Once a primary researcher has performed the initial analysis of the data, collaborators can engage in experiment analysis and make their own figure layouts using the gated, compensated experiment files. Cytobank is available to the scientific community at http://www.cytobank.org.

WebFlow: a software package for high-throughput analysis of flow cytometry data.

Flow cytometry has emerged as a powerful tool for quantitative, single-cell analysis of both surface markers and intracellular antigens, including phosphoproteins and kinase signaling cascades, with the flexibility to process hundreds of samples in multiwell plate format. Quantitative flow cytometric analysis is being applied in many areas of biology, from the study of immunology in animal models or human patients to high-content drug screening of pharmacologically active compounds. However, these experiments generate thousands of data points per sample, each with multiple measured parameters, leading to data management and analysis challenges. We developed WebFlow (http://webflow.stanford.edu), a web server-based software package to manage, analyze, and visualize data from flow cytometry experiments. WebFlow is accessible via standard web browsers and does not require users to install software on their personal computers. The software enables plate-based annotation of large data sets, which provides the basis for exploratory data analysis tools and rapid visualization of multiple different parameters. These tools include custom user-defined statistics to normalize data to other wells or other channels, as well as interactive, user-selectable heat maps for viewing the underlying single-cell data. The web-based approach of WebFlow allows for sharing of data with collaborators or the general public. WebFlow provides a novel platform for quantitative analysis of flow cytometric data from high-throughput drug screening or disease profiling experiments.

Single-cell profiling identifies aberrant STAT5 activation in myeloid malignancies with specific clinical and biologic correlates.

Progress in understanding the molecular pathogenesis of human myeloproliferative disorders (MPDs) has led to guidelines incorporating genetic assays with histopathology during diagnosis. Advances in flow cytometry have made it possible to simultaneously measure cell type and signaling abnormalities arising as a consequence of genetic pathologies. Using flow cytometry, we observed a specific evoked STAT5 signaling signature in a subset of samples from patients suspected of having juvenile myelomonocytic leukemia (JMML), an aggressive MPD with a challenging clinical presentation during active disease. This signature was a specific feature involving JAK-STAT signaling, suggesting a critical role of this pathway in the biological mechanism of this disorder and indicating potential targets for future therapies.

K-RasG12D expression induces hyperproliferation and aberrant signaling in primary hematopoietic stem/progenitor cells.

Defining how cancer-associated mutations perturb signaling networks in stem/progenitor populations that are integral to tumor formation and maintenance is a fundamental problem with biologic and clinical implications. Point mutations in RAS genes contribute to many cancers, including myeloid malignancies. We investigated the effects of an oncogenic Kras(G12D) allele on phosphorylated signaling molecules in primary c-kit(+) lin(-/low) hematopoietic stem/progenitor cells. Comparison of wild-type and Kras(G12D) c-kit(+) lin(-/low) cells shows that K-Ras(G12D) expression causes hyperproliferation in vivo and results in abnormal levels of phosphorylated STAT5, ERK, and S6 under basal and stimulated conditions. Whereas Kras(G12D) cells demonstrate hyperactive signaling after exposure to granulocyte-macrophage colony-stimulating factor, we unexpectedly observe a paradoxical attenuation of ERK and S6 phosphorylation in response to stem cell factor. These studies provide direct biochemical evidence that cancer stem/progenitor cells remodel signaling networks in response to oncogenic stress and demonstrate that multi-parameter flow cytometry can be used to monitor the effects of targeted therapeutics in vivo. This strategy has broad implications for defining the architecture of signaling networks in primary cancer cells and for implementing stem cell-targeted interventions.

Mapping normal and cancer cell signalling networks: towards single-cell proteomics.

Oncogenesis and tumour progression are supported by alterations in cell signalling. Using flow cytometry, it is now possible to track and analyse signalling events in individual cancer cells. Data from this type of analysis can be used to create a network map of signalling in each cell and to link specific signalling profiles with clinical outcomes. This form of 'single-cell proteomics' can identify pathways that are activated in therapy-resistant cells and can provide biomarkers for cancer diagnosis and for determining patient prognosis.