A critical assessment of using ChatGPT for extracting structured data from clinical notes.

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT's capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral "prompt engineering" process, leveraging OpenAI's API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

Deep Learning-Based H-Score Quantification of Immunohistochemistry-Stained Images.

Immunohistochemistry (IHC) is a well-established and commonly used staining method for clinical diagnosis and biomedical research. In most IHC images, the target protein is conjugated with a specific antibody and stained using diaminobenzidine (DAB), resulting in a brown coloration, whereas hematoxylin serves as a blue counterstain for cell nuclei. The protein expression level is quantified through the H-score, calculated from DAB staining intensity within the target cell region. Traditionally, this process requires evaluation by 2 expert pathologists, which is both time consuming and subjective. To enhance the efficiency and accuracy of this process, we have developed an automatic algorithm for quantifying the H-score of IHC images. To characterize protein expression in specific cell regions, a deep learning model for region recognition was trained based on hematoxylin staining only, achieving pixel accuracy for each class ranging from 0.92 to 0.99. Within the desired area, the algorithm categorizes DAB intensity of each pixel as negative, weak, moderate, or strong staining and calculates the final H-score based on the percentage of each intensity category. Overall, this algorithm takes an IHC image as input and directly outputs the H-score within a few seconds, significantly enhancing the speed of IHC image analysis. This automated tool provides H-score quantification with precision and consistency comparable to experienced pathologists but at a significantly reduced cost during IHC diagnostic workups. It holds significant potential to advance biomedical research reliant on IHC staining for protein expression quantification.

Deep learning of cell spatial organizations identifies clinically relevant insights in tissue images.

Recent advancements in tissue imaging techniques have facilitated the visualization and identification of various cell types within physiological and pathological contexts. Despite the emergence of cell-cell interaction studies, there is a lack of methods for evaluating individual spatial interactions. In this study, we introduce Ceograph, a cell spatial organization-based graph convolutional network designed to analyze cell spatial organization (for example,. the cell spatial distribution, morphology, proximity, and interactions) derived from pathology images. Ceograph identifies key cell spatial organization features by accurately predicting their influence on patient clinical outcomes. In patients with oral potentially malignant disorders, our model highlights reduced structural concordance and increased closeness in epithelial substrata as driving features for an elevated risk of malignant transformation. In lung cancer patients, Ceograph detects elongated tumor nuclei and diminished stroma-stroma closeness as biomarkers for insensitivity to EGFR tyrosine kinase inhibitors. With its potential to predict various clinical outcomes, Ceograph offers a deeper understanding of biological processes and supports the development of personalized therapeutic strategies.

Unsupervised domain adaptation for nuclei segmentation: Adapting from hematoxylin & eosin stained slides to immunohistochemistry stained slides using a curriculum approach.

BACKGROUND AND OBJECTIVE: Unsupervised domain adaptation (UDA) is a powerful approach in tackling domain discrepancies and reducing the burden of laborious and error-prone pixel-level annotations for instance segmentation. However, the domain adaptation strategies utilized in previous instance segmentation models pool all the labeled/detected instances together to train the instance-level GAN discriminator, which neglects the differences among multiple instance categories. Such pooling prevents UDA instance segmentation models from learning categorical correspondence between source and target domains for accurate instance classification; METHODS: To tackle this challenge, we propose an Instance Segmentation CycleGAN (ISC-GAN) algorithm for UDA multiclass-instance segmentation. We conduct extensive experiments on the multiclass nuclei recognition task to transfer knowledge from hematoxylin and eosin to immunohistochemistry stained pathology images. Specifically, we fuse CycleGAN with Mask R-CNN to learn categorical correspondence with image-level domain adaptation and virtual supervision. Moreover, we utilize Curriculum Learning to separate the learning process into two steps: (1) learning segmentation only on labeled source data, and (2) learning target domain segmentation with paired virtual labels generated by ISC-GAN. The performance was further improved through experiments with other strategies, including Shared Weights, Knowledge Distillation, and Expanded Source Data. RESULTS: Comparing to the baseline model or the three UDA instance detection and segmentation models, ISC-GAN illustrates the state-of-the-art performance, with 39.1% average precision and 48.7% average recall. The source codes of ISC-GAN are available at https://github.com/sdw95927/InstanceSegmentation-CycleGAN. CONCLUSION: ISC-GAN adapted knowledge from hematoxylin and eosin to immunohistochemistry stained pathology images, suggesting the potential for reducing the need for large annotated pathological image datasets in deep learning and computer vision tasks.

ScopeViewer: A Browser-Based Solution for Visualizing Spatial Transcriptomics Data.

Motivation: Spatial transcriptomics (ST) enables a high-resolution interrogation of molecular characteristics within specific spatial contexts and tissue morphology. Despite its potential, visualization of ST data is a challenging task due to the complexities in handling, sharing and visualizing large image datasets together with molecular information. Results: We introduce ScopeViewer, a browser-based software designed to overcome these challenges. ScopeViewer offers the following functionalities: (1) It visualizes large image data and associated annotations at various zoom levels, allowing for intricate exploration of the data; (2) It enables dual interactive viewing of the original images along with their annotations, providing a comprehensive understanding of the context; (3) It displays spatial molecular features with optimized bandwidth, ensuring a smooth user experience; and (4) It bolsters data security by circumventing data transfers. Availability: ScopeViewer is available at: https://datacommons.swmed.edu/scopeviewer.

Deep Learning of Cell Spatial Organizations Identifies Clinically Relevant Insights in Tissue Images.

Recent advancements in tissue imaging techniques have facilitated the visualization and identification of various cell types within physiological and pathological contexts. Despite the emergence of cell-cell interaction studies, there is a lack of methods for evaluating individual spatial interactions. In this study, we introduce Ceograph, a novel cell spatial organization-based graph convolutional network designed to analyze cell spatial organization (i.e. the cell spatial distribution, morphology, proximity, and interactions) derived from pathology images. Ceograph identifies key cell spatial organization features by accurately predicting their influence on patient clinical outcomes. In patients with oral potentially malignant disorders, our model highlights reduced structural concordance and increased closeness in epithelial substrata as driving features for an elevated risk of malignant transformation. In lung cancer patients, Ceograph detects elongated tumor nuclei and diminished stroma-stroma closeness as biomarkers for insensitivity to EGFR tyrosine kinase inhibitors. With its potential to predict various clinical outcomes, Ceograph offers a deeper understanding of biological processes and supports the development of personalized therapeutic strategies.

Image-based quantification of histological features as a function of spatial location using the Tissue Positioning System.

BACKGROUND: Tissues such as the liver lobule, kidney nephron, and intestinal gland exhibit intricate patterns of zonated gene expression corresponding to distinct cell types and functions. To quantitatively understand zonation, it is important to measure cellular or genetic features as a function of position along a zonal axis. While it is possible to manually count, characterize, and locate features in relation to the zonal axis, it is labor-intensive and difficult to do manually while maintaining precision and accuracy. METHODS: We addressed this challenge by developing a deep-learning-based quantification method called the "Tissue Positioning System" (TPS), which can automatically analyze zonation in the liver lobule as a model system. FINDINGS: By using algorithms that identified vessels, classified vessels, and segmented zones based on the relative position along the portal vein to central vein axis, TPS was able to spatially quantify gene expression in mice with zone specific reporters. INTERPRETATION: TPS could discern expression differences between zonal reporter strains, ages, and disease states. TPS could also reveal the zonal distribution of cells previously thought to be positioned randomly. The design principles of TPS could be generalized to other tissues to explore the biology of zonation. FUNDING: CPRIT (RP190208, RP220614, RP230330) and NIH (P30CA142543, R01AA028791, R01CA251928, R01DK1253961, R01GM140012, 1R01GM141519, 1R01DE030656, 1U01CA249245). The Pollack Foundation, Simmons Comprehensive Cancer Center Cancer & Obesity Translational Pilot Award, and the Emerging Leader Award from the Mark Foundation For Cancer Research (#21-003-ELA).

Deep-Learning-Based Hepatic Ploidy Quantification Using H&E Histopathology Images.

Polyploidy, the duplication of the entire genome within a single cell, is a significant characteristic of cells in many tissues, including the liver. The quantification of hepatic ploidy typically relies on flow cytometry and immunofluorescence (IF) imaging, which are not widely available in clinical settings due to high financial and time costs. To improve accessibility for clinical samples, we developed a computational algorithm to quantify hepatic ploidy using hematoxylin-eosin (H&E) histopathology images, which are commonly obtained during routine clinical practice. Our algorithm uses a deep learning model to first segment and classify different types of cell nuclei in H&E images. It then determines cellular ploidy based on the relative distance between identified hepatocyte nuclei and determines nuclear ploidy using a fitted Gaussian mixture model. The algorithm can establish the total number of hepatocytes and their detailed ploidy information in a region of interest (ROI) on H&E images. This is the first successful attempt to automate ploidy analysis on H&E images. Our algorithm is expected to serve as an important tool for studying the role of polyploidy in human liver disease.

A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization.

Microscopic examination of pathology slides is essential to disease diagnosis and biomedical research. However, traditional manual examination of tissue slides is laborious and subjective. Tumor whole-slide image (WSI) scanning is becoming part of routine clinical procedures and produces massive data that capture tumor histologic details at high resolution. Furthermore, the rapid development of deep learning algorithms has significantly increased the efficiency and accuracy of pathology image analysis. In light of this progress, digital pathology is fast becoming a powerful tool to assist pathologists. Studying tumor tissue and its surrounding microenvironment provides critical insight into tumor initiation, progression, metastasis, and potential therapeutic targets. Nucleus segmentation and classification are critical to pathology image analysis, especially in characterizing and quantifying the tumor microenvironment (TME). Computational algorithms have been developed for nucleus segmentation and TME quantification within image patches. However, existing algorithms are computationally intensive and time consuming for WSI analysis. This study presents Histology-based Detection using Yolo (HD-Yolo), a new method that significantly accelerates nucleus segmentation and TME quantification. We demonstrate that HD-Yolo outperforms existing WSI analysis methods in nucleus detection, classification accuracy, and computation time. We validated the advantages of the system on 3 different tissue types: lung cancer, liver cancer, and breast cancer. For breast cancer, nucleus features by HD-Yolo were more prognostically significant than both the estrogen receptor status by immunohistochemistry and the progesterone receptor status by immunohistochemistry. The WSI analysis pipeline and a real-time nucleus segmentation viewer are available at https://github.com/impromptuRong/hd_wsi.

Features of tumor-microenvironment images predict targeted therapy survival benefit in patients with EGFR-mutant lung cancer.

Tyrosine kinase inhibitors (TKIs) targeting epidermal growth factor receptor (EGFR) are effective for many patients with lung cancer with EGFR mutations. However, not all patients are responsive to EGFR TKIs, including even those harboring EGFR-sensitizing mutations. In this study, we quantified the cells and cellular interaction features of the tumor microenvironment (TME) using routine H&E-stained biopsy sections. These TME features were used to develop a prediction model for survival benefit from EGFR TKI therapy in patients with lung adenocarcinoma and EGFR-sensitizing mutations in the Lung Cancer Mutation Consortium 1 (LCMC1) and validated in an independent LCMC2 cohort. In the validation data set, EGFR TKI treatment prolonged survival in the predicted-to-benefit group but not in the predicted-not-to-benefit group. Among patients treated with EGFR TKIs, the predicted-to-benefit group had prolonged survival outcomes compared with the predicted not-to-benefit group. The EGFR TKI survival benefit positively correlated with tumor-tumor interaction image features and negatively correlated with tumor-stroma interaction. Moreover, the tumor-stroma interaction was associated with higher activation of the hepatocyte growth factor/MET-mediated PI3K/AKT signaling pathway and epithelial-mesenchymal transition process, supporting the hypothesis of fibroblast-involved resistance to EGFR TKI treatment.

Enhanced Pathology Image Quality with Restore-Generative Adversarial Network.

Whole slide imaging is becoming a routine procedure in clinical diagnosis. Advanced image analysis techniques have been developed to assist pathologists in disease diagnosis, staging, subtype classification, and risk stratification. Recently, deep learning algorithms have achieved state-of-the-art performances in various imaging analysis tasks, including tumor region segmentation, nuclei detection, and disease classification. However, widespread clinical use of these algorithms is hampered by their performances often degrading due to image quality issues commonly seen in real-world pathology imaging data such as low resolution, blurring regions, and staining variation. Restore-Generative Adversarial Network (GAN), a deep learning model, was developed to improve the imaging qualities by restoring blurred regions, enhancing low resolution, and normalizing staining colors. The results demonstrate that Restore-GAN can significantly improve image quality, which leads to improved model robustness and performance for existing deep learning algorithms in pathology image analysis. Restore-GAN has the potential to be used to facilitate the applications of deep learning models in digital pathology analyses.

MB-SupCon: Microbiome-based Predictive Models via Supervised Contrastive Learning.

Human microbiome consists of trillions of microorganisms. Microbiota can modulate the host physiology through molecule and metabolite interactions. Integrating microbiome and metabolomics data have the potential to predict different diseases more accurately. Yet, most datasets only measure microbiome data but without paired metabolome data. Here, we propose a novel integrative modeling framework, Microbiome-based Supervised Contrastive Learning Framework (MB-SupCon). MB-SupCon integrates microbiome and metabolome data to generate microbiome embeddings, which can be used to improve the prediction accuracy in datasets that only measure microbiome data. As a proof of concept, we applied MB-SupCon on 720 samples with paired 16S microbiome data and metabolomics data from patients with type 2 diabetes. MB-SupCon outperformed existing prediction methods and achieved high average prediction accuracies for insulin resistance status (84.62%), sex (78.98%), and race (80.04%). Moreover, the microbiome embeddings form separable clusters for different covariate groups in the lower-dimensional space, which enhances data visualization. We also applied MB-SupCon on a large inflammatory bowel disease study and observed similar advantages. Thus, MB-SupCon could be broadly applicable to improve microbiome prediction models in multi-omics disease studies.

Deep Learning of Rhabdomyosarcoma Pathology Images for Classification and Survival Outcome Prediction.

Rhabdomyosarcoma (RMS), the most common malignant soft tissue tumor in children, has several histologic subtypes that influence treatment and predict patient outcomes. Assistance with histologic classification for pathologists as well as discovery of optimized predictive biomarkers is needed. A convolutional neural network for RMS histology subtype classification was developed using digitized pathology images from 80 patients collected at time of diagnosis. A subsequent embryonal rhabdomyosarcoma (eRMS) prognostic model was also developed in a cohort of 60 eRMS patients. The RMS classification model reached a performance of an area under the receiver operating curve of 0.94 for alveolar rhabdomyosarcoma and an area under the receiver operating curve of 0.92 for eRMS at slide level in the test data set (n = 192). The eRMS prognosis model separated the patients into predicted high- and low-risk groups with significantly different event-free survival outcome (likelihood ratio test; P = 0.02) in the test data set (n = 136). The predicted risk group is significantly associated with patient event-free survival outcome after adjusting for patient age and sex (predicted high- versus low-risk group hazard ratio, 4.64; 95% CI, 1.05-20.57; P = 0.04). This is the first comprehensive study to develop computational algorithms for subtype classification and prognosis prediction for RMS histopathology images. Such models can aid pathology evaluation and provide additional parameters for risk stratification.

MB-GAN: Microbiome Simulation via Generative Adversarial Network.

BACKGROUND: Trillions of microbes inhabit the human body and have a profound effect on human health. The recent development of metagenome-wide association studies and other quantitative analysis methods accelerate the discovery of the associations between human microbiome and diseases. To assess the strengths and limitations of these analytical tools, simulating realistic microbiome datasets is critically important. However, simulating the real microbiome data is challenging because it is difficult to model their correlation structure using explicit statistical models. RESULTS: To address the challenge of simulating realistic microbiome data, we designed a novel simulation framework termed MB-GAN, by using a generative adversarial network (GAN) and utilizing methodology advancements from the deep learning community. MB-GAN can automatically learn from given microbial abundances and compute simulated abundances that are indistinguishable from them. In practice, MB-GAN showed the following advantages. First, MB-GAN avoids explicit statistical modeling assumptions, and it only requires real datasets as inputs. Second, unlike the traditional GANs, MB-GAN is easily applicable and can converge efficiently. CONCLUSIONS: By applying MB-GAN to a case-control gut microbiome study of 396 samples, we demonstrated that the simulated data and the original data had similar first-order and second-order properties, including sparsity, diversities, and taxa-taxa correlations. These advantages are suitable for further microbiome methodology development where high-fidelity microbiome data are needed.

A deep learning-based model for screening and staging pneumoconiosis.

This study aims to develop an artificial intelligence (AI)-based model to assist radiologists in pneumoconiosis screening and staging using chest radiographs. The model, based on chest radiographs, was developed using a training cohort and validated using an independent test cohort. Every image in the training and test datasets were labeled by experienced radiologists in a double-blinded fashion. The computational model started by segmenting the lung field into six subregions. Then, convolutional neural network classification model was used to predict the opacity level for each subregion respectively. Finally, the diagnosis for each subject (normal, stage I, II, or III pneumoconiosis) was determined by summarizing the subregion-based prediction results. For the independent test cohort, pneumoconiosis screening accuracy was 0.973, with both sensitivity and specificity greater than 0.97. The accuracy for pneumoconiosis staging was 0.927, better than that achieved by two groups of radiologists (0.87 and 0.84, respectively). This study develops a deep learning-based model for screening and staging of pneumoconiosis using man-annotated chest radiographs. The model outperformed two groups of radiologists in the accuracy of pneumoconiosis staging. This pioneer work demonstrates the feasibility and efficiency of AI-assisted radiography screening and diagnosis in occupational lung diseases.

Computational Staining of Pathology Images to Study the Tumor Microenvironment in Lung Cancer.

The spatial organization of different types of cells in tumor tissues reveals important information about the tumor microenvironment (TME). To facilitate the study of cellular spatial organization and interactions, we developed Histology-based Digital-Staining, a deep learning-based computation model, to segment the nuclei of tumor, stroma, lymphocyte, macrophage, karyorrhexis, and red blood cells from standard hematoxylin and eosin-stained pathology images in lung adenocarcinoma. Using this tool, we identified and classified cell nuclei and extracted 48 cell spatial organization-related features that characterize the TME. Using these features, we developed a prognostic model from the National Lung Screening Trial dataset, and independently validated the model in The Cancer Genome Atlas lung adenocarcinoma dataset, in which the predicted high-risk group showed significantly worse survival than the low-risk group (P = 0.001), with a HR of 2.23 (1.37-3.65) after adjusting for clinical variables. Furthermore, the image-derived TME features significantly correlated with the gene expression of biological pathways. For example, transcriptional activation of both the T-cell receptor and programmed cell death protein 1 pathways positively correlated with the density of detected lymphocytes in tumor tissues, while expression of the extracellular matrix organization pathway positively correlated with the density of stromal cells. In summary, we demonstrate that the spatial organization of different cell types is predictive of patient survival and associated with the gene expression of biological pathways. SIGNIFICANCE: These findings present a deep learning-based analysis tool to study the TME in pathology images and demonstrate that the cell spatial organization is predictive of patient survival and is associated with gene expression.See related commentary by Rodriguez-Antolin, p. 1912.

Artificial Intelligence in Lung Cancer Pathology Image Analysis.

OBJECTIVE: Accurate diagnosis and prognosis are essential in lung cancer treatment selection and planning. With the rapid advance of medical imaging technology, whole slide imaging (WSI) in pathology is becoming a routine clinical procedure. An interplay of needs and challenges exists for computer-aided diagnosis based on accurate and efficient analysis of pathology images. Recently, artificial intelligence, especially deep learning, has shown great potential in pathology image analysis tasks such as tumor region identification, prognosis prediction, tumor microenvironment characterization, and metastasis detection. MATERIALS AND METHODS: In this review, we aim to provide an overview of current and potential applications for AI methods in pathology image analysis, with an emphasis on lung cancer. RESULTS: We outlined the current challenges and opportunities in lung cancer pathology image analysis, discussed the recent deep learning developments that could potentially impact digital pathology in lung cancer, and summarized the existing applications of deep learning algorithms in lung cancer diagnosis and prognosis. DISCUSSION AND CONCLUSION: With the advance of technology, digital pathology could have great potential impacts in lung cancer patient care. We point out some promising future directions for lung cancer pathology image analysis, including multi-task learning, transfer learning, and model interpretation.

Pathology Image Analysis Using Segmentation Deep Learning Algorithms.

With the rapid development of image scanning techniques and visualization software, whole slide imaging (WSI) is becoming a routine diagnostic method. Accelerating clinical diagnosis from pathology images and automating image analysis efficiently and accurately remain significant challenges. Recently, deep learning algorithms have shown great promise in pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many machine learning algorithms, including convolutional neural networks, have been proposed to automatically segment pathology images. Among these algorithms, segmentation deep learning algorithms such as fully convolutional networks stand out for their accuracy, computational efficiency, and generalizability. Thus, deep learning-based pathology image segmentation has become an important tool in WSI analysis. In this review, the pathology image segmentation process using deep learning algorithms is described in detail. The goals are to provide quick guidance for implementing deep learning into pathology image analysis and to provide some potential ways of further improving segmentation performance. Although there have been previous reviews on using machine learning methods in digital pathology image analysis, this is the first in-depth review of the applications of deep learning algorithms for segmentation in WSI analysis.

Skin Lesion Segmentation with C-UNet.

Malignant melanoma is one of the leading cancers around the world. It is critical to timely diagnose and treat melanoma to improve patient survival. This paper proposes a deep learning model C-UNet for skin lesion segmentation. The C-UNet incorporates the Inception-like convolutional block, the recurrent convolutional block and dilated convolutional layers. We also apply a finetune technique using Dice loss after training the model with commonly used cross-entropy loss. The conditional random field was used to further smooth predicted label maps. Experiment results show that the proposed method achieves better accuracy and more robust segmentation results than UNet.

Disease correlation network: a computational package for identifying temporal correlations between disease states from Large-Scale longitudinal medical records.

OBJECTIVE: To provide an open-source software package for determining temporal correlations between disease states using longitudinal electronic medical records (EMR). MATERIALS AND METHODS: We have developed an R-based package, Disease Correlation Network (DCN), which builds retrospective matched cohorts from longitudinal medical records to assess for significant temporal correlations between diseases using two independent methodologies: Cox proportional hazards regression and random forest survival analysis. This optimizable package has the potential to control for relevant confounding factors such as age, gender, and other demographic and medical characteristics. Output is presented as a DCN which may be analyzed using a JavaScript-based interactive visualization tool for users to explore statistically significant correlations between disease states of interest using graph-theory-based network topology. RESULTS: We have applied this package to a longitudinal dataset at Loyola University Chicago Medical Center with 654 084 distinct initial diagnoses of 51 conditions in 175 539 patients. Over 90% of disease correlations identified are supported by literature review. DCN is available for download at https://github.com/qunfengdong/DCN. CONCLUSIONS: DCN allows screening of EMR data to identify potential relationships between chronic disease states. This data may then be used to formulate novel research hypotheses for further characterization of these relationships.