A critical assessment of using ChatGPT for extracting structured data from clinical notes.

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT's capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral "prompt engineering" process, leveraging OpenAI's API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

Semi-supervised generative adversarial learning for denoising adaptive optics retinal images.

This study presents denoiseGAN, a novel semi-supervised generative adversarial network, for denoising adaptive optics (AO) retinal images. By leveraging both synthetic and real-world data, denoiseGAN effectively addresses various noise sources, including blur, motion artifacts, and electronic noise, commonly found in AO retinal imaging. Experimental results demonstrate that denoiseGAN outperforms traditional image denoising methods and the state-of-the-art conditional GAN model, preserving retinal cell structures and enhancing image contrast. Moreover, denoiseGAN aids downstream analysis, improving cell segmentation accuracy. Its 30% faster computational efficiency makes it a potential choice for real-time AO image processing in ophthalmology research and clinical practice.

Correction: Validation and applicability of the music ear test on a large Chinese sample.

[This corrects the article DOI: 10.1371/journal.pone.0297073.].

Validation and applicability of the music ear test on a large Chinese sample.

In the context of extensive disciplinary integration, researchers worldwide have increasingly focused on musical ability. However, despite the wide range of available music ability tests, there remains a dearth of validated tests applicable to China. The Music Ear Test (MET) is a validated scale that has been reported to be potentially suitable for cross-cultural distribution in a Chinese sample. However, no formal translation and cross-cultural reliability/validity tests have been conducted for the Chinese population in any of the studies using the Music Ear Test. This study aims to assess the factor structure, convergence, predictiveness, and validity of the Chinese version of the MET, based on a large sample of Chinese participants (n≥1235). Furthermore, we seek to determine whether variables such as music training level, response pattern, and demographic data such as gender and age have intervening effects on the results. In doing so, we aim to provide clear indications of musical aptitude and expertise by validating an existing instrument, the Music Ear Test, and provide a valid method for further understanding the musical abilities of the Chinese sample.

Deep Learning-Based H-Score Quantification of Immunohistochemistry-Stained Images.

Immunohistochemistry (IHC) is a well-established and commonly used staining method for clinical diagnosis and biomedical research. In most IHC images, the target protein is conjugated with a specific antibody and stained using diaminobenzidine (DAB), resulting in a brown coloration, whereas hematoxylin serves as a blue counterstain for cell nuclei. The protein expression level is quantified through the H-score, calculated from DAB staining intensity within the target cell region. Traditionally, this process requires evaluation by 2 expert pathologists, which is both time consuming and subjective. To enhance the efficiency and accuracy of this process, we have developed an automatic algorithm for quantifying the H-score of IHC images. To characterize protein expression in specific cell regions, a deep learning model for region recognition was trained based on hematoxylin staining only, achieving pixel accuracy for each class ranging from 0.92 to 0.99. Within the desired area, the algorithm categorizes DAB intensity of each pixel as negative, weak, moderate, or strong staining and calculates the final H-score based on the percentage of each intensity category. Overall, this algorithm takes an IHC image as input and directly outputs the H-score within a few seconds, significantly enhancing the speed of IHC image analysis. This automated tool provides H-score quantification with precision and consistency comparable to experienced pathologists but at a significantly reduced cost during IHC diagnostic workups. It holds significant potential to advance biomedical research reliant on IHC staining for protein expression quantification.

Deep learning of cell spatial organizations identifies clinically relevant insights in tissue images.

Recent advancements in tissue imaging techniques have facilitated the visualization and identification of various cell types within physiological and pathological contexts. Despite the emergence of cell-cell interaction studies, there is a lack of methods for evaluating individual spatial interactions. In this study, we introduce Ceograph, a cell spatial organization-based graph convolutional network designed to analyze cell spatial organization (for example,. the cell spatial distribution, morphology, proximity, and interactions) derived from pathology images. Ceograph identifies key cell spatial organization features by accurately predicting their influence on patient clinical outcomes. In patients with oral potentially malignant disorders, our model highlights reduced structural concordance and increased closeness in epithelial substrata as driving features for an elevated risk of malignant transformation. In lung cancer patients, Ceograph detects elongated tumor nuclei and diminished stroma-stroma closeness as biomarkers for insensitivity to EGFR tyrosine kinase inhibitors. With its potential to predict various clinical outcomes, Ceograph offers a deeper understanding of biological processes and supports the development of personalized therapeutic strategies.

Reconstructing Spatial Transcriptomics at the Single-cell Resolution with BayesDeep.

Spatially resolved transcriptomics (SRT) techniques have revolutionized the characterization of molecular profiles while preserving spatial and morphological context. However, most next-generation sequencing-based SRT techniques are limited to measuring gene expression in a confined array of spots, capturing only a fraction of the spatial domain. Typically, these spots encompass gene expression from a few to hundreds of cells, underscoring a critical need for more detailed, single-cell resolution SRT data to enhance our understanding of biological functions within the tissue context. Addressing this challenge, we introduce BayesDeep, a novel Bayesian hierarchical model that leverages cellular morphological data from histology images, commonly paired with SRT data, to reconstruct SRT data at the single-cell resolution. BayesDeep effectively model count data from SRT studies via a negative binomial regression model. This model incorporates explanatory variables such as cell types and nuclei-shape information for each cell extracted from the paired histology image. A feature selection scheme is integrated to examine the association between the morphological and molecular profiles, thereby improving the model robustness. We applied BayesDeep to two real SRT datasets, successfully demonstrating its capability to reconstruct SRT data at the single-cell resolution. This advancement not only yields new biological insights but also significantly enhances various downstream analyses, such as pseudotime and cell-cell communication.

Comprehensive characterization of patient-derived xenograft models of pediatric leukemia.

Patient-derived xenografts (PDX) remain valuable models for understanding the biology and for developing novel therapeutics. To expand current PDX models of childhood leukemia, we have developed new PDX models from Hispanic patients, a subgroup with a poorer overall outcome. Of 117 primary leukemia samples obtained, successful engraftment and serial passage in mice were achieved in 82 samples (70%). Hispanic patient samples engrafted at a rate (51/73, 70%) that was similar to non-Hispanic patient samples (31/45, 70%). With a new algorithm to remove mouse contamination in multi-omics datasets including methylation data, we found PDX models faithfully reflected somatic mutations, copy-number alterations, RNA expression, gene fusions, whole-genome methylation patterns, and immunophenotypes found in primary tumor (PT) samples in the first 50 reported here. This cohort of characterized PDX childhood leukemias represents a valuable resource in that germline DNA sequencing has allowed the unambiguous determination of somatic mutations in both PT and PDX.

Depression prediction based on LassoNet-RNN model: A longitudinal study.

Depression has become a widespread health concern today. Understanding the influencing factors can promote human mental health as well as provide a basis for exploring preventive measures. Combining LassoNet with recurrent neural network (RNN), this study constructed a screening model ,LassoNet-RNN, for identifying influencing factors of individual depression. Based on multi-wave surveys of China Health and Retirement Longitudinal Study (CHARLS) dataset (11,661 observations), we analyzed the multivariate time series data and recognized 27 characteristic variables selected from four perspectives: demographics, health-related risk factors, household economic status, and living environment. Additionally, the importance rankings of the characteristic variables were obtained. These results offered insightful recommendations for theoretical developments and practical decision making in public health.

Unsupervised domain adaptation for nuclei segmentation: Adapting from hematoxylin & eosin stained slides to immunohistochemistry stained slides using a curriculum approach.

BACKGROUND AND OBJECTIVE: Unsupervised domain adaptation (UDA) is a powerful approach in tackling domain discrepancies and reducing the burden of laborious and error-prone pixel-level annotations for instance segmentation. However, the domain adaptation strategies utilized in previous instance segmentation models pool all the labeled/detected instances together to train the instance-level GAN discriminator, which neglects the differences among multiple instance categories. Such pooling prevents UDA instance segmentation models from learning categorical correspondence between source and target domains for accurate instance classification; METHODS: To tackle this challenge, we propose an Instance Segmentation CycleGAN (ISC-GAN) algorithm for UDA multiclass-instance segmentation. We conduct extensive experiments on the multiclass nuclei recognition task to transfer knowledge from hematoxylin and eosin to immunohistochemistry stained pathology images. Specifically, we fuse CycleGAN with Mask R-CNN to learn categorical correspondence with image-level domain adaptation and virtual supervision. Moreover, we utilize Curriculum Learning to separate the learning process into two steps: (1) learning segmentation only on labeled source data, and (2) learning target domain segmentation with paired virtual labels generated by ISC-GAN. The performance was further improved through experiments with other strategies, including Shared Weights, Knowledge Distillation, and Expanded Source Data. RESULTS: Comparing to the baseline model or the three UDA instance detection and segmentation models, ISC-GAN illustrates the state-of-the-art performance, with 39.1% average precision and 48.7% average recall. The source codes of ISC-GAN are available at https://github.com/sdw95927/InstanceSegmentation-CycleGAN. CONCLUSION: ISC-GAN adapted knowledge from hematoxylin and eosin to immunohistochemistry stained pathology images, suggesting the potential for reducing the need for large annotated pathological image datasets in deep learning and computer vision tasks.

ScopeViewer: A Browser-Based Solution for Visualizing Spatial Transcriptomics Data.

Motivation: Spatial transcriptomics (ST) enables a high-resolution interrogation of molecular characteristics within specific spatial contexts and tissue morphology. Despite its potential, visualization of ST data is a challenging task due to the complexities in handling, sharing and visualizing large image datasets together with molecular information. Results: We introduce ScopeViewer, a browser-based software designed to overcome these challenges. ScopeViewer offers the following functionalities: (1) It visualizes large image data and associated annotations at various zoom levels, allowing for intricate exploration of the data; (2) It enables dual interactive viewing of the original images along with their annotations, providing a comprehensive understanding of the context; (3) It displays spatial molecular features with optimized bandwidth, ensuring a smooth user experience; and (4) It bolsters data security by circumventing data transfers. Availability: ScopeViewer is available at: https://datacommons.swmed.edu/scopeviewer.

A Deep Learning Onion Peeling Approach to Measure Oral Epithelium Layer Number.

Head and neck squamous cell carcinoma (HNSCC), specifically in the oral cavity (oral squamous cell carcinoma, OSCC), is a common, complex cancer that significantly affects patients' quality of life. Early diagnosis typically improves prognoses yet relies on pathologist examination of histology images that exhibit high inter- and intra-observer variation. The advent of deep learning has automated this analysis, notably with object segmentation. However, techniques for automated oral dysplasia diagnosis have been limited to shape or cell stain information, without addressing the diagnostic potential in counting the number of cell layers in the oral epithelium. Our study attempts to address this gap by combining the existing U-Net and HD-Staining architectures for segmenting the oral epithelium and introducing a novel algorithm that we call Onion Peeling for counting the epithelium layer number. Experimental results show a close correlation between our algorithmic and expert manual layer counts, demonstrating the feasibility of automated layer counting. We also show the clinical relevance of oral epithelial layer number to grading oral dysplasia severity through survival analysis. Overall, our study shows that automated counting of oral epithelium layers can represent a potential addition to the digital pathology toolbox. Model generalizability and accuracy could be improved further with a larger training dataset.

Deep Learning of Cell Spatial Organizations Identifies Clinically Relevant Insights in Tissue Images.

Recent advancements in tissue imaging techniques have facilitated the visualization and identification of various cell types within physiological and pathological contexts. Despite the emergence of cell-cell interaction studies, there is a lack of methods for evaluating individual spatial interactions. In this study, we introduce Ceograph, a novel cell spatial organization-based graph convolutional network designed to analyze cell spatial organization (i.e. the cell spatial distribution, morphology, proximity, and interactions) derived from pathology images. Ceograph identifies key cell spatial organization features by accurately predicting their influence on patient clinical outcomes. In patients with oral potentially malignant disorders, our model highlights reduced structural concordance and increased closeness in epithelial substrata as driving features for an elevated risk of malignant transformation. In lung cancer patients, Ceograph detects elongated tumor nuclei and diminished stroma-stroma closeness as biomarkers for insensitivity to EGFR tyrosine kinase inhibitors. With its potential to predict various clinical outcomes, Ceograph offers a deeper understanding of biological processes and supports the development of personalized therapeutic strategies.

Deep-Learning-Based Hepatic Ploidy Quantification Using H&E Histopathology Images.

Polyploidy, the duplication of the entire genome within a single cell, is a significant characteristic of cells in many tissues, including the liver. The quantification of hepatic ploidy typically relies on flow cytometry and immunofluorescence (IF) imaging, which are not widely available in clinical settings due to high financial and time costs. To improve accessibility for clinical samples, we developed a computational algorithm to quantify hepatic ploidy using hematoxylin-eosin (H&E) histopathology images, which are commonly obtained during routine clinical practice. Our algorithm uses a deep learning model to first segment and classify different types of cell nuclei in H&E images. It then determines cellular ploidy based on the relative distance between identified hepatocyte nuclei and determines nuclear ploidy using a fitted Gaussian mixture model. The algorithm can establish the total number of hepatocytes and their detailed ploidy information in a region of interest (ROI) on H&E images. This is the first successful attempt to automate ploidy analysis on H&E images. Our algorithm is expected to serve as an important tool for studying the role of polyploidy in human liver disease.

A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization.

Microscopic examination of pathology slides is essential to disease diagnosis and biomedical research. However, traditional manual examination of tissue slides is laborious and subjective. Tumor whole-slide image (WSI) scanning is becoming part of routine clinical procedures and produces massive data that capture tumor histologic details at high resolution. Furthermore, the rapid development of deep learning algorithms has significantly increased the efficiency and accuracy of pathology image analysis. In light of this progress, digital pathology is fast becoming a powerful tool to assist pathologists. Studying tumor tissue and its surrounding microenvironment provides critical insight into tumor initiation, progression, metastasis, and potential therapeutic targets. Nucleus segmentation and classification are critical to pathology image analysis, especially in characterizing and quantifying the tumor microenvironment (TME). Computational algorithms have been developed for nucleus segmentation and TME quantification within image patches. However, existing algorithms are computationally intensive and time consuming for WSI analysis. This study presents Histology-based Detection using Yolo (HD-Yolo), a new method that significantly accelerates nucleus segmentation and TME quantification. We demonstrate that HD-Yolo outperforms existing WSI analysis methods in nucleus detection, classification accuracy, and computation time. We validated the advantages of the system on 3 different tissue types: lung cancer, liver cancer, and breast cancer. For breast cancer, nucleus features by HD-Yolo were more prognostically significant than both the estrogen receptor status by immunohistochemistry and the progesterone receptor status by immunohistochemistry. The WSI analysis pipeline and a real-time nucleus segmentation viewer are available at https://github.com/impromptuRong/hd_wsi.

Deep learning in digital pathology for personalized treatment plans of cancer patients.

Over the past decade, many new cancer treatments have been developed and made available to patients. However, in most cases, these treatments only benefit a specific subgroup of patients, making the selection of treatment for a specific patient an essential but challenging task for oncologists. Although some biomarkers were found to associate with treatment response, manual assessment is time-consuming and subjective. With the rapid developments and expanded implementation of artificial intelligence (AI) in digital pathology, many biomarkers can be quantified automatically from histopathology images. This approach allows for a more efficient and objective assessment of biomarkers, aiding oncologists in formulating personalized treatment plans for cancer patients. This review presents an overview and summary of the recent studies on biomarker quantification and treatment response prediction using hematoxylin-eosin (H&E) stained pathology images. These studies have shown that an AI-based digital pathology approach can be practical and will become increasingly important in improving the selection of cancer treatments for patients.

Deep learning-based pathology image analysis predicts cancer progression risk in patients with oral leukoplakia.

BACKGROUND: Oral leukoplakia (OL) is associated with an increased risk for oral cancer (OC) development. Prediction of OL cancer progression may contribute to decreased OC morbidity and mortality by favoring early intervention. Current OL progression risk assessment approaches face large interobserver variability and is weakly prognostic. We hypothesized that convolutional neural networks (CNN)-based histology image analyses could accelerate the discovery of better OC progression risk models. METHODS: Our CNN-based oral mucosa risk stratification model (OMRS) was trained to classify a set of nondysplastic oral mucosa (OM) and a set of OC H&E slides. As a result, the OMRS model could identify abnormal morphological features of the oral epithelium. By applying this model to OL slides, we hypothesized that the extent of OC-like features identified in the OL epithelium would correlate with its progression risk. The OMRS model scored and categorized the OL cohort (n = 62) into high- and low-risk groups. RESULTS: OL patients classified as high-risk (n = 31) were 3.98 (95% CI 1.36-11.7) times more likely to develop OC than low-risk ones (n = 31). Time-to-progression significantly differed between high- and low-risk groups (p = 0.003). The 5-year OC development probability was 21.3% for low-risk and 52.5% for high-risk patients. The predictive power of the OMRS model was sustained even after adjustment for age, OL site, and OL dysplasia grading (HR = 4.52, 1.5-13.7). CONCLUSION: The ORMS model successfully identified OL patients with a high risk of OC development and can potentially benefit OC early diagnosis and prevention policies.

Features of tumor-microenvironment images predict targeted therapy survival benefit in patients with EGFR-mutant lung cancer.

Tyrosine kinase inhibitors (TKIs) targeting epidermal growth factor receptor (EGFR) are effective for many patients with lung cancer with EGFR mutations. However, not all patients are responsive to EGFR TKIs, including even those harboring EGFR-sensitizing mutations. In this study, we quantified the cells and cellular interaction features of the tumor microenvironment (TME) using routine H&E-stained biopsy sections. These TME features were used to develop a prediction model for survival benefit from EGFR TKI therapy in patients with lung adenocarcinoma and EGFR-sensitizing mutations in the Lung Cancer Mutation Consortium 1 (LCMC1) and validated in an independent LCMC2 cohort. In the validation data set, EGFR TKI treatment prolonged survival in the predicted-to-benefit group but not in the predicted-not-to-benefit group. Among patients treated with EGFR TKIs, the predicted-to-benefit group had prolonged survival outcomes compared with the predicted not-to-benefit group. The EGFR TKI survival benefit positively correlated with tumor-tumor interaction image features and negatively correlated with tumor-stroma interaction. Moreover, the tumor-stroma interaction was associated with higher activation of the hepatocyte growth factor/MET-mediated PI3K/AKT signaling pathway and epithelial-mesenchymal transition process, supporting the hypothesis of fibroblast-involved resistance to EGFR TKI treatment.

Enhanced Pathology Image Quality with Restore-Generative Adversarial Network.

Whole slide imaging is becoming a routine procedure in clinical diagnosis. Advanced image analysis techniques have been developed to assist pathologists in disease diagnosis, staging, subtype classification, and risk stratification. Recently, deep learning algorithms have achieved state-of-the-art performances in various imaging analysis tasks, including tumor region segmentation, nuclei detection, and disease classification. However, widespread clinical use of these algorithms is hampered by their performances often degrading due to image quality issues commonly seen in real-world pathology imaging data such as low resolution, blurring regions, and staining variation. Restore-Generative Adversarial Network (GAN), a deep learning model, was developed to improve the imaging qualities by restoring blurred regions, enhancing low resolution, and normalizing staining colors. The results demonstrate that Restore-GAN can significantly improve image quality, which leads to improved model robustness and performance for existing deep learning algorithms in pathology image analysis. Restore-GAN has the potential to be used to facilitate the applications of deep learning models in digital pathology analyses.

Single-cell level deconvolution, convolution, and clustering in spatial transcriptomics by aligning spot level transcriptome to nuclear morphology.

In spot-based spatial transcriptomics, spots that are of the same size and printed at the fixed location cannot precisely capture the actual randomly located single cells, therefore failing to profile the transcriptome at the single-cell level. The current studies primarily focused on enhancing the spot resolution in size via computational imputation or technical improvement, however, they largely overlooked that single-cell resolution, i.e., resolution in cellular or even smaller size, does not equal single-cell level. Using both real and simulated spatial transcriptomics data, we demonstrated that even the high-resolution spatial transcriptomics still has a large number of spots partially covering multiple cells simultaneously, revealing the intrinsic non-single-cell level of spot-based spatial transcriptomics regardless of spot size. To this end, we present STIE, an EM algorithm that aligns the spatial transcriptome to its matched histology image-based nuclear morphology and recovers missing cells from up to ~70% gap area between spots via the nuclear morphological similarity and neighborhood information, thereby achieving the real single-cell level and whole-slide scale deconvolution/convolution and clustering for both low- and high-resolution spots. On both real and simulation spatial transcriptomics data, STIE characterizes the cell-type specific gene expression variation and demonstrates the outperforming concordance with the single-cell RNAseq-derived cell type transcriptomic signatures compared to the other spot- and subspot-level methods. Furthermore, STIE enabled us to gain novel insights that failed to be revealed by the existing methods due to the lack of single-cell level, for instance, lower actual spot resolution than its reported spot size, the additional contribution of cellular morphology to cell typing beyond transcriptome, unbiased evaluation of cell type colocalization, superior power of high-resolution spot in distinguishing nuanced cell types, and spatially resolved cell-cell interactions at the single-cell level other than spot level. The STIE code is publicly available as an R package at https://github.com/zhushijia/STIE.