RESUMEN
OBJECTIVE: Traditional knowledge-based and machine learning diagnostic decision support systems have benefited from integrating the medical domain knowledge encoded in the Unified Medical Language System (UMLS). The emergence of Large Language Models (LLMs) to supplant traditional systems poses questions of the quality and extent of the medical knowledge in the models' internal knowledge representations and the need for external knowledge sources. The objective of this study is three-fold: to probe the diagnosis-related medical knowledge of popular LLMs, to examine the benefit of providing the UMLS knowledge to LLMs (grounding the diagnosis predictions), and to evaluate the correlations between human judgments and the UMLS-based metrics for generations by LLMs. METHODS: We evaluated diagnoses generated by LLMs from consumer health questions and daily care notes in the electronic health records using the ConsumerQA and Problem Summarization datasets. Probing LLMs for the UMLS knowledge was performed by prompting the LLM to complete the diagnosis-related UMLS knowledge paths. Grounding the predictions was examined in an approach that integrated the UMLS graph paths and clinical notes in prompting the LLMs. The results were compared to prompting without the UMLS paths. The final experiments examined the alignment of different evaluation metrics, UMLS-based and non-UMLS, with human expert evaluation. RESULTS: In probing the UMLS knowledge, GPT-3.5 significantly outperformed Llama2 and a simple baseline yielding an F1 score of 10.9% in completing one-hop UMLS paths for a given concept. Grounding diagnosis predictions with the UMLS paths improved the results for both models on both tasks, with the highest improvement (4%) in SapBERT score. There was a weak correlation between the widely used evaluation metrics (ROUGE and SapBERT) and human judgments. CONCLUSION: We found that while popular LLMs contain some medical knowledge in their internal representations, augmentation with the UMLS knowledge provides performance gains around diagnosis generation. The UMLS needs to be tailored for the task to improve the LLMs predictions. Finding evaluation metrics that are aligned with human judgments better than the traditional ROUGE and BERT-based scores remains an open research question.
RESUMEN
In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.
RESUMEN
PURPOSE: Salivary dysfunction is a significant side effect of radiation therapy for head and neck cancer (HNC). Preliminary data suggests that mesenchymal stromal cells (MSCs) can improve salivary function. Whether MSCs from HNC patients who have completed chemoradiation are functionally similar to those from healthy patients is unknown. We performed a pilot clinical study to determine whether bone marrow-derived MSCs [MSC(M)] from HNC patients could be used for the treatment of RT-induced salivary dysfunction. METHODS: An IRB-approved pilot clinical study was undertaken on HNC patients with xerostomia who had completed treatment two or more years prior. Patients underwent iliac crest bone marrow aspirate and MSC(M) were isolated and cultured. Culture-expanded MSC(M) were stimulated with IFNγ and cryopreserved prior to reanimation and profiling for functional markers by flow cytometry and ELISA. MSC(M) were additionally injected into mice with radiation-induced xerostomia and the changes in salivary gland histology and salivary production were examined. RESULTS: A total of six subjects were enrolled. MSC(M) from all subjects were culture expanded to > 20 million cells in a median of 15.5 days (range 8-20 days). Flow cytometry confirmed that cultured cells from HNC patients were MSC(M). Functional flow cytometry demonstrated that these IFNγ-stimulated MSC(M) acquired an immunosuppressive phenotype. IFNγ-stimulated MSC(M) from HNC patients were found to express GDNF, WNT1, and R-spondin 1 as well as pro-angiogenesis and immunomodulatory cytokines. In mice, IFNγ-stimulated MSC(M) injection after radiation decreased the loss of acinar cells, decreased the formation of fibrosis, and increased salivary production. CONCLUSIONS: MSC (M) from previously treated HNC patients can be expanded for auto-transplantation and are functionally active. Furthermore IFNγ-stimulated MSC(M) express proteins implicated in salivary gland regeneration. This study provides preliminary data supporting the feasibility of using autologous MSC(M) from HNC patients to treat RT-induced salivary dysfunction.