Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM).

BACKGROUND: Evaluating patients' experiences is essential when incorporating the patients' perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. METHODS: We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. RESULTS: The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. CONCLUSIONS: The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study.

BACKGROUND: Physicians spend approximately half of their time on administrative tasks, which is one of the leading causes of physician burnout and decreased work satisfaction. The implementation of natural language processing-assisted clinical documentation tools may provide a solution. OBJECTIVE: This study investigates the impact of a commercially available Dutch digital scribe system on clinical documentation efficiency and quality. METHODS: Medical students with experience in clinical practice and documentation (n=22) created a total of 430 summaries of mock consultations and recorded the time they spent on this task. The consultations were summarized using 3 methods: manual summaries, fully automated summaries, and automated summaries with manual editing. We then randomly reassigned the summaries and evaluated their quality using a modified version of the Physician Documentation Quality Instrument (PDQI-9). We compared the differences between the 3 methods in descriptive statistics, quantitative text metrics (word count and lexical diversity), the PDQI-9, Recall-Oriented Understudy for Gisting Evaluation scores, and BERTScore. RESULTS: The median time for manual summarization was 202 seconds against 186 seconds for editing an automatic summary. Without editing, the automatic summaries attained a poorer PDQI-9 score than manual summaries (median PDQI-9 score 25 vs 31, P<.001, ANOVA test). Automatic summaries were found to have higher word counts but lower lexical diversity than manual summaries (P<.001, independent t test). The study revealed variable impacts on PDQI-9 scores and summarization time across individuals. Generally, students viewed the digital scribe system as a potentially useful tool, noting its ease of use and time-saving potential, though some criticized the summaries for their greater length and rigid structure. CONCLUSIONS: This study highlights the potential of digital scribes in improving clinical documentation processes by offering a first summary draft for physicians to edit, thereby reducing documentation time without compromising the quality of patient records. Furthermore, digital scribes may be more beneficial to some physicians than to others and could play a role in improving the reusability of clinical documentation. Future studies should focus on the impact and quality of such a system when used by physicians in clinical practice.

Applying natural language processing to patient messages to identify depression concerns in cancer patients.

OBJECTIVE: This study aims to explore and develop tools for early identification of depression concerns among cancer patients by leveraging the novel data source of messages sent through a secure patient portal. MATERIALS AND METHODS: We developed classifiers based on logistic regression (LR), support vector machines (SVMs), and 2 Bidirectional Encoder Representations from Transformers (BERT) models (original and Reddit-pretrained) on 6600 patient messages from a cancer center (2009-2022), annotated by a panel of healthcare professionals. Performance was compared using AUROC scores, and model fairness and explainability were examined. We also examined correlations between model predictions and depression diagnosis and treatment. RESULTS: BERT and RedditBERT attained AUROC scores of 0.88 and 0.86, respectively, compared to 0.79 for LR and 0.83 for SVM. BERT showed bigger differences in performance across sex, race, and ethnicity than RedditBERT. Patients who sent messages classified as concerning had a higher chance of receiving a depression diagnosis, a prescription for antidepressants, or a referral to the psycho-oncologist. Explanations from BERT and RedditBERT differed, with no clear preference from annotators. DISCUSSION: We show the potential of BERT and RedditBERT in identifying depression concerns in messages from cancer patients. Performance disparities across demographic groups highlight the need for careful consideration of potential biases. Further research is needed to address biases, evaluate real-world impacts, and ensure responsible integration into clinical settings. CONCLUSION: This work represents a significant methodological advancement in the early identification of depression concerns among cancer patients. Our work contributes to a route to reduce clinical burden while enhancing overall patient care, leveraging BERT-based models.

Predicting Depression Risk in Patients With Cancer Using Multimodal Data: Algorithm Development Study.

BACKGROUND: Patients with cancer starting systemic treatment programs, such as chemotherapy, often develop depression. A prediction model may assist physicians and health care workers in the early identification of these vulnerable patients. OBJECTIVE: This study aimed to develop a prediction model for depression risk within the first month of cancer treatment. METHODS: We included 16,159 patients diagnosed with cancer starting chemo- or radiotherapy treatment between 2008 and 2021. Machine learning models (eg, least absolute shrinkage and selection operator [LASSO] logistic regression) and natural language processing models (Bidirectional Encoder Representations from Transformers [BERT]) were used to develop multimodal prediction models using both electronic health record data and unstructured text (patient emails and clinician notes). Model performance was assessed in an independent test set (n=5387, 33%) using area under the receiver operating characteristic curve (AUROC), calibration curves, and decision curve analysis to assess initial clinical impact use. RESULTS: Among 16,159 patients, 437 (2.7%) received a depression diagnosis within the first month of treatment. The LASSO logistic regression models based on the structured data (AUROC 0.74, 95% CI 0.71-0.78) and structured data with email classification scores (AUROC 0.74, 95% CI 0.71-0.78) had the best discriminative performance. The BERT models based on clinician notes and structured data with email classification scores had AUROCs around 0.71. The logistic regression model based on email classification scores alone performed poorly (AUROC 0.54, 95% CI 0.52-0.56), and the model based solely on clinician notes had the worst performance (AUROC 0.50, 95% CI 0.49-0.52). Calibration was good for the logistic regression models, whereas the BERT models produced overly extreme risk estimates even after recalibration. There was a small range of decision thresholds for which the best-performing model showed promising clinical effectiveness use. The risks were underestimated for female and Black patients. CONCLUSIONS: The results demonstrated the potential and limitations of machine learning and multimodal models for predicting depression risk in patients with cancer. Future research is needed to further validate these models, refine the outcome label and predictors related to mental health, and address biases across subgroups.

Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review.

Importance: The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow. Objectives: To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle. Evidence Review: PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores. Findings: The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%). Conclusions and Relevance: The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms' quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.

Natural Language Processing Methods to Identify Oncology Patients at High Risk for Acute Care with Clinical Notes.

Clinical notes are an essential component of a health record. This paper evaluates how natural language processing (NLP) can be used to identify the risk of acute care use (ACU) in oncology patients, once chemotherapy starts. Risk prediction using structured health data (SHD) is now standard, but predictions using free-text formats are complex. This paper explores the use of free-text notes for the prediction of ACU in leu of SHD. Deep Learning models were compared to manually engineered language features. Results show that SHD models minimally outperform NLP models; an ℓ1-penalised logistic regression with SHD achieved a C-statistic of 0.748 (95%-CI: 0.735, 0.762), while the same model with language features achieved 0.730 (95%-CI: 0.717, 0.745) and a transformer-based model achieved 0.702 (95%-CI: 0.688, 0.717). This paper shows how language models can be used in clinical applications and underlines how risk bias is different for diverse patient groups, even using only free-text data.

[Making a diagnosis with the aid of the internet in times of large language models]. / Diagnose stellen met online tools.

The internet is an excellent aid in making diagnoses. One can retrieve diagnostic information from a reliable source such as a continuously updated textbook or search specifically for a diagnosis in bibliographic databases such as PubMed. Entry of a patient summary in a general search engine or a large language model such as ChatGPT can suggest differential-diagnoses to the expert user,but one must be conscious of the limitations of current large language models. There seems little room left for the traditional differential-diagnosis generators. Ideally, large language models will be combined with transparent algorithms with which medical data can be retrieved, to create a new generation of diagnostic decision support systems.

Artificial Intelligence-Enabled Analysis of Statin-Related Topics and Sentiments on Social Media.

Importance: Despite compelling evidence that statins are safe, are generally well tolerated, and reduce cardiovascular events, statins are underused even in patients with the highest risk. Social media may provide contemporary insights into public perceptions about statins. Objective: To characterize and classify public perceptions about statins that were gleaned from more than a decade of statin-related discussions on Reddit, a widely used social media platform. Design, Setting, and Participants: This qualitative study analyzed all statin-related discussions on the social media platform that were dated between January 1, 2009, and July 12, 2022. Statin- and cholesterol-focused communities, were identified to create a list of statin-related discussions. An artificial intelligence (AI) pipeline was developed to cluster these discussions into specific topics and overarching thematic groups. The pipeline consisted of a semisupervised natural language processing model (BERT [Bidirectional Encoder Representations from Transformers]), a dimensionality reduction technique, and a clustering algorithm. The sentiment for each discussion was labeled as positive, neutral, or negative using a pretrained BERT model. Exposures: Statin-related posts and comments containing the terms statin and cholesterol. Main Outcomes and Measures: Statin-related topics and thematic groups. Results: A total of 10 233 unique statin-related discussions (961 posts and 9272 comments) from 5188 unique authors were identified. The number of statin-related discussions increased by a mean (SD) of 32.9% (41.1%) per year. A total of 100 discussion topics were identified and were classified into 6 overarching thematic groups: (1) ketogenic diets, diabetes, supplements, and statins; (2) statin adverse effects; (3) statin hesitancy; (4) clinical trial appraisals; (5) pharmaceutical industry bias and statins; and (6) red yeast rice and statins. The sentiment analysis revealed that most discussions had a neutral (66.6%) or negative (30.8%) sentiment. Conclusions and Relevance: Results of this study demonstrated the potential of an AI approach to analyze large, contemporary, publicly available social media data and generate insights into public perceptions about statins. This information may help guide strategies for addressing barriers to statin use and adherence.

Diagnosis Classification in the Emergency Room Using Natural Language Processing.

Diagnosis classification in the emergency room (ER) is a complex task. We developed several natural language processing classification models, looking both at the full classification task of 132 diagnostic categories and at several clinically applicable samples consisting of two diagnoses that are hard to distinguish.

Predicting Depression Risk in Patients with Cancer Using Multimodal Data.

When patients with cancer develop depression, it is often left untreated. We developed a prediction model for depression risk within the first month after starting cancer treatment using machine learning and Natural Language Processing (NLP) models. The LASSO logistic regression model based on structured data performed well, whereas the NLP model based on only clinician notes did poorly. After further validation, prediction models for depression risk could lead to earlier identification and treatment of vulnerable patients, ultimately improving cancer care and treatment adherence.

Picture a data scientist: a call to action for increasing diversity, equity, and inclusion in the age of AI.

The lack of diversity, equity, and inclusion continues to hamper the artificial intelligence (AI) field and is especially problematic for healthcare applications. In this article, we expand on the need for diversity, equity, and inclusion, specifically focusing on the composition of AI teams. We call to action leaders at all levels to make team inclusivity and diversity the centerpieces of AI development, not the afterthought. These recommendations take into consideration mitigation at several levels, including outreach programs at the local level, diversity statements at the academic level, and regulatory steps at the federal level.

The digital scribe in clinical practice: a scoping review and research agenda.

The number of clinician burnouts is increasing and has been linked to a high administrative burden. Automatic speech recognition (ASR) and natural language processing (NLP) techniques may address this issue by creating the possibility of automating clinical documentation with a "digital scribe". We reviewed the current status of the digital scribe in development towards clinical practice and present a scope for future research. We performed a literature search of four scientific databases (Medline, Web of Science, ACL, and Arxiv) and requested several companies that offer digital scribes to provide performance data. We included articles that described the use of models on clinical conversational data, either automatically or manually transcribed, to automate clinical documentation. Of 20 included articles, three described ASR models for clinical conversations. The other 17 articles presented models for entity extraction, classification, or summarization of clinical conversations. Two studies examined the system's clinical validity and usability, while the other 18 studies only assessed their model's technical validity on the specific NLP task. One company provided performance data. The most promising models use context-sensitive word embeddings in combination with attention-based neural networks. However, the studies on digital scribes only focus on technical validity, while companies offering digital scribes do not publish information on any of the research phases. Future research should focus on more extensive reporting, iteratively studying technical validity and clinical validity and usability, and investigating the clinical utility of digital scribes.

Multitracer model for staging cortical amyloid deposition using PET imaging.

OBJECTIVE: To develop and evaluate a model for staging cortical amyloid deposition using PET with high generalizability. METHODS: Three thousand twenty-seven individuals (1,763 cognitively unimpaired [CU], 658 impaired, 467 with Alzheimer disease [AD] dementia, 111 with non-AD dementia, and 28 with missing diagnosis) from 6 cohorts (European Medical Information Framework for AD, Alzheimer's and Family, Alzheimer's Biomarkers in Daily Practice, Amsterdam Dementia Cohort, Open Access Series of Imaging Studies [OASIS]-3, Alzheimer's Disease Neuroimaging Initiative [ADNI]) who underwent amyloid PET were retrospectively included; 1,049 individuals had follow-up scans. With application of dataset-specific cutoffs to global standard uptake value ratio (SUVr) values from 27 regions, single-tracer and pooled multitracer regional rankings were constructed from the frequency of abnormality across 400 CU individuals (100 per tracer). The pooled multitracer ranking was used to create a staging model consisting of 4 clusters of regions because it displayed a high and consistent correlation with each single-tracer ranking. Relationships between amyloid stage, clinical variables, and longitudinal cognitive decline were investigated. RESULTS: SUVr abnormality was most frequently observed in cingulate, followed by orbitofrontal, precuneal, and insular cortices and then the associative, temporal, and occipital regions. Abnormal amyloid levels based on binary global SUVr classification were observed in 1.0%, 5.5%, 17.9%, 90.0%, and 100.0% of individuals in stage 0 to 4, respectively. Baseline stage predicted decline in Mini-Mental State Examination (MMSE) score (ADNI: n = 867, F = 67.37, p < 0.001; OASIS: n = 475, F = 9.12, p < 0.001) and faster progression toward an MMSE score ≤25 (ADNI: n = 787, hazard ratio [HR]stage1 2.00, HRstage2 3.53, HRstage3 4.55, HRstage4 9.91, p < 0.001; OASIS: n = 469, HRstage4 4.80, p < 0.001). CONCLUSION: The pooled multitracer staging model successfully classified the level of amyloid burden in >3,000 individuals across cohorts and radiotracers and detects preglobal amyloid burden and distinct risk profiles of cognitive decline within globally amyloid-positive individuals.

Development and Usability of ADappt: Web-Based Tool to Support Clinicians, Patients, and Caregivers in the Diagnosis of Mild Cognitive Impairment and Alzheimer Disease.

BACKGROUND: As a result of advances in diagnostic testing in the field of Alzheimer disease (AD), patients are diagnosed in earlier stages of the disease, for example, in the stage of mild cognitive impairment (MCI). This poses novel challenges for a clinician during the diagnostic workup with regard to diagnostic testing itself, namely, which tests are to be performed, but also on how to engage patients in this decision and how to communicate test results. As a result, tools to support decision making and improve risk communication could be valuable for clinicians and patients. OBJECTIVE: The aim of this study was to present the design, development, and testing of a Web-based tool for clinicians in a memory clinic setting and to ascertain whether this tool can (1) facilitate the interpretation of biomarker results in individual patients with MCI regarding their risk of progression to dementia, (2) support clinicians in communicating biomarker test results and risks to MCI patients and their caregivers, and (3) support clinicians in a process of shared decision making regarding the diagnostic workup of AD. METHODS: A multiphase mixed-methods approach was used. Phase 1 consisted of a qualitative needs assessment among professionals, patients, and caregivers; phase 2, consisted of an iterative process of development and the design of the tool (ADappt); and phase 3 consisted of a quantitative and qualitative assessment of usability and acceptability of ADappt. Across these phases, co-creation was realized via a user-centered qualitative approach with clinicians, patients, and caregivers. RESULTS: In phase 1, clinicians indicated the need for risk calculation tools and visual aids to communicate test results to patients. Patients and caregivers expressed their needs for more specific information on their risk for developing AD and related consequences. In phase 2, we developed the content and graphical design of ADappt encompassing 3 modules: a risk calculation tool, a risk communication tool including a summary sheet for patients and caregivers, and a conversation starter to support shared decision making regarding the diagnostic workup. In phase 3, ADappt was considered to be clear and user-friendly. CONCLUSIONS: Clinicians in a memory clinic setting can use ADappt, a Web-based tool, developed using multiphase design and co-creation, for support that includes an individually tailored interpretation of biomarker test results, communication of test results and risks to patients and their caregivers, and shared decision making on diagnostic testing.

Disclosure of amyloid positron emission tomography results to individuals without dementia: a systematic review.

BACKGROUND: Disclosure of amyloid positron emission tomography (PET) results to individuals without dementia has become standard practice in secondary prevention trials and also increasingly occurs in clinical practice. However, this is controversial given the current lack of understanding of the predictive value of a PET result at the individual level and absence of disease-modifying treatments. In this study, we systematically reviewed the literature on the disclosure of amyloid PET in cognitively normal (CN) individuals and patients with mild cognitive impairment (MCI) in both research and clinical settings. METHODS: We performed a systematic literature search of four scientific databases. Two independent reviewers screened the identified records and selected relevant articles. Included articles presented either empirical data or theoretical data (i.e. arguments in favor or against amyloid status disclosure). Results from the theoretical data were aggregated and presented per theme. RESULTS: Of the seventeen included studies, eleven reported empirical data and six provided theoretical arguments. There was a large variation in the design of the empirical studies, which were almost exclusively in the context of cognitively normal trial participants, comprising only two prospective cohort studies quantitatively assessing the psychological impact of PET result disclosure which showed a low risk of psychological harm after disclosure. Four studies showed that both professionals and cognitively normal individuals support amyloid PET result disclosure and underlined the need for clear disclosure protocols. From the articles presenting theoretical data, we identified 51 'pro' and 'contra' arguments. Theoretical arguments in favor or against disclosure were quite consistent across population groups and settings. Arguments against disclosure focused on the principle of non-maleficence, whereas its psychological impact and predictive value is unknown. Important arguments in favor of amyloid disclosure are the patients right to know (patient autonomy) and that it enables early future decision making. DISCUSSION: Before amyloid PET result disclosure in individuals without dementia in a research or clinical setting is ready for widespread application, more research is needed about its psychological impact, and its predictive value at an individual level. Finally, communication materials and strategies to support disclosure of amyloid PET results should be further developed and prospectively evaluated.

Association of Amyloid Positron Emission Tomography With Changes in Diagnosis and Patient Treatment in an Unselected Memory Clinic Cohort: The ABIDE Project.

Importance: Previous studies have evaluated the diagnostic effect of amyloid positron emission tomography (PET) in selected research cohorts. However, these research populations do not reflect daily practice, thus hampering clinical implementation of amyloid imaging. Objective: To evaluate the association of amyloid PET with changes in diagnosis, diagnostic confidence, treatment, and patients' experiences in an unselected memory clinic cohort. Design, Setting, and Participants: Amyloid PET using fluoride-18 florbetaben was offered to 866 patients who visited the tertiary memory clinic at the VU University Medical Center between January 2015 and December 2016 as part of their routine diagnostic dementia workup. Of these patients, 476 (55%) were included, 32 (4%) were excluded, and 358 (41%) did not participate. To enrich this sample, 31 patients with mild cognitive impairment from the University Medical Center Utrecht memory clinic were included. For each patient, neurologists determined a preamyloid and postamyloid PET diagnosis that existed of both a clinical syndrome (dementia, mild cognitive impairment, or subjective cognitive decline) and a suspected etiology (Alzheimer disease [AD] or non-AD), with a confidence level ranging from 0% to 100%. In addition, the neurologist determined patient treatment in terms of ancillary investigations, medication, and care. Each patient received a clinical follow-up 1 year after being scanned. Main Outcomes and Measures: Primary outcome measures were post-PET changes in diagnosis, diagnostic confidence, and patient treatment. Results: Of the 507 patients (mean [SD] age, 65 (8) years; 201 women [39%]; mean [SD] Mini-Mental State Examination score, 25 [4]), 164 (32%) had AD dementia, 70 (14%) non-AD dementia, 114 (23%) mild cognitive impairment, and 159 (31%) subjective cognitive decline. Amyloid PET results were positive for 242 patients (48%). The suspected etiology changed for 125 patients (25%) after undergoing amyloid PET, more often due to a negative (82 of 265 [31%]) than a positive (43 of 242 [18%]) PET result (P < .01). Post-PET changes in suspected etiology occurred more frequently in patients older (>65 years) than younger (<65 years) than the typical age at onset of 65 years (74 of 257 [29%] vs 51 of 250 [20%]; P < .05). Mean diagnostic confidence (SD) increased from 80 (13) to 89 (13%) (P < .001). In 123 patients (24%), there was a change in patient treatment post-PET, mostly related to additional investigations and therapy. Conclusions and Relevance: This prospective diagnostic study provides a bridge between validating amyloid PET in a research setting and implementing this diagnostic tool in daily clinical practice. Both amyloid-positive and amyloid-negative results had substantial associations with changes in diagnosis and treatment, both in patients with and without dementia.