Automatic ICD-10-CM coding via Lambda-Scaled attention based deep learning model.

The International Classification of Diseases (ICD) serves as a global healthcare administration standard, with one of its editions being ICD-10-CM, an enhanced diagnostic classification system featuring numerous new codes for specific anatomic sites, co-morbidities, and causes. These additions facilitate conveying the complexities of various diseases. Currently, ICD-10 coding is widely adopted worldwide. However, public hospitals in Pakistan have yet to implement it and automate the coding process. In this research, we implemented ICD-10-CM coding for a private database and named it Clinical Pool of Liver Transplant (CPLT). Additionally, we proposed a novel deep learning model called Deep Recurrent-Convolution Neural Network with a lambda-scaled Attention module (DRCNN-ATT) using the CPLT database to achieve automatic ICD-10-CM coding. DRCNN-ATT combines a bi-directional long short-term memory network (bi-LSTM), a multi-scale convolutional neural network (MS-CNN), and a lambda-scaled attention module. Experimental results demonstrate that deep recurrent convolutional neural network (DRCNN) faces attention score vanishing problem with a standard attention module for automatic ICD coding. However, adding a lambda-scaled attention module resolves this issue. We evaluated DRCNN-ATT model using two distinct datasets: a private CPLT dataset and a public MIMIC III top 50 dataset. The results indicate that the DRCNN-ATT model outperformed various baselines by generating 0.862 micro F1 and 0.25 macro F1 scores on CPLT dataset and 0.705 micro F1 and 0.655 macro F1 scores on MIMIC III top 50 dataset. Furthermore, we also deployed our model for automatic ICD-10-CM coding using ngrok and the Flask APIs, which receives input, processes it, and then returns the results.

Psychiatrists' perceptions of conditions and consequences associated with the implementation of open notes: qualitative investigation.

OBJECTIVE: In a growing list of countries, patients are granted access to their clinical notes ("open notes") as part of their online record access. Especially in the field of mental health, open notes remain controversial with some clinicians perceiving open notes as a tool for improving therapeutic outcomes by increasing patient involvement, while others fear that patients might experience psychological distress and perceived stigmatization, particularly when reading clinicians' notes. More research is needed to optimize the benefits and mitigate the risks. METHODS: Using a qualitative research design, we conducted semi-structured interviews with psychiatrists practicing in Germany, to explore what conditions they believe need to be in place to ensure successful implementation of open notes in psychiatric practice as well as expected subsequent changes to their workload and treatment outcomes. Data were analyzed using thematic analysis. RESULTS: We interviewed 18 psychiatrists; interviewees believed four key conditions needed to be in place prior to implementation of open notes including careful consideration of (1) diagnoses and symptom severity, (2) the availability of additional time for writing clinical notes and discussing them with patients, (3) available resources and system compatibility, and (4) legal and data protection aspects. As a result of introducing open notes, interviewees expected changes in documentation, treatment processes, and doctor-physician interaction. While open notes were expected to improve transparency and trust, participants anticipated negative unintended consequences including the risk of deteriorating therapeutic relationships due to note access-related misunderstandings and conflicts. CONCLUSION: Psychiatrists practiced in Germany where open notes have not yet been established as part of the healthcare data infrastructure. Interviewees were supportive of open notes but had some reservations. They found open notes to be generally beneficial but anticipated effects to vary depending on patient characteristics. Clear guidelines for managing access, time constraints, usability, and privacy are crucial. Open notes were perceived to increase transparency and patient involvement but were also believed to raise issues of stigmatization and conflicts.

Moving Biosurveillance Beyond Coded Data Using AI for Symptom Detection From Physician Notes: Retrospective Cohort Study.

BACKGROUND: Real-time surveillance of emerging infectious diseases necessitates a dynamically evolving, computable case definition, which frequently incorporates symptom-related criteria. For symptom detection, both population health monitoring platforms and research initiatives primarily depend on structured data extracted from electronic health records. OBJECTIVE: This study sought to validate and test an artificial intelligence (AI)-based natural language processing (NLP) pipeline for detecting COVID-19 symptoms from physician notes in pediatric patients. We specifically study patients presenting to the emergency department (ED) who can be sentinel cases in an outbreak. METHODS: Subjects in this retrospective cohort study are patients who are 21 years of age and younger, who presented to a pediatric ED at a large academic children's hospital between March 1, 2020, and May 31, 2022. The ED notes for all patients were processed with an NLP pipeline tuned to detect the mention of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria. For a gold standard, 3 subject matter experts labeled 226 ED notes and had strong agreement (F1-score=0.986; positive predictive value [PPV]=0.972; and sensitivity=1.0). F1-score, PPV, and sensitivity were used to compare the performance of both NLP and the International Classification of Diseases, 10th Revision (ICD-10) coding to the gold standard chart review. As a formative use case, variations in symptom patterns were measured across SARS-CoV-2 variant eras. RESULTS: There were 85,678 ED encounters during the study period, including 4% (n=3420) with patients with COVID-19. NLP was more accurate at identifying encounters with patients that had any of the COVID-19 symptoms (F1-score=0.796) than ICD-10 codes (F1-score =0.451). NLP accuracy was higher for positive symptoms (sensitivity=0.930) than ICD-10 (sensitivity=0.300). However, ICD-10 accuracy was higher for negative symptoms (specificity=0.994) than NLP (specificity=0.917). Congestion or runny nose showed the highest accuracy difference (NLP: F1-score=0.828 and ICD-10: F1-score=0.042). For encounters with patients with COVID-19, prevalence estimates of each NLP symptom differed across variant eras. Patients with COVID-19 were more likely to have each NLP symptom detected than patients without this disease. Effect sizes (odds ratios) varied across pandemic eras. CONCLUSIONS: This study establishes the value of AI-based NLP as a highly effective tool for real-time COVID-19 symptom detection in pediatric patients, outperforming traditional ICD-10 methods. It also reveals the evolving nature of symptom prevalence across different virus variants, underscoring the need for dynamic, technology-driven approaches in infectious disease surveillance.

Enhancing post-traumatic stress disorder patient assessment: leveraging natural language processing for research of domain criteria identification using electronic medical records.

BACKGROUND: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities. METHODS: In our study, we created a natural language processing (NLP) workflow to analyze electronic medical record (EMR) data and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, all-mpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories. RESULTS: The sentence transformer model demonstrated high F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. We found that 60.6% of PTSD women have at least one abnormal instance of the six RDoC domains as compared to PTSD men (51.3%), with 45.1% of PTSD women with higher levels of sensorimotor disturbances compared to men (41.3%). We also found that 57.3% of PTSD patients have at least one abnormal instance of the six RDoC domains based on our records. Also, veterans had the higher abnormalities of negative and positive valence systems (60% and 51.9% of veterans respectively) compared to non-veterans (59.1% and 49.2% respectively). The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption. CONCLUSIONS: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes.

Documentation of social determinants of health across individuals from different racial and ethnic groups in home healthcare.

INTRODUCTION: Home healthcare (HHC) enables patients to receive healthcare services within their homes to manage chronic conditions and recover from illnesses. Recent research has identified disparities in HHC based on race or ethnicity. Social determinants of health (SDOH) describe the external factors influencing a patient's health, such as access to care and social support. Individuals from racially or ethnically minoritized communities are known to be disproportionately affected by SDOH. Existing evidence suggests that SDOH are documented in clinical notes. However, no prior study has investigated the documentation of SDOH across individuals from different racial or ethnic backgrounds in the HHC setting. This study aimed to (1) describe frequencies of SDOH documented in clinical notes by race or ethnicity and (2) determine associations between race or ethnicity and SDOH documentation. DESIGN: Retrospective data analysis. METHODS: We conducted a cross-sectional secondary data analysis of 86,866 HHC episodes representing 65,693 unique patients from one large HHC agency in New York collected between January 1, 2015, and December 31, 2017. We reported the frequency of six SDOH (physical environment, social environment, housing and economic circumstances, food insecurity, access to care, and education and literacy) documented in clinical notes across individuals reported as Asian/Pacific Islander, Black, Hispanic, multi-racial, Native American, or White. We analyzed differences in SDOH documentation by race or ethnicity using logistic regression models. RESULTS: Compared to patients reported as White, patients across other racial or ethnic groups had higher frequencies of SDOH documented in their clinical notes. Our results suggest that race or ethnicity is associated with SDOH documentation in HHC. CONCLUSION: As the study of SDOH in HHC continues to evolve, our results provide a foundation to evaluate social information in the HHC setting and understand how it influences the quality of care provided. CLINICAL RELEVANCE: The results of this exploratory study can help clinicians understand the differences in SDOH across individuals from different racial and ethnic groups and serve as a foundation for future research aimed at fostering more inclusive HHC documentation practices.

A phrase-based questionnaire-answering approach for automatic initial frailty assessment based on clinical notes.

Frailty stands out as a particularly challenging multidimensional geriatric syndrome in the elderly population, often resulting in diminished quality of life and heightened mortality risk. Negative consequences encompass a heightened likelihood of hospitalization and institutionalization, as well as suboptimal post-hospitalization outcomes and elevated mortality rates. Using a questionnaire-based approach for assessing frailty has been shown to be an effective method for early diagnosis of frailty. Nonetheless, the majority of current frailty assessment tools necessitate in-person consultations. This poses a significant challenge for elderly patients residing in rural areas, who often encounter difficulties in accessing healthcare compared to their urban or suburban counterparts. Additionally, elderly patients face an elevated risk of contracting diseases as a result of frequent hospital visits, given that many of them are immunocompromised. An automated initial frailty assessment approach can help mitigate the challenges mentioned above and conserve clinical resources by circumventing the need for extensive manual assessments. The primary aim of this paper is to introduce an automatic initial frailty assessment method. This method efficiently identifies individuals who may necessitate further frailty evaluation by automatically extracting relevant information from a patient's clinical notes and using it to complete the Tillburg Frailty Indicator (TFI) questionnaire. The introduced phrase-based query expansion technique is designed to identify the most pertinent phrases related to the frailty assessment questionnaire using Unified Medical Language System (UMLS) ontology and incorporates information from clinical notes to enhance its accuracy. Additionally, a method for retrieving pertinent clinical notes to automatically facilitate the frailty assessment process based on the identified phrases was also proposed. The proposed approaches are evaluated using a dataset containing a collection of clinical notes from elderly patients, assessing their effectiveness in terms of automating frailty assessment and question-answering tasks. This research underscores the significance of incorporating phrases as features in the automated frailty assessment process using clinical notes. The research empowers clinicians to conduct automatic frailty assessments utilizing medical data, thereby reducing the need for frequent hospital visits and in-patient consultations. This becomes particularly valuable during unusual or unexpected situations, such as the COVID-19 pandemic, where minimizing in-person interactions is crucial.

Similarity Analysis of Clinical Notes for Revealing Writing Styles in an Electronic Medical Record System.

BACKGROUND: One of the disadvantages of medical documentation in Electronic Medical Record (EMR) systems is that records tend to be redundant by "copying and pasting", a writing style to duplicate and revise previous records. In this study, we analyzed the similarity between records to identify the factors affecting the writing style of clinical notes. METHOD: We analyzed 98,038 records of 4,149 patients from two years in the Department of Obstetricians and Gynecology at Kyoto University Hospital, Japan. We observed the correlation between the distribution of the record similarity and string amounts, as well as the disease codes and ratios of outpatient visit. RESULTS: The patient group with high record similarity and large number of strings was the group with reproductive medicine, followed by the group of malignant tumor follow-up or Women's Healthcare. DISCUSSION: In reproductive medicine, physicians have a demand for an overarching evaluation, and in follow-up malignancies or in Women's Healthcare, they have a demand to check for subtle differences from the last time. These facts along with our data insist that the writing style in EMR systems is related to the patient's status. CONCLUSION: We declared that the writing style in EMR systems is affected by the patient's status. The writing style of duplicating and revising is preferred (1) when there is a clinical demand for an overarching evaluation, and (2) when there is a clinical demand to check for subtle differences from the last time.

Assessing the Proficiency of LLMs with Various Tasks and Evaluators.

Previous studies have been limited to giving one or two tasks to Large Language Models (LLMs) and involved a small number of evaluators within a single domain to evaluate the LLM's answer. We assessed the proficiency of four LLMs by applying eight tasks and evaluating 32 results with 17 evaluators from diverse domains, demonstrating the significance of various tasks and evaluators on LLMs.

Accelerating Clinical Text Annotation in Underrepresented Languages: A Case Study on Text De-Identification.

Clinical notes contain valuable information for research and monitoring quality of care. Named Entity Recognition (NER) is the process for identifying relevant pieces of information such as diagnoses, treatments, side effects, etc., and bring them to a more structured form. Although recent advancements in deep learning have facilitated automated recognition, particularly in English, NER can still be challenging due to limited specialized training data. This exacerbated in hospital settings where annotations are costly to obtain without appropriate incentives and often dependent on local specificities. In this work, we study whether this annotation process can be effectively accelerated by combining two practical strategies. First, we convert usually passive annotation tasks into a proactive contest to motivate human annotators in performing a task often considered tedious and time-consuming. Second, we provide pre-annotations for the participants to evaluate how recall and precision of the pre-annotations can boost or deteriorate annotation performance. We applied both strategies to a text de-identification task on French clinical notes and discharge summaries at a large Swiss university hospital. Our results show that proactive contest and average quality pre-annotations can significantly speed up annotation time and increase annotation quality, enabling us to develop a text de-identification model for French clinical notes with high performance (F1 score 0.94).

HR-BGCN : Predicting readmission for heart failure from electronic health records.

Heart failure has become a huge public health problem, and failure to accurately predict readmission will further lead to the disease's high cost and high mortality. The construction of readmission prediction model can assist doctors in making decisions to prevent patients from deteriorating and reduce the cost burden. This paper extracts the patient discharge records from the MIMIC-III database. It divides the patients into three research categories: no readmission, readmission within 30 days, and readmission after 30 days, to predict the readmission of patients. We propose the HR-BGCN model to predict the readmission of patients. First, we use the Adaptive-TMix to improve the prediction indicators of a few categories and reduce the impact of unbalanced categories. Then, the knowledge-informed graph attention mechanism is proposed. By introducing a document-level explicit diagram structure, the coding ability of graph node features is significantly improved. The paragraph-level representation obtained through graph learning is combined with the context token-level representation of BERT, and finally, the multi-classification task is carried out. We also compare several typical graph learning classification models to verify the model's effectiveness, such as the IA-GCN model, GAT model, etc. The results show that the average F1 score of the HR-BGCN model proposed in this paper for 30-day readmission of heart failure patients is 88.26%, and the average accuracy is 90.47%. The HR-BGCN model is significantly better than the graph learning classification model for predicting heart failure readmission. It can help doctors predict the 30-day readmission of patients, then reduce the readmission rate of patients.

Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline.

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children's hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring.

Identification of an ANCA-Associated Vasculitis Cohort Using Deep Learning and Electronic Health Records.

Background: ANCA-associated vasculitis (AAV) is a rare but serious disease. Traditional case-identification methods using claims data can be time-intensive and may miss important subgroups. We hypothesized that a deep learning model analyzing electronic health records (EHR) can more accurately identify AAV cases. Methods: We examined the Mass General Brigham (MGB) repository of clinical documentation from 12/1/1979 to 5/11/2021, using expert-curated keywords and ICD codes to identify a large cohort of potential AAV cases. Three labeled datasets (I, II, III) were created, each containing note sections. We trained and evaluated a range of machine learning and deep learning algorithms for note-level classification, using metrics like positive predictive value (PPV), sensitivity, F-score, area under the receiver operating characteristic curve (AUROC), and area under the precision and recall curve (AUPRC). The deep learning model was further evaluated for its ability to classify AAV cases at the patient-level, compared with rule-based algorithms in 2,000 randomly chosen samples. Results: Datasets I, II, and III comprised 6,000, 3,008, and 7,500 note sections, respectively. Deep learning achieved the highest AUROC in all three datasets, with scores of 0.983, 0.991, and 0.991. The deep learning approach also had among the highest PPVs across the three datasets (0.941, 0.954, and 0.800, respectively). In a test cohort of 2,000 cases, the deep learning model achieved a PPV of 0.262 and an estimated sensitivity of 0.975. Compared to the best rule-based algorithm, the deep learning model identified six additional AAV cases, representing 13% of the total. Conclusion: The deep learning model effectively classifies clinical note sections for AAV diagnosis. Its application to EHR notes can potentially uncover additional cases missed by traditional rule-based methods.

Enhancing post-traumatic stress disorder patient assessment: Leveraging Natural Language Processing for Research of Domain Criteria Identification using electronic medical records.

Background: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities. Methods: In our study, we created an NLP workflow to analyze electronic medical record (EMR) data, and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, allmpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories. Results: The sentence transformer model demonstrated superior F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. Women had the highest abnormalities of sensorimotor systems, while veterans had the highest abnormalities of negative and positive valence systems. The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption. Conclusions: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes.

Extraction of sleep information from clinical notes of Alzheimer's disease patients using natural language processing.

OBJECTIVES: Alzheimer's disease (AD) is the most common form of dementia in the United States. Sleep is one of the lifestyle-related factors that has been shown critical for optimal cognitive function in old age. However, there is a lack of research studying the association between sleep and AD incidence. A major bottleneck for conducting such research is that the traditional way to acquire sleep information is time-consuming, inefficient, non-scalable, and limited to patients' subjective experience. We aim to automate the extraction of specific sleep-related patterns, such as snoring, napping, poor sleep quality, daytime sleepiness, night wakings, other sleep problems, and sleep duration, from clinical notes of AD patients. These sleep patterns are hypothesized to play a role in the incidence of AD, providing insight into the relationship between sleep and AD onset and progression. MATERIALS AND METHODS: A gold standard dataset is created from manual annotation of 570 randomly sampled clinical note documents from the adSLEEP, a corpus of 192 000 de-identified clinical notes of 7266 AD patients retrieved from the University of Pittsburgh Medical Center (UPMC). We developed a rule-based natural language processing (NLP) algorithm, machine learning models, and large language model (LLM)-based NLP algorithms to automate the extraction of sleep-related concepts, including snoring, napping, sleep problem, bad sleep quality, daytime sleepiness, night wakings, and sleep duration, from the gold standard dataset. RESULTS: The annotated dataset of 482 patients comprised a predominantly White (89.2%), older adult population with an average age of 84.7 years, where females represented 64.1%, and a vast majority were non-Hispanic or Latino (94.6%). Rule-based NLP algorithm achieved the best performance of F1 across all sleep-related concepts. In terms of positive predictive value (PPV), the rule-based NLP algorithm achieved the highest PPV scores for daytime sleepiness (1.00) and sleep duration (1.00), while the machine learning models had the highest PPV for napping (0.95) and bad sleep quality (0.86), and LLAMA2 with finetuning had the highest PPV for night wakings (0.93) and sleep problem (0.89). DISCUSSION: Although sleep information is infrequently documented in the clinical notes, the proposed rule-based NLP algorithm and LLM-based NLP algorithms still achieved promising results. In comparison, the machine learning-based approaches did not achieve good results, which is due to the small size of sleep information in the training data. CONCLUSION: The results show that the rule-based NLP algorithm consistently achieved the best performance for all sleep concepts. This study focused on the clinical notes of patients with AD but could be extended to general sleep information extraction for other diseases.

Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT.

To enhance phenotype recognition in clinical notes of genetic diseases, we developed two models-PhenoBCBERT and PhenoGPT-for expanding the vocabularies of Human Phenotype Ontology (HPO) terms. While HPO offers a standardized vocabulary for phenotypes, existing tools often fail to capture the full scope of phenotypes due to limitations from traditional heuristic or rule-based approaches. Our models leverage large language models to automate the detection of phenotype terms, including those not in the current HPO. We compare these models with PhenoTagger, another HPO recognition tool, and found that our models identify a wider range of phenotype concepts, including previously uncharacterized ones. Our models also show strong performance in case studies on biomedical literature. We evaluate the strengths and weaknesses of BERT- and GPT-based models in aspects such as architecture and accuracy. Overall, our models enhance automated phenotype detection from clinical texts, improving downstream analyses on human diseases.

Social Determinants Documented Across Individuals from Different Racial and Ethnic Groups in Home Healthcare.

Home healthcare (HHC) enables patients to receive health services within their homes. Social determinants of health (SDOH) influence a patient's health and may disproportionately affect patients from racially and ethnically minoritized groups. This study describes differences in SDOH documentation in clinical notes among individuals from different racial or ethnic groups from one HHC agency in the northeastern United States. Compared to White patients, HHC episodes for patients across racially and ethnically minoritized groups had higher frequencies of SDOH documented. Further, our results suggest that race or ethnicity is significantly associated with SDOH documentation.

Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness.

OBJECTIVES: Active learning (AL) has rarely integrated diversity-based and uncertainty-based strategies into a dynamic sampling framework for clinical named entity recognition (NER). Machine-assisted annotation is becoming popular for creating gold-standard labels. This study investigated the effectiveness of dynamic AL strategies under simulated machine-assisted annotation scenarios for clinical NER. MATERIALS AND METHODS: We proposed 3 new AL strategies: a diversity-based strategy (CLUSTER) based on Sentence-BERT and 2 dynamic strategies (CLC and CNBSE) capable of switching from diversity-based to uncertainty-based strategies. Using BioClinicalBERT as the foundational NER model, we conducted simulation experiments on 3 medication-related clinical NER datasets independently: i2b2 2009, n2c2 2018 (Track 2), and MADE 1.0. We compared the proposed strategies with uncertainty-based (LC and NBSE) and passive-learning (RANDOM) strategies. Performance was primarily measured by the number of edits made by the annotators to achieve a desired target effectiveness evaluated on independent test sets. RESULTS: When aiming for 98% overall target effectiveness, on average, CLUSTER required the fewest edits. When aiming for 99% overall target effectiveness, CNBSE required 20.4% fewer edits than NBSE did. CLUSTER and RANDOM could not achieve such a high target under the pool-based simulation experiment. For high-difficulty entities, CNBSE required 22.5% fewer edits than NBSE to achieve 99% target effectiveness, whereas neither CLUSTER nor RANDOM achieved 93% target effectiveness. DISCUSSION AND CONCLUSION: When the target effectiveness was set high, the proposed dynamic strategy CNBSE exhibited both strong learning capabilities and low annotation costs in machine-assisted annotation. CLUSTER required the fewest edits when the target effectiveness was set low.

Developing a Framework to Infer Opioid Use Disorder Severity From Clinical Notes to Inform Natural Language Processing Methods: Characterization Study.

BACKGROUND: Information regarding opioid use disorder (OUD) status and severity is important for patient care. Clinical notes provide valuable information for detecting and characterizing problematic opioid use, necessitating development of natural language processing (NLP) tools, which in turn requires reliably labeled OUD-relevant text and understanding of documentation patterns. OBJECTIVE: To inform automated NLP methods, we aimed to develop and evaluate an annotation schema for characterizing OUD and its severity, and to document patterns of OUD-relevant information within clinical notes of heterogeneous patient cohorts. METHODS: We developed an annotation schema to characterize OUD severity based on criteria from the Diagnostic and Statistical Manual of Mental Disorders, 5th edition. In total, 2 annotators reviewed clinical notes from key encounters of 100 adult patients with varied evidence of OUD, including patients with and those without chronic pain, with and without medication treatment for OUD, and a control group. We completed annotations at the sentence level. We calculated severity scores based on annotation of note text with 18 classes aligned with criteria for OUD severity and determined positive predictive values for OUD severity. RESULTS: The annotation schema contained 27 classes. We annotated 1436 sentences from 82 patients; notes of 18 patients (11 of whom were controls) contained no relevant information. Interannotator agreement was above 70% for 11 of 15 batches of reviewed notes. Severity scores for control group patients were all 0. Among noncontrol patients, the mean severity score was 5.1 (SD 3.2), indicating moderate OUD, and the positive predictive value for detecting moderate or severe OUD was 0.71. Progress notes and notes from emergency department and outpatient settings contained the most and greatest diversity of information. Substance misuse and psychiatric classes were most prevalent and highly correlated across note types with high co-occurrence across patients. CONCLUSIONS: Implementation of the annotation schema demonstrated strong potential for inferring OUD severity based on key information in a small set of clinical notes and highlighting where such information is documented. These advancements will facilitate NLP tool development to improve OUD prevention, diagnosis, and treatment.

Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study.

Background: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes. Objective: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students' free-text history and physical notes. Methods: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct. Results: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002). Conclusions: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.

Reasons for Acceptance or Rejection of Online Record Access Among Patients Affected by a Severe Mental Illness: Mixed Methods Study.

BACKGROUND: Over the past few years, online record access (ORA) has been established through secure patient portals in various countries, allowing patients to access their health data, including clinical notes ("open notes"). Previous research indicates that ORA in mental health, particularly among patients with severe mental illness (SMI), has been rarely offered. Little is known about the expectations and motivations of patients with SMI when reading what their clinicians share via ORA. OBJECTIVE: The aim of this study is to explore the reasons why patients with SMI consider or reject ORA and whether sociodemographic characteristics may influence patient decisions. METHODS: ORA was offered to randomly selected patients at 3 university outpatient clinics in Brandenburg, Germany, which exclusively treat patients with SMI. Within the framework of a mixed methods evaluation, qualitative interviews were conducted with patients who chose to participate in ORA and those who declined, aiming to explore the underlying reasons for their decisions. The interviews were transcribed and analyzed using thematic analysis. Sociodemographic characteristics of patients were examined using descriptive statistics to identify predictors of acceptance or rejection of ORA. RESULTS: Out of 103 included patients, 58% (n=60) wished to read their clinical notes. The reasons varied, ranging from a desire to engage more actively in their treatment to critically monitoring it and using the accessible data for third-party purposes. Conversely, 42% (n=43) chose not to use ORA, voicing concerns about possibly harming the trustful relationship with their clinicians as well as potential personal distress or uncertainty arising from reading the notes. Practical barriers such as a lack of digital literacy or suspected difficult-to-understand medical language were also named as contributing factors. Correlation analysis revealed that the majority of patients with depressive disorder desired to read the clinical notes (P<.001), while individuals with psychotic disorders showed a higher tendency to decline ORA (P<.05). No significant group differences were observed for other patient groups or characteristics. CONCLUSIONS: The adoption of ORA is influenced by a wide range of motivational factors, while patients also present a similar variety of reasons for declining its use. The results emphasize the urgent need for knowledge and patient education regarding factors that may hinder the decision to use ORA, including its practical usage, its application possibilities, and concerns related to data privacy. Further research is needed to explore approaches for adequately preparing individuals with SMI to transition from their inherent interest to active engagement with ORA. TRIAL REGISTRATION: German Clinical Trial Register DRKS00030188; https://drks.de/search/en/trial/DRKS00030188.