Leveraging a Large Language Model to Assess Quality-of-Care: Monitoring ADHD Medication Side Effects.

Objective: To assess the accuracy of a large language model (LLM) in measuring clinician adherence to practice guidelines for monitoring side effects after prescribing medications for children with attention-deficit/hyperactivity disorder (ADHD). Methods: Retrospective population-based cohort study of electronic health records. Cohort included children aged 6-11 years with ADHD diagnosis and ≥2 ADHD medication encounters (stimulants or non-stimulants prescribed) between 2015-2022 in a community-based primary healthcare network (n=1247). To identify documentation of side effects inquiry, we trained, tested, and deployed an open-source LLM (LLaMA) on all clinical notes from ADHD-related encounters (ADHD diagnosis or ADHD medication prescription), including in-clinic/telehealth and telephone encounters (n=15,593 notes). Model performance was assessed using holdout and deployment test sets, compared to manual chart review. Results: The LLaMA model achieved excellent performance in classifying notes that contain side effects inquiry (sensitivity= 87.2%, specificity=86.3/90.3%, area under curve (AUC)=0.93/0.92 on holdout/deployment test sets). Analyses revealed no model bias in relation to patient age, sex, or insurance. Mean age (SD) at first prescription was 8.8 (1.6) years; patient characteristics were similar across patients with and without documented side effects inquiry. Rates of documented side effects inquiry were lower in telephone encounters than in-clinic/telehealth encounters (51.9% vs. 73.0%, p<0.01). Side effects inquiry was documented in 61% of encounters following stimulant prescriptions and 48% of encounters following non-stimulant prescriptions (p<0.01). Conclusions: Deploying an LLM on a variable set of clinical notes, including telephone notes, offered scalable measurement of quality-of-care and uncovered opportunities to improve psychopharmacological medication management in primary care.

Patient Characteristics Associated With Phone and Video Visits at a Tele-Urgent Care Center During the Initial COVID-19 Response: Cross-Sectional Study.

BACKGROUND: Health systems rapidly adopted telemedicine as an alternative health care delivery modality in response to the COVID-19 pandemic. Demographic factors, such as age and gender, may play a role in patients' choice of a phone or video visit. However, it is unknown whether there are differences in utilization between phone and video visits. OBJECTIVE: This study aimed to investigate patients' characteristics, patient utilization, and service characteristics of a tele-urgent care clinic during the initial response to the pandemic. METHODS: We conducted a cross-sectional study of urgent care patients using a statewide, on-demand telemedicine clinic with board-certified physicians during the initial phases of the pandemic. The study data were collected from March 3, 2020, through May 3, 2020. RESULTS: Of 1803 telemedicine visits, 1278 (70.9%) patients were women, 730 (40.5%) were aged 18 to 34 years, and 1423 (78.9%) were uninsured. There were significant differences between telemedicine modalities and gender (P<.001), age (P<.001), insurance status (P<.001), prescriptions given (P<.001), and wait times (P<.001). Phone visits provided significantly more access to rural areas than video visits (P<.001). CONCLUSIONS: Our findings suggest that offering patients a combination of phone and video options provided additional flexibility for various patient subgroups, particularly patients living in rural regions with limited internet bandwidth. Differences in utilization were significant based on patient gender, age, and insurance status. We also found differences in prescription administration between phone and video visits that require additional investigation.

Measuring quality-of-care in treatment of young children with attention-deficit/hyperactivity disorder using pre-trained language models.

OBJECTIVE: To measure pediatrician adherence to evidence-based guidelines in the treatment of young children with attention-deficit/hyperactivity disorder (ADHD) in a diverse healthcare system using natural language processing (NLP) techniques. MATERIALS AND METHODS: We extracted structured and free-text data from electronic health records (EHRs) of all office visits (2015-2019) of children aged 4-6 years in a community-based primary healthcare network in California, who had ≥1 visits with an ICD-10 diagnosis of ADHD. Two pediatricians annotated clinical notes of the first ADHD visit for 423 patients. Inter-annotator agreement (IAA) was assessed for the recommendation for the first-line behavioral treatment (F-measure = 0.89). Four pre-trained language models, including BioClinical Bidirectional Encoder Representations from Transformers (BioClinicalBERT), were used to identify behavioral treatment recommendations using a 70/30 train/test split. For temporal validation, we deployed BioClinicalBERT on 1,020 unannotated notes from other ADHD visits and well-care visits; all positively classified notes (n = 53) and 5% of negatively classified notes (n = 50) were manually reviewed. RESULTS: Of 423 patients, 313 (74%) were male; 298 (70%) were privately insured; 138 (33%) were White; 61 (14%) were Hispanic. The BioClinicalBERT model trained on the first ADHD visits achieved F1 = 0.76, precision = 0.81, recall = 0.72, and AUC = 0.81 [0.72-0.89]. Temporal validation achieved F1 = 0.77, precision = 0.68, and recall = 0.88. Fairness analysis revealed low model performance in publicly insured patients (F1 = 0.53). CONCLUSION: Deploying pre-trained language models on a variable set of clinical notes accurately captured pediatrician adherence to guidelines in the treatment of children with ADHD. Validating this approach in other patient populations is needed to achieve equitable measurement of quality of care at scale and improve clinical care for mental health conditions.

Toward Community-Based Natural Language Processing (CBNLP): Cocreating With Communities.

Rapid development and adoption of natural language processing (NLP) techniques has led to a multitude of exciting and innovative societal and health care applications. These advancements have also generated concerns around perpetuation of historical injustices and that these tools lack cultural considerations. While traditional health care NLP techniques typically include clinical subject matter experts to extract health information or aid in interpretation, few NLP tools involve community stakeholders with lived experiences. In this perspective paper, we draw upon the field of community-based participatory research, which gathers input from community members for development of public health interventions, to identify and examine ways to equitably involve communities in developing health care NLP tools. To realize the potential of community-based NLP (CBNLP), research and development teams must thoughtfully consider mechanisms and resources needed to effectively collaborate with community members for maximal societal and ethical impact of NLP-based tools.

Augmenting Quality Assurance Measures in Treatment Review with Machine Learning in Radiation Oncology.

Purpose: Pretreatment quality assurance (QA) of treatment plans often requires a high cognitive workload and considerable time expenditure. This study explores the use of machine learning to classify pretreatment chart check QA for a given radiation plan as difficult or less difficult, thereby alerting the physicists to increase scrutiny on difficult plans. Methods and Materials: Pretreatment QA data were collected for 973 cases between July 2018 and October 2020. The outcome variable, a degree of difficulty, was collected as a subjective rating by physicists who performed the pretreatment chart checks. Potential features were identified based on clinical relevance, contribution to plan complexity, and QA metrics. Five machine learning models were developed: support vector machine, random forest classifier, adaboost classifier, decision tree classifier, and neural network. These were incorporated into a voting classifier, where at least 2 algorithms needed to predict a case as difficult for it to be classified as such. Sensitivity analyses were conducted to evaluate feature importance. Results: The voting classifier achieved an overall accuracy of 77.4% on the test set, with 76.5% accuracy on difficult cases and 78.4% accuracy on less difficult cases. Sensitivity analysis showed features associated with plan complexity (number of fractions, dose per monitor unit, number of planning structures, and number of image sets) and clinical relevance (patient age) were sensitive across at least 3 algorithms. Conclusions: This approach can be used to equitably allocate plans to physicists rather than randomly allocate them, potentially improving pretreatment chart check effectiveness by reducing errors propagating downstream.

Feature Engineering for Interpretable Machine Learning for Quality Assurance in Radiation Oncology.

Chart checking is a time intensive process with high cognitive workload for physicists. Previous studies have partially automated and standardized chart checking, but limited studies implement data-driven approaches to reduce cognitive workload for quality assurance processes. This study aims to evaluate feature selection methods to improve the interpretability and transparency of machine learning models in predicting the degree of difficulty for a pretreatment physics chart check. We compare chi-square, mutual information, feature importance thresholding, and greedy feature selection for four different classifiers. Random forest has the highest performance with SMOTE oversampling using mutual information for feature selection (accuracy 84.0%, AUC 87.0%, precision 80.0%, recall 80.0%). This study demonstrates that feature selection methods can improve model interpretability and transparency.

Predicting Objective Performance Using Perceived Cognitive Workload Data in Healthcare Professionals: A Machine Learning Study.

Cognitive Workload (CWL) is a fundamental concept in predicting healthcare professionals' (HCPs) objective performance. The study aims to compare the accuracy of the classical model (utilizes all six dimensions of the National Aeronautics and Space Administration Task Load Index (NASA-TLX)) and novel models (utilize four or five dimensions of NASA-TLX) in predicting HCPs' objective performance. We use a dataset from our previous human factors research studies and apply a broad selection of supervised machine learning classification techniques to develop data-driven computational models and predict objective performance. The study findings confirm that classical models are better predictors of objective performance than novel models. This has practical implications for research in health informatics, human factors and ergonomics, and human-computer interaction in healthcare. Findings, although promising, cannot be generalized as they are based on a small dataset. Future studies may investigate additional subjective and physiological measures of CWL to predict HCPs' objective performance.

Using Explainable Supervised Machine Learning to Predict Burnout in Healthcare Professionals.

Burnout in healthcare professionals (HCPs) is a multi-factorial problem. There are limited studies utilizing machine learning approaches to predict HCPs' burnout during the COVID-19 pandemic. A survey consisting of demographic characteristics and work system factors was administered to 450 HCPs during the pandemic (participation rate: 59.3%). The highest performing machine learning model had an area under the receiver operating curve of 0.81. The eight key features that best predicted burnout are excessive workload, inadequate staffing, administrative burden, professional relationships, organizational culture, values and expectations, intrinsic motivation, and work-life integration. These findings provide evidence for resource allocation and implementation of interventions to reduce HCPs' burnout and improve the quality of care.

Recommendations for design of a mobile application to support management of anxiety and depression among Black American women.

Black American women experience adverse health outcomes due to anxiety and depression. They face systemic barriers to accessing culturally appropriate mental health care leading to the underutilization of mental health services and resources. Mobile technology can be leveraged to increase access to culturally relevant resources, however, the specific needs and preferences that Black women feel are useful in an app to support management of anxiety and depression are rarely reflected in existing digital health tools. This study aims to assess what types of content, features, and important considerations should be included in the design of a mobile app tailored to support management of anxiety and depression among Black women. Focus groups were conducted with 20 women (mean age 36.6 years, SD 17.8 years), with 5 participants per group. Focus groups were led by a moderator, with notetaker present, using an interview guide to discuss topics, such as participants' attitudes and perceptions towards mental health and use of mental health services, and content, features, and concerns for design of a mobile app to support management of anxiety and depression. Descriptive qualitative content analysis was conducted. Recommendations for content were either informational (e.g., information to find a Black woman therapist) or inspirational (e.g., encouraging stories about overcoming adversity). Suggested features allow users to monitor their progress, practice healthy coping techniques, and connect with others. The importance of feeling "a sense of community" was emphasized. Transparency about who created and owns the app, and how users' data will be used and protected was recommended to establish trust. The findings from this study were consistent with previous literature which highlighted the need for educational, psychotherapy, and personal development components for mental health apps. There has been exponential growth in the digital mental health space due to the COVID-19 pandemic; however, a one-size-fits-all approach may lead to more options but continued disparity in receiving mental health care. Designing a mental health app for and with Black women may help to advance digital health equity by providing a tool that addresses their specific needs and preferences, and increase engagement.

Improved Metabolite Prediction Using Microbiome Data-Based Elastic Net Models.

Microbiome data are becoming increasingly available in large health cohorts, yet metabolomics data are still scant. While many studies generate microbiome data, they lack matched metabolomics data or have considerable missing proportions of metabolites. Since metabolomics is key to understanding microbial and general biological activities, the possibility of imputing individual metabolites or inferring metabolomics pathways from microbial taxonomy or metagenomics is intriguing. Importantly, current metabolomics profiling methods such as the HMP Unified Metabolic Analysis Network (HUMAnN) have unknown accuracy and are limited in their ability to predict individual metabolites. To address this gap, we developed a novel metabolite prediction method, and we present its application and evaluation in an oral microbiome study. The new method for predicting metabolites using microbiome data (ENVIM) is based on the elastic net model (ENM). ENVIM introduces an extra step to ENM to consider variable importance (VI) scores, and thus, achieves better prediction power. We investigate the metabolite prediction performance of ENVIM using metagenomic and metatranscriptomic data in a supragingival biofilm multi-omics dataset of 289 children ages 3-5 who were participants of a community-based study of early childhood oral health (ZOE 2.0) in North Carolina, United States. We further validate ENVIM in two additional publicly available multi-omics datasets generated from studies of gut health. We select gene family sets based on variable importance scores and modify the existing ENM strategy used in the MelonnPan prediction software to accommodate the unique features of microbiome and metabolome data. We evaluate metagenomic and metatranscriptomic predictors and compare the prediction performance of ENVIM to the standard ENM employed in MelonnPan. The newly developed ENVIM method showed superior metabolite predictive accuracy than MelonnPan when trained with metatranscriptomics data only, metagenomics data only, or both. Better metabolite prediction is achieved in the gut microbiome compared with the oral microbiome setting. We report the best-predictable compounds in all these three datasets from two different body sites. For example, the metabolites trehalose, maltose, stachyose, and ribose are all well predicted by the supragingival microbiome.

Evaluating the Telehealth Experience of Patients With COVID-19 Symptoms: Recommendations on Best Practices.

Positive patient experiences are associated with illness recovery and adherence to medication. To evaluate the virtual care experience for patients with COVID-19 symptoms as their chief complaints. We conducted a cross-sectional study of the first cohort of patients with COVID-19 symptoms in a virtual clinic. The main end points of this study were visit volume, wait times, visit duration, patient diagnosis, prescriptions received, and satisfaction. Of the 1139 total virtual visits, 212 (24.6%) patients had COVID-19 symptoms. The average wait time (SD) for all visits was 75.5 (121.6) minutes. The average visit duration for visits was 10.5 (4.9) minutes. The highest volume of virtual visits was on Saturdays (39), and the lowest volume was on Friday (19). Patients experienced shorter wait times (SD) on the weekdays 67.1 (106.8) minutes compared to 90.3 (142.6) minutes on the weekends. The most common diagnoses for patients with COVID-19 symptoms were upper respiratory infection. Patient wait times for a telehealth visit varied depending on the time and day of appointment. Long wait times were a major drawback in the patient experience. Based on patient-reported experience, we proposed a list of general, provider, and patient telehealth best practices.

Chemical Entity Recognition for MEDLINE Indexing.

Chemical entity recognition is essential for indexing scientific literature in the MEDLINE database at the National Library of Medicine. However, the tool currently used to suggest terms for indexing, the Medical Text Indexer, was not originally conceived as a chemical recognition tool. It has instead been adapted to the task via its use of MetaMap and the addition of in-house patterns and rules. In order to develop a tool more suitable for chemical recognition, we have created a collection of 200 MEDLINE titles and abstracts annotated with genes, proteins, inorganic and organic chemicals, as well as other biological molecules. We use this collection to evaluate eleven chemical entity recognition systems, where we seek to identify a tool that effectively recognizes chemical entities for indexing and also performs well on chemical recognition beyond the indexing task. We observe the highest performance with a SciBERT ensemble.

Using Artificial Intelligence to Improve the Quality and Safety of Radiation Therapy.

Within artificial intelligence, machine learning (ML) efforts in radiation oncology have augmented the transition from generalized to personalized treatment delivery. Although their impact on quality and safety of radiation therapy has been limited, they are increasingly being used throughout radiation therapy workflows. Various data-driven approaches have been used for outcome prediction, CT simulation, clinical decision support, knowledge-based planning, adaptive radiation therapy, plan validation, machine quality assurance, and process quality assurance; however, there are many challenges that need to be addressed with the creation and usage of ML algorithms as well as the interpretation and dissemination of findings. In this review, the authors present current applications of ML in radiation oncology quality and safety initiatives, discuss challenges faced by the radiation oncology community, and suggest future directions.