Identifying Inequities in Video and Audio Telehealth Services for Primary Care Encounters During COVID-19: Repeated Cross-Sectional, Observational Study.

BACKGROUND: The COVID-19 pandemic resulted in rapid changes in how patient care was provided, particularly through the expansion of telehealth and audio-only phone-based care. OBJECTIVE: The goal of this study was to evaluate inequities in video and audio-only care during various time points including the initial wave of the COVID-19 pandemic, later stages of the pandemic, and a historical control. We sought to understand the characteristics of care during this time for a variety of different groups of patients that may experience health care inequities. METHODS: We conducted a retrospective analysis of electronic health record (EHR) data from encounters from 34 family medicine and internal medicine primary care clinics in a large, Midwestern health system, using a repeated cross-sectional, observational study design. These data included patient demographic data, as well as encounter, diagnosis, and procedure records. Data were obtained for all in-person and telehealth encounters (including audio-only phone-based care) that occurred during 3 separate time periods: an initial COVID-19 period (T2: March 16, 2020, to May 3, 2020), a later COVID-19 period (T3: May 4, 2020, to September 30, 2020), and a historical control period from the previous year (T1: March 16, 2019, to September 30, 2019). Primary analysis focused on the status of each encounter in terms of whether it was completed as scheduled, it was canceled, or the patient missed the appointment. A secondary analysis was performed to evaluate the likelihood of an encounter being completed based on visit modality (phone, video, in-person). RESULTS: In total, there were 938,040 scheduled encounters during the 3 time periods, with 178,747 unique patients, that were included for analysis. Patients with completed encounters were more likely to be younger than 65 years old (71.8%-74.1%), be female (58.8%-61.8%), be White (75.6%-76.7%), and have no significant comorbidities (63.2%-66.8%) or disabilities (53.2%-61.1%) in all time periods than those who had only canceled or missed encounters. Effects on different subpopulations are discussed herein. CONCLUSIONS: Findings from this study demonstrate that primary care utilization across delivery modalities (in person, video, and phone) was not equivalent across all groups before and during the COVID-19 pandemic and different groups were differentially impacted at different points. Understanding how different groups of patients responded to these rapid changes and how health care inequities may have been affected is an important step in better understanding implementation strategies for digital solutions in the future.

Human versus machine editing of electronic prescription directions.

BACKGROUND: Pharmacy staff are responsible for editing poor-quality and difficult-to-read electronic prescription (e-prescription) directions. Machine translation (MT) models are capable of translating free text from 1 sequence into another. However, the quality of MTs of e-prescriptions into pharmacy label directions is unknown. OBJECTIVE: To determine the types and frequencies of e-prescription direction component errors made by an MT model, pharmacy staff, and prescribers. METHODS: A prospective evaluation was conducted on a random sample of 300 patient directions in a test set of e-prescriptions from a mail-order pharmacy. Each row included directions produced by (1) prescribers on e-prescriptions, (2) pharmacy staff on prescription labels, and (3) an open neural MT model. Annotators labeled direction sets for missing direction components, use of abbreviations and medical jargon, and incorrect information (e.g., changing the number of tablets to be taken). The longest common subsequence (LCS) compared the amount of pharmacy staff editing with and without MT. RESULTS: Out of 279 direction sets labeled, the MT model directions contained no quality issues in 196 (70.3%) samples compared with 187 (67.0%) and 83 (29.8%) samples for pharmacy staff directions and prescriber directions, respectively. The MT model directions contained more incorrect components (n = 23). Median LCS was greater without MT (30.0 vs. 18.5, P < 0.01, Wilcoxon signed-rank test), indicating more editing was needed. CONCLUSION: MT could be used to improve the quality of e-prescription directions; however, MT makes high-risk mistakes such as incorrectly predicting the tapering regimen for prednisone. The use of semiautomated MT, where pharmacy staff can review model predictions to detect and resolve quality issues, should be considered to improve safety and decrease total work time compared with current practice. MT has strengths and weaknesses for improving the editing process of the patient directions compared with pharmacy staff alone.

"Baby Wants Tacos": Analysis of Health-Related Facebook Posts from Young Pregnant Women.

Objectives Pregnant young women gain more weight than recommended by the National Academy of Medicine, increasing the likelihood of adverse maternal and fetal outcomes. The purpose of this study is to use online social media to understand beliefs and practices surrounding weight gain, diet and exercise during pregnancy among young women. Methods Facebook posts were mined from young women ages 16 to 24 during pregnancy who were consented from two Midwest primary care clinics serving low-income communities. Natural language processing was used to identify posts related to weight gain, exercise and diet by keyword searching. Two investigators iteratively coded the mined posts and identified major themes around health behaviors. Outcome measures included the frequency of posts and major themes regarding health behaviors during pregnancy. Results Participants (n = 43) had a mean age of 21 (SD 2.3), and the largest subgroups identified as black (49%; 26% white, 16% Hispanic, 9% other) and having graduated from high school (49%; 24% completed some high school and 24% completed at least some post-secondary education). Among the 2899 pregnancy posts analyzed, 311 were related to weight. Major themes included eating behaviors and cravings (58% of identified posts), body image (24%), the influence of family, partners and friends (14%), and the desire to exercise (4%). Conclusions for practice Facebook posts revealed that young women often frame their thoughts and feelings regarding weight gain in pregnancy in the context of food cravings and body image and that friends and family are important influencers to these behaviors.

Identifying peer experts in online health forums.

BACKGROUND: Online health forums have become increasingly popular over the past several years. They provide members with a platform to network with peers and share information, experiential advice, and support. Among the members of health forums, we define "peer experts" as a set of lay users who have gained expertise on the particular health topic through personal experience, and who demonstrate credibility in responding to questions from other members. This paper aims to motivate the need to identify peer experts in health forums and study their characteristics. METHODS: We analyze profiles and activity of members of a popular online health forum and characterize the interaction behavior of peer experts. We study the temporal patterns of comments posted by lay users and peer experts to uncover how peer expertise is developed. We further train a supervised classifier to identify peer experts based on their activity level, textual features, and temporal progression of posts. RESULT: A support vector machine classifier with radial basis function kernel was found to be the most suitable model among those studied. Features capturing the key semantic word classes and higher mean user activity were found to be most significant features. CONCLUSION: We define a new class of members of health forums called peer experts, and present preliminary, yet promising, approaches to distinguish peer experts from novice users. Identifying such peer expertise could potentially help improve the perceived reliability and trustworthiness of information in community health forums.

Complexities, variations, and errors of numbering within clinical notes: the potential impact on information extraction and cohort-identification.

BACKGROUND: Numbers and numerical concepts appear frequently in free text clinical notes from electronic health records. Knowledge of the frequent lexical variations of these numerical concepts, and their accurate identification, is important for many information extraction tasks. This paper describes an analysis of the variation in how numbers and numerical concepts are represented in clinical notes. METHODS: We used an inverted index of approximately 100 million notes to obtain the frequency of various permutations of numbers and numerical concepts, including the use of Roman numerals, numbers spelled as English words, and invalid dates, among others. Overall, twelve types of lexical variants were analyzed. RESULTS: We found substantial variation in how these concepts were represented in the notes, including multiple data quality issues. We also demonstrate that not considering these variations could have substantial real-world implications for cohort identification tasks, with one case missing > 80% of potential patients. CONCLUSIONS: Numbering within clinical notes can be variable, and not taking these variations into account could result in missing or inaccurate information for natural language processing and information retrieval tasks.

Describing the patient experience from Yelp reviews of community pharmacies.

OBJECTIVES: To examine the characteristics of patient experience in community pharmacies through pattern exploration techniques of the unstructured free-text data from an online review website. DESIGN: Retrospective observational study design using structural topic model (STM) and term frequency-inverse document frequency (tf-idf) to categorize free-text data. Tf-idf scores words in terms of importance, and STM extracts latent themes from free-text data based on the co-occurrence of words in a review. Human labels were assigned to STM output, with each topic's prevalence mapped to each level of the 1- to 5-star review ratings. SETTING AND PARTICIPANTS: Data were obtained from the Yelp Academic data set from April 2006 through December 2017. These data were available for analysis from certain cities in the United States, Canada, and Europe. Included reviews were filtered based on the presence of pharmacy-specific character strings (e.g., "prescri"). MAIN OUTCOME MEASURES: Descriptive statistics of Yelp review characteristics, tf-idf scores, and topics produced from STM were used to characterize the content of Yelp reviews at each star-rating level. RESULTS: The filtered data set contained 4463 reviews from 964 pharmacies in 8 U.S. states. The mean (±SD) review rating was 2.97 ± 0.91. The mean number of words in a review was 135 ± 116. STM revealed 9 topics that influenced patient experiences at community pharmacies, including waiting time, service attitude, and physical store characteristics. Friendly and helpful staff accounted for 28.3% of content in 5-star ratings, whereas waiting time accounted for 19.4% of 1-star ratings. CONCLUSION: Yelp reviews provide a public look into patient experience at community pharmacies, and the reviews likely influence other patients' decisions to use the pharmacy. Pharmacies should focus their efforts on enabling pharmacy staff to provide high-quality care and minimizing unnecessary waiting times for patients.

User acceptance of location-tracking technologies in health research: Implications for study design and data quality.

Research regarding place and health has undergone a revolution due to the availability of consumer-focused location-tracking devices that reveal fine-grained details of human mobility. Such research requires that participants accept such devices enough to use them in their daily lives. There is a need for a theoretically grounded understanding of acceptance of different location-tracking technology options, and its research implications. Guided by an extended Unified Theory of Acceptance and Use of Technology (UTAUT), we conducted a 28-day field study comparing 21 chronically ill people's acceptance of two leading, consumer-focused location-tracking technologies deployed for research purposes: (1) a location-enabled smartphone, and (2) a GPS watch/activity tracker. Participants used both, and completed two surveys and qualitative interviews. Findings revealed that all participants exerted effort to facilitate data capture, such as by incorporating devices into daily routines and developing workarounds to keep devices functioning. Nevertheless, the smartphone was perceived to be significantly easier and posed fewer usability challenges for participants than the watch. Older participants found the watch significantly more difficult to use. For both devices, effort expectancy was significantly associated with future willingness to participate in research although prosocial motivations overcame some concerns. Social influence, performance expectancy and use behavior were significantly associated with intentions to use the devices in participants' personal lives. Data gathered via the smartphone was significantly more complete than data gathered via the watch, primarily due to usability challenges. To make longer-term participation in location tracking research a reality, and to achieve complete data capture, researchers must minimize the effort involved in participation; this requires usable devices. For long-term location-tracking studies using similar devices, findings indicate that only smartphone-based tracking is up to the challenge.

Augmenting Qualitative Text Analysis with Natural Language Processing: Methodological Study.

BACKGROUND: Qualitative research methods are increasingly being used across disciplines because of their ability to help investigators understand the perspectives of participants in their own words. However, qualitative analysis is a laborious and resource-intensive process. To achieve depth, researchers are limited to smaller sample sizes when analyzing text data. One potential method to address this concern is natural language processing (NLP). Qualitative text analysis involves researchers reading data, assigning code labels, and iteratively developing findings; NLP has the potential to automate part of this process. Unfortunately, little methodological research has been done to compare automatic coding using NLP techniques and qualitative coding, which is critical to establish the viability of NLP as a useful, rigorous analysis procedure. OBJECTIVE: The purpose of this study was to compare the utility of a traditional qualitative text analysis, an NLP analysis, and an augmented approach that combines qualitative and NLP methods. METHODS: We conducted a 2-arm cross-over experiment to compare qualitative and NLP approaches to analyze data generated through 2 text (short message service) message survey questions, one about prescription drugs and the other about police interactions, sent to youth aged 14-24 years. We randomly assigned a question to each of the 2 experienced qualitative analysis teams for independent coding and analysis before receiving NLP results. A third team separately conducted NLP analysis of the same 2 questions. We examined the results of our analyses to compare (1) the similarity of findings derived, (2) the quality of inferences generated, and (3) the time spent in analysis. RESULTS: The qualitative-only analysis for the drug question (n=58) yielded 4 major findings, whereas the NLP analysis yielded 3 findings that missed contextual elements. The qualitative and NLP-augmented analysis was the most comprehensive. For the police question (n=68), the qualitative-only analysis yielded 4 primary findings and the NLP-only analysis yielded 4 slightly different findings. Again, the augmented qualitative and NLP analysis was the most comprehensive and produced the highest quality inferences, increasing our depth of understanding (ie, details and frequencies). In terms of time, the NLP-only approach was quicker than the qualitative-only approach for the drug (120 vs 270 minutes) and police (40 vs 270 minutes) questions. An approach beginning with qualitative analysis followed by qualitative- or NLP-augmented analysis took longer time than that beginning with NLP for both drug (450 vs 240 minutes) and police (390 vs 220 minutes) questions. CONCLUSIONS: NLP provides both a foundation to code qualitatively more quickly and a method to validate qualitative findings. NLP methods were able to identify major themes found with traditional qualitative analysis but were not useful in identifying nuances. Traditional qualitative text analysis added important details and context.

Development and empirical user-centered evaluation of semantically-based query recommendation for an electronic health record search engine.

OBJECTIVE: The utility of biomedical information retrieval environments can be severely limited when users lack expertise in constructing effective search queries. To address this issue, we developed a computer-based query recommendation algorithm that suggests semantically interchangeable terms based on an initial user-entered query. In this study, we assessed the value of this approach, which has broad applicability in biomedical information retrieval, by demonstrating its application as part of a search engine that facilitates retrieval of information from electronic health records (EHRs). MATERIALS AND METHODS: The query recommendation algorithm utilizes MetaMap to identify medical concepts from search queries and indexed EHR documents. Synonym variants from UMLS are used to expand the concepts along with a synonym set curated from historical EHR search logs. The empirical study involved 33 clinicians and staff who evaluated the system through a set of simulated EHR search tasks. User acceptance was assessed using the widely used technology acceptance model. RESULTS: The search engine's performance was rated consistently higher with the query recommendation feature turned on vs. off. The relevance of computer-recommended search terms was also rated high, and in most cases the participants had not thought of these terms on their own. The questions on perceived usefulness and perceived ease of use received overwhelmingly positive responses. A vast majority of the participants wanted the query recommendation feature to be available to assist in their day-to-day EHR search tasks. DISCUSSION AND CONCLUSION: Challenges persist for users to construct effective search queries when retrieving information from biomedical documents including those from EHRs. This study demonstrates that semantically-based query recommendation is a viable solution to addressing this challenge.

Ease of adoption of clinical natural language processing software: An evaluation of five systems.

OBJECTIVE: In recognition of potential barriers that may inhibit the widespread adoption of biomedical software, the 2014 i2b2 Challenge introduced a special track, Track 3 - Software Usability Assessment, in order to develop a better understanding of the adoption issues that might be associated with the state-of-the-art clinical NLP systems. This paper reports the ease of adoption assessment methods we developed for this track, and the results of evaluating five clinical NLP system submissions. MATERIALS AND METHODS: A team of human evaluators performed a series of scripted adoptability test tasks with each of the participating systems. The evaluation team consisted of four "expert evaluators" with training in computer science, and eight "end user evaluators" with mixed backgrounds in medicine, nursing, pharmacy, and health informatics. We assessed how easy it is to adopt the submitted systems along the following three dimensions: communication effectiveness (i.e., how effective a system is in communicating its designed objectives to intended audience), effort required to install, and effort required to use. We used a formal software usability testing tool, TURF, to record the evaluators' interactions with the systems and 'think-aloud' data revealing their thought processes when installing and using the systems and when resolving unexpected issues. RESULTS: Overall, the ease of adoption ratings that the five systems received are unsatisfactory. Installation of some of the systems proved to be rather difficult, and some systems failed to adequately communicate their designed objectives to intended adopters. Further, the average ratings provided by the end user evaluators on ease of use and ease of interpreting output are -0.35 and -0.53, respectively, indicating that this group of users generally deemed the systems extremely difficult to work with. While the ratings provided by the expert evaluators are higher, 0.6 and 0.45, respectively, these ratings are still low indicating that they also experienced considerable struggles. DISCUSSION: The results of the Track 3 evaluation show that the adoptability of the five participating clinical NLP systems has a great margin for improvement. Remedy strategies suggested by the evaluators included (1) more detailed and operation system specific use instructions; (2) provision of more pertinent onscreen feedback for easier diagnosis of problems; (3) including screen walk-throughs in use instructions so users know what to expect and what might have gone wrong; (4) avoiding jargon and acronyms in materials intended for end users; and (5) packaging prerequisites required within software distributions so that prospective adopters of the software do not have to obtain each of the third-party components on their own.

Automated-detection of risky alcohol use prior to surgery using natural language processing.

BACKGROUND: Preoperative risky alcohol use is one of the most common surgical risk factors. Accurate and early identification of risky alcohol use could enhance surgical safety. Artificial Intelligence-based approaches, such as natural language processing (NLP), provide an innovative method to identify alcohol-related risks from patients' electronic health records (EHR) before surgery. METHODS: Clinical notes (n = 53,629) from pre-operative patients in a tertiary care facility were analyzed for evidence of risky alcohol use and alcohol use disorder. One hundred of these records were reviewed by experts and labeled for comparison. A rule-based NLP model was built, and we assessed the clinical notes for the entire population. Additionally, we assessed each record for the presence or absence of alcohol-related International Classification of Diseases (ICD) diagnosis codes as an additional comparator. RESULTS: NLP correctly identified 87% of the human-labeled patients classified with risky alcohol use. In contrast, diagnosis codes alone correctly identified only 29% of these patients. In terms of specificity, NLP correctly identified 84% of the non-risky cohort, while diagnosis codes correctly identified 90% of this cohort. In the analysis of the full dataset, the NLP-based approach identified three times more patients with risky alcohol use than ICD codes. CONCLUSIONS: NLP, an artificial intelligence-based approach, efficiently and accurately identifies alcohol-related risk in patients' EHRs. This approach could supplement other alcohol screening tools to identify patients in need of intervention, treatment, and/or postoperative withdrawal prophylaxis. Alcohol-related ICD diagnosis had limited utility relative to NLP, which extracts richer information within clinical notes to classify patients.

Social Acceptability of Health Behavior Posts on Social Media: An Experiment.

INTRODUCTION: Social media sites like Twitter (now X) are increasingly used to create health behavior metrics for public health surveillance. Yet little is known about social norms that may bias the content of posts about health behaviors. Social norms for posts about four health behaviors (smoking tobacco, drinking alcohol, physical activity, eating food) on Twitter/X were evaluated. METHODS: This was a randomized experiment delivered via web-based survey to adult, English-speaking Twitter/X users in three Michigan, USA, counties from 2020 to 2022 (n=559). Each participant viewed 24 posts presenting experimental manipulations regarding four health behaviors and answered questions about each post's social acceptability. Principal component analysis was used to combine survey responses into one perceived social acceptability measure. Linear mixed models with the Benjamini-Hochberg correction were implemented to test seven study hypotheses in 2023. RESULTS: Supporting six hypotheses, posts presenting healthier (CI: 0.028, 0.454), less stigmatized behaviors (CI: 0.552, 0.157) were more socially acceptable than posts regarding unhealthier, stigmatized behaviors. Unhealthy (CI: -0.268, -0.109) and stigmatized behavior (CI: -0.261, -0.103) posts were less acceptable for more educated participants. Posts about collocated activities (CI: 0.410, 0.573) and accompanied by expressions of liking (CI: 0.906, 1.11) were more acceptable than activities undertaken alone or disliked. Contrary to one hypothesis, posts reporting unusual activities were less acceptable than usual ones (CI: -0.472, 0.312). CONCLUSIONS: Perceived social acceptability may be associated with the frequency and content of health behavior posts. Users of Twitter/X and other social media platform posts to estimate health behavior prevalence should account for potential estimation biases from perceived social acceptability of posts.

Virtual Care: Perspectives From Family Physicians.

BACKGROUND: During the COVID-19 pandemic, virtual care expanded rapidly at Michigan Medicine and other health systems. From family physicians' perspectives, this shift to virtual care has the potential to affect workflow, job satisfaction, and patient communication. As clinics reopened and care delivery models shifted to a combination of in-person and virtual care, the need to understand physician experiences with virtual care arose in order to improve both patient and provider experiences. This study investigated Michigan Medicine family medicine physicians' perceptions of virtual care through qualitative interviews to better understand how to improve the quality and effectiveness of virtual care for both patients and physicians. METHODS: We employed a qualitative descriptive design to examine physician perspectives through semistructured interviews. We coded and analyzed transcripts using thematic analysis, facilitated by MAXQDA (VERBI) software. RESULTS: The results of the analysis identified four major themes: (a) chief concerns that are appropriate for virtual evaluation, (b) physician perceptions of patient benefits, (c) focused but contextually enriched patient-physician communication, and (d) structural support needed for high-quality virtual care. CONCLUSIONS: These findings can help further direct the discussion of how to make use of resources to improve the quality and effectiveness of virtual care.

A Systematic Temporal Extraction Pipeline for Medical Concepts in Clinical Notes.

With increased application of natural language processing (NLP) in medicine, many NLP models are being developed for uncovering relevant clinical features from electronic health records. Temporal information plays a key role in understanding the context, significance, and interpretation of medical concepts extracted from clinical notes. This is particularly true in situations where the behavior, value, or status of a medical concept changes over time. In this paper, we introduce a systematic framework, NLP annotation-Relaxation-Generation (NRG). NRG compiles incidents of medical concept changes from status annotations and timestamps of multiple clinical notes. We demonstrate the effectiveness of the NRG pipeline by applying it to two medical concepts related to patients with inflammatory bowel disease: extra-intestinal manifestations and medications. We show that the NRG pipeline offers not only insights into medical concept changes over time, but can help convey longitudinal changes in clinical features at both individual and population level.

Dementia and electronic health record phenotypes: a scoping review of available phenotypes and opportunities for future research.

OBJECTIVE: We performed a scoping review of algorithms using electronic health record (EHR) data to identify patients with Alzheimer's disease and related dementias (ADRD), to advance their use in research and clinical care. MATERIALS AND METHODS: Starting with a previous scoping review of EHR phenotypes, we performed a cumulative update (April 2020 through March 1, 2023) using Pubmed, PheKB, and expert review with exclusive focus on ADRD identification. We included algorithms using EHR data alone or in combination with non-EHR data and characterized whether they identified patients at high risk of or with a current diagnosis of ADRD. RESULTS: For our cumulative focused update, we reviewed 271 titles meeting our search criteria, 49 abstracts, and 26 full text papers. We identified 8 articles from the original systematic review, 8 from our new search, and 4 recommended by an expert. We identified 20 papers describing 19 unique EHR phenotypes for ADRD: 7 algorithms identifying patients with diagnosed dementia and 12 algorithms identifying patients at high risk of dementia that prioritize sensitivity over specificity. Reference standards range from only using other EHR data to in-person cognitive screening. CONCLUSION: A variety of EHR-based phenotypes are available for use in identifying populations with or at high-risk of developing ADRD. This review provides comparative detail to aid in choosing the best algorithm for research, clinical care, and population health projects based on the use case and available data. Future research may further improve the design and use of algorithms by considering EHR data provenance.

Developing a Semantically Based Query Recommendation for an Electronic Medical Record Search Engine: Query Log Analysis and Design Implications.

BACKGROUND: An effective and scalable information retrieval (IR) system plays a crucial role in enabling clinicians and researchers to harness the valuable information present in electronic health records. In a previous study, we developed a prototype medical IR system, which incorporated a semantically based query recommendation (SBQR) feature. The system was evaluated empirically and demonstrated high perceived performance by end users. To delve deeper into the factors contributing to this perceived performance, we conducted a follow-up study using query log analysis. OBJECTIVE: One of the primary challenges faced in IR is that users often have limited knowledge regarding their specific information needs. Consequently, an IR system, particularly its user interface, needs to be thoughtfully designed to assist users through the iterative process of refining their queries as they encounter relevant documents during their search. To address these challenges, we incorporated "query recommendation" into our Electronic Medical Record Search Engine (EMERSE), drawing inspiration from the success of similar features in modern IR systems for general purposes. METHODS: The query log data analyzed in this study were collected during our previous experimental study, where we developed EMERSE with the SBQR feature. We implemented a logging mechanism to capture user query behaviors and the output of the IR system (retrieved documents). In this analysis, we compared the initial query entered by users with the query formulated with the assistance of the SBQR. By examining the results of this comparison, we could examine whether the use of SBQR helped in constructing improved queries that differed from the original ones. RESULTS: Our findings revealed that the first query entered without SBQR and the final query with SBQR assistance were highly similar (Jaccard similarity coefficient=0.77). This suggests that the perceived positive performance of the system was primarily attributed to the automatic query expansion facilitated by the SBQR rather than users manually manipulating their queries. In addition, through entropy analysis, we observed that search results converged in scenarios of moderate difficulty, and the degree of convergence correlated strongly with the perceived system performance. CONCLUSIONS: The study demonstrated the potential contribution of the SBQR in shaping participants' positive perceptions of system performance, contingent upon the difficulty of the search scenario. Medical IR systems should therefore consider incorporating an SBQR as a user-controlled option or a semiautomated feature. Future work entails redesigning the experiment in a more controlled manner and conducting multisite studies to demonstrate the effectiveness of EMERSE with SBQR for patient cohort identification. By further exploring and validating these findings, we can enhance the usability and functionality of medical IR systems in real-world settings.

Applying AI and Guidelines to Assist Medical Students in Recognizing Patients With Heart Failure: Protocol for a Randomized Trial.

BACKGROUND: The integration of artificial intelligence (AI) into clinical practice is transforming both clinical practice and medical education. AI-based systems aim to improve the efficacy of clinical tasks, enhancing diagnostic accuracy and tailoring treatment delivery. As it becomes increasingly prevalent in health care for high-quality patient care, it is critical for health care providers to use the systems responsibly to mitigate bias, ensure effective outcomes, and provide safe clinical practices. In this study, the clinical task is the identification of heart failure (HF) prior to surgery with the intention of enhancing clinical decision-making skills. HF is a common and severe disease, but detection remains challenging due to its subtle manifestation, often concurrent with other medical conditions, and the absence of a simple and effective diagnostic test. While advanced HF algorithms have been developed, the use of these AI-based systems to enhance clinical decision-making in medical education remains understudied. OBJECTIVE: This research protocol is to demonstrate our study design, systematic procedures for selecting surgical cases from electronic health records, and interventions. The primary objective of this study is to measure the effectiveness of interventions aimed at improving HF recognition before surgery, the second objective is to evaluate the impact of inaccurate AI recommendations, and the third objective is to explore the relationship between the inclination to accept AI recommendations and their accuracy. METHODS: Our study used a 3 × 2 factorial design (intervention type × order of prepost sets) for this randomized trial with medical students. The student participants are asked to complete a 30-minute e-learning module that includes key information about the intervention and a 5-question quiz, and a 60-minute review of 20 surgical cases to determine the presence of HF. To mitigate selection bias in the pre- and posttests, we adopted a feature-based systematic sampling procedure. From a pool of 703 expert-reviewed surgical cases, 20 were selected based on features such as case complexity, model performance, and positive and negative labels. This study comprises three interventions: (1) a direct AI-based recommendation with a predicted HF score, (2) an indirect AI-based recommendation gauged through the area under the curve metric, and (3) an HF guideline-based intervention. RESULTS: As of July 2023, 62 of the enrolled medical students have fulfilled this study's participation, including the completion of a short quiz and the review of 20 surgical cases. The subject enrollment commenced in August 2022 and will end in December 2023, with the goal of recruiting 75 medical students in years 3 and 4 with clinical experience. CONCLUSIONS: We demonstrated a study protocol for the randomized trial, measuring the effectiveness of interventions using AI and HF guidelines among medical students to enhance HF recognition in preoperative care with electronic health record data. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/49842.

Automatic Extraction of Medication Mentions from Tweets-Overview of the BioCreative VII Shared Task 3 Competition.

This study presents the outcomes of the shared task competition BioCreative VII (Task 3) focusing on the extraction of medication names from a Twitter user's publicly available tweets (the user's 'timeline'). In general, detecting health-related tweets is notoriously challenging for natural language processing tools. The main challenge, aside from the informality of the language used, is that people tweet about any and all topics, and most of their tweets are not related to health. Thus, finding those tweets in a user's timeline that mention specific health-related concepts such as medications requires addressing extreme imbalance. Task 3 called for detecting tweets in a user's timeline that mentions a medication name and, for each detected mention, extracting its span. The organizers made available a corpus consisting of 182 049 tweets publicly posted by 212 Twitter users with all medication mentions manually annotated. The corpus exhibits the natural distribution of positive tweets, with only 442 tweets (0.2%) mentioning a medication. This task was an opportunity for participants to evaluate methods that are robust to class imbalance beyond the simple lexical match. A total of 65 teams registered, and 16 teams submitted a system run. This study summarizes the corpus created by the organizers and the approaches taken by the participating teams for this challenge. The corpus is freely available at https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vii/track-3/. The methods and the results of the competing systems are analyzed with a focus on the approaches taken for learning from class-imbalanced data.

An Assessment of Mentions of Adverse Drug Events on Social Media With Natural Language Processing: Model Development and Analysis.

BACKGROUND: Adverse reactions to drugs attract significant concern in both clinical practice and public health monitoring. Multiple measures have been put into place to increase postmarketing surveillance of the adverse effects of drugs and to improve drug safety. These measures include implementing spontaneous reporting systems and developing automated natural language processing systems based on data from electronic health records and social media to collect evidence of adverse drug events that can be further investigated as possible adverse reactions. OBJECTIVE: While using social media for collecting evidence of adverse drug events has potential, it is not clear whether social media are a reliable source for this information. Our work aims to (1) develop natural language processing approaches to identify adverse drug events on social media and (2) assess the reliability of social media data to identify adverse drug events. METHODS: We propose a collocated long short-term memory network model with attentive pooling and aggregated, contextual representation generated by a pretrained model. We applied this model on large-scale Twitter data to identify adverse drug event-related tweets. We conducted a qualitative content analysis of these tweets to validate the reliability of social media data as a means to collect such information. RESULTS: The model outperformed a variant without contextual representation during both the validation and evaluation phases. Through the content analysis of adverse drug event tweets, we observed that adverse drug event-related discussions had 7 themes. Mental health-related, sleep-related, and pain-related adverse drug event discussions were most frequent. We also contrast known adverse drug reactions to those mentioned in tweets. CONCLUSIONS: We observed a distinct improvement in the model when it used contextual information. However, our results reveal weak generalizability of the current systems to unseen data. Additional research is needed to fully utilize social media data and improve the robustness and reliability of natural language processing systems. The content analysis, on the other hand, showed that Twitter covered a sufficiently wide range of adverse drug events, as well as known adverse reactions, for the drugs mentioned in tweets. Our work demonstrates that social media can be a reliable data source for collecting adverse drug event mentions.