Medical Text Prediction and Suggestion Using Generative Pretrained Transformer Models with Dental Medical Notes.

BACKGROUND: Generative pretrained transformer (GPT) models are one of the latest large pretrained natural language processing models that enables model training with limited datasets and reduces dependency on large datasets, which are scarce and costly to establish and maintain. There is a rising interest to explore the use of GPT models in health care. OBJECTIVE: We investigate the performance of GPT-2 and GPT-Neo models for medical text prediction using 374,787 free-text dental notes. METHODS: We fine-tune pretrained GPT-2 and GPT-Neo models for next word prediction on a dataset of over 374,000 manually written sections of dental clinical notes. Each model was trained on 80% of the dataset, validated on 10%, and tested on the remaining 10%. We report model performance in terms of next word prediction accuracy and loss. Additionally, we analyze the performance of the models on different types of prediction tokens for categories. For comparison, we also fine-tuned a non-GPT pretrained neural network model, XLNet (large), for next word prediction. We annotate each token in 100 randomly sampled notes by category (e.g., names, abbreviations, clinical terms, punctuation, etc.) and compare the performance of each model by token category. RESULTS: Models present acceptable accuracy scores (GPT-2: 76%; GPT-Neo: 53%), and the GPT-2 model also performs better in manual evaluations, especially for names, abbreviations, and punctuation. Both GPT models outperformed XLNet in terms of accuracy. The results suggest that pretrained models have the potential to assist medical charting in the future. We share the lessons learned, insights, and suggestions for future implementations. CONCLUSION: The results suggest that pretrained models have the potential to assist medical charting in the future. Our study presented one of the first implementations of the GPT model used with medical notes.

A large collection of real-world pediatric sleep studies.

Despite being crucial to health and quality of life, sleep-especially pediatric sleep-is not yet well understood. This is exacerbated by lack of access to sufficient pediatric sleep data with clinical annotation. In order to accelerate research on pediatric sleep and its connection to health, we create the Nationwide Children's Hospital (NCH) Sleep DataBank and publish it at Physionet and the National Sleep Research Resource (NSRR), which is a large sleep data common with physiological data, clinical data, and tools for analyses. The NCH Sleep DataBank consists of 3,984 polysomnography studies and over 5.6 million clinical observations on 3,673 unique patients between 2017 and 2019 at NCH. The novelties of this dataset include: (1) large-scale sleep dataset suitable for discovering new insights via data mining, (2) explicit focus on pediatric patients, (3) gathered in a real-world clinical setting, and (4) the accompanying rich set of clinical data. The NCH Sleep DataBank is a valuable resource for advancing automatic sleep scoring and real-time sleep disorder prediction, among many other potential scientific discoveries.

Prevalence of Sensitive Terms in Clinical Notes Using Natural Language Processing Techniques: Observational Study.

BACKGROUND: With the increased sharing of electronic health information as required by the US 21st Century Cures Act, there is an increased risk of breaching patient, parent, or guardian confidentiality. The prevalence of sensitive terms in clinical notes is not known. OBJECTIVE: The aim of this study is to define sensitive terms that represent the documentation of content that may be private and determine the prevalence and characteristics of provider notes that contain sensitive terms. METHODS: Using keyword expansion, we defined a list of 781 sensitive terms. We searched all provider history and physical, progress, consult, and discharge summary notes for patients aged 0-21 years written between January 1, 2019, and December 31, 2019, for a direct string match of sensitive terms. We calculated the prevalence of notes with sensitive terms and characterized clinical encounters and patient characteristics. RESULTS: Sensitive terms were present in notes from every clinical context in all pediatric ages. Terms related to the mental health category were most used overall (254,975/1,338,297, 19.5%), but terms related to substance abuse and reproductive health were most common in patients aged 0-3 years. History and physical notes (19,854/34,771, 57.1%) and ambulatory progress notes (265,302/563,273, 47.1%) were most likely to include sensitive terms. The highest prevalence of notes with sensitive terms was found in pain management (950/1112, 85.4%) and child abuse (1092/1282, 85.2%) clinics. CONCLUSIONS: Notes containing sensitive terms are not limited to adolescent patients, specific note types, or certain specialties. Recognition of sensitive terms across all ages and clinical settings complicates efforts to protect patient and caregiver privacy in the era of information-blocking regulations.

Characterization of Electronic Health Record Use Outside Scheduled Clinic Hours Among Primary Care Pediatricians: Retrospective Descriptive Task Analysis of Electronic Health Record Access Log Data.

BACKGROUND: Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels because of a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting that physicians spend considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician clinic schedules to define the outside work time. OBJECTIVE: This study aimed to characterize EHR work outside scheduled clinic hours among primary care pediatricians using a retrospective descriptive task analysis of EHR access log data and actual physician clinic schedules to define work time. METHODS: We conducted a retrospective, exploratory, descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large Midwestern pediatric health center to quantify and identify actions completed outside scheduled clinic hours. Mixed-effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR work during scheduled clinic hours on the use of EHRs outside scheduled clinic hours. RESULTS: Primary care pediatricians (n=56) in this study generated 1,523,872 access log data points (across 1069 physician workdays) and spent an average of 4.4 (SD 2.0) hours and 0.8 (SD 0.8) hours per physician per workday engaged in EHRs during and outside scheduled clinic hours, respectively. Approximately three-quarters of the time working in EHR during or outside scheduled clinic hours was spent reviewing data and reports. Mixed-effects regression revealed no associations of age, sex, or clinical full-time equivalent status with EHR use during or outside scheduled clinic hours. CONCLUSIONS: For every hour primary care pediatricians spent engaged with the EHR during scheduled clinic hours, they spent approximately 10 minutes interacting with the EHR outside scheduled clinic hours. Most of their time (during and outside scheduled clinic hours) was spent reviewing data, records, and other information in EHR.

Pretrained transformer framework on pediatric claims data for population specific tasks.

The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset, followed by a discriminative fine-tuning on each population-specific task. The semantic meaning of medical events can be captured in the pre-training stage, and the effective knowledge transfer is completed through the task-aware fine-tuning stage. The fine-tuning process requires minimal parameter modification without changing the model architecture, which mitigates the data scarcity issue and helps train the deep learning model adequately on small patient cohorts. We conducted experiments on a real-world pediatric dataset with more than one million patient records. Experimental results on two downstream tasks demonstrated the effectiveness of our method: our general task-agnostic pre-training framework outperformed tailored task-specific models, achieving more than 10% higher in model performance as compared to baselines. In addition, our framework showed a potential to transfer learned knowledge from one institution to another, which may pave the way for future healthcare model pre-training across institutions.

Locating Youth Exposed to Parental Justice Involvement in the Electronic Health Record: Development of a Natural Language Processing Model.

BACKGROUND: Parental justice involvement (eg, prison, jail, parole, or probation) is an unfortunately common and disruptive household adversity for many US youths, disproportionately affecting families of color and rural families. Data on this adversity has not been captured routinely in pediatric health care settings, and if it is, it is not discrete nor able to be readily analyzed for purposes of research. OBJECTIVE: In this study, we outline our process training a state-of-the-art natural language processing model using unstructured clinician notes of one large pediatric health system to identify patients who have experienced a justice-involved parent. METHODS: Using the electronic health record database of a large Midwestern pediatric hospital-based institution from 2011-2019, we located clinician notes (of any type and written by any type of provider) that were likely to contain such evidence of family justice involvement via a justice-keyword search (eg, prison and jail). To train and validate the model, we used a labeled data set of 7500 clinician notes identifying whether the patient was ever exposed to parental justice involvement. We calculated the precision and recall of the model and compared those rates to the keyword search. RESULTS: The development of the machine learning model increased the precision (positive predictive value) of locating children affected by parental justice involvement in the electronic health record from 61% (a simple keyword search) to 92%. CONCLUSIONS: The use of machine learning may be a feasible approach to addressing the gaps in our understanding of the health and health services of underrepresented youth who encounter childhood adversities not routinely captured-particularly for children of justice-involved parents.

Operationalizing and Implementing Pretrained, Large Artificial Intelligence Linguistic Models in the US Health Care System: Outlook of Generative Pretrained Transformer 3 (GPT-3) as a Service Model.

Generative pretrained transformer models have been popular recently due to their enhanced capabilities and performance. In contrast to many existing artificial intelligence models, generative pretrained transformer models can perform with very limited training data. Generative pretrained transformer 3 (GPT-3) is one of the latest releases in this pipeline, demonstrating human-like logical and intellectual responses to prompts. Some examples include writing essays, answering complex questions, matching pronouns to their nouns, and conducting sentiment analyses. However, questions remain with regard to its implementation in health care, specifically in terms of operationalization and its use in clinical practice and research. In this viewpoint paper, we briefly introduce GPT-3 and its capabilities and outline considerations for its implementation and operationalization in clinical practice through a use case. The implementation considerations include (1) processing needs and information systems infrastructure, (2) operating costs, (3) model biases, and (4) evaluation metrics. In addition, we outline the following three major operational factors that drive the adoption of GPT-3 in the US health care system: (1) ensuring Health Insurance Portability and Accountability Act compliance, (2) building trust with health care providers, and (3) establishing broader access to the GPT-3 tools. This viewpoint can inform health care practitioners, developers, clinicians, and decision makers toward understanding the use of the powerful artificial intelligence tools integrated into hospital systems and health care.

Rapid Development of a Telehealth Patient Satisfaction Survey Using a Multi-Stakeholder Approach.

Introduction: The COVID-19 pandemic has hastened the adoption of telehealth and the drastic shift to an unfamiliar process may impose significant impact to the quality-of-care delivery. Many providers are interested in understanding the quality of their telehealth services from the patients' experience. Materials and Methods: A telehealth patient satisfaction survey (TPSS) was developed by using an iterative stakeholder-centered design approach, incorporating elements from validated telemedicine and customer service survey instruments, and meeting the operational needs and constraints. A cross-sectional study design was employed to collect survey responses from patients and families of a large pediatric hospital. Finally, we performed exploratory factor analysis (EFA) to extract latent constructs and factor loadings of the survey items to further explain relationships. Results: A 22-item TPSS closely matched the existing in-person patient satisfaction survey and mapped to a revised SERVPERF conceptual model that was proposed by the interdisciplinary committee. Survey was implemented in the HIPAA-compliant online platform REDCap® with survey link embedded in an automated Epic MyChart (Verona, WI) visit follow-up message. In total, 2,394 survey responses were collected between July 7, 2020, and September 2, 2020. EFA revealed three constructs (with factor loadings >0.30): admission process, perceived quality of services, and telehealth satisfaction. Conclusions: We reported the development of TPSS that met the operational needs of compatibility with existing data and possible comparison to in-person survey. The survey is short and yet covers both the clinical experience and telehealth usability, with acceptable survey validity.

Identifying the Factors Influencing Patients' Telehealth Visit Satisfaction: Survey Validation Through a Structural Equation Modeling Approach.

Introduction: The COVID-19 pandemic accelerated the adoption of telehealth as an alternative to in-person hospital visits. To understand the factors impacting the quality of telehealth services, there is a need for validated survey instruments and conceptual frameworks. The objective of this study is to validate a telehealth patient satisfaction survey by structural equation modeling (SEM) and determine the relationship between the factors in the proposed telehealth patient satisfaction model (TPSM). Methods: We conducted a cross-sectional survey of pediatric patients and families receiving care from a comprehensive pediatric hospital in the Midwest between September 2020 and January 2021. In total, 2,039 usable responses were collected. We used an SEM approach by performing confirmatory factor analysis with Diagonally Weighted Least Squares modeling and Partial Least Squares-Path Modeling to establish the structural validity and examined the relationships among the constructs of "Admission Process" (AP), "Perceived Quality of Service" (PQS), and "Telehealth Satisfaction" (TS). Results: Participants were predominantly White (75%) and English-speaking (95%) parents (85%) of patients (mean age of patients was 10.2 years old). The survey responses were collected from patients visiting 43 department specialties, whereas 50% were behavioral and occupational therapy patients. The structural model showed that the admission process (AP) had a strong positive impact on perceived quality of service (PQS) (p = 0.67, t = 36.1, p < 0.001). The PQS had a strong positive impact on telehealth satisfaction (TS) (p = 0.66, t = 31.8, p < 0.001). The AP had a low positive direct impact on TS (p = 0.16, t = 7.46, p < 0.05). Overall, AP and PQS explained 61% variances (R2) of TS. Conclusions: We validated a newly proposed TS assessment model by using SEM. The TPSM will inform researchers to better understand the influencing factors in TS and help health care systems to improve telehealth patient satisfaction through a validated model.

Measuring Telehealth Visit Length and Schedule Adherence Using Videoconferencing Data.

Background: The ability to measure clinical visit length is critical for operational efficiency, patient experience, and accurate billing. Despite the unprecedented surge in telehealth use in 2020, studies on visit length and schedule adherence in the telehealth setting are nonexistent in the literature. This article aims to demonstrate the use of videoconferencing data to measure telehealth visit length and schedule adherence. Materials and Methods: We used data from telehealth video visits at four clinical specialties at Nationwide Children's Hospital, including behavioral health (BH), speech pathology (SP), physical therapy/occupational therapy (PT/OT), and primary care (PC). We combined videoconferencing timestamp data with visit scheduling data to calculate the total visit length, examination length, and patient wait times. We also assessed schedule adherence, including patient on-time performance, examination on-time performance, provider schedule deviations, and schedule length deviations. Results: The analyses included a total of 175,876 telehealth video visits. On average, children with BH appointments spent a total of 57.2 min for each visit, followed by PT/OT (50.8 min), SP (42.1 min), and PC (25.0 min). The average patient wait times were 4.1 min (BH), 2.7 min (PT/OT), 2.8 min (SP), and 3.1 min (PC). The average examination lengths were 48.8 min (BH), 44.5 min (PT/OT), 34.9 min (SP), and 16.6 min (PC). Regardless of clinical specialty, actual examination lengths of most visits were shorter than the scheduled lengths, except that appointments scheduled for 15 min tended to run overtime. Conclusions: Videoconferencing data provide a low-cost, accurate, and readily available resource for measuring telehealth visit length and schedule adherence.