Chatbot for Social Need Screening and Resource Sharing With Vulnerable Families: Iterative Design and Evaluation Study.

BACKGROUND: Health outcomes are significantly influenced by unmet social needs. Although screening for social needs has become common in health care settings, there is often poor linkage to resources after needs are identified. The structural barriers (eg, staffing, time, and space) to helping address social needs could be overcome by a technology-based solution. OBJECTIVE: This study aims to present the design and evaluation of a chatbot, DAPHNE (Dialog-Based Assistant Platform for Healthcare and Needs Ecosystem), which screens for social needs and links patients and families to resources. METHODS: This research used a three-stage study approach: (1) an end-user survey to understand unmet needs and perception toward chatbots, (2) iterative design with interdisciplinary stakeholder groups, and (3) a feasibility and usability assessment. In study 1, a web-based survey was conducted with low-income US resident households (n=201). Following that, in study 2, web-based sessions were held with an interdisciplinary group of stakeholders (n=10) using thematic and content analysis to inform the chatbot's design and development. Finally, in study 3, the assessment on feasibility and usability was completed via a mix of a web-based survey and focus group interviews following scenario-based usability testing with community health workers (family advocates; n=4) and social workers (n=9). We reported descriptive statistics and chi-square test results for the household survey. Content analysis and thematic analysis were used to analyze qualitative data. Usability score was descriptively reported. RESULTS: Among the survey participants, employed and younger individuals reported a higher likelihood of using a chatbot to address social needs, in contrast to the oldest age group. Regarding designing the chatbot, the stakeholders emphasized the importance of provider-technology collaboration, inclusive conversational design, and user education. The participants found that the chatbot's capabilities met expectations and that the chatbot was easy to use (System Usability Scale score=72/100). However, there were common concerns about the accuracy of suggested resources, electronic health record integration, and trust with a chatbot. CONCLUSIONS: Chatbots can provide personalized feedback for families to identify and meet social needs. Our study highlights the importance of user-centered iterative design and development of chatbots for social needs. Future research should examine the efficacy, cost-effectiveness, and scalability of chatbot interventions to address social needs.

Systematic data quality assessment of electronic health record data to evaluate study-specific fitness: Report from the PRESERVE research study.

Study-specific data quality testing is an essential part of minimizing analytic errors, particularly for studies making secondary use of clinical data. We applied a systematic and reproducible approach for study-specific data quality testing to the analysis plan for PRESERVE, a 15-site, EHR-based observational study of chronic kidney disease in children. This approach integrated widely adopted data quality concepts with healthcare-specific evaluation methods. We implemented two rounds of data quality assessment. The first produced high-level evaluation using aggregate results from a distributed query, focused on cohort identification and main analytic requirements. The second focused on extended testing of row-level data centralized for analysis. We systematized reporting and cataloguing of data quality issues, providing institutional teams with prioritized issues for resolution. We tracked improvements and documented anomalous data for consideration during analyses. The checks we developed identified 115 and 157 data quality issues in the two rounds, involving completeness, data model conformance, cross-variable concordance, consistency, and plausibility, extending traditional data quality approaches to address more complex stratification and temporal patterns. Resolution efforts focused on higher priority issues, given finite study resources. In many cases, institutional teams were able to correct data extraction errors or obtain additional data, avoiding exclusion of 2 institutions entirely and resolving 123 other gaps. Other results identified complexities in measures of kidney function, bearing on the study's outcome definition. Where limitations such as these are intrinsic to clinical data, the study team must account for them in conducting analyses. This study rigorously evaluated fitness of data for intended use. The framework is reusable and built on a strong theoretical underpinning. Significant data quality issues that would have otherwise delayed analyses or made data unusable were addressed. This study highlights the need for teams combining subject-matter and informatics expertise to address data quality when working with real world data.

A machine learning-based phenotype for long COVID in children: An EHR-based study from the RECOVER program.

As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data. In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS- CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values. The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses.

Extracting Medical Information From Free-Text and Unstructured Patient-Generated Health Data Using Natural Language Processing Methods: Feasibility Study With Real-world Data.

BACKGROUND: Patient-generated health data (PGHD) captured via smart devices or digital health technologies can reflect an individual health journey. PGHD enables tracking and monitoring of personal health conditions, symptoms, and medications out of the clinic, which is crucial for self-care and shared clinical decisions. In addition to self-reported measures and structured PGHD (eg, self-screening, sensor-based biometric data), free-text and unstructured PGHD (eg, patient care note, medical diary) can provide a broader view of a patient's journey and health condition. Natural language processing (NLP) is used to process and analyze unstructured data to create meaningful summaries and insights, showing promise to improve the utilization of PGHD. OBJECTIVE: Our aim is to understand and demonstrate the feasibility of an NLP pipeline to extract medication and symptom information from real-world patient and caregiver data. METHODS: We report a secondary data analysis, using a data set collected from 24 parents of children with special health care needs (CSHCN) who were recruited via a nonrandom sampling approach. Participants used a voice-interactive app for 2 weeks, generating free-text patient notes (audio transcription or text entry). We built an NLP pipeline using a zero-shot approach (adaptive to low-resource settings). We used named entity recognition (NER) and medical ontologies (RXNorm and SNOMED CT [Systematized Nomenclature of Medicine Clinical Terms]) to identify medication and symptoms. Sentence-level dependency parse trees and part-of-speech tags were used to extract additional entity information using the syntactic properties of a note. We assessed the data; evaluated the pipeline with the patient notes; and reported the precision, recall, and F1 scores. RESULTS: In total, 87 patient notes are included (audio transcriptions n=78 and text entries n=9) from 24 parents who have at least one CSHCN. The participants were between the ages of 26 and 59 years. The majority were White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid- or upper-mid household income (n=15, 62.5%), and had higher level education (n=24, 58%). Out of 87 notes, 30 were drug and medication related, and 46 were symptom related. We captured medication instances (medication, unit, quantity, and date) and symptoms satisfactorily (precision >0.65, recall >0.77, F1>0.72). These results indicate the potential when using NER and dependency parsing through an NLP pipeline on information extraction from unstructured PGHD. CONCLUSIONS: The proposed NLP pipeline was found to be feasible for use with real-world unstructured PGHD to accomplish medication and symptom extraction. Unstructured PGHD can be leveraged to inform clinical decision-making, remote monitoring, and self-care including medical adherence and chronic disease management. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings (eg, a limited number of patient notes or training data).

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study.

BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

The Current State of Mobile Apps Owned by Large Pediatric Hospitals in the United States: Systematic Search and Analysis on Google Play and Apple App Stores.

BACKGROUND: Pediatric hospitals in the United States are increasingly leveraging patient-facing mobile apps as their digital front doors for patients, families, and caretakers. These mobile health apps are sanctioned by pediatric hospitals to inform the public or populations about pediatric care to provide individualized information, to enhance communication, and to improve patient experience. Yet the functionalities and user feedback of these hospital mobile apps have not been systematically investigated. OBJECTIVE: Our aim was to understand the current state of hospital-owned mobile apps provided by large pediatric hospitals, comparatively analyze and report the services provided, and identify potential gaps to inform developers and providers. The American Hospital Association defines large hospitals as those having a bed count of more than 400. METHODS: We conducted a systematic search on Google Play and Apple App Store to identify all hospital-owned mobile apps from the large pediatric hospitals included in our review. Our inclusion criteria were (1) apps provided by large pediatric hospitals; (2) hospital-owned apps available in Apple App Store and Google Play; and (3) apps that are provided for general populations. Specialty apps that serve specific user groups or populations focusing on education, telehealth, specific conditions or procedures, or apps intended for research or clinician use were excluded. The features and functionality of the included apps were examined. RESULTS: Of the 16 pediatric hospitals included in our review, 4 (25%) had no general patient-facing apps, 4 (25%) had one app, and 8 (50%) had more than one app available on Google Play or Apple App Store. The 12 hospitals with at least one mobile app had a combined total of 72 apps. Of these 72 apps, 61 (85%) were considered specialty and were excluded from our review, leaving a total of 11 (15%) apps to analyze. Among the 11 apps analyzed, the most common feature was appointment scheduling or reminder (n=9, 82%). Doctor search (n=8, 73%) and patient resources (n=8, 73%) were the second most common, followed by payment, billing, or claims (n=7, 64%), patient portal integration (n=6, 55%), personal health management (n=6, 55%), hospital way finding (n=5, 45%), message a provider (n=4, 36%), urgent care wait times (n=4, 36%), video chat (n=4, 36%), and health information access (n=4, 36%). Parking information (n=3, 27%) was the least common. CONCLUSIONS: Out of the 16 pediatric hospitals identified for our review, 75% (n=12) offer mobile apps. Based on the most common features, these apps were intended to help improve accessibility for patients and families in terms of finding providers, scheduling appointments, and accessing patient resources. We believe the findings will inform pediatric hospital administrators, developers, and other stakeholders to improve app feature offerings and increase their impact on service accessibility and patient experience.

A large collection of real-world pediatric sleep studies.

Despite being crucial to health and quality of life, sleep-especially pediatric sleep-is not yet well understood. This is exacerbated by lack of access to sufficient pediatric sleep data with clinical annotation. In order to accelerate research on pediatric sleep and its connection to health, we create the Nationwide Children's Hospital (NCH) Sleep DataBank and publish it at Physionet and the National Sleep Research Resource (NSRR), which is a large sleep data common with physiological data, clinical data, and tools for analyses. The NCH Sleep DataBank consists of 3,984 polysomnography studies and over 5.6 million clinical observations on 3,673 unique patients between 2017 and 2019 at NCH. The novelties of this dataset include: (1) large-scale sleep dataset suitable for discovering new insights via data mining, (2) explicit focus on pediatric patients, (3) gathered in a real-world clinical setting, and (4) the accompanying rich set of clinical data. The NCH Sleep DataBank is a valuable resource for advancing automatic sleep scoring and real-time sleep disorder prediction, among many other potential scientific discoveries.

Prevalence of Sensitive Terms in Clinical Notes Using Natural Language Processing Techniques: Observational Study.

BACKGROUND: With the increased sharing of electronic health information as required by the US 21st Century Cures Act, there is an increased risk of breaching patient, parent, or guardian confidentiality. The prevalence of sensitive terms in clinical notes is not known. OBJECTIVE: The aim of this study is to define sensitive terms that represent the documentation of content that may be private and determine the prevalence and characteristics of provider notes that contain sensitive terms. METHODS: Using keyword expansion, we defined a list of 781 sensitive terms. We searched all provider history and physical, progress, consult, and discharge summary notes for patients aged 0-21 years written between January 1, 2019, and December 31, 2019, for a direct string match of sensitive terms. We calculated the prevalence of notes with sensitive terms and characterized clinical encounters and patient characteristics. RESULTS: Sensitive terms were present in notes from every clinical context in all pediatric ages. Terms related to the mental health category were most used overall (254,975/1,338,297, 19.5%), but terms related to substance abuse and reproductive health were most common in patients aged 0-3 years. History and physical notes (19,854/34,771, 57.1%) and ambulatory progress notes (265,302/563,273, 47.1%) were most likely to include sensitive terms. The highest prevalence of notes with sensitive terms was found in pain management (950/1112, 85.4%) and child abuse (1092/1282, 85.2%) clinics. CONCLUSIONS: Notes containing sensitive terms are not limited to adolescent patients, specific note types, or certain specialties. Recognition of sensitive terms across all ages and clinical settings complicates efforts to protect patient and caregiver privacy in the era of information-blocking regulations.

Characterization of Electronic Health Record Use Outside Scheduled Clinic Hours Among Primary Care Pediatricians: Retrospective Descriptive Task Analysis of Electronic Health Record Access Log Data.

BACKGROUND: Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels because of a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting that physicians spend considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician clinic schedules to define the outside work time. OBJECTIVE: This study aimed to characterize EHR work outside scheduled clinic hours among primary care pediatricians using a retrospective descriptive task analysis of EHR access log data and actual physician clinic schedules to define work time. METHODS: We conducted a retrospective, exploratory, descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large Midwestern pediatric health center to quantify and identify actions completed outside scheduled clinic hours. Mixed-effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR work during scheduled clinic hours on the use of EHRs outside scheduled clinic hours. RESULTS: Primary care pediatricians (n=56) in this study generated 1,523,872 access log data points (across 1069 physician workdays) and spent an average of 4.4 (SD 2.0) hours and 0.8 (SD 0.8) hours per physician per workday engaged in EHRs during and outside scheduled clinic hours, respectively. Approximately three-quarters of the time working in EHR during or outside scheduled clinic hours was spent reviewing data and reports. Mixed-effects regression revealed no associations of age, sex, or clinical full-time equivalent status with EHR use during or outside scheduled clinic hours. CONCLUSIONS: For every hour primary care pediatricians spent engaged with the EHR during scheduled clinic hours, they spent approximately 10 minutes interacting with the EHR outside scheduled clinic hours. Most of their time (during and outside scheduled clinic hours) was spent reviewing data, records, and other information in EHR.

"Hey Siri, Help Me Take Care of My Child": A Feasibility Study With Caregivers of Children With Special Healthcare Needs Using Voice Interaction and Automatic Speech Recognition in Remote Care Management.

Background: About 23% of households in the United States have at least one child who has special healthcare needs. As most care activities occur at home, there is often a disconnect and lack of communication between families, home care nurses, and healthcare providers. Digital health technologies may help bridge this gap. Objective: We conducted a pre-post study with a voice-enabled medical note taking (diary) app (SpeakHealth) in a real world setting with caregivers (parents, family members) of children with special healthcare needs (CSHCN) to understand feasibility of voice interaction and automatic speech recognition (ASR) for medical note taking at home. Methods: In total, 41 parents of CSHCN were recruited. Participants completed a pre-study survey collecting demographic details, technology and care management preferences. Out of 41, 24 participants completed the study, using the app for 2 weeks and completing an exit survey. The app facilitated caregiver note-taking using voice interaction and ASR. An exit survey was conducted to collect feedback on technology adoption and changes in technology preferences in care management. We assessed the feasibility of the app by descriptively analyzing survey responses and user data following the key focus areas of acceptability, demand, implementation and integration, adaptation and expansion. In addition, perceived effectiveness of the app was assessed by comparing perceived changes in mobile app preferences among participants. In addition, the voice data, notes, and transcriptions were descriptively analyzed for understanding the feasibility of the app. Results: The majority of the recruited parents were 35-44 years old (22, 53.7%), part of a two-parent household (30, 73.2%), white (37, 90.2%), had more than one child (31, 75.6%), lived in Ohio (37, 90.2%), used mobile health apps, mobile note taking apps or calendar apps (28, 68.3%) and patient portal apps (22, 53.7%) to track symptoms and health events at home. Caregivers had experience with voice technology as well (32, 78%). Among those completed the post-study survey (in Likert Scale 1-5), ~80% of the caregivers agreed or strongly agreed that using the app would enhance their performance in completing tasks (perceived usefulness; mean = 3.4, SD = 0.8), the app is free of effort (perceived ease of use; mean = 3.2, SD = 0.9), and they would use the app in the future (behavioral intention; mean = 3.1, SD = 0.9). In total, 88 voice interactive patient notes were generated with the majority of the voice recordings being less than 20 s in length (66%). Most noted symptoms and conditions, medications, treatment and therapies, and patient behaviors. More than half of the caregivers reported that voice interaction with the app and using transcribed notes positively changed their preference of technology to use and methods for tracking symptoms and health events at home. Conclusions: Our findings suggested that voice interaction and ASR use in mobile apps are feasible and effective in keeping track of symptoms and health events at home. Future work is suggested toward using integrated and intelligent systems with voice interactions with broader populations.

Associations of Telehealth Care Delivery with Pediatric Health Care Provider Well-Being.

BACKGROUND: The rapid, large-scale deployment of new health technologies can introduce challenges to clinicians who are already under stress. The novel coronavirus disease 19 (COVID-19) pandemic transformed health care in the United States to include a telehealth model of care delivery. Clarifying paths through which telehealth technology use is associated with change in provider well-being and interest in sustaining virtual care delivery can inform planning and optimization efforts. OBJECTIVE: This study aimed to characterize provider-reported changes in well-being and daily work associated with the pandemic-accelerated expansion of telehealth and assess the relationship of provider perceptions of telehealth effectiveness, efficiency, and work-life balance with desire for future telehealth. METHODS: A cross-sectional survey study was conducted October through November 2020, 6 months after the outbreak of COVID-19 at three children's hospitals. Factor analysis and structural equation modeling (SEM) were used to examine telehealth factors associated with reported change in well-being and desire for future telehealth. RESULTS: A total of 947 nontrainee physicians, advanced practice providers, and psychologists were surveyed. Of them, 502 (53.0%) providers responded and 467 (49.3%) met inclusion criteria of telehealth use during the study period. Of these, 325 (69.6%) were female, 301 (65.6%) were physicians, and 220 (47.1%) were medical subspecialists. Providers were 4.77 times as likely (95% confidence interval [CI]: 3.29-7.06) to report improved versus worsened well-being associated with telehealth. Also, 95.5% of providers (95% CI: 93.2-97.2%) wish to continue performing telehealth postpandemic. Our model explains 66% of the variance in telehealth-attributed provider well-being and 59% of the variance for future telehealth preference and suggests telehealth resources significantly influence provider-perceived telehealth care effectiveness which in turn significantly influences provider well-being and desire to perform telehealth. CONCLUSION: Telehealth has potential to promote provider well-being; telehealth-related changes in provider well-being are associated with both provider-perceived effectiveness of telemedicine for patients and adequacy of telehealth resources.

Rapid Development of a Telehealth Patient Satisfaction Survey Using a Multi-Stakeholder Approach.

Introduction: The COVID-19 pandemic has hastened the adoption of telehealth and the drastic shift to an unfamiliar process may impose significant impact to the quality-of-care delivery. Many providers are interested in understanding the quality of their telehealth services from the patients' experience. Materials and Methods: A telehealth patient satisfaction survey (TPSS) was developed by using an iterative stakeholder-centered design approach, incorporating elements from validated telemedicine and customer service survey instruments, and meeting the operational needs and constraints. A cross-sectional study design was employed to collect survey responses from patients and families of a large pediatric hospital. Finally, we performed exploratory factor analysis (EFA) to extract latent constructs and factor loadings of the survey items to further explain relationships. Results: A 22-item TPSS closely matched the existing in-person patient satisfaction survey and mapped to a revised SERVPERF conceptual model that was proposed by the interdisciplinary committee. Survey was implemented in the HIPAA-compliant online platform REDCap® with survey link embedded in an automated Epic MyChart (Verona, WI) visit follow-up message. In total, 2,394 survey responses were collected between July 7, 2020, and September 2, 2020. EFA revealed three constructs (with factor loadings >0.30): admission process, perceived quality of services, and telehealth satisfaction. Conclusions: We reported the development of TPSS that met the operational needs of compatibility with existing data and possible comparison to in-person survey. The survey is short and yet covers both the clinical experience and telehealth usability, with acceptable survey validity.

Identifying the Factors Influencing Patients' Telehealth Visit Satisfaction: Survey Validation Through a Structural Equation Modeling Approach.

Introduction: The COVID-19 pandemic accelerated the adoption of telehealth as an alternative to in-person hospital visits. To understand the factors impacting the quality of telehealth services, there is a need for validated survey instruments and conceptual frameworks. The objective of this study is to validate a telehealth patient satisfaction survey by structural equation modeling (SEM) and determine the relationship between the factors in the proposed telehealth patient satisfaction model (TPSM). Methods: We conducted a cross-sectional survey of pediatric patients and families receiving care from a comprehensive pediatric hospital in the Midwest between September 2020 and January 2021. In total, 2,039 usable responses were collected. We used an SEM approach by performing confirmatory factor analysis with Diagonally Weighted Least Squares modeling and Partial Least Squares-Path Modeling to establish the structural validity and examined the relationships among the constructs of "Admission Process" (AP), "Perceived Quality of Service" (PQS), and "Telehealth Satisfaction" (TS). Results: Participants were predominantly White (75%) and English-speaking (95%) parents (85%) of patients (mean age of patients was 10.2 years old). The survey responses were collected from patients visiting 43 department specialties, whereas 50% were behavioral and occupational therapy patients. The structural model showed that the admission process (AP) had a strong positive impact on perceived quality of service (PQS) (p = 0.67, t = 36.1, p < 0.001). The PQS had a strong positive impact on telehealth satisfaction (TS) (p = 0.66, t = 31.8, p < 0.001). The AP had a low positive direct impact on TS (p = 0.16, t = 7.46, p < 0.05). Overall, AP and PQS explained 61% variances (R2) of TS. Conclusions: We validated a newly proposed TS assessment model by using SEM. The TPSM will inform researchers to better understand the influencing factors in TS and help health care systems to improve telehealth patient satisfaction through a validated model.

Turning Digital Trails into a Telehealth Competitive Edge.

Digital trails, data collections of individuals' traceable digital activities online or on digital devices, have been utilized by many industries to provide valuable insights to enhance customer experience, improve operation efficiency, and increase revenues. Despite the abundance of digital trails among health care data, health care has lagged behind other industries in extracting their values. Recently, telehealth's accelerated adoption due to the COVID-19 pandemic provides an unprecedented opportunity for health care providers to take advantage of digital trails. In this study, we describe digital trails generated from the telehealth workflow and discuss a few use cases to demonstrate how telehealth digital trails can be used to improve clinical service quality, streamline patient care workflow, and enhance the patient experience.

Measuring Telehealth Visit Length and Schedule Adherence Using Videoconferencing Data.

Background: The ability to measure clinical visit length is critical for operational efficiency, patient experience, and accurate billing. Despite the unprecedented surge in telehealth use in 2020, studies on visit length and schedule adherence in the telehealth setting are nonexistent in the literature. This article aims to demonstrate the use of videoconferencing data to measure telehealth visit length and schedule adherence. Materials and Methods: We used data from telehealth video visits at four clinical specialties at Nationwide Children's Hospital, including behavioral health (BH), speech pathology (SP), physical therapy/occupational therapy (PT/OT), and primary care (PC). We combined videoconferencing timestamp data with visit scheduling data to calculate the total visit length, examination length, and patient wait times. We also assessed schedule adherence, including patient on-time performance, examination on-time performance, provider schedule deviations, and schedule length deviations. Results: The analyses included a total of 175,876 telehealth video visits. On average, children with BH appointments spent a total of 57.2 min for each visit, followed by PT/OT (50.8 min), SP (42.1 min), and PC (25.0 min). The average patient wait times were 4.1 min (BH), 2.7 min (PT/OT), 2.8 min (SP), and 3.1 min (PC). The average examination lengths were 48.8 min (BH), 44.5 min (PT/OT), 34.9 min (SP), and 16.6 min (PC). Regardless of clinical specialty, actual examination lengths of most visits were shorter than the scheduled lengths, except that appointments scheduled for 15 min tended to run overtime. Conclusions: Videoconferencing data provide a low-cost, accurate, and readily available resource for measuring telehealth visit length and schedule adherence.

A machine learning-based phenotype for long COVID in children: an EHR-based study from the RECOVER program.

Background: As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data. Methods and Findings: In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS-CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values. Conclusions: The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses. Funding Source: This research was funded by the National Institutes of Health (NIH) Agreement OT2HL161847-01 as part of the Researching COVID to Enhance Recovery (RECOVER) program of research. Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the RECOVER Program, the NIH or other funders.

A natural language processing pipeline to synthesize patient-generated notes toward improving remote care and chronic disease management: a cystic fibrosis case study.

OBJECTIVES: Patient-generated health data (PGHD) are important for tracking and monitoring out of clinic health events and supporting shared clinical decisions. Unstructured text as PGHD (eg, medical diary notes and transcriptions) may encapsulate rich information through narratives which can be critical to better understand a patient's condition. We propose a natural language processing (NLP) supported data synthesis pipeline for unstructured PGHD, focusing on children with special healthcare needs (CSHCN), and demonstrate it with a case study on cystic fibrosis (CF). MATERIALS AND METHODS: The proposed unstructured data synthesis and information extraction pipeline extract a broad range of health information by combining rule-based approaches with pretrained deep-learning models. Particularly, we build upon the scispaCy biomedical model suite, leveraging its named entity recognition capabilities to identify and link clinically relevant entities to established ontologies such as Systematized Nomenclature of Medicine (SNOMED) and RXNORM. We then use scispaCy's syntax (grammar) parsing tools to retrieve phrases associated with the entities in medication, dose, therapies, symptoms, bowel movements, and nutrition ontological categories. The pipeline is illustrated and tested with simulated CF patient notes. RESULTS: The proposed hybrid deep-learning rule-based approach can operate over a variety of natural language note types and allow customization for a given patient or cohort. Viable information was successfully extracted from simulated CF notes. This hybrid pipeline is robust to misspellings and varied word representations and can be tailored to accommodate the needs of a specific patient, cohort, or clinician. DISCUSSION: The NLP pipeline can extract predefined or ontology-based entities from free-text PGHD, aiming to facilitate remote care and improve chronic disease management. Our implementation makes use of open source models, allowing for this solution to be easily replicated and integrated in different health systems. Outside of the clinic, the use of the NLP pipeline may increase the amount of clinical data recorded by families of CSHCN and ease the process to identify health events from the notes. Similarly, care coordinators, nurses and clinicians would be able to track adherence with medications, identify symptoms, and effectively intervene to improve clinical care. Furthermore, visualization tools can be applied to digest the structured data produced by the pipeline in support of the decision-making process for a patient, caregiver, or provider. CONCLUSION: Our study demonstrated that an NLP pipeline can be used to create an automated analysis and reporting mechanism for unstructured PGHD. Further studies are suggested with real-world data to assess pipeline performance and further implications.

Assessment of Prevalence of Adolescent Patient Portal Account Access by Guardians.

Importance: Patient portals can be configured to allow confidential communication for adolescents' sensitive health care information. Guardian access of adolescent patient portal accounts could compromise adolescents' confidentiality. Objective: To estimate the prevalence of guardian access to adolescent patient portals at 3 academic children's hospitals. Design, Setting, and Participants: A cross-sectional study to estimate the prevalence of guardian access to adolescent patient portal accounts was conducted at 3 academic children's hospitals. Adolescent patients (aged 13-18 years) with access to their patient portal account with at least 1 outbound message from their portal during the study period were included. A rule-based natural language processing algorithm was used to analyze all portal messages from June 1, 2014, to February 28, 2020, and identify any message sent by guardians. The sensitivity and specificity of the algorithm at each institution was estimated through manual review of a stratified subsample of patient accounts. The overall proportion of accounts with guardian access was estimated after correcting for the sensitivity and specificity of the natural language processing algorithm. Exposures: Use of patient portal. Main Outcome and Measures: Percentage of adolescent portal accounts indicating guardian access. Results: A total of 3429 eligible adolescent accounts containing 25 642 messages across 3 institutions were analyzed. A total of 1797 adolescents (52%) were female and mean (SD) age was 15.6 (1.6) years. The percentage of adolescent portal accounts with apparent guardian access ranged from 52% to 57% across the 3 institutions. After correcting for the sensitivity and specificity of the algorithm based on manual review of 200 accounts per institution, an estimated 64% (95% CI, 59%-69%) to 76% (95% CI, 73%-88%) of accounts with outbound messages were accessed by guardians across the 3 institutions. Conclusions and Relevance: In this study, more than half of adolescent accounts with outbound messages were estimated to have been accessed by guardians at least once. These findings have implications for health systems intending to rely on separate adolescent accounts to protect adolescent confidentiality.

Feasibility of a Voice-Enabled Medical Diary App (SpeakHealth) for Caregivers of Children With Special Health Care Needs and Health Care Providers: Mixed Methods Study.

BACKGROUND: Children with special health care needs (CSHCN) require more than the usual care management and coordination efforts from caregivers and health care providers (HCPs). Health information and communication technologies can potentially facilitate these efforts to increase the quality of care received by CSHCN. OBJECTIVE: In this study, we aim to assess the feasibility of a voice-enabled medical diary app (SpeakHealth) by investigating its potential use among caregivers and HCPs. METHODS: Following a mixed methods approach, caregivers of CSHCN were interviewed (n=10) and surveyed (n=86) about their care management and communication technology use. Only interviewed participants were introduced to the SpeakHealth app prototype, and they tested the app during the interview session. In addition, we interviewed complex care HCPs (n=15) to understand their perception of the value of a home medical diary such as the SpeakHealth app. Quantitative data were analyzed using descriptive statistics and correlational analyses. Theoretical thematic analysis was used to analyze qualitative data. RESULTS: The survey results indicated a positive attitude toward voice-enabled technology and features; however, there was no strong correlation among the measured items. The caregivers identified communication, information sharing, tracking medication, and appointments as fairly and highly important features of the app. Qualitative analysis revealed the following two overarching themes: enablers and barriers in care communication and enablers and barriers in communication technologies. The subthemes included parent roles, care communication technologies, and challenges. HCPs found the SpeakHealth app to be a promising tool for timely information collection that could be available for sharing information with the health system. Overall, the findings demonstrated a variety of needs and challenges for caregivers of CSHCN and opportunities for voice-enabled, interactive medical diary apps in care management and coordination. Caregivers fundamentally look for better information sharing and communication with HCPs. Health care and communication technologies can potentially improve care communication and coordination in addressing the patient and caregiver needs. CONCLUSIONS: The perspectives of caregivers and providers suggested both benefits and challenges in using the SpeakHealth app for medical note-taking and tracking health events at home. Our findings could inform researchers and developers about the potential development and use of a voice-enabled medical diary app.

Delivering Perinatal Health Information via a Voice Interactive App (SMILE): Mixed Methods Feasibility Study.

BACKGROUND: Perinatal health care is critically important for maternal health outcomes in infants. The United States fares considerably worse than comparable countries for maternal and infant mortality rates. As such, alternative models of care or engagement are warranted. Ubiquitous digital devices and increased use of digital health tools have the potential to extend the reach to women and infants in their everyday lives and make a positive impact on their health outcomes. As voice technology becomes more mainstream, research is prudent to establish evidence-based practice on how to best leverage voice technology to promote maternal-infant health. OBJECTIVE: The aim of this study is to assess the feasibility of using voice technology to support perinatal health and infant care practices. METHODS: Perinatal women were recruited from a large Midwest Children's Hospital via hospital email announcements and word of mouth. Owing to the technical aspects of the intervention, participants were required to speak English and use an iPhone. Demographics, patterns of technology use, and technology use specific to perinatal health or self-care practices were assessed at baseline. Next, participants were onboarded and asked to use the intervention, Self-Management Intervention-Life Essentials (SMILE), over the course of 2 weeks. SMILE provided users with perinatal health content delivered through mini podcasts (ranging from 3 to 8 minutes in duration). After each podcast, SMILE prompted users to provide immediate verbal feedback to the content. An exit interview was conducted with participants to gather feedback on the intervention and further explore participants' perceptions of voice technology as a means to support perinatal health in the future. RESULTS: In total, 19 pregnant women (17 to 36 weeks pregnant) were consented. Themes identified as important for perinatal health information include establishing routines, expected norms, and realistic expectations and providing key takeaways. Themes identified as important for voice interaction include customization and user preferences, privacy, family and friends, and context and convenience. Qualitative analysis suggested that perinatal health promotion content delivered by voice should be accurate and succinctly delivered and highlight key takeaways. Perinatal health interventions that use voice should provide users with the ability to customize the intervention but also provide opportunities to engage family members, particularly spouses. As a number of women multitasked while the intervention was being deployed, future interventions should leverage the convenience of voice technology while also balancing the influence of user context (eg, timing or ability to listen or talk versus nonvoice interaction with the system). CONCLUSIONS: Our findings demonstrate the short-term feasibility of disseminating evidence-based perinatal support via podcasts and curate voice-captured data from perinatal women. However, key areas of improvement have been identified specifically for perinatal interventions leveraging voice technology. Findings contribute to future content, design, and delivery considerations of perinatal digital health interventions.