Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study.

BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

Turning Digital Trails into a Telehealth Competitive Edge.

Digital trails, data collections of individuals' traceable digital activities online or on digital devices, have been utilized by many industries to provide valuable insights to enhance customer experience, improve operation efficiency, and increase revenues. Despite the abundance of digital trails among health care data, health care has lagged behind other industries in extracting their values. Recently, telehealth's accelerated adoption due to the COVID-19 pandemic provides an unprecedented opportunity for health care providers to take advantage of digital trails. In this study, we describe digital trails generated from the telehealth workflow and discuss a few use cases to demonstrate how telehealth digital trails can be used to improve clinical service quality, streamline patient care workflow, and enhance the patient experience.

Measuring Telehealth Visit Length and Schedule Adherence Using Videoconferencing Data.

Background: The ability to measure clinical visit length is critical for operational efficiency, patient experience, and accurate billing. Despite the unprecedented surge in telehealth use in 2020, studies on visit length and schedule adherence in the telehealth setting are nonexistent in the literature. This article aims to demonstrate the use of videoconferencing data to measure telehealth visit length and schedule adherence. Materials and Methods: We used data from telehealth video visits at four clinical specialties at Nationwide Children's Hospital, including behavioral health (BH), speech pathology (SP), physical therapy/occupational therapy (PT/OT), and primary care (PC). We combined videoconferencing timestamp data with visit scheduling data to calculate the total visit length, examination length, and patient wait times. We also assessed schedule adherence, including patient on-time performance, examination on-time performance, provider schedule deviations, and schedule length deviations. Results: The analyses included a total of 175,876 telehealth video visits. On average, children with BH appointments spent a total of 57.2 min for each visit, followed by PT/OT (50.8 min), SP (42.1 min), and PC (25.0 min). The average patient wait times were 4.1 min (BH), 2.7 min (PT/OT), 2.8 min (SP), and 3.1 min (PC). The average examination lengths were 48.8 min (BH), 44.5 min (PT/OT), 34.9 min (SP), and 16.6 min (PC). Regardless of clinical specialty, actual examination lengths of most visits were shorter than the scheduled lengths, except that appointments scheduled for 15 min tended to run overtime. Conclusions: Videoconferencing data provide a low-cost, accurate, and readily available resource for measuring telehealth visit length and schedule adherence.

Identifying the Factors Influencing Patients' Telehealth Visit Satisfaction: Survey Validation Through a Structural Equation Modeling Approach.

Introduction: The COVID-19 pandemic accelerated the adoption of telehealth as an alternative to in-person hospital visits. To understand the factors impacting the quality of telehealth services, there is a need for validated survey instruments and conceptual frameworks. The objective of this study is to validate a telehealth patient satisfaction survey by structural equation modeling (SEM) and determine the relationship between the factors in the proposed telehealth patient satisfaction model (TPSM). Methods: We conducted a cross-sectional survey of pediatric patients and families receiving care from a comprehensive pediatric hospital in the Midwest between September 2020 and January 2021. In total, 2,039 usable responses were collected. We used an SEM approach by performing confirmatory factor analysis with Diagonally Weighted Least Squares modeling and Partial Least Squares-Path Modeling to establish the structural validity and examined the relationships among the constructs of "Admission Process" (AP), "Perceived Quality of Service" (PQS), and "Telehealth Satisfaction" (TS). Results: Participants were predominantly White (75%) and English-speaking (95%) parents (85%) of patients (mean age of patients was 10.2 years old). The survey responses were collected from patients visiting 43 department specialties, whereas 50% were behavioral and occupational therapy patients. The structural model showed that the admission process (AP) had a strong positive impact on perceived quality of service (PQS) (p = 0.67, t = 36.1, p < 0.001). The PQS had a strong positive impact on telehealth satisfaction (TS) (p = 0.66, t = 31.8, p < 0.001). The AP had a low positive direct impact on TS (p = 0.16, t = 7.46, p < 0.05). Overall, AP and PQS explained 61% variances (R2) of TS. Conclusions: We validated a newly proposed TS assessment model by using SEM. The TPSM will inform researchers to better understand the influencing factors in TS and help health care systems to improve telehealth patient satisfaction through a validated model.

Rapid Development of a Telehealth Patient Satisfaction Survey Using a Multi-Stakeholder Approach.

Introduction: The COVID-19 pandemic has hastened the adoption of telehealth and the drastic shift to an unfamiliar process may impose significant impact to the quality-of-care delivery. Many providers are interested in understanding the quality of their telehealth services from the patients' experience. Materials and Methods: A telehealth patient satisfaction survey (TPSS) was developed by using an iterative stakeholder-centered design approach, incorporating elements from validated telemedicine and customer service survey instruments, and meeting the operational needs and constraints. A cross-sectional study design was employed to collect survey responses from patients and families of a large pediatric hospital. Finally, we performed exploratory factor analysis (EFA) to extract latent constructs and factor loadings of the survey items to further explain relationships. Results: A 22-item TPSS closely matched the existing in-person patient satisfaction survey and mapped to a revised SERVPERF conceptual model that was proposed by the interdisciplinary committee. Survey was implemented in the HIPAA-compliant online platform REDCap® with survey link embedded in an automated Epic MyChart (Verona, WI) visit follow-up message. In total, 2,394 survey responses were collected between July 7, 2020, and September 2, 2020. EFA revealed three constructs (with factor loadings >0.30): admission process, perceived quality of services, and telehealth satisfaction. Conclusions: We reported the development of TPSS that met the operational needs of compatibility with existing data and possible comparison to in-person survey. The survey is short and yet covers both the clinical experience and telehealth usability, with acceptable survey validity.

Graph embedding on biomedical networks: methods, applications and evaluations.

MOTIVATION: Graph embedding learning that aims to automatically learn low-dimensional node representations, has drawn increasing attention in recent years. To date, most recent graph embedding methods are evaluated on social and information networks and are not comprehensively studied on biomedical networks under systematic experiments and analyses. On the other hand, for a variety of biomedical network analysis tasks, traditional techniques such as matrix factorization (which can be seen as a type of graph embedding methods) have shown promising results, and hence there is a need to systematically evaluate the more recent graph embedding methods (e.g. random walk-based and neural network-based) in terms of their usability and potential to further the state-of-the-art. RESULTS: We select 11 representative graph embedding methods and conduct a systematic comparison on 3 important biomedical link prediction tasks: drug-disease association (DDA) prediction, drug-drug interaction (DDI) prediction, protein-protein interaction (PPI) prediction; and 2 node classification tasks: medical term semantic type classification, protein function prediction. Our experimental results demonstrate that the recent graph embedding methods achieve promising results and deserve more attention in the future biomedical graph analysis. Compared with three state-of-the-art methods for DDAs, DDIs and protein function predictions, the recent graph embedding methods achieve competitive performance without using any biological features and the learned embeddings can be treated as complementary representations for the biological features. By summarizing the experimental results, we provide general guidelines for properly selecting graph embedding methods and setting their hyper-parameters for different biomedical tasks. AVAILABILITY AND IMPLEMENTATION: As part of our contributions in the paper, we develop an easy-to-use Python package with detailed instructions, BioNEV, available at: https://github.com/xiangyue9607/BioNEV, including all source code and datasets, to facilitate studying various graph embedding methods on biomedical tasks. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Documented Reasons of Cancellation and Rescheduling of Telehealth Appointments During the Pandemic.

Background and Objective: The COVID-19 pandemic increased the use of telehealth around the world. The aim is to minimize health care service disruption as well as reducing COVID-19 exposure. However, one of the major operational concerns is cancellations and rescheduling (C/Rs). C/Rs may create additional burden and cost to the patient, provider, and the health system. Our aim is to understand the reasons for C/Rs of the telehealth session after the scheduled start time. Materials and Methods: We reviewed electronic health records (EHRs) to identify the C/R reasons for behavioral health and speech language pathology departments. Documented C/Rs in the medical charts were identified from EHR by using a keyword-based and Natural Language Processing (NLP)-supported EHR search engine. From the search results, we randomly selected 200 notes and conducted a thematic analysis. Results: We identified four themes explaining C/R reasons. Most frequent theme was "technicality" (47, 36%), followed by "engagement" (34, 25%), "scheduling" (31, 24%), and "unspecified" (20, 15%). The findings showed that technical reasons are the leading cause of C/Rs, constituting 36% of the cases (95% confidence interval [CI]: 29-43%). Notably, "engagement" constituted a sizeable 25% (95% CI: 19-31%) of C/Rs, as a result of the inability to engage a patient to complete the telehealth session. Conclusions: The study shows that engagement is one of the new challenges to the pediatric telehealth visits. Future studies of new engagement models are needed for the success of telehealth. Our findings will help fill the literature gaps and may help with enhancing the digital experience for both caregivers and providers, reducing wasted time and resources due to preventable C/Rs, improving clinical operation efficiency, and treatment adherence.

Capturing At-Home Health and Care Information for Children With Medical Complexity Using Voice Interactive Technologies: Multi-Stakeholder Viewpoint.

Digital health tools and technologies are transforming health care and making significant impacts on how health and care information are collected, used, and shared to achieve best outcomes. As most of the efforts are still focused on clinical settings, the wealth of health information generated outside of clinical settings is not being fully tapped. This is especially true for children with medical complexity (CMC) and their families, as they frequently spend significant hours providing hands-on medical care within the home setting and coordinating activities among multiple providers and other caregivers. In this paper, a multidisciplinary team of stakeholders discusses the value of health information generated at home, how technology can enhance care coordination, and challenges of technology adoption from a patient-centered perspective. Voice interactive technology has been identified to have the potential to transform care coordination for CMC. This paper shares opinions on the promises, limitations, recommended approaches, and challenges of adopting voice technology in health care, especially for the targeted patient population of CMC.

Comorbidities in Childhood Celiac Disease: A Phenome Wide Association Study Using the Electronic Health Record.

OBJECTIVES: Celiac disease (CD) is associated with a variety of extraintestinal autoimmune and inflammatory findings that manifest clinically as symptoms and comorbidities. Understanding these comorbidities may improve identification of the disease and prevent sequelae. In this study, we use an unbiased electronic health record (EHR)-based Phenome Wide Association Study (PheWAS) method to confirm known comorbidities, discover novel associations and enhance characterization of the clinical presentation of CD in children. METHODS: Data were extracted from the Nationwide Children's Hospital EHR. Confirmed CD cases (n = 433) were matched with 4330 randomly selected controls. Utilizing an EHR-based PheWAS method to analyze associations of phenotypes with CD, we conducted an unbiased screening of all International Classification of Diseases, 10th revision diagnostic codes and examined significance by performing Fisher's Exact tests. We further tested for the association between CD and 14 previously identified comorbidities in an a priori fashion. RESULTS: We found 45 International Classification of Diseases, 10th revision codes significantly associated with CD. Thirteen are known comorbidities and nine are expected symptoms of CD, thus validating our study methods. Further investigation found symptoms that characterized CD clinically and discovered a significant association between eosinophilic disorders of the esophagus and CD. Of 14 previously identified comorbidities, 8 were significantly associated with CD. CONCLUSIONS: An EHR-based PheWAS method is a powerful, efficient, and cost-effective method to screen for possible CD comorbidities and validate associations at the population level. Ours is the first PheWAS of CD to confirm a significant association of eosinophilic disorders of the esophagus with CD in a controlled study.

Simplified Readability Metric Drives Improvement of Radiology Reports: an Experiment on Ultrasound Reports at a Pediatric Hospital.

Highly complex medical documents, including ultrasound reports, are greatly mismatched with patient literacy levels. While improving radiology reports for readability is a longstanding concern, few articles objectively measure the effectiveness of physician training for readability improvement. We hypothesized that writing styles may be evaluated using an objective two-dimensional measure and writing training could improve the writing styles of radiologists. To test it, a simplified "grade vs. length" readability metric is developed based on results from factor analysis of ten readability metrics applied to more than 500,000 radiology reports. To test the short-term effectiveness of a writing workshop, we measured the writing style improvement before and after the training. Statistically significant writing style improvement occurred as a result of the training. Although the degree of improvement varied for different measures, it is evident that targeted training could provide potential benefits to improve readability due to our statistically significant results. The simplified grade vs. length metric enables future clinical decision support systems to quantitatively guide physicians to improve writing styles through writing workshops.

Systematic data quality assessment of electronic health record data to evaluate study-specific fitness: Report from the PRESERVE research study.

Study-specific data quality testing is an essential part of minimizing analytic errors, particularly for studies making secondary use of clinical data. We applied a systematic and reproducible approach for study-specific data quality testing to the analysis plan for PRESERVE, a 15-site, EHR-based observational study of chronic kidney disease in children. This approach integrated widely adopted data quality concepts with healthcare-specific evaluation methods. We implemented two rounds of data quality assessment. The first produced high-level evaluation using aggregate results from a distributed query, focused on cohort identification and main analytic requirements. The second focused on extended testing of row-level data centralized for analysis. We systematized reporting and cataloguing of data quality issues, providing institutional teams with prioritized issues for resolution. We tracked improvements and documented anomalous data for consideration during analyses. The checks we developed identified 115 and 157 data quality issues in the two rounds, involving completeness, data model conformance, cross-variable concordance, consistency, and plausibility, extending traditional data quality approaches to address more complex stratification and temporal patterns. Resolution efforts focused on higher priority issues, given finite study resources. In many cases, institutional teams were able to correct data extraction errors or obtain additional data, avoiding exclusion of 2 institutions entirely and resolving 123 other gaps. Other results identified complexities in measures of kidney function, bearing on the study's outcome definition. Where limitations such as these are intrinsic to clinical data, the study team must account for them in conducting analyses. This study rigorously evaluated fitness of data for intended use. The framework is reusable and built on a strong theoretical underpinning. Significant data quality issues that would have otherwise delayed analyses or made data unusable were addressed. This study highlights the need for teams combining subject-matter and informatics expertise to address data quality when working with real world data.

Extracting Medical Information From Free-Text and Unstructured Patient-Generated Health Data Using Natural Language Processing Methods: Feasibility Study With Real-world Data.

BACKGROUND: Patient-generated health data (PGHD) captured via smart devices or digital health technologies can reflect an individual health journey. PGHD enables tracking and monitoring of personal health conditions, symptoms, and medications out of the clinic, which is crucial for self-care and shared clinical decisions. In addition to self-reported measures and structured PGHD (eg, self-screening, sensor-based biometric data), free-text and unstructured PGHD (eg, patient care note, medical diary) can provide a broader view of a patient's journey and health condition. Natural language processing (NLP) is used to process and analyze unstructured data to create meaningful summaries and insights, showing promise to improve the utilization of PGHD. OBJECTIVE: Our aim is to understand and demonstrate the feasibility of an NLP pipeline to extract medication and symptom information from real-world patient and caregiver data. METHODS: We report a secondary data analysis, using a data set collected from 24 parents of children with special health care needs (CSHCN) who were recruited via a nonrandom sampling approach. Participants used a voice-interactive app for 2 weeks, generating free-text patient notes (audio transcription or text entry). We built an NLP pipeline using a zero-shot approach (adaptive to low-resource settings). We used named entity recognition (NER) and medical ontologies (RXNorm and SNOMED CT [Systematized Nomenclature of Medicine Clinical Terms]) to identify medication and symptoms. Sentence-level dependency parse trees and part-of-speech tags were used to extract additional entity information using the syntactic properties of a note. We assessed the data; evaluated the pipeline with the patient notes; and reported the precision, recall, and F1 scores. RESULTS: In total, 87 patient notes are included (audio transcriptions n=78 and text entries n=9) from 24 parents who have at least one CSHCN. The participants were between the ages of 26 and 59 years. The majority were White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid- or upper-mid household income (n=15, 62.5%), and had higher level education (n=24, 58%). Out of 87 notes, 30 were drug and medication related, and 46 were symptom related. We captured medication instances (medication, unit, quantity, and date) and symptoms satisfactorily (precision >0.65, recall >0.77, F1>0.72). These results indicate the potential when using NER and dependency parsing through an NLP pipeline on information extraction from unstructured PGHD. CONCLUSIONS: The proposed NLP pipeline was found to be feasible for use with real-world unstructured PGHD to accomplish medication and symptom extraction. Unstructured PGHD can be leveraged to inform clinical decision-making, remote monitoring, and self-care including medical adherence and chronic disease management. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings (eg, a limited number of patient notes or training data).

A machine learning-based phenotype for long COVID in children: An EHR-based study from the RECOVER program.

As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data. In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS- CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values. The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses.

KELVIN: a software package for rigorous measurement of statistical evidence in human genetics.

This paper describes the software package KELVIN, which supports the PPL (posterior probability of linkage) framework for the measurement of statistical evidence in human (or more generally, diploid) genetic studies. In terms of scope, KELVIN supports two-point (trait-marker or marker-marker) and multipoint linkage analysis, based on either sex-averaged or sex-specific genetic maps, with an option to allow for imprinting; trait-marker linkage disequilibrium (LD), or association analysis, in case-control data, trio data, and/or multiplex family data, with options for joint linkage and trait-marker LD or conditional LD given linkage; dichotomous trait, quantitative trait and quantitative trait threshold models; and certain types of gene-gene interactions and covariate effects. Features and data (pedigree) structures can be freely mixed and matched within analyses. The statistical framework is specifically tailored to accumulate evidence in a mathematically rigorous way across multiple data sets or data subsets while allowing for multiple sources of heterogeneity, and KELVIN itself utilizes sophisticated software engineering to provide a powerful and robust platform for studying the genetics of complex disorders.

The Current State of Mobile Apps Owned by Large Pediatric Hospitals in the United States: Systematic Search and Analysis on Google Play and Apple App Stores.

BACKGROUND: Pediatric hospitals in the United States are increasingly leveraging patient-facing mobile apps as their digital front doors for patients, families, and caretakers. These mobile health apps are sanctioned by pediatric hospitals to inform the public or populations about pediatric care to provide individualized information, to enhance communication, and to improve patient experience. Yet the functionalities and user feedback of these hospital mobile apps have not been systematically investigated. OBJECTIVE: Our aim was to understand the current state of hospital-owned mobile apps provided by large pediatric hospitals, comparatively analyze and report the services provided, and identify potential gaps to inform developers and providers. The American Hospital Association defines large hospitals as those having a bed count of more than 400. METHODS: We conducted a systematic search on Google Play and Apple App Store to identify all hospital-owned mobile apps from the large pediatric hospitals included in our review. Our inclusion criteria were (1) apps provided by large pediatric hospitals; (2) hospital-owned apps available in Apple App Store and Google Play; and (3) apps that are provided for general populations. Specialty apps that serve specific user groups or populations focusing on education, telehealth, specific conditions or procedures, or apps intended for research or clinician use were excluded. The features and functionality of the included apps were examined. RESULTS: Of the 16 pediatric hospitals included in our review, 4 (25%) had no general patient-facing apps, 4 (25%) had one app, and 8 (50%) had more than one app available on Google Play or Apple App Store. The 12 hospitals with at least one mobile app had a combined total of 72 apps. Of these 72 apps, 61 (85%) were considered specialty and were excluded from our review, leaving a total of 11 (15%) apps to analyze. Among the 11 apps analyzed, the most common feature was appointment scheduling or reminder (n=9, 82%). Doctor search (n=8, 73%) and patient resources (n=8, 73%) were the second most common, followed by payment, billing, or claims (n=7, 64%), patient portal integration (n=6, 55%), personal health management (n=6, 55%), hospital way finding (n=5, 45%), message a provider (n=4, 36%), urgent care wait times (n=4, 36%), video chat (n=4, 36%), and health information access (n=4, 36%). Parking information (n=3, 27%) was the least common. CONCLUSIONS: Out of the 16 pediatric hospitals identified for our review, 75% (n=12) offer mobile apps. Based on the most common features, these apps were intended to help improve accessibility for patients and families in terms of finding providers, scheduling appointments, and accessing patient resources. We believe the findings will inform pediatric hospital administrators, developers, and other stakeholders to improve app feature offerings and increase their impact on service accessibility and patient experience.

"Hey Siri, Help Me Take Care of My Child": A Feasibility Study With Caregivers of Children With Special Healthcare Needs Using Voice Interaction and Automatic Speech Recognition in Remote Care Management.

Background: About 23% of households in the United States have at least one child who has special healthcare needs. As most care activities occur at home, there is often a disconnect and lack of communication between families, home care nurses, and healthcare providers. Digital health technologies may help bridge this gap. Objective: We conducted a pre-post study with a voice-enabled medical note taking (diary) app (SpeakHealth) in a real world setting with caregivers (parents, family members) of children with special healthcare needs (CSHCN) to understand feasibility of voice interaction and automatic speech recognition (ASR) for medical note taking at home. Methods: In total, 41 parents of CSHCN were recruited. Participants completed a pre-study survey collecting demographic details, technology and care management preferences. Out of 41, 24 participants completed the study, using the app for 2 weeks and completing an exit survey. The app facilitated caregiver note-taking using voice interaction and ASR. An exit survey was conducted to collect feedback on technology adoption and changes in technology preferences in care management. We assessed the feasibility of the app by descriptively analyzing survey responses and user data following the key focus areas of acceptability, demand, implementation and integration, adaptation and expansion. In addition, perceived effectiveness of the app was assessed by comparing perceived changes in mobile app preferences among participants. In addition, the voice data, notes, and transcriptions were descriptively analyzed for understanding the feasibility of the app. Results: The majority of the recruited parents were 35-44 years old (22, 53.7%), part of a two-parent household (30, 73.2%), white (37, 90.2%), had more than one child (31, 75.6%), lived in Ohio (37, 90.2%), used mobile health apps, mobile note taking apps or calendar apps (28, 68.3%) and patient portal apps (22, 53.7%) to track symptoms and health events at home. Caregivers had experience with voice technology as well (32, 78%). Among those completed the post-study survey (in Likert Scale 1-5), ~80% of the caregivers agreed or strongly agreed that using the app would enhance their performance in completing tasks (perceived usefulness; mean = 3.4, SD = 0.8), the app is free of effort (perceived ease of use; mean = 3.2, SD = 0.9), and they would use the app in the future (behavioral intention; mean = 3.1, SD = 0.9). In total, 88 voice interactive patient notes were generated with the majority of the voice recordings being less than 20 s in length (66%). Most noted symptoms and conditions, medications, treatment and therapies, and patient behaviors. More than half of the caregivers reported that voice interaction with the app and using transcribed notes positively changed their preference of technology to use and methods for tracking symptoms and health events at home. Conclusions: Our findings suggested that voice interaction and ASR use in mobile apps are feasible and effective in keeping track of symptoms and health events at home. Future work is suggested toward using integrated and intelligent systems with voice interactions with broader populations.

A large collection of real-world pediatric sleep studies.

Despite being crucial to health and quality of life, sleep-especially pediatric sleep-is not yet well understood. This is exacerbated by lack of access to sufficient pediatric sleep data with clinical annotation. In order to accelerate research on pediatric sleep and its connection to health, we create the Nationwide Children's Hospital (NCH) Sleep DataBank and publish it at Physionet and the National Sleep Research Resource (NSRR), which is a large sleep data common with physiological data, clinical data, and tools for analyses. The NCH Sleep DataBank consists of 3,984 polysomnography studies and over 5.6 million clinical observations on 3,673 unique patients between 2017 and 2019 at NCH. The novelties of this dataset include: (1) large-scale sleep dataset suitable for discovering new insights via data mining, (2) explicit focus on pediatric patients, (3) gathered in a real-world clinical setting, and (4) the accompanying rich set of clinical data. The NCH Sleep DataBank is a valuable resource for advancing automatic sleep scoring and real-time sleep disorder prediction, among many other potential scientific discoveries.

Prevalence of Sensitive Terms in Clinical Notes Using Natural Language Processing Techniques: Observational Study.

BACKGROUND: With the increased sharing of electronic health information as required by the US 21st Century Cures Act, there is an increased risk of breaching patient, parent, or guardian confidentiality. The prevalence of sensitive terms in clinical notes is not known. OBJECTIVE: The aim of this study is to define sensitive terms that represent the documentation of content that may be private and determine the prevalence and characteristics of provider notes that contain sensitive terms. METHODS: Using keyword expansion, we defined a list of 781 sensitive terms. We searched all provider history and physical, progress, consult, and discharge summary notes for patients aged 0-21 years written between January 1, 2019, and December 31, 2019, for a direct string match of sensitive terms. We calculated the prevalence of notes with sensitive terms and characterized clinical encounters and patient characteristics. RESULTS: Sensitive terms were present in notes from every clinical context in all pediatric ages. Terms related to the mental health category were most used overall (254,975/1,338,297, 19.5%), but terms related to substance abuse and reproductive health were most common in patients aged 0-3 years. History and physical notes (19,854/34,771, 57.1%) and ambulatory progress notes (265,302/563,273, 47.1%) were most likely to include sensitive terms. The highest prevalence of notes with sensitive terms was found in pain management (950/1112, 85.4%) and child abuse (1092/1282, 85.2%) clinics. CONCLUSIONS: Notes containing sensitive terms are not limited to adolescent patients, specific note types, or certain specialties. Recognition of sensitive terms across all ages and clinical settings complicates efforts to protect patient and caregiver privacy in the era of information-blocking regulations.

Characterization of Electronic Health Record Use Outside Scheduled Clinic Hours Among Primary Care Pediatricians: Retrospective Descriptive Task Analysis of Electronic Health Record Access Log Data.

BACKGROUND: Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels because of a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting that physicians spend considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician clinic schedules to define the outside work time. OBJECTIVE: This study aimed to characterize EHR work outside scheduled clinic hours among primary care pediatricians using a retrospective descriptive task analysis of EHR access log data and actual physician clinic schedules to define work time. METHODS: We conducted a retrospective, exploratory, descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large Midwestern pediatric health center to quantify and identify actions completed outside scheduled clinic hours. Mixed-effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR work during scheduled clinic hours on the use of EHRs outside scheduled clinic hours. RESULTS: Primary care pediatricians (n=56) in this study generated 1,523,872 access log data points (across 1069 physician workdays) and spent an average of 4.4 (SD 2.0) hours and 0.8 (SD 0.8) hours per physician per workday engaged in EHRs during and outside scheduled clinic hours, respectively. Approximately three-quarters of the time working in EHR during or outside scheduled clinic hours was spent reviewing data and reports. Mixed-effects regression revealed no associations of age, sex, or clinical full-time equivalent status with EHR use during or outside scheduled clinic hours. CONCLUSIONS: For every hour primary care pediatricians spent engaged with the EHR during scheduled clinic hours, they spent approximately 10 minutes interacting with the EHR outside scheduled clinic hours. Most of their time (during and outside scheduled clinic hours) was spent reviewing data, records, and other information in EHR.

Associations of Telehealth Care Delivery with Pediatric Health Care Provider Well-Being.

BACKGROUND: The rapid, large-scale deployment of new health technologies can introduce challenges to clinicians who are already under stress. The novel coronavirus disease 19 (COVID-19) pandemic transformed health care in the United States to include a telehealth model of care delivery. Clarifying paths through which telehealth technology use is associated with change in provider well-being and interest in sustaining virtual care delivery can inform planning and optimization efforts. OBJECTIVE: This study aimed to characterize provider-reported changes in well-being and daily work associated with the pandemic-accelerated expansion of telehealth and assess the relationship of provider perceptions of telehealth effectiveness, efficiency, and work-life balance with desire for future telehealth. METHODS: A cross-sectional survey study was conducted October through November 2020, 6 months after the outbreak of COVID-19 at three children's hospitals. Factor analysis and structural equation modeling (SEM) were used to examine telehealth factors associated with reported change in well-being and desire for future telehealth. RESULTS: A total of 947 nontrainee physicians, advanced practice providers, and psychologists were surveyed. Of them, 502 (53.0%) providers responded and 467 (49.3%) met inclusion criteria of telehealth use during the study period. Of these, 325 (69.6%) were female, 301 (65.6%) were physicians, and 220 (47.1%) were medical subspecialists. Providers were 4.77 times as likely (95% confidence interval [CI]: 3.29-7.06) to report improved versus worsened well-being associated with telehealth. Also, 95.5% of providers (95% CI: 93.2-97.2%) wish to continue performing telehealth postpandemic. Our model explains 66% of the variance in telehealth-attributed provider well-being and 59% of the variance for future telehealth preference and suggests telehealth resources significantly influence provider-perceived telehealth care effectiveness which in turn significantly influences provider well-being and desire to perform telehealth. CONCLUSION: Telehealth has potential to promote provider well-being; telehealth-related changes in provider well-being are associated with both provider-perceived effectiveness of telemedicine for patients and adequacy of telehealth resources.