Chatbot for Social Need Screening and Resource Sharing With Vulnerable Families: Iterative Design and Evaluation Study.

BACKGROUND: Health outcomes are significantly influenced by unmet social needs. Although screening for social needs has become common in health care settings, there is often poor linkage to resources after needs are identified. The structural barriers (eg, staffing, time, and space) to helping address social needs could be overcome by a technology-based solution. OBJECTIVE: This study aims to present the design and evaluation of a chatbot, DAPHNE (Dialog-Based Assistant Platform for Healthcare and Needs Ecosystem), which screens for social needs and links patients and families to resources. METHODS: This research used a three-stage study approach: (1) an end-user survey to understand unmet needs and perception toward chatbots, (2) iterative design with interdisciplinary stakeholder groups, and (3) a feasibility and usability assessment. In study 1, a web-based survey was conducted with low-income US resident households (n=201). Following that, in study 2, web-based sessions were held with an interdisciplinary group of stakeholders (n=10) using thematic and content analysis to inform the chatbot's design and development. Finally, in study 3, the assessment on feasibility and usability was completed via a mix of a web-based survey and focus group interviews following scenario-based usability testing with community health workers (family advocates; n=4) and social workers (n=9). We reported descriptive statistics and chi-square test results for the household survey. Content analysis and thematic analysis were used to analyze qualitative data. Usability score was descriptively reported. RESULTS: Among the survey participants, employed and younger individuals reported a higher likelihood of using a chatbot to address social needs, in contrast to the oldest age group. Regarding designing the chatbot, the stakeholders emphasized the importance of provider-technology collaboration, inclusive conversational design, and user education. The participants found that the chatbot's capabilities met expectations and that the chatbot was easy to use (System Usability Scale score=72/100). However, there were common concerns about the accuracy of suggested resources, electronic health record integration, and trust with a chatbot. CONCLUSIONS: Chatbots can provide personalized feedback for families to identify and meet social needs. Our study highlights the importance of user-centered iterative design and development of chatbots for social needs. Future research should examine the efficacy, cost-effectiveness, and scalability of chatbot interventions to address social needs.

Acceptance of voice assistant technology in dental practice: A cross sectional study with dentists and validation using structural equation modeling.

Voice assistant technologies (VAT) has been part of our daily lives, as a virtual assistant to complete requested tasks. The integration of VAT in dental offices has the potential to augment productivity and hygiene practices. Prior to the adoption of such innovations in dental settings, it is crucial to evaluate their applicability. This study aims to assess dentists' perceptions and the factors influencing their intention to use VAT in a clinical setting. A survey and research model were designed based on an extended Unified Theory of Acceptance and Use of Technology (UTAUT). The survey was sent to 7,544 Ohio-licensed dentists through email. The data was analyzed and reported using descriptive statistics, model reliability testing, and partial least squares regression (PLSR) to explain dentists' behavioral intention (BI) to use VAT. In total, 257 participants completed the survey. The model accounted for 74.2% of the variance in BI to use VAT. Performance expectancy and perceived enjoyment had significant positive influence on BI to use VAT. Perceived risk had significant negative influence on BI to use VAT. Self-efficacy had significantly influenced perceived enjoyment, accounting for 35.5% of the variance of perceived enjoyment. This investigation reveals that performance efficiency and user enjoyment are key determinants in dentists' decision to adopt VAT. Concerns regarding the privacy of VAT also play a crucial role in its acceptance. This study represents the first documented inquiry into dentists' reception of VAT, laying groundwork for future research and implementation strategies.

Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls.

OBJECTIVE: We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance. MATERIALS AND METHODS: We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries. RESULTS: The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate. DISCUSSION: The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories. CONCLUSION: The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.

Redefining Virtual Assistants in Health Care: The Future With Large Language Models.

This editorial explores the evolving and transformative role of large language models (LLMs) in enhancing the capabilities of virtual assistants (VAs) in the health care domain, highlighting recent research on the performance of VAs and LLMs in health care information sharing. Focusing on recent research, this editorial unveils the marked improvement in the accuracy and clinical relevance of responses from LLMs, such as GPT-4, compared to current VAs, especially in addressing complex health care inquiries, like those related to postpartum depression. The improved accuracy and clinical relevance with LLMs mark a paradigm shift in digital health tools and VAs. Furthermore, such LLM applications have the potential to dynamically adapt and be integrated into existing VA platforms, offering cost-effective, scalable, and inclusive solutions. These suggest a significant increase in the applicable range of VA applications, as well as the increased value, risk, and impact in health care, moving toward more personalized digital health ecosystems. However, alongside these advancements, it is necessary to develop and adhere to ethical guidelines, regulatory frameworks, governance principles, and privacy and safety measures. We need a robust interdisciplinary collaboration to navigate the complexities of safely and effectively integrating LLMs into health care applications, ensuring that these emerging technologies align with the diverse needs and ethical considerations of the health care domain.

Mobile Application Measurement of Menstrual Cycle Characteristics and Their Association with Dysmenorrhea and Activity Limitation in Early Adolescents.

OBJECTIVE: Real-time tracking of menstrual bleeding is a barrier to research due to limitations with traditional data collection tools. This prospective cohort study utilized a mobile application (TDot app) in young adolescents aged 10-14 years to assess the relationship between heavy menstrual bleeding (HMB), dysmenorrhea, and activity limitation. METHODS: Menstrual cycles were captured over six months in real-time using the Pictorial Blood loss Assessment Chart (PBAC). A median PBAC score of >100 was used to identify participants with HMB. Participants also completed a modified WaLIDD (Working ability, Location, Intensity, Days of pain, Dysmenorrhea) scale. Impact of menses on daily activities was collected for each cycle. RESULTS: A total of 160 participants enrolled and 100 (63%) participants with ≥3 cycles recorded in the mobile app were analyzed. HMB was noted in 41% of participants. Median modified WaLIDD score was significantly higher in participants with HMB than those without HMB (p=0.01). No significant differences were found in activity limitations between participants with and without HMB (p=0.34). Median modified WaLIDD score for participants with activity limitation was significantly higher than those without activity limitation (p=0.01). CONCLUSION: Utilizing mobile app technology, we were able to gather real-time menstrual outcome data from young adolescents on heaviness of flow, dysmenorrhea and activity limitations. While we did not find that patients with HMB were more likely to have activity limitations, we did find that those with limitations had modestly higher dysmenorrhea scores. Future studies should focus on identifying additional variables that impact activity limitation during menstruation.

Development and Evaluation of a Digital Scribe: Conversation Summarization Pipeline for Emergency Department Counseling Sessions towards Reducing Documentation Burden.

Objective: We present a proof-of-concept digital scribe system as an ED clinical conversation summarization pipeline and report its performance. Materials and Methods: We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries. Results: The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate. Discussion: The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories. Conclusion: The study provides evidence towards the potential of AI-assisted tools in reducing clinical documentation burden. Future work is suggested on expanding the research scope with larger language models, and comparative analysis to measure documentation efforts and time.

Objectively Quantifying Pediatric Psychiatric Severity Using Artificial Intelligence, Voice Recognition Technology, and Universal Emotions: Pilot Study for Artificial Intelligence-Enabled Innovation to Address Youth Mental Health Crisis.

BACKGROUND: Providing Psychotherapy, particularly for youth, is a pressing challenge in the health care system. Traditional methods are resource-intensive, and there is a need for objective benchmarks to guide therapeutic interventions. Automated emotion detection from speech, using artificial intelligence, presents an emerging approach to address these challenges. Speech can carry vital information about emotional states, which can be used to improve mental health care services, especially when the person is suffering. OBJECTIVE: This study aims to develop and evaluate automated methods for detecting the intensity of emotions (anger, fear, sadness, and happiness) in audio recordings of patients' speech. We also demonstrate the viability of deploying the models. Our model was validated in a previous publication by Alemu et al with limited voice samples. This follow-up study used significantly more voice samples to validate the previous model. METHODS: We used audio recordings of patients, specifically children with high adverse childhood experience (ACE) scores; the average ACE score was 5 or higher, at the highest risk for chronic disease and social or emotional problems; only 1 in 6 have a score of 4 or above. The patients' structured voice sample was collected by reading a fixed script. In total, 4 highly trained therapists classified audio segments based on a scoring process of 4 emotions and their intensity levels for each of the 4 different emotions. We experimented with various preprocessing methods, including denoising, voice-activity detection, and diarization. Additionally, we explored various model architectures, including convolutional neural networks (CNNs) and transformers. We trained emotion-specific transformer-based models and a generalized CNN-based model to predict emotion intensities. RESULTS: The emotion-specific transformer-based model achieved a test-set precision and recall of 86% and 79%, respectively, for binary emotional intensity classification (high or low). In contrast, the CNN-based model, generalized to predict the intensity of 4 different emotions, achieved test-set precision and recall of 83% for each. CONCLUSIONS: Automated emotion detection from patients' speech using artificial intelligence models is found to be feasible, leading to a high level of accuracy. The transformer-based model exhibited better performance in emotion-specific detection, while the CNN-based model showed promise in generalized emotion detection. These models can serve as valuable decision-support tools for pediatricians and mental health providers to triage youth to appropriate levels of mental health care services. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR1-10.2196/51912.

Identification of Child Survivors of Sex Trafficking From Electronic Health Records: An Artificial Intelligence Guided Approach.

Survivors of child sex trafficking (SCST) experience high rates of adverse health outcomes. Amidst the duration of their victimization, survivors regularly seek healthcare yet fail to be identified. This study sought to utilize artificial intelligence (AI) to identify SCST and describe the elements of their healthcare presentation. An AI-supported keyword search was conducted to identify SCST within the electronic medical records (EMR) of ∼1.5 million patients at a large midwestern pediatric hospital. Descriptive analyses were used to evaluate associated diagnoses and clinical presentation. A sex trafficking-related keyword was identified in .18% of patient charts. Among this cohort, the most common associated diagnostic codes were for Confirmed Sexual/Physical Assault; Trauma and Stress-Related Disorders; Depressive Disorders; Anxiety Disorders; and Suicidal Ideation. Our findings are consistent with the myriad of known adverse physical and psychological outcomes among SCST and illuminate the future potential of AI technology to improve screening and research efforts surrounding all aspects of this vulnerable population.

Improving medication adherence monitoring and clinical outcomes through mHealth: A randomized controlled trial protocol in pediatric stem cell transplant.

Medication non-adherence rates in children range between 50% and 80% in the United States. Due to multifaceted outpatient routines, children receiving hematopoietic stem cell transplant (HCT) are at especially high risk of non-adherence, which can be life-threatening. Although digital health interventions have been effective in improving non-adherence in many pediatric conditions, limited research has examined their benefits among families of children receiving HCT. To address this gap, we created the BMT4me© mobile health app, an innovative intervention serving as a "virtual assistant" to send medication-taking reminders for caregivers and to track, in real-time, the child's medication taking, barriers to missed doses, symptoms or side effects, and other notes regarding their child's treatment. In this randomized controlled trial, caregivers will be randomized to either the control (standard of care) group or the intervention (BMT4me© app) group at initial discharge post-HCT. Both groups will receive an electronic adherence monitoring device (i.e., medication event monitoring system "MEMS" cap, Medy Remote Patient Management "MedyRPM" medication adherence box) to store their child's immunosuppressant medication. Caregivers who agree to participate will be asked to complete enrollment, weekly, and monthly parent-proxy measures of their child's medication adherence until the child reaches Day 100 or complete taper from immunosuppression. Caregivers will also participate in a 15 to 30-minute exit interview at the conclusion of the study. Descriptive statistics and correlations will be used to assess phone activity and use behavior over time. Independent samples t-tests will examine the efficacy of the intervention to improve adherence monitoring and reduce readmission rates. The primary expected outcome of this study is that the BMT4me© app will improve the real-time monitoring and medication adherence in children receiving hematopoietic stem cell transplant following discharge, thus improving clinical outcomes.

Artificial intelligence in healthcare: Complementing, not replacing, doctors and healthcare providers.

The utilization of artificial intelligence (AI) in clinical practice has increased and is evidently contributing to improved diagnostic accuracy, optimized treatment planning, and improved patient outcomes. The rapid evolution of AI, especially generative AI and large language models (LLMs), have reignited the discussions about their potential impact on the healthcare industry, particularly regarding the role of healthcare providers. Concerning questions, "can AI replace doctors?" and "will doctors who are using AI replace those who are not using it?" have been echoed. To shed light on this debate, this article focuses on emphasizing the augmentative role of AI in healthcare, underlining that AI is aimed to complement, rather than replace, doctors and healthcare providers. The fundamental solution emerges with the human-AI collaboration, which combines the cognitive strengths of healthcare providers with the analytical capabilities of AI. A human-in-the-loop (HITL) approach ensures that the AI systems are guided, communicated, and supervised by human expertise, thereby maintaining safety and quality in healthcare services. Finally, the adoption can be forged further by the organizational process informed by the HITL approach to improve multidisciplinary teams in the loop. AI can create a paradigm shift in healthcare by complementing and enhancing the skills of healthcare providers, ultimately leading to improved service quality, patient outcomes, and a more efficient healthcare system.

Detecting Clinically Relevant Emotional Distress and Functional Impairment in Children and Adolescents: Protocol for an Automated Speech Analysis Algorithm Development Study.

BACKGROUND: Even before the onset of the COVID-19 pandemic, children and adolescents were experiencing a mental health crisis, partly due to a lack of quality mental health services. The rate of suicide for Black youth has increased by 80%. By 2025, the health care system will be short of 225,000 therapists, further exacerbating the current crisis. Therefore, it is of utmost importance for providers, schools, youth mental health, and pediatric medical providers to integrate innovation in digital mental health to identify problems proactively and rapidly for effective collaboration with other health care providers. Such approaches can help identify robust, reproducible, and generalizable predictors and digital biomarkers of treatment response in psychiatry. Among the multitude of digital innovations to identify a biomarker for psychiatric diseases currently, as part of the macrolevel digital health transformation, speech stands out as an attractive candidate with features such as affordability, noninvasive, and nonintrusive. OBJECTIVE: The protocol aims to develop speech-emotion recognition algorithms leveraging artificial intelligence/machine learning, which can establish a link between trauma, stress, and voice types, including disrupting speech-based characteristics, and detect clinically relevant emotional distress and functional impairments in children and adolescents. METHODS: Informed by theoretical foundations (the Theory of Psychological Trauma Biomarkers and Archetypal Voice Categories), we developed our methodology to focus on 5 emotions: anger, happiness, fear, neutral, and sadness. Participants will be recruited from 2 local mental health centers that serve urban youths. Speech samples, along with responses to the Symptom and Functioning Severity Scale, Patient Health Questionnaire 9, and Adverse Childhood Experiences scales, will be collected using an Android mobile app. Our model development pipeline is informed by Gaussian mixture model (GMM), recurrent neural network, and long short-term memory. RESULTS: We tested our model with a public data set. The GMM with 128 clusters showed an evenly distributed accuracy across all 5 emotions. Using utterance-level features, GMM achieved an accuracy of 79.15% overall, while frame selection increased accuracy to 85.35%. This demonstrates that GMM is a robust model for emotion classification of all 5 emotions and that emotion frame selection enhances accuracy, which is significant for scientific evaluation. Recruitment and data collection for the study were initiated in August 2021 and are currently underway. The study results are likely to be available and published in 2024. CONCLUSIONS: This study contributes to the literature as it addresses the need for speech-focused digital health tools to detect clinically relevant emotional distress and functional impairments in children and adolescents. The preliminary results show that our algorithm has the potential to improve outcomes. The findings will contribute to the broader digital health transformation. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/46970.

Extracting Medical Information From Free-Text and Unstructured Patient-Generated Health Data Using Natural Language Processing Methods: Feasibility Study With Real-world Data.

BACKGROUND: Patient-generated health data (PGHD) captured via smart devices or digital health technologies can reflect an individual health journey. PGHD enables tracking and monitoring of personal health conditions, symptoms, and medications out of the clinic, which is crucial for self-care and shared clinical decisions. In addition to self-reported measures and structured PGHD (eg, self-screening, sensor-based biometric data), free-text and unstructured PGHD (eg, patient care note, medical diary) can provide a broader view of a patient's journey and health condition. Natural language processing (NLP) is used to process and analyze unstructured data to create meaningful summaries and insights, showing promise to improve the utilization of PGHD. OBJECTIVE: Our aim is to understand and demonstrate the feasibility of an NLP pipeline to extract medication and symptom information from real-world patient and caregiver data. METHODS: We report a secondary data analysis, using a data set collected from 24 parents of children with special health care needs (CSHCN) who were recruited via a nonrandom sampling approach. Participants used a voice-interactive app for 2 weeks, generating free-text patient notes (audio transcription or text entry). We built an NLP pipeline using a zero-shot approach (adaptive to low-resource settings). We used named entity recognition (NER) and medical ontologies (RXNorm and SNOMED CT [Systematized Nomenclature of Medicine Clinical Terms]) to identify medication and symptoms. Sentence-level dependency parse trees and part-of-speech tags were used to extract additional entity information using the syntactic properties of a note. We assessed the data; evaluated the pipeline with the patient notes; and reported the precision, recall, and F1 scores. RESULTS: In total, 87 patient notes are included (audio transcriptions n=78 and text entries n=9) from 24 parents who have at least one CSHCN. The participants were between the ages of 26 and 59 years. The majority were White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid- or upper-mid household income (n=15, 62.5%), and had higher level education (n=24, 58%). Out of 87 notes, 30 were drug and medication related, and 46 were symptom related. We captured medication instances (medication, unit, quantity, and date) and symptoms satisfactorily (precision >0.65, recall >0.77, F1>0.72). These results indicate the potential when using NER and dependency parsing through an NLP pipeline on information extraction from unstructured PGHD. CONCLUSIONS: The proposed NLP pipeline was found to be feasible for use with real-world unstructured PGHD to accomplish medication and symptom extraction. Unstructured PGHD can be leveraged to inform clinical decision-making, remote monitoring, and self-care including medical adherence and chronic disease management. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings (eg, a limited number of patient notes or training data).

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study.

BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

The Current State of Mobile Apps Owned by Large Pediatric Hospitals in the United States: Systematic Search and Analysis on Google Play and Apple App Stores.

BACKGROUND: Pediatric hospitals in the United States are increasingly leveraging patient-facing mobile apps as their digital front doors for patients, families, and caretakers. These mobile health apps are sanctioned by pediatric hospitals to inform the public or populations about pediatric care to provide individualized information, to enhance communication, and to improve patient experience. Yet the functionalities and user feedback of these hospital mobile apps have not been systematically investigated. OBJECTIVE: Our aim was to understand the current state of hospital-owned mobile apps provided by large pediatric hospitals, comparatively analyze and report the services provided, and identify potential gaps to inform developers and providers. The American Hospital Association defines large hospitals as those having a bed count of more than 400. METHODS: We conducted a systematic search on Google Play and Apple App Store to identify all hospital-owned mobile apps from the large pediatric hospitals included in our review. Our inclusion criteria were (1) apps provided by large pediatric hospitals; (2) hospital-owned apps available in Apple App Store and Google Play; and (3) apps that are provided for general populations. Specialty apps that serve specific user groups or populations focusing on education, telehealth, specific conditions or procedures, or apps intended for research or clinician use were excluded. The features and functionality of the included apps were examined. RESULTS: Of the 16 pediatric hospitals included in our review, 4 (25%) had no general patient-facing apps, 4 (25%) had one app, and 8 (50%) had more than one app available on Google Play or Apple App Store. The 12 hospitals with at least one mobile app had a combined total of 72 apps. Of these 72 apps, 61 (85%) were considered specialty and were excluded from our review, leaving a total of 11 (15%) apps to analyze. Among the 11 apps analyzed, the most common feature was appointment scheduling or reminder (n=9, 82%). Doctor search (n=8, 73%) and patient resources (n=8, 73%) were the second most common, followed by payment, billing, or claims (n=7, 64%), patient portal integration (n=6, 55%), personal health management (n=6, 55%), hospital way finding (n=5, 45%), message a provider (n=4, 36%), urgent care wait times (n=4, 36%), video chat (n=4, 36%), and health information access (n=4, 36%). Parking information (n=3, 27%) was the least common. CONCLUSIONS: Out of the 16 pediatric hospitals identified for our review, 75% (n=12) offer mobile apps. Based on the most common features, these apps were intended to help improve accessibility for patients and families in terms of finding providers, scheduling appointments, and accessing patient resources. We believe the findings will inform pediatric hospital administrators, developers, and other stakeholders to improve app feature offerings and increase their impact on service accessibility and patient experience.

Considerations for Conducting Bring Your Own "Device" (BYOD) Clinical Studies.

Background: Digital health technologies are attracting attention as novel tools for data collection in clinical research. They present alternative methods compared to in-clinic data collection, which often yields snapshots of the participants' physiology, behavior, and function that may be prone to biases and artifacts, e.g., white coat hypertension, and not representative of the data in free-living conditions. Modern digital health technologies equipped with multi-modal sensors combine different data streams to derive comprehensive endpoints that are important to study participants and are clinically meaningful. Used for data collection in clinical trials, they can be deployed as provisioned products where technology is given at study start or in a bring your own "device" (BYOD) manner where participants use their technologies to generate study data. Summary: The BYOD option has the potential to be more user-friendly, allowing participants to use technologies that they are familiar with, ensuring better participant compliance, and potentially reducing the bias that comes with introducing new technologies. However, this approach presents different technical, operational, regulatory, and ethical challenges to study teams. For example, BYOD data can be more heterogeneous, and recruiting historically underrepresented populations with limited access to technology and the internet can be challenging. Despite the rapid increase in digital health technologies for clinical and healthcare research, BYOD use in clinical trials is limited, and regulatory guidance is still evolving. Key Messages: We offer considerations for academic researchers, drug developers, and patient advocacy organizations on the design and deployment of BYOD models in clinical research. These considerations address: (1) early identification and engagement with internal and external stakeholders; (2) study design including informed consent and recruitment strategies; (3) outcome, endpoint, and technology selection; (4) data management including compliance and data monitoring; (5) statistical considerations to meet regulatory requirements. We believe that this article acts as a primer, providing insights into study design and operational requirements to ensure the successful implementation of BYOD clinical studies.

An mHealth App to Promote Adherence to Immunosuppressant Medication and Track Symptoms in Children After Hematopoietic Stem Cell Transplant: Protocol for a Mixed Methods Usability Study.

BACKGROUND: In the United States, poor adherence accounts for up to 70% of all medication-related hospital admissions, resulting in $100 billion in health care costs annually. In pediatrics, adherence is largely dependent on caregivers. In a high-risk hematopoietic stem cell transplant (HSCT) population, caregivers are isolated with their child due to infection risk and must manage challenging treatment regimens at home, often with limited time and support. Complex behavioral interventions, typically employed to address adherence, are difficult to deliver and manage in the context of these daily tasks. The most successful adherence interventions, and thus improved clinical outcomes, have included mobile health (mHealth) reminder approaches and a direct measure of adherence. OBJECTIVE: This is a 3-phase project, with this protocol describing phase 2, to determine the usability and feasibility of an mHealth app (BMT4me) designed to promote adherence to immunosuppressant medication and to track symptoms among children who received HSCT. METHODS: This study uses an iterative convergent mixed methods design to develop and assess the usability and feasibility of an adherence digital health intervention. We will recruit 15 caregivers of pediatric patients receiving HSCT to complete user testing. Qualitative and quantitative data will be integrated to enhance and expand upon study findings. RESULTS: Enrollment began in September 2021 and is ongoing. A total of 7 caregivers have enrolled. We anticipate completion by fall 2022. We anticipate high usability scores and a better understanding of unique features within the app that are needed for HSCT families post transplant. To date, usability scores among enrolled participants are greater than 70%. Feedback from qualitative interviews is being used to further adapt the app by adding specific weekly logs, call provider options, and voice to text. CONCLUSIONS: This protocol describes a mixed methods usability and feasibility study to develop and implement a smartphone app for caregivers of children receiving HSCT. The app was designed to improve immunosuppressant adherence and to track symptoms in the acute phase post discharge. Study findings will inform further refinement of the app and the feasibility of a pilot randomized controlled trial examining efficacy on clinical outcomes. TRIAL REGISTRATION: ClinicalTrials.gov NCT04976933; https://clinicaltrials.gov/ct2/show/NCT04976933. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/39098.

Medical Text Prediction and Suggestion Using Generative Pretrained Transformer Models with Dental Medical Notes.

BACKGROUND: Generative pretrained transformer (GPT) models are one of the latest large pretrained natural language processing models that enables model training with limited datasets and reduces dependency on large datasets, which are scarce and costly to establish and maintain. There is a rising interest to explore the use of GPT models in health care. OBJECTIVE: We investigate the performance of GPT-2 and GPT-Neo models for medical text prediction using 374,787 free-text dental notes. METHODS: We fine-tune pretrained GPT-2 and GPT-Neo models for next word prediction on a dataset of over 374,000 manually written sections of dental clinical notes. Each model was trained on 80% of the dataset, validated on 10%, and tested on the remaining 10%. We report model performance in terms of next word prediction accuracy and loss. Additionally, we analyze the performance of the models on different types of prediction tokens for categories. For comparison, we also fine-tuned a non-GPT pretrained neural network model, XLNet (large), for next word prediction. We annotate each token in 100 randomly sampled notes by category (e.g., names, abbreviations, clinical terms, punctuation, etc.) and compare the performance of each model by token category. RESULTS: Models present acceptable accuracy scores (GPT-2: 76%; GPT-Neo: 53%), and the GPT-2 model also performs better in manual evaluations, especially for names, abbreviations, and punctuation. Both GPT models outperformed XLNet in terms of accuracy. The results suggest that pretrained models have the potential to assist medical charting in the future. We share the lessons learned, insights, and suggestions for future implementations. CONCLUSION: The results suggest that pretrained models have the potential to assist medical charting in the future. Our study presented one of the first implementations of the GPT model used with medical notes.