Supporting Adolescent Engagement with Artificial Intelligence-Driven Digital Health Behavior Change Interventions.

Understanding and optimizing adolescent-specific engagement with behavior change interventions will open doors for providers to promote healthy changes in an age group that is simultaneously difficult to engage and especially important to affect. For digital interventions, there is untapped potential in combining the vastness of process-level data with the analytical power of artificial intelligence (AI) to understand not only how adolescents engage but also how to improve upon interventions with the goal of increasing engagement and, ultimately, efficacy. Rooted in the example of the INSPIRE narrative-centered digital health behavior change intervention (DHBCI) for adolescent risky behaviors around alcohol use, we propose a framework for harnessing AI to accomplish 4 goals that are pertinent to health care providers and software developers alike: measurement of adolescent engagement, modeling of adolescent engagement, optimization of current interventions, and generation of novel interventions. Operationalization of this framework with youths must be situated in the ethical use of this technology, and we have outlined the potential pitfalls of AI with particular attention to privacy concerns for adolescents. Given how recently AI advances have opened up these possibilities in this field, the opportunities for further investigation are plenty.

The Association Between Temperament and Characteristics, Smartphone App Use Patterns and Academic Performance of University Students.

BACKGROUND: Smartphone use patterns may predict daily life efficacy and performance improvements in sports. Additionally, personal characteristics may be associated with smartphone overuse. METHODS: We investigated the correlation between the temperament and character inventory (TCI) and academic performance using smartphone log data. We hypothesized that the elite and general groups, divided based on academic performance, differed according to the TCI and downloadable smartphone apps (applications). Additionally, we hypothesized a correlation between smartphone app usage patterns and TCI. A total of 151 students provided smartphone log data of the previous four weeks. They also completed the TCI and provided academic records of the previous year. RESULTS: The first and second most frequently used apps by both groups of students were social networking and entertainment, respectively. Elite students scored higher on novelty seeking, reward dependence, persistence, self-directedness, and self-transcendence than general students. In all participants, the usage time of serious apps was correlated with the scores for novelty seeking (r = 0.32, P < 0.007), reward dependence (r = 0.32, P < 0.007), and self-transcendence (r = 0.35, P < 0.006). In the elite group, the usage time of serious apps was correlated with the scores for novelty seeking (r = 0.45, P < 0.001), reward dependence (r = 0.39, P = 0.022), and self-transcendence (r = 0.35, P = 0.031). In the general group, the usage time of serious apps was correlated only with self-transcendence (r = 0.32, P < 0.007). CONCLUSION: High usage time of serious apps can help sports majors to excel academically. Particularly among sports majors, serious apps are related to activity, the desire for rewards and recognition, and the tendency to transcend themselves.

Commencement of and Retention in Web-Based Interventions and Response to Prompts and Reminders: Longitudinal Observational Study Based on Two Randomized Controlled Trials.

BACKGROUND: Web-based interventions are effective for several psychological problems. However, recruitment, adherence, and missing data are challenges when evaluating these interventions. OBJECTIVE: This study aimed to describe the use patterns during the commencement phase, possible retention patterns (continuation of data provision), and responses to prompts and reminders among participants in 2 randomized controlled trials (RCTs) evaluating web-based interventions. METHODS: Data on use patterns logged in 2 RCTs aiming to reduce symptoms of anxiety and depression among adult patients recently diagnosed with cancer (AdultCan RCT) and patients with a recent myocardial infarction (Heart RCT) were analyzed. The web-based intervention in the AdultCan trial consisted of unguided self-help and psychoeducation and that in the Heart trial consisted of therapist-supported cognitive behavioral therapy. In total, 2360 participants' use patterns at first log-in, including data collection at baseline (ie, commencement) and at 2 follow-ups, were analyzed. Both the intervention and comparison groups were analyzed. RESULTS: At commencement, 70.85% (909/1283) and 86.82% (935/1077) of the participants in AdultCan and Heart RCTs, respectively, logged in and completed baseline data collection after receiving a welcome email with log-in credentials. The median duration of the first log-in was 44 minutes and 38 minutes in AdultCan and Heart RCTs, respectively. Slightly less than half of the participants' first log-ins were completed outside standard office hours. More than 80% (92/114 and 103/111) of the participants in both trials explored the intervention within 2 weeks of being randomized to the treatment group, with a median duration of 7 minutes and 47 minutes in AdultCan and Heart RCTs, respectively. There was a significant association between intervention exploration time during the first 2 weeks and retention in the Heart trial but not in the AdultCan trial. However, the control group was most likely to retain and provide complete follow-up data. Across the 3 time points of data collection explored in this study, the proportion of participants responding to all questionnaires within 1 week from the prompt, without a reminder, varied between 35.45% (413/1165) and 66.3% (112/169). After 2 reminders, up to 97.6% (165/169) of the participants responded. CONCLUSIONS: Most participants in both RCTs completed the baseline questionnaires within 1 week of receiving the welcome email. Approximately half of them answered questions at baseline data collection outside office hours, suggesting that the time flexibility inherent in web-based interventions contributes to commencement and use. In contrast to what was expected, the intervention groups generally had lower completion rates than the comparison groups. About half of the participants completed the questionnaires without a reminder, but thereafter, reminders contributed to both baseline and follow-up retention, suggesting they were effective. Strategies to increase commencement of and retention in eHealth interventions are important for the future development of effective interventions and relevant research.

Lifelog Data-Based Prediction Model of Digital Health Care App Customer Churn: Retrospective Observational Study.

BACKGROUND: Customer churn is the rate at which customers stop doing business with an entity. In the field of digital health care, user churn prediction is important not only in terms of company revenue but also for improving the health of users. Churn prediction has been previously studied, but most studies applied time-invariant model structures and used structured data. However, additional unstructured data have become available; therefore, it has become essential to process daily time-series log data for churn predictions. OBJECTIVE: We aimed to apply a recurrent neural network structure to accept time-series patterns using lifelog data and text message data to predict the churn of digital health care users. METHODS: This study was based on the use data of a digital health care app that provides interactive messages with human coaches regarding food, exercise, and weight logs. Among the users in Korea who enrolled between January 1, 2017 and January 1, 2019, we defined churn users according to the following criteria: users who received a refund before the paid program ended and users who received a refund 7 days after the trial period. We used long short-term memory with a masking layer to receive sequence data with different lengths. We also performed topic modeling to vectorize text messages. To interpret the contributions of each variable to model predictions, we used integrated gradients, which is an attribution method. RESULTS: A total of 1868 eligible users were included in this study. The final performance of churn prediction was an F1 score of 0.89; that score decreased by 0.12 when the data of the final week were excluded (F1 score 0.77). Additionally, when text data were included, the mean predicted performance increased by approximately 0.085 at every time point. Steps per day had the largest contribution (0.1085). Among the topic variables, poor habits (eg, drinking alcohol, overeating, and late-night eating) showed the largest contribution (0.0875). CONCLUSIONS: The model with a recurrent neural network architecture that used log data and message data demonstrated high performance for churn classification. Additionally, the analysis of the contribution of the variables is expected to help identify signs of user churn in advance and improve the adherence in digital health care.

EHR audit logs: A new goldmine for health services research?

A byproduct of the transition to electronic health records (EHRs) is the associated observational data that capture EHR users' granular interactions with the medical record. Often referred to as audit log data or event log data, these datasets capture and timestamp user activity while they are logged in to the EHR. These data - alone and in combination with other datasets - offer a new source of insights, which cannot be gleaned from claims data or clinical data, to support health services research and those studying healthcare processes and outcomes. In this commentary, we seek to promote broader awareness of EHR audit log data and to stimulate their use in many contexts. We do so by describing EHR audit log data and offering a framework for their potential uses in quality domains (as defined by the National Academy of Medicine). The framework is illustrated with select examples in the safety and efficiency domains, along with their accompanying methodologies, which serve as a proof of concept. This article also discusses insights and challenges from working with EHR audit log data. Ensuring that researchers are aware of such data, and the new opportunities they offer, is one way to assure that our healthcare system benefits from the digital revolution.

Integrating People, Context, and Technology in the Implementation of a Web-Based Intervention in Forensic Mental Health Care: Mixed-Methods Study.

BACKGROUND: While eMental health interventions can have many potential benefits for mental health care, implementation outcomes are often disappointing. In order to improve these outcomes, there is a need for a better understanding of complex, dynamic interactions between a broad range of implementation-related factors. These interactions and processes should be studied holistically, paying attention to factors related to context, technology, and people. OBJECTIVE: The main objective of this mixed-method study was to holistically evaluate the implementation strategies and outcomes of an eMental health intervention in an organization for forensic mental health care. METHODS: First, desk research was performed on 18 documents on the implementation process. Second, the intervention's use by 721 patients and 172 therapists was analyzed via log data. Third, semistructured interviews were conducted with all 18 therapists of one outpatient clinic to identify broad factors that influence implementation outcomes. The interviews were analyzed via a combination of deductive analysis using the nonadoption, abandonment, scale-up, spread, and sustainability framework and inductive, open coding. RESULTS: The timeline generated via desk research showed that implementation strategies focused on technical skills training of therapists. Log data analyses demonstrated that 1019 modules were started, and 18.65% (721/3865) of patients of the forensic hospital started at least one module. Of these patients, 18.0% (130/721) completed at least one module. Of the therapists using the module, 54.1% (93/172 sent at least one feedback message to a patient. The median number of feedback messages sent per therapist was 1, with a minimum of 0 and a maximum of 460. Interviews showed that therapists did not always introduce the intervention to patients and using the intervention was not part of their daily routine. Also, therapists indicated patients often did not have the required conscientiousness and literacy levels. Furthermore, they had mixed opinions about the design of the intervention. Important organization-related factors were the need for more support and better integration in organizational structures. Finally, therapists stated that despite its current low use, the intervention had the potential to improve the quality of treatment. CONCLUSIONS: Synthesis of different types of data showed that implementation outcomes were mostly disappointing. Implementation strategies focused on technical training of therapists, while little attention was paid to changes in the organization, design of the technology, and patient awareness. A more holistic approach toward implementation strategies-with more attention to the organization, patients, technology, and training therapists-might have resulted in better implementation outcomes. Overall, adaptivity appears to be an important concept in eHealth implementation: a technology should be easily adaptable to an individual patient, therapists should be trained to deal flexibly with an eMental health intervention in their treatment, and organizations should adapt their implementation strategies and structures to embed a new eHealth intervention.

Predictors and Effects of Usage of an Online Mindfulness Intervention for Distressed Cancer Patients: Usability Study.

BACKGROUND: One in three cancer patients experience high psychological distress. Mindfulness-based interventions are effective in reducing psychological distress in this patient group. However, these interventions lack availability and flexibility, which may compromise participation in the intervention for cancer patients experiencing late symptoms like fatigue or pain. Therefore, mindfulness-based interventions are increasingly offered via the internet. However, little is known about the usage of these online mindfulness-based interventions. OBJECTIVE: The aim of this study was to (1) predict uptake of and adherence to online mindfulness-based cognitive therapy (eMBCT) using baseline patient characteristics (demographic, cancer-related, personality, and psychological variables) and (2) examine the relations between adherence and treatment outcomes in eMBCT for cancer patients. METHODS: A total of 125 cancer patients were assigned to eMBCT in a parent randomized controlled trial comparing MBCT and eMBCT with treatment as usual in distressed cancer patients. Various usage measures of eMBCT were automatically tracked within the online program. Based on activity of use, participants were classified as nonusers, minimal users, low users, and intended users. Questionnaires were used to assess baseline characteristics (preintervention) and outcomes (pre- and postintervention). To answer the research questions, data were analyzed with t tests, χ2 tests, and linear regression models. RESULTS: Based on weekly activity, participants were classified as nonusers (n=17, 13.6%), who completed no exercises in MBCT; minimal users (n=31, 24.8%), who completed at least one exercise of one to three sessions; low users (n=12, 9.6%), who completed at least one exercise of four to seven sessions; and intended users (n=65, 52.0%), who completed at least one exercise of eight to nine sessions. Nonusers had more fear of cancer recurrence at baseline than users (uptake), and intended users were more conscientious than minimal and low users (adherence). Intended users reported a larger reduction in psychological distress and more improvement of positive mental health (ie, emotional, psychological, and social well-being) after the intervention than other participants. CONCLUSIONS: This study showed that adherence was related to improved patient outcomes. Patients with strong fear of recurrence or low levels of conscientiousness should receive extra attention, as they are less likely to respectively start or complete eMBCT. Future research may focus on the development of flexible and adaptive eMBCT programs to fit individual needs.

A multi-faceted approach to characterizing user behavior and experience in a digital mental health intervention.

Digital interventions offer great promise for supporting health-related behavior change. However, there is much that we have yet to learn about how people respond to them. In this study, we present a novel mixed-methods approach to analysis of the complex and rich data that digital interventions collect. We perform secondary analysis of IntelliCare, an intervention in which participants are able to try 14 different mental health apps over the course of eight weeks. The goal of our analysis is to characterize users' app use behavior and experiences, and is rooted in theoretical conceptualizations of engagement as both usage and user experience. In the first aim, we employ cluster analysis to identify subgroups of participants that share similarities in terms of the frequency of their usage of particular apps, and then employ other engagement measures to compare the clusters. We identified four clusters with different app usage patterns: Low Usage, High Usage, Daily Feats Users, and Day to Day users. Each cluster was distinguished by its overall frequency of app use, or the main app that participants used. In the second aim, we developed a computer-assisted text analysis and visualization method - message highlighting - to facilitate comparison of the clusters. Last, we performed a qualitative analysis using participant messages to better understand the mechanisms of change and usability of salient apps from the cluster analysis. Our novel approach, integrating text and visual analytics with more traditional qualitative analysis techniques, can be used to generate insights concerning the behavior and experience of users in digital health contexts, for subsequent personalization and to identify areas for improvement of intervention technologies.

Data Mining in the Development of Mobile Health Apps: Assessing In-App Navigation Through Markov Chain Analysis.

BACKGROUND: Mobile apps generate vast amounts of user data. In the mobile health (mHealth) domain, researchers are increasingly discovering the opportunities of log data to assess the usage of their mobile apps. To date, however, the analysis of these data are often limited to descriptive statistics. Using data mining techniques, log data can offer significantly deeper insights. OBJECTIVE: The purpose of this study was to assess how Markov Chain and sequence clustering analysis can be used to find meaningful usage patterns of mHealth apps. METHODS: Using the data of a 25-day field trial (n=22) of the Start2Cycle app, an app developed to encourage recreational cycling in adults, a transition matrix between the different pages of the app was composed. From this matrix, a Markov Chain was constructed, enabling intuitive user behavior analysis. RESULTS: Through visual inspection of the transitions, 3 types of app use could be distinguished (route tracking, gamification, and bug reporting). Markov Chain-based sequence clustering was subsequently used to demonstrate how clusters of session types can otherwise be obtained. CONCLUSIONS: Using Markov Chains to assess in-app navigation presents a sound method to evaluate use of mHealth interventions. The insights can be used to evaluate app use and improve user experience.

The Service of Research Analytics to Optimize Digital Health Evidence Generation: Multilevel Case Study.

BACKGROUND: The widespread adoption of digital health interventions for chronic disease self-management has catalyzed a paradigm shift in the selection of methodologies used to evidence them. Recently, the application of digital health research analytics has emerged as an efficient approach to evaluate these data-rich interventions. However, there is a growing mismatch between the promising evidence base emerging from analytics mediated trials and the complexity of introducing these novel research methods into evaluative practice. OBJECTIVE: This study aimed to generate transferable insights into the process of implementing research analytics to evaluate digital health interventions. We sought to answer the following two research questions: (1) how should the service of research analytics be designed to optimize digital health evidence generation? and (2) what are the challenges and opportunities to scale, spread, and sustain this service in evaluative practice? METHODS: We conducted a qualitative multilevel embedded single case study of implementing research analytics in evaluative practice that comprised a review of the policy and regulatory climate in Ontario (macro level), a field study of introducing a digital health analytics platform into evaluative practice (meso level), and interviews with digital health innovators on their perceptions of analytics and evaluation (microlevel). RESULTS: The practice of research analytics is an efficient and effective means of supporting digital health evidence generation. The introduction of a research analytics platform to evaluate effective engagement with digital health interventions into a busy research lab was ultimately accepted by research staff, became routinized in their evaluative practice, and optimized their existing mechanisms of log data analysis and interpretation. The capacity for research analytics to optimize digital health evaluations is highest when there is (1) a collaborative working relationship between research client and analytics service provider, (2) a data-driven research agenda, (3) a robust data infrastructure with clear documentation of analytic tags, (4) in-house software development expertise, and (5) a collective tolerance for methodological change. CONCLUSIONS: Scientific methods and practices that can facilitate the agile trials needed to iterate and improve digital health interventions warrant continued implementation. The service of research analytics may help to accelerate the pace of digital health evidence generation and build a data-rich research infrastructure that enables continuous learning and evaluation.

Internet Searches for Medical Symptoms Before Seeking Information on 12-Step Addiction Treatment Programs: A Web-Search Log Analysis.

BACKGROUND: Brief intervention is a critical method for identifying patients with problematic substance use in primary care settings and for motivating them to consider treatment options. However, despite considerable evidence of delay discounting in patients with substance use disorders, most brief advice by physicians focuses on the long-term negative medical consequences, which may not be the best way to motivate patients to seek treatment information. OBJECTIVE: Identification of the specific symptoms that most motivate individuals to seek treatment information may offer insights for further improving brief interventions. To this end, we used anonymized internet search engine data to investigate which medical conditions and symptoms preceded searches for 12-step meeting locators and general 12-step information. METHODS: We extracted all queries made by people in the United States on the Bing search engine from November 2016 to July 2017. These queries were filtered for those who mentioned seeking Alcoholics Anonymous (AA) or Narcotics Anonymous (NA); in addition, queries that contained a medical symptom or condition or a synonym thereof were analyzed. We identified medical symptoms and conditions that predicted searches for seeking treatment at different time lags. Specifically, symptom queries were first determined to be significantly predictive of subsequent 12-step queries if the probability of querying a medical symptom by those who later sought information about the 12-step program exceeded the probability of that same query being made by a comparison group of all other Bing users in the United States. Second, we examined symptom queries preceding queries on the 12-step program at time lags of 0-7 days, 7-14 days, and 14-30 days, where the probability of asking about a medical symptom was greater in the 30-day time window preceding 12-step program information-seeking as compared to all previous times that the symptom was queried. RESULTS: In our sample of 11,784 persons, we found 10 medical symptoms that predicted AA information seeking and 9 symptoms that predicted NA information seeking. Of these symptoms, a substantial number could be categorized as nonsevere in nature. Moreover, when medical symptom persistence was examined across a 1-month time period, a substantial number of nonsevere, yet persistent, symptoms were identified. CONCLUSIONS: Our results suggest that many common or nonsevere medical symptoms and conditions motivate subsequent interest in AA and NA programs. In addition to highlighting severe long-term consequences, brief interventions could be restructured to highlight how increasing substance misuse can worsen discomfort from common medical symptoms in the short term, as well as how these worsening symptoms could exacerbate social embarrassment or decrease physical attractiveness.

Quality assurance of geometric accuracy based on an electronic portal imaging device and log data analysis for Dynamic WaveArc irradiation.

The purpose of this study was to develop a simple verification method for the routine quality assurance (QA) of Dynamic WaveArc (DWA) irradiation using electronic portal imaging device (EPID) images and log data analysis. First, an automatic calibration method utilizing the outermost multileaf collimator (MLC) slits was developed to correct the misalignment between the center of the EPID and the beam axis. Moreover, to verify the detection accuracy of the MLC position according to the EPID images, various positions of the MLC with intentional errors in the range 0.1-1 mm were assessed. Second, to validate the geometric accuracy during DWA irradiation, tests were designed in consideration of three indices. Test 1 evaluated the accuracy of the MLC position. Test 2 assessed dose output consistency with variable dose rate (160-400 MU/min), gantry speed (2.2-6°/s), and ring speed (0.5-2.7°/s). Test 3 validated dose output consistency with variable values of the above parameters plus MLC speed (1.6-4.2 cm/s). All tests were delivered to the EPID and compared with those obtained using a stationary radiation beam with a 0° gantry angle. Irradiation log data were recorded simultaneously. The 0.1-mm intentional error on the MLC position could be detected by the EPID, which is smaller than the EPID pixel size. In Test 1, the MLC slit widths agreed within 0.20 mm of their exposed values. The averaged root-mean-square error (RMSE) of the dose outputs was less than 0.8% in Test 2 and Test 3. Using log data analysis in Test 3, the RMSE between the planned and recorded data was 0.1 mm, 0.12°, and 0.07° for the MLC position, gantry angle, and ring angle, respectively. The proposed method is useful for routine QA of the accuracy of DWA.

Analyzing engagement in a web-based intervention platform through visualizing log-data.

BACKGROUND: Engagement has emerged as a significant cross-cutting concern within the development of Web-based interventions. There have been calls to institute a more rigorous approach to the design of Web-based interventions, to increase both the quantity and quality of engagement. One approach would be to use log-data to better understand the process of engagement and patterns of use. However, an important challenge lies in organizing log-data for productive analysis. OBJECTIVE: Our aim was to conduct an initial exploration of the use of visualizations of log-data to enhance understanding of engagement with Web-based interventions. METHODS: We applied exploratory sequential data analysis to highlight sequential aspects of the log data, such as time or module number, to provide insights into engagement. After applying a number of processing steps, a range of visualizations were generated from the log-data. We then examined the usefulness of these visualizations for understanding the engagement of individual users and the engagement of cohorts of users. The visualizations created are illustrated with two datasets drawn from studies using the SilverCloud Platform: (1) a small, detailed dataset with interviews (n=19) and (2) a large dataset (n=326) with 44,838 logged events. RESULTS: We present four exploratory visualizations of user engagement with a Web-based intervention, including Navigation Graph, Stripe Graph, Start-Finish Graph, and Next Action Heat Map. The first represents individual usage and the last three, specific aspects of cohort usage. We provide examples of each with a discussion of salient features. CONCLUSIONS: Log-data analysis through data visualization is an alternative way of exploring user engagement with Web-based interventions, which can yield different insights than more commonly used summative measures. We describe how understanding the process of engagement through visualizations can support the development and evaluation of Web-based interventions. Specifically, we show how visualizations can (1) allow inspection of content or feature usage in a temporal relationship to the overall program at different levels of granularity, (2) detect different patterns of use to consider personalization in the design process, (3) detect usability issues, (4) enable exploratory analysis to support the design of statistical queries to summarize the data, (5) provide new opportunities for real-time evaluation, and (6) examine assumptions about interactivity that underlie many summative measures in this field.

Testing the validity of the smartphone pervasiveness scale for adolescents with self-reported objective smartphone use data.

An ongoing and heated scientific debate pertains to the conceptualization and quantification of adolescents' problematic smartphone use (PSU). To address the limitations of existing surveys, the smartphone pervasiveness scale for adolescents (SPS-A) has been designed to measure the subjective frequency of smartphone usage during significant moments within daily routines. Given the weak correlations in prior literature between self-reported PSU metrics and objective use data, this study investigates the relationships between diverse self-reported objective metrics of smartphone engagement-that is duration, frequency, and count of notifications-and the SPS-A scale, employing a cohort of Swiss adolescents (N = 1396; Mage = 15.8, SDage = 0.81; 59% female). The findings reveal a substantial correlation between the total objectively measured duration of smartphone engagement and the SPS-A scale (r = .41 for iOS users and r = .42 for Android users). Moreover, a similar trend emerges as users are categorized by their level of objective use, with each category displaying a linear augmentation in smartphone pervasiveness levels. Instead, modest correlations emerge when considering the quantity of device unlocks and notifications. Noteworthy, no gender disparities emerged. These results add to our knowledge about the usefulness of the concept and measurement of smartphone pervasiveness: not only the SPS-A is a valid alternative to scales on "smartphone addiction" to capture non-pathological PSU, but it is also a better predictor of smartphone objective duration of use than self-reported measures. The correlation found between self-reported pervasiveness and actual use is discussed in light of the debate about the relevance of screen time in the study of PSU.

Development and application of Breadth-Depth-Context (BDC), a conceptual framework for measuring technology engagement with a qualified clinical data registry.

Objectives: Despite the proliferation of dashboards that display performance data derived from Qualified Clinical Data Registries (QCDR), the degree to which clinicians and practices engage with such dashboards has not been well described. We aimed to develop a conceptual framework for assessing user engagement with dashboard technology and to demonstrate its application to a rheumatology QCDR. Materials and Methods: We developed the BDC (Breadth-Depth-Context) framework, which included concepts of breadth (derived from dashboard sessions), depth (derived from dashboard actions), and context (derived from practice characteristics). We demonstrated its application via user log data from the American College of Rheumatology's Rheumatology Informatics System for Effectiveness (RISE) registry to define engagement profiles and characterize practice-level factors associated with different profiles. Results: We applied the BDC framework to 213 ambulatory practices from the RISE registry in 2020-2021, and classified practices into 4 engagement profiles: not engaged (8%), minimally engaged (39%), moderately engaged (34%), and most engaged (19%). Practices with more patients and with specific electronic health record vendors (eClinicalWorks and eMDs) had a higher likelihood of being in the most engaged group, even after adjusting for other factors. Discussion: We developed the BDC framework to characterize user engagement with a registry dashboard and demonstrated its use in a specialty QCDR. The application of the BDC framework revealed a wide range of breadth and depth of use and that specific contextual factors were associated with nature of engagement. Conclusion: Going forward, the BDC framework can be used to study engagement with similar dashboards.

The relation between usage of an eHealth intervention for stress urinary incontinence and treatment outcomes: an observational study.

BACKGROUND: Stress urinary incontinence (SUI), though a prevalent condition among women, is undertreated in primary care. EHealth with pelvic floor muscle training is an evidence-based alternative to care-as-usual. It is unknown, however, how eHealth usage is related to treatment outcome, and this knowledge is required for general practitioners to implement eHealth in their practice. This study examines the relation between usage of eHealth for SUI and treatment outcomes by examining log data. Baseline factors were also explored for associations with treatment success. METHOD: In this pre-post study, women with SUI participated in "Baasoverjeblaas.nl", a web-based intervention translated from the Swedish internet intervention "Tät®-treatment of stress urinary incontinence". Usage was based on log data and divided into three user groups (low, intermediate and high). Online questionnaires were sent before, after treatment and at six-months follow-up. The relation between usage and the primary outcome - treatment success (PGI-) - was studied with a binomial logistic regression analysis. Changes in the secondary outcomes - symptom severity (ICIQ-UI SF) and quality of life (ICIQ-LUTSqol) - were studied per user group with linear mixed model analysis. RESULTS: Included were 515 users with a mean age of 50.5 years (12.0 SD). The majority were low users (n = 295, 57.3%). Treatment success (PGI-I) was reached by one in four women and was more likely in high and intermediate users than in low users (OR 13.2, 95% CI 6.1-28.5, p < 0.001 and OR 2.92, 95% CI 1.35-6.34, p = 0.007, respectively). Symptom severity decreased and quality of life improved significantly over time, especially among high users. The women's expected ability to train their pelvic floor muscles and the frequency of pelvic floor muscle exercises at baseline were associated with treatment success. CONCLUSION: This study shows that usage of eHealth for SUI is related to all treatment outcomes. High users are more likely to have treatment success. Treatment success is more likely in women with higher expectations and pelvic floor muscle training at baseline. These findings indicate that general practitioners can select patients that would be more likely to benefit from eHealth treatment, and they can enhance treatment effect by stimulating eHealth usage. TRIAL REGISTRATION: Landelijk Trial Register NL6570;  https://onderzoekmetmensen.nl/nl/trial/25463 .

Identification of parameters for electronic distance examinations.

Introduction: This study investigates the log data and response behavior from invigilated in-person electronic timed exams at the University of Veterinary Medicine Hannover, Foundation, Germany. The primary focus is on understanding how various factors influence the time needed per exam item, including item format, item difficulty, item discrimination and character count. The aim was to use these results to derive recommendations for designing timed online distance examinations, an examination format that has become increasingly important in recent years. Methods: Data from 216,625 log entries of five electronic exams, taken by a total of 1,241 veterinary medicine students in 2021 and 2022, were analyzed. Various statistical methods were employed to assess the correlations between the recorded parameters. Results: The analysis revealed that different item formats require varying amounts of time. For instance, image-based question formats and Kprim necessitated more than 60 s per item, whereas one-best-answer multiple-choice questions (MCQs) and individual Key Feature items were effectively completed in less than 60 s. Furthermore, there was a positive correlation between character count and response time, suggesting that longer items require more time. A negative correlation could be verified for the parameters "difficulty" and "discrimination index" towards response time, indicating that more challenging items and those that are less able to differentiate between high- and low-performing students take longer to answer. Conclusion: The findings highlight the need for careful consideration of the ratio of item formats when defining time limits for exams. Regarding exam design, the literature mentions that time pressure is a critical factor, since it can negatively impact students' exam performance and some students, such as those with disabilities, are particularly disadvantaged. Therefore, this study emphasizes finding the right time limits to provide sufficient time for answering questions and reducing time pressure. In the context of unsupervised online exams, the findings of this study support previous recommendations that implementation of a stringent time limit might be a useful strategy to reduce cheating.

Use and Engagement With Low-Intensity Cognitive Behavioral Therapy Techniques Used Within an App to Support Worry Management: Content Analysis of Log Data.

BACKGROUND: Low-intensity cognitive behavioral therapy (LICBT) has been implemented by the Improving Access to Psychological Therapies services across England to manage excessive worry associated with generalized anxiety disorder and support emotional well-being. However, barriers to access limit scalability. A solution has been to incorporate LICBT techniques derived from an evidence-based protocol within the Iona Mind Well-being app for Worry management (IMWW) with support provided through an algorithmically driven conversational agent. OBJECTIVE: This study aims to examine engagement with a mobile phone app to support worry management with specific attention directed toward interaction with specific LICBT techniques and examine the potential to reduce symptoms of anxiety. METHODS: Log data were examined with respect to a sample of "engaged" users who had completed at least 1 lesson related to the Worry Time and Problem Solving in-app modules that represented the "minimum dose." Paired sample 2-tailed t tests were undertaken to examine the potential for IMWW to reduce worry and anxiety, with multivariate linear regressions examining the extent to which completion of each of the techniques led to reductions in worry and anxiety. RESULTS: There was good engagement with the range of specific LICBT techniques included within IMWW. The vast majority of engaged users were able to interact with the cognitive behavioral therapy model and successfully record types of worry. When working through Problem Solving, the conversational agent was successfully used to support the user with lower levels of engagement. Several users engaged with Worry Time outside of the app. Forgetting to use the app was the most common reason for lack of engagement, with features of the app such as completion of routine outcome measures and weekly reflections having lower levels of engagement. Despite difficulties in the collection of end point data, there was a significant reduction in severity for both anxiety (t53=5.5; P<.001; 95% CI 2.4-5.2) and low mood (t53=2.3; P=.03; 95% CI 0.2-3.3). A statistically significant linear model was also fitted to the Generalized Anxiety Disorder-7 (F2,51=6.73; P<.001), while the model predicting changes in the Patient Health Questionnaire-8 did not reach significance (F2,51=2.33; P=.11). This indicates that the reduction in these measures was affected by in-app engagement with Worry Time and Problem Solving. CONCLUSIONS: Engaged users were able to successfully interact with the LICBT-specific techniques informed by an evidence-based protocol although there were lower completion rates of routine outcome measures and weekly reflections. Successful interaction with the specific techniques potentially contributes to promising data, indicating that IMWW may be effective in the management of excessive worry. A relationship between dose and improvement justifies the use of log data to inform future developments. However, attention needs to be directed toward enhancing interaction with wider features of the app given that larger improvements were associated with greater engagement.

Effect of restricting electronic health records on clinician efficiency: substudy of a randomized clinical trial.

A prior randomized controlled trial (RCT) showed no significant difference in wrong-patient errors between clinicians assigned to a restricted electronic health record (EHR) configuration (limiting to 1 record open at a time) versus an unrestricted EHR configuration (allowing up to 4 records open concurrently). However, it is unknown whether an unrestricted EHR configuration is more efficient. This substudy of the RCT compared clinician efficiency between EHR configurations using objective measures. All clinicians who logged onto the EHR during the substudy period were included. The primary outcome measure of efficiency was total active minutes per day. Counts were extracted from audit log data, and mixed-effects negative binomial regression was performed to determine differences between randomized groups. Incidence rate ratios (IRRs) were calculated with 95% confidence intervals (CIs). Among a total of 2556 clinicians, there was no significant difference between unrestricted and restricted groups in total active minutes per day (115.1 vs 113.3 min, respectively; IRR, 0.99; 95% CI, 0.93-1.06), overall or by clinician type and practice area.

Using Browser Data to Understand Desires to Spend Time Online.

There is growing recognition that many people feel the need to regulate their use of the internet and other digital technologies to support their wellbeing. In this study, we used Mozilla Firefox browser telemetry to investigate the role played by various usage factors in desires to regulate time spent online. In particular, we investigated how six metrics pertaining to time spent on the internet, and the diversity and intensity of use, predict participants' (n = 8,094) desires to spend more or less time online. Across all six metrics, we did not find evidence for a relationship between browser usage metrics and participants wanting to spend more or less time online. This finding was robust across various analytical pathways. The study highlights a number of considerations and concerns that need to be addressed in future industry-academia collaborations that draw on trace data or usage telemetry.