HCI-modelling for improving the clinical usability of digital health technologies.

INTRODUCTION: Digital Health Technologies (DHTs) have been shown to have variable usability as measured by efficiency, effectiveness and user satisfaction despite large-scale government projects to regulate and standardise user interface (UI) design. We hypothesised that Human-Computer Interaction (HCI) modelling could improve the methodology for DHT design and regulation, and support the creation of future evidence-based UI standards and guidelines for DHTs. METHODOLOGY: Using a Design Science Research (DSR) framework, we developed novel UI components that adhered to existing standards and guidelines (combining the NHS Common User Interface (CUI) standard and the NHS Design System). We firstly evaluated the Patient Banner UI component for compliance with the two guidelines and then used HCI-modelling to evaluate the "Add New Patient" workflow to measure time to task completion and cognitive load. RESULTS: Combining the two guidelines to produce new UI elements is technically feasible for the Patient Banner and the Patient Name Input components. There are some inconsistencies between the NHS Design System and the NHS CUI when implementing the Patient Banner. HCI-modelling successfully quantified challenges adhering to the NHS CUI and the NHS Design system for the "Add New Patient" workflow. DISCUSSION: We successfully developed new design artefacts combing two major design guidelines for DHTs. By quantifying usability issues using HCI-modelling, we have demonstrated the feasibility of a methodology that combines HCI-modelling into a human-centred design (HCD) process could enable the development of standardised UI elements for DHTs that is more scientifically robust than HCD alone. CONCLUSION: Combining HCI-modelling and Human-Centred Design could improve scientific progress towards developing safer and more user-friendly DHTs.

Detecting Emotional Context for Safer Digital Mental Health Agents.

Digital tools for mental health show great promise, but concerns arise when they fail to recognize the user state. We train a classifier to detect the emotional context of dialogs among 6 categories, achieving 78% accuracy on top choice. Importantly greatest areas of confusion (excited-hopeful, angry-sad) are not of the most unsafe kind. Such a classifier could serve as a resource to the dialog managers of future digital mental health agents.

Use of Consumer Wearables in Health Research: Issues and Considerations.

As wearable devices, which allow individuals to track and self-manage their health, become more ubiquitous, the opportunities are growing for researchers to use these sensors within interventions and for data collection. They offer access to data that are captured continuously, passively, and pragmatically with minimal user burden, providing huge advantages for health research. However, the growth in their use must be coupled with consideration of their potential limitations, in particular, digital inclusion, data availability, privacy, ethics of third-party involvement, data quality, and potential for adverse consequences. In this paper, we discuss these issues and strategies used to prevent or mitigate them and recommendations for researchers using wearables as part of interventions or for data collection.

The Impact of Patient Preference on Attendance and Completion Rates at Centre-Based and mHealth Pulmonary Rehabilitation: A Non-Inferiority Pragmatic Clinical Trial.

Purpose: Pulmonary rehabilitation (PR) is vital in the management of chronic respiratory disorders (CRDs) although uptake, attendance and completion are poor. Differing models of delivering PR are emerging in an attempt to increase the uptake and completion of this intervention. This study aimed to evaluate participant rate of attendance and completion of PR when given a preference regarding model of delivery (centre-based and mPR). Secondary aims were to evaluate the factors affecting patient preference for model of delivery and determine whether mPR is non-inferior to centre-based PR in health outcomes. Methods: A multi-centre non-inferiority preference based clinical trial in Auckland, New Zealand. Participants with a CRD referred for PR were offered the choice of centre-based or mHealth PR (mPR). The primary outcome was completion rate of chosen intervention. Results: A total of 105 participants were recruited to the study with 67 (64%) preferring centre-based and 38 (36%) mPR. The odds of completing the PR programme were higher in the centre-based group compared to mPR (odds ratio 1.90 95% CI [0.83-4.35]). Participants opting for mPR were significantly younger (p = 0.002) and significantly more likely to be working (p = 0.0001). Results showed that mPR was not inferior to centre-based regarding changes in symptom scores (CAT) or time spent in sedentary behaviour (SBQ). When services were forced to transition to telehealth services during COVID-19 restrictions, the attendance and completion rates were higher with telephone calls and video conferencing compared to mPR - suggesting that synchronous interpersonal interactions with clinicians may facilitate the best attendance and completion rates. Conclusion: When offered the choice of PR delivery method, the majority of participants preferred centre-based PR and this facilitated the best completion rates. mPR was the preferred choice for younger, working participants suggesting that mPR may offer a viable alternative to centre-based PR for some participants, especially younger, employed participants.

Multivariate Sequential Analytics for Cardiovascular Disease Event Prediction.

BACKGROUND: Automated clinical decision support for risk assessment is a powerful tool in combating cardiovascular disease (CVD), enabling targeted early intervention that could avoid issues of overtreatment or undertreatment. However, current CVD risk prediction models use observations at baseline without explicitly representing patient history as a time series. OBJECTIVE: The aim of this study is to examine whether by explicitly modelling the temporal dimension of patient history event prediction may be improved. METHODS: This study investigates methods for multivariate sequential modelling with a particular emphasis on long short-term memory (LSTM) recurrent neural networks. Data from a CVD decision support tool is linked to routinely collected national datasets including pharmaceutical dispensing, hospitalization, laboratory test results, and deaths. The study uses a 2-year observation and a 5-year prediction window. Selected methods are applied to the linked dataset. The experiments performed focus on CVD event prediction. CVD death or hospitalization in a 5-year interval was predicted for patients with history of lipid-lowering therapy. RESULTS: The results of the experiments showed temporal models are valuable for CVD event prediction over a 5-year interval. This is especially the case for LSTM, which produced the best predictive performance among all models compared achieving AUROC of 0.801 and average precision of 0.425. The non-temporal model comparator ridge classifier (RC) trained using all quarterly data or by aggregating quarterly data (averaging time-varying features) was highly competitive achieving AUROC of 0.799 and average precision of 0.420 and AUROC of 0.800 and average precision of 0.421, respectively. CONCLUSION: This study provides evidence that the use of deep temporal models particularly LSTM in clinical decision support for chronic disease would be advantageous with LSTM significantly improving on commonly used regression models such as logistic regression and Cox proportional hazards on the task of CVD event prediction.

A Chatbot to Support Young People During the COVID-19 Pandemic in New Zealand: Evaluation of the Real-World Rollout of an Open Trial.

BACKGROUND: The number of young people in New Zealand (Aotearoa) who experience mental health challenges is increasing. As those in Aotearoa went into the initial COVID-19 lockdown, an ongoing digital mental health project was adapted and underwent rapid content authoring to create the Aroha chatbot. This dynamic digital support was designed with and for young people to help manage pandemic-related worry. OBJECTIVE: Aroha was developed to provide practical evidence-based tools for anxiety management using cognitive behavioral therapy and positive psychology. The chatbot included practical ideas to maintain social and cultural connection, and to stay active and well. METHODS: Stay-at-home orders under Aotearoa's lockdown commenced on March 20, 2020. By leveraging previously developed chatbot technology and broader existing online trial infrastructure, the Aroha chatbot was launched promptly on April 7, 2020. Dissemination of the chatbot for an open trial was via a URL, and feedback on the experience of the lockdown and the experience of Aroha was gathered via online questionnaires and a focus group, and from community members. RESULTS: In the 2 weeks following the launch of the chatbot, there were 393 registrations, and 238 users logged into the chatbot, of whom 127 were in the target age range (13-24 years). Feedback guided iterative and responsive content authoring to suit the dynamic situation and motivated engineering to dynamically detect and react to a range of conversational intents. CONCLUSIONS: The experience of the implementation of the Aroha chatbot highlights the feasibility of providing timely event-specific digital mental health support and the technology requirements for a flexible and enabling chatbot architectural framework.

Current ecotoxicity testing needs among selected U.S. federal agencies.

U.S. regulatory and research agencies use ecotoxicity test data to assess the hazards associated with substances that may be released into the environment, including but not limited to industrial chemicals, pharmaceuticals, pesticides, food additives, and color additives. These data are used to conduct hazard assessments and evaluate potential risks to aquatic life (e.g., invertebrates, fish), birds, wildlife species, or the environment. To identify opportunities for regulatory uses of non-animal replacements for ecotoxicity tests, the needs and uses for data from tests utilizing animals must first be clarified. Accordingly, the objective of this review was to identify the ecotoxicity test data relied upon by U.S. federal agencies. The standards, test guidelines, guidance documents, and/or endpoints that are used to address each of the agencies' regulatory and research needs regarding ecotoxicity testing are described in the context of their application to decision-making. Testing and information use, needs, and/or requirements relevant to the regulatory or programmatic mandates of the agencies taking part in the Interagency Coordinating Committee on the Validation of Alternative Methods Ecotoxicology Workgroup are captured. This information will be useful for coordinating efforts to develop and implement alternative test methods to reduce, refine, or replace animal use in chemical safety evaluations.

Human Versus Machine: How Do We Know Who Is Winning? ROC Analysis for Comparing Human and Machine Performance under Varying Cost-Prevalence Assumptions.

BACKGROUND: Receiver operating characteristic (ROC) analysis is commonly used for comparing models and humans; however, the exact analytical techniques vary and some are flawed. OBJECTIVES: The aim of the study is to identify common flaws in ROC analysis for human versus model performance, and address them. METHODS: We review current use and identify common errors. We also review the ROC analysis literature for more appropriate techniques. RESULTS: We identify concerns in three techniques: (1) using mean human sensitivity and specificity; (2) assuming humans can be approximated by ROCs; and (3) matching sensitivity and specificity. We identify a technique from Provost et al using dominance tables and cost-prevalence gradients that can be adapted to address these concerns. CONCLUSION: Dominance tables and cost-prevalence gradients provide far greater detail when comparing performances of models and humans, and address common failings in other approaches. This should be the standard method for such analyses moving forward.

A mental models approach for defining explainable artificial intelligence.

BACKGROUND: Wide-ranging concerns exist regarding the use of black-box modelling methods in sensitive contexts such as healthcare. Despite performance gains and hype, uptake of artificial intelligence (AI) is hindered by these concerns. Explainable AI is thought to help alleviate these concerns. However, existing definitions for explainable are not forming a solid foundation for this work. METHODS: We critique recent reviews on the literature regarding: the agency of an AI within a team; mental models, especially as they apply to healthcare, and the practical aspects of their elicitation; and existing and current definitions of explainability, especially from the perspective of AI researchers. On the basis of this literature, we create a new definition of explainable, and supporting terms, providing definitions that can be objectively evaluated. Finally, we apply the new definition of explainable to three existing models, demonstrating how it can apply to previous research, and providing guidance for future research on the basis of this definition. RESULTS: Existing definitions of explanation are premised on global applicability and don't address the question 'understandable by whom?'. Eliciting mental models can be likened to creating explainable AI if one considers the AI as a member of a team. On this basis, we define explainability in terms of the context of the model, comprising the purpose, audience, and language of the model and explanation. As examples, this definition is applied to regression models, neural nets, and human mental models in operating-room teams. CONCLUSIONS: Existing definitions of explanation have limitations for ensuring that the concerns for practical applications are resolved. Defining explainability in terms of the context of their application forces evaluations to be aligned with the practical goals of the model. Further, it will allow researchers to explicitly distinguish between explanations for technical and lay audiences, allowing different evaluations to be applied to each.

Evaluation Metrics for Health Chatbots: A Delphi Study.

BACKGROUND: In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear. OBJECTIVES: The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics. METHODS: We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics). RESULTS: Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon-only moderate or low consensus was achieved for those metrics. CONCLUSION: The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multifaceted to ensure acceptability.

Improving the Usability and Safety of Digital Health Systems: The Role of Predictive Human-Computer Interaction Modeling.

In this paper, we describe techniques for predictive modeling of human-computer interaction (HCI) and discuss how they could be used in the development and evaluation of user interfaces for digital health systems such as electronic health record systems. Predictive HCI modeling has the potential to improve the generalizability of usability evaluations of digital health interventions beyond specific contexts, especially when integrated with models of distributed cognition and higher-level sociotechnical frameworks. Evidence generated from building and testing HCI models of the user interface (UI) components for different types of digital health interventions could be valuable for informing evidence-based UI design guidelines to support the development of safer and more effective UIs for digital health interventions.

Self-Harm Detection for Mental Health Chatbots.

Chatbots potentially address deficits in availability of the traditional health workforce and could help to stem concerning rates of youth mental health issues including high suicide rates. While chatbots have shown some positive results in helping people cope with mental health issues, there are yet deep concerns regarding such chatbots in terms of their ability to identify emergency situations and act accordingly. Risk of suicide/self-harm is one such concern which we have addressed in this project. A chatbot decides its response based on the text input from the user and must correctly recognize the significance of a given input. We have designed a self-harm classifier which could use the user's response to the chatbot and predict whether the response indicates intent for self-harm. With the difficulty to access confidential counselling data, we looked for alternate data sources and found Twitter and Reddit to provide data similar to what we would expect to get from a chatbot user. We trained a sentiment analysis classifier on Twitter data and a self-harm classifier on the Reddit data. We combined the results of the two models to improve the model performance. We got the best results from a LSTM-RNN classifier using BERT encoding. The best model accuracy achieved was 92.13%. We tested the model on new data from Reddit and got an impressive result with an accuracy of 97%. Such a model is promising for future embedding in mental health chatbots to improve their safety through accurate detection of self-harm talk by users.

VeCHAT: a proof-of-concept study on screening and managing veterans.

INTRODUCTION New Zealand veterans may have complex mental and physical complaints related to multiple exposures to war environments. They are entitled to, but often do not, access a range of physical, mental health and social services funded through Veterans' Affairs New Zealand. eCHAT (electronic Case-finding and Help Assessment Tool) is a self-completed electronic holistic screen for substance misuse, problem gambling, anger control, physical inactivity, depression, anxiety, exposure to abuse; and assesses whether help is wanted for identified issues. AIM A proof-of-concept study was conducted to develop a modified version of eCHAT (VeCHAT) with remote functionality for clinical assessment of mental health and lifestyle issues of contemporary veterans, and assesses acceptability by veterans and Veterans' Affairs staff, and feasibility of implementation. METHODS We used a co-design approach to develop VeCHAT. Veterans' Affairs and service organisations invited veterans to remotely complete VeCHAT and a subsequent short online acceptability survey. Veterans' Affairs medical and case manager staff underwent semi-structured interviews on feasibility and acceptability of VeCHAT use. RESULTS Thirty-four veterans completed VeCHAT. The tool proved acceptable to veterans and Veterans' Affairs staff. Key emergent themes related to tool functionality, design, ways and barriers to use, and suggested improvements. Veterans' Affairs staff considered VeCHAT use to be feasible with much potential. DISCUSSION Capacity of Veterans' Affairs to respond if their engagement with veterans increases and employment of VeCHAT is scaled up, is unknown. Work is needed to assess how introducing VeCHAT as a standard procedure might influence Veterans' Affairs case management processes.

Debate: Supporting the mental health of school students in the COVID-19 pandemic in New Zealand - a digital ecosystem approach.

The pandemic is creating unprecedented demand for mental health support for young people. While schools often facilitate mental health support for their students, the demands for online teaching and the uncertainty created by the pandemic make traditional delivery of support through schools challenging. Technology provides a potential way forward. We have developed a digital ecosystem, HABITS, that can be integrated into school and healthcare systems. This has allowed us to deploy specific evidence-based interventions directly, and through schools, to students and to parents in New Zealand during the current pandemic. Chatbot architecture is particularly suited to rapid iteration to provide specific information while apps can provide more generalised support. While technology can provide some solutions, it is important to be aware of the potential to increase current inequities, with those facing the greatest challenges to health and well-being, also least able to afford the resources to access digital interventions. Development of an integrated and equitable digital system will take time and collaboration.

Predicting Lipid-Lowering Medication Persistence after the First Cardiovascular Disease Hospitalization.

OBJECTIVES: This study analyzed patient factors in medication persistence after discharge from the first hospitalization for cardiovascular disease (CVD) with the aim of predicting persistence to lipid-lowering therapy for 1 to 2 years. METHODS: A subcohort having a first CVD hospitalization was selected from 313,207 patients for proportional hazard model analysis. Logistic regression, support vector machine, artificial neural networks, and boosted regression tree (BRT) models were used to predict 1- and 2-year medication persistence. RESULTS: Proportional hazard modeling found significant association of persistence with age, diabetes history, complication and comorbidity level, days stayed in hospital, CVD diagnosis type, in-patient procedures, and being new to therapy. BRT had the best predictive performance with c-statistic of 0.811 (0.799-0.824) for 1-year and 0.793 (0.772-0.814) for 2-year prediction using variables potentially available shortly after discharge. CONCLUSION: The results suggest that development of a machine learning-based clinical decision support tool to focus improvements in secondary prevention of CVD is feasible.

How to Evaluate Health Applications with Conversational User Interface?

Application of conversational user interfaces (CUI) or chatbots to healthcare is gaining interest fueled by the rising power of artificial intelligence, increasing popularity of mobile health applications and the desire for engagement and usability. While their use is mainly justified by increasing adherence to mobile health applications and facilitating interactions with the system, the question arises: How can such systems be evaluated in a reliable manner? This paper introduces an evaluation framework for health systems whose core interaction principle is a CUI. We derive quality dimensions and attributes by collecting relevant evaluation aspects from applications that have been developed in previous work and from literature on health chatbots. The collected aspects are aggregated into six thematic categories for chatbot quality, including user experience, linguistic, task-oriented and artificial intelligence perspectives, but also healthcare quality and system quality perspectives. The framework is intended to support developers and researchers in the domain of chatbots in healthcare in selecting relevant quality attributes to be assessed before their systems are distributed to patients.

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review.

BACKGROUND: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. OBJECTIVE: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. METHODS: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. RESULTS: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). CONCLUSIONS: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Interrupted time series analysis on first cardiovascular disease hospitalization for adherence to lipid-lowering therapy.

PURPOSE: We analysed lipid-lowering medication adherence before and after the first hospitalization for cardiovascular disease (CVD) to explore the influence hospitalization has on patient medication adherence. METHODS: We extracted a sub-cohort for analysis from 313,207 patients who had primary CVD risk assessment. Adherence was assessed as proportion of days covered (PDC) ≥ 80% based on community dispensing records. Adherence in the 4 quarters (360 days) before the first CVD hospitalization and 8 quarters (720 days) after hospital discharge was assessed for each individual in the sub-cohort. An interrupted time series design using generalized estimating equations was applied to compare the differences of population-level medication adherence rates before and after the first CVD hospitalization. RESULTS: Overall, a significant improvement in medication adherence rate from before to after the hospitalization was observed (odds ratio (OR) 2.49 [1.74-3.57]) among the 946 patients included in the analysis. Patients having diabetes history had a higher OR of adherence before the hospitalization than patients without diabetes (1.50 [1.03-2.22]) but no significant difference after the hospitalization (OR 1.13 [0.89-1.43]). Before the first hospitalization, we observed that quarterly medication adherence rate was steady at around 55% (OR 0.97 [0.93-1.01), whereas the trend in adherence over the post-hospitalization period decreased significantly per quarter (OR 0.97 [0.94-0.99]). CONCLUSIONS: Patients were more likely to adhere to lipid-lowering therapy after experiencing a first CVD hospitalization. The change in medication adherence rate is consistent with patients having heightened perception of disease severity following the hospitalization.

Understanding End-User Perspectives of Mobile Pulmonary Rehabilitation (mPR): Cross-Sectional Survey and Interviews.

BACKGROUND: Pulmonary rehabilitation (PR) is an effective intervention for the management of people with chronic respiratory diseases, but the uptake of and adherence to PR programs is low. There is potential for mobile health (mHealth) to provide an alternative modality for the delivery of PR, overcoming many of the barriers contributing to poor attendance to current services. OBJECTIVE: The objective of this study was to understand the needs, preferences, and priorities of end users for the development of an adaptive mobile PR (mPR) support program. METHODS: A mixed methods (qualitative and quantitative) approach was used to assess the needs, preferences, and priorities of the end users (ie, patients with chronic respiratory disorders) and key stakeholders (ie, clinicians working with patients with chronic respiratory disorders and running PR). The formative studies included the following: (1) a survey to understand the preferences and priorities of patients for PR and how mobile technology could be used to provide PR support, (2) ethnographic semistructured interviews with patients with chronic respiratory disorders to gain perspectives on their understanding of their health and potential features that could be included in an mPR program, and (3) key informant interviews with health care providers to understand the needs, preferences, and priorities for the development of an mPR support program. RESULTS: Across all formative studies (patient survey, n=30; patient interviews, n=8; and key stakeholder interviews, n=8), the participants were positive about the idea of an mPR program but raised concerns related to digital literacy and confidence in using technology, access to technology, and loss of social support currently gained from traditional programs. Key stakeholders highlighted the need for patient safety to be maintained and ensuring appropriate programs for different groups within the population. Finding a balance between ensuring safety and maximizing access was seen to be essential in the success of an mPR program. CONCLUSIONS: These formative studies found high interest in mHealth-based PR intervention and detailed the potential for an mPR program to overcome current barriers to accessing traditional PR programs. Key considerations and features were identified, including the importance of technology access and digital literacy being considered in utilizing technology with this population.

Junior Doctor Communication Systems and the Deterioration Communication Management Theory.

Inadequate communication is a factor in suboptimal junior doctor management of deteriorating ward patients. Junior doctors' information and communication technology (ICT) systems are not the sole cause or cure for this. However, junior doctors are already dissatisfied with existing technologies for general hospital communication. The Deterioration Communication Management Theory (DCMT) provides a means to approach these issues by uniting two themes: 1) factors affecting the properties of ICT used to communicate to junior doctors; and 2) factors affecting junior doctor interpretation of communication about deteriorating hospital patients. ICT factors include how the combination of physical devices and mode of usage affect user perception of system reliability and efficiency. Junior doctors interpret clinician communication about patient deterioration in terms of risk, which is affected by their contextual responsibility and experience. Perceived risk and contextual experience in turn affects their communication efficiency. Combining these themes gives more options to explain junior doctor communication in this clinical context and to design ICT systems to improve it.