HCI-modelling for improving the clinical usability of digital health technologies.

INTRODUCTION: Digital Health Technologies (DHTs) have been shown to have variable usability as measured by efficiency, effectiveness and user satisfaction despite large-scale government projects to regulate and standardise user interface (UI) design. We hypothesised that Human-Computer Interaction (HCI) modelling could improve the methodology for DHT design and regulation, and support the creation of future evidence-based UI standards and guidelines for DHTs. METHODOLOGY: Using a Design Science Research (DSR) framework, we developed novel UI components that adhered to existing standards and guidelines (combining the NHS Common User Interface (CUI) standard and the NHS Design System). We firstly evaluated the Patient Banner UI component for compliance with the two guidelines and then used HCI-modelling to evaluate the "Add New Patient" workflow to measure time to task completion and cognitive load. RESULTS: Combining the two guidelines to produce new UI elements is technically feasible for the Patient Banner and the Patient Name Input components. There are some inconsistencies between the NHS Design System and the NHS CUI when implementing the Patient Banner. HCI-modelling successfully quantified challenges adhering to the NHS CUI and the NHS Design system for the "Add New Patient" workflow. DISCUSSION: We successfully developed new design artefacts combing two major design guidelines for DHTs. By quantifying usability issues using HCI-modelling, we have demonstrated the feasibility of a methodology that combines HCI-modelling into a human-centred design (HCD) process could enable the development of standardised UI elements for DHTs that is more scientifically robust than HCD alone. CONCLUSION: Combining HCI-modelling and Human-Centred Design could improve scientific progress towards developing safer and more user-friendly DHTs.

Use of Consumer Wearables in Health Research: Issues and Considerations.

As wearable devices, which allow individuals to track and self-manage their health, become more ubiquitous, the opportunities are growing for researchers to use these sensors within interventions and for data collection. They offer access to data that are captured continuously, passively, and pragmatically with minimal user burden, providing huge advantages for health research. However, the growth in their use must be coupled with consideration of their potential limitations, in particular, digital inclusion, data availability, privacy, ethics of third-party involvement, data quality, and potential for adverse consequences. In this paper, we discuss these issues and strategies used to prevent or mitigate them and recommendations for researchers using wearables as part of interventions or for data collection.

Current ecotoxicity testing needs among selected U.S. federal agencies.

U.S. regulatory and research agencies use ecotoxicity test data to assess the hazards associated with substances that may be released into the environment, including but not limited to industrial chemicals, pharmaceuticals, pesticides, food additives, and color additives. These data are used to conduct hazard assessments and evaluate potential risks to aquatic life (e.g., invertebrates, fish), birds, wildlife species, or the environment. To identify opportunities for regulatory uses of non-animal replacements for ecotoxicity tests, the needs and uses for data from tests utilizing animals must first be clarified. Accordingly, the objective of this review was to identify the ecotoxicity test data relied upon by U.S. federal agencies. The standards, test guidelines, guidance documents, and/or endpoints that are used to address each of the agencies' regulatory and research needs regarding ecotoxicity testing are described in the context of their application to decision-making. Testing and information use, needs, and/or requirements relevant to the regulatory or programmatic mandates of the agencies taking part in the Interagency Coordinating Committee on the Validation of Alternative Methods Ecotoxicology Workgroup are captured. This information will be useful for coordinating efforts to develop and implement alternative test methods to reduce, refine, or replace animal use in chemical safety evaluations.

A Chatbot to Support Young People During the COVID-19 Pandemic in New Zealand: Evaluation of the Real-World Rollout of an Open Trial.

BACKGROUND: The number of young people in New Zealand (Aotearoa) who experience mental health challenges is increasing. As those in Aotearoa went into the initial COVID-19 lockdown, an ongoing digital mental health project was adapted and underwent rapid content authoring to create the Aroha chatbot. This dynamic digital support was designed with and for young people to help manage pandemic-related worry. OBJECTIVE: Aroha was developed to provide practical evidence-based tools for anxiety management using cognitive behavioral therapy and positive psychology. The chatbot included practical ideas to maintain social and cultural connection, and to stay active and well. METHODS: Stay-at-home orders under Aotearoa's lockdown commenced on March 20, 2020. By leveraging previously developed chatbot technology and broader existing online trial infrastructure, the Aroha chatbot was launched promptly on April 7, 2020. Dissemination of the chatbot for an open trial was via a URL, and feedback on the experience of the lockdown and the experience of Aroha was gathered via online questionnaires and a focus group, and from community members. RESULTS: In the 2 weeks following the launch of the chatbot, there were 393 registrations, and 238 users logged into the chatbot, of whom 127 were in the target age range (13-24 years). Feedback guided iterative and responsive content authoring to suit the dynamic situation and motivated engineering to dynamically detect and react to a range of conversational intents. CONCLUSIONS: The experience of the implementation of the Aroha chatbot highlights the feasibility of providing timely event-specific digital mental health support and the technology requirements for a flexible and enabling chatbot architectural framework.

Improving the Usability and Safety of Digital Health Systems: The Role of Predictive Human-Computer Interaction Modeling.

In this paper, we describe techniques for predictive modeling of human-computer interaction (HCI) and discuss how they could be used in the development and evaluation of user interfaces for digital health systems such as electronic health record systems. Predictive HCI modeling has the potential to improve the generalizability of usability evaluations of digital health interventions beyond specific contexts, especially when integrated with models of distributed cognition and higher-level sociotechnical frameworks. Evidence generated from building and testing HCI models of the user interface (UI) components for different types of digital health interventions could be valuable for informing evidence-based UI design guidelines to support the development of safer and more effective UIs for digital health interventions.

A mental models approach for defining explainable artificial intelligence.

BACKGROUND: Wide-ranging concerns exist regarding the use of black-box modelling methods in sensitive contexts such as healthcare. Despite performance gains and hype, uptake of artificial intelligence (AI) is hindered by these concerns. Explainable AI is thought to help alleviate these concerns. However, existing definitions for explainable are not forming a solid foundation for this work. METHODS: We critique recent reviews on the literature regarding: the agency of an AI within a team; mental models, especially as they apply to healthcare, and the practical aspects of their elicitation; and existing and current definitions of explainability, especially from the perspective of AI researchers. On the basis of this literature, we create a new definition of explainable, and supporting terms, providing definitions that can be objectively evaluated. Finally, we apply the new definition of explainable to three existing models, demonstrating how it can apply to previous research, and providing guidance for future research on the basis of this definition. RESULTS: Existing definitions of explanation are premised on global applicability and don't address the question 'understandable by whom?'. Eliciting mental models can be likened to creating explainable AI if one considers the AI as a member of a team. On this basis, we define explainability in terms of the context of the model, comprising the purpose, audience, and language of the model and explanation. As examples, this definition is applied to regression models, neural nets, and human mental models in operating-room teams. CONCLUSIONS: Existing definitions of explanation have limitations for ensuring that the concerns for practical applications are resolved. Defining explainability in terms of the context of their application forces evaluations to be aligned with the practical goals of the model. Further, it will allow researchers to explicitly distinguish between explanations for technical and lay audiences, allowing different evaluations to be applied to each.

Interrupted time series analysis on first cardiovascular disease hospitalization for adherence to lipid-lowering therapy.

PURPOSE: We analysed lipid-lowering medication adherence before and after the first hospitalization for cardiovascular disease (CVD) to explore the influence hospitalization has on patient medication adherence. METHODS: We extracted a sub-cohort for analysis from 313,207 patients who had primary CVD risk assessment. Adherence was assessed as proportion of days covered (PDC) ≥ 80% based on community dispensing records. Adherence in the 4 quarters (360 days) before the first CVD hospitalization and 8 quarters (720 days) after hospital discharge was assessed for each individual in the sub-cohort. An interrupted time series design using generalized estimating equations was applied to compare the differences of population-level medication adherence rates before and after the first CVD hospitalization. RESULTS: Overall, a significant improvement in medication adherence rate from before to after the hospitalization was observed (odds ratio (OR) 2.49 [1.74-3.57]) among the 946 patients included in the analysis. Patients having diabetes history had a higher OR of adherence before the hospitalization than patients without diabetes (1.50 [1.03-2.22]) but no significant difference after the hospitalization (OR 1.13 [0.89-1.43]). Before the first hospitalization, we observed that quarterly medication adherence rate was steady at around 55% (OR 0.97 [0.93-1.01), whereas the trend in adherence over the post-hospitalization period decreased significantly per quarter (OR 0.97 [0.94-0.99]). CONCLUSIONS: Patients were more likely to adhere to lipid-lowering therapy after experiencing a first CVD hospitalization. The change in medication adherence rate is consistent with patients having heightened perception of disease severity following the hospitalization.

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review.

BACKGROUND: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. OBJECTIVE: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. METHODS: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. RESULTS: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). CONCLUSIONS: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Debate: Supporting the mental health of school students in the COVID-19 pandemic in New Zealand - a digital ecosystem approach.

The pandemic is creating unprecedented demand for mental health support for young people. While schools often facilitate mental health support for their students, the demands for online teaching and the uncertainty created by the pandemic make traditional delivery of support through schools challenging. Technology provides a potential way forward. We have developed a digital ecosystem, HABITS, that can be integrated into school and healthcare systems. This has allowed us to deploy specific evidence-based interventions directly, and through schools, to students and to parents in New Zealand during the current pandemic. Chatbot architecture is particularly suited to rapid iteration to provide specific information while apps can provide more generalised support. While technology can provide some solutions, it is important to be aware of the potential to increase current inequities, with those facing the greatest challenges to health and well-being, also least able to afford the resources to access digital interventions. Development of an integrated and equitable digital system will take time and collaboration.

Cardiovascular disease risk prediction equations in 400 000 primary care patients in New Zealand: a derivation and validation study.

BACKGROUND: Most cardiovascular disease risk prediction equations in use today were derived from cohorts established last century and with participants at higher risk but less socioeconomically and ethnically diverse than patients they are now applied to. We recruited a nationally representative cohort in New Zealand to develop equations relevant to patients in contemporary primary care and compared the performance of these new equations to equations that are recommended in the USA. METHODS: The PREDICT study automatically recruits participants in routine primary care when general practitioners in New Zealand use PREDICT software to assess their patients' risk profiles for cardiovascular disease, which are prospectively linked to national ICD-coded hospitalisation and mortality databases. The study population included male and female patients in primary care who had no prior cardiovascular disease, renal disease, or congestive heart failure. New equations predicting total cardiovascular disease risk were developed using Cox regression models, which included clinical predictors plus an area-based deprivation index and self-identified ethnicity. Calibration and discrimination performance of the equations were assessed and compared with 2013 American College of Cardiology/American Heart Association Pooled Cohort Equations (PCEs). The additional predictors included in new PREDICT equations were also appended to the PCEs to determine whether they were independent predictors in the equations from the USA. FINDINGS: Outcome events were derived for 401 752 people aged 30-74 years at the time of their first PREDICT risk assessment between Aug 27, 2002, and Oct 12, 2015, representing about 90% of the eligible population. The mean follow-up was 4·2 years, and a third of participants were followed for 5 years or more. 15 386 (4%) people had cardiovascular disease events (1507 [10%] were fatal, and 8549 [56%] met the PCEs definition of hard atherosclerotic cardiovascular disease) during 1 685 521 person-years follow-up. The median 5-year risk of total cardiovascular disease events predicted by the new equations was 2·3% in women and 3·2% in men. Multivariable adjusted risk increased by about 10% per quintile of socioeconomic deprivation. Maori, Pacific, and Indian patients were at 13-48% higher risk of cardiovascular disease than Europeans, and Chinese or other Asians were at 25-33% lower risk of cardiovascular disease than Europeans. The PCEs overestimated of hard atherosclerotic cardiovascular disease by about 40% in men and by 60% in women, and the additional predictors in the new equations were also independent predictors in the PCEs. The new equations were significantly better than PCEs on all performance metrics. INTERPRETATION: We constructed a large prospective cohort study representing typical patients in primary care in New Zealand who were recommended for cardiovascular disease risk assessment. Most patients are now at low risk of cardiovascular disease, which explains why the PCEs based mainly on old cohorts substantially overestimate risk. Although the PCEs and many other equations will need to be recalibrated to mitigate overtreatment of the healthy majority, they also need new predictors that include measures of socioeconomic deprivation and multiple ethnicities to identify vulnerable high-risk subpopulations that might otherwise be undertreated. FUNDING: Health Research Council of New Zealand, Heart Foundation of New Zealand, and Healthier Lives National Science Challenge.

Burden of atrial fibrillation in Maori and Pacific people in New Zealand: a cohort study.

BACKGROUND: Atrial fibrillation (AF) is a major risk factor for ischaemic stroke and cardiovascular events. In New Zealand (NZ), Maori (indigenous New Zealanders) and Pacific people experience higher rates of AF compared with non-Maori/non-Pacific people. AIM: To describe a primary care population with AF in NZ. Stroke risk and medication adherence according to ethnicity are also detailed. METHODS: Electronic medical records for adults (≥20 years, n = 135 840, including 19 918 Maori and 43 634 Pacific people) enrolled at 37 NZ general practices were analysed for AF diagnosis and associated medication prescription information. RESULTS: The overall prevalence of non-valvular AF (NVAF) in this population was 1.3% (1769), and increased with age (4.4% in people ≥55 years). Maori aged ≥55 years were more likely to be diagnosed with NVAF (7.3%) than Pacific (4.0%) and non-Maori/non-Pacific people (4.1%, P < 0.001). Maori and Pacific NVAF patients were diagnosed with AF 10 years earlier than non-Maori/non-Pacific patients (median age of diagnosis: Maori = 60 years, Pacific = 61 years, non-Maori/non-Pacific = 71 years, P < 0.001). Overall, 67% of NVAF patients were at high risk for stroke (CHA2 DS2 -VASc ≥ 2) at the time of AF diagnosis. Almost half (48%) of Maori and Pacific NVAF patients aged <65 years were at high risk for stroke, compared with 22% of non-Maori/non-Pacific (P < 0.001). Irrespective of ethnic group, adherence to AF medication was suboptimal in those NVAF patients with a high risk of stroke or with stroke history. CONCLUSION: AF screening and stroke thromboprophylaxis in Maori and Pacific people could start below the age of 65 years in NZ.

Exploring reasoning mechanisms in ward rounds: a critical realist multiple case study.

BACKGROUND: Ward rounds are an important and ubiquitous element of hospital care with a history extending well over a century. Although originally intended as a means of educating medical trainees and junior doctors, over time they have become focused on supporting clinical practice. Surprisingly, given their ubiquity and importance, they are under-researched and inadequately understood. This study aims to contribute knowledge in human reasoning within medical teams, meeting a pressing need for research concerning the reasoning occurring in rounds. METHODS: The research reported here aimed to improve the understanding of ward round reasoning by conducting a critical realist case study exploring the collaborative group reasoning mechanisms in the ward rounds of two hospitals in Victoria, Australia. The data collection involved observing rounds, interviewing medical practitioners and holding focus group meetings. RESULTS: Nine group reasoning mechanisms concerning sharing, agreeing and recording information in the categories of information accumulation, sense-making and decision-making were identified, together forming a program theory of ward round reasoning. In addition, themes spanning across mechanisms were identified, further explaining ward round reasoning and suggesting avenues for future exploration. Themes included the use of various criteria, tensions involving mechanisms, time factors, medical roles and hierarchies. CONCLUSIONS: This paper contributes to the literature by representing rounds in a manner that strengthens understanding of the form of the group reasoning occurring within, thus supporting theory-based evaluation strategies, redesigned practices and training enhancements.

Detecting Emotional Context for Safer Digital Mental Health Agents.

Digital tools for mental health show great promise, but concerns arise when they fail to recognize the user state. We train a classifier to detect the emotional context of dialogs among 6 categories, achieving 78% accuracy on top choice. Importantly greatest areas of confusion (excited-hopeful, angry-sad) are not of the most unsafe kind. Such a classifier could serve as a resource to the dialog managers of future digital mental health agents.

eCHAT for lifestyle and mental health screening in primary care.

PURPOSE: Early detection and management of unhealthy behaviors and mental health issues in primary care has the potential to prevent or ameliorate many chronic diseases and increase patients' well-being. This study aimed to assess the feasibility and acceptability of the systematic use of a Web-based eCHAT (electronic Case-finding and Help Assessment Tool) screening patients for problematic drinking, smoking, and other drug use, gambling, exposure to abuse, anxiety, depression, anger control, and physical inactivity, and whether they want help with these issues. Patients self-administered eCHAT on an iPad in the waiting room and received summarized results, including relevant scores and interpretations, which could be by a family physician on the website and in the electronic health record (EHR) at the point of care. METHODS: We conducted a mixed method feasibility and acceptability study in 2 general practices in Auckland, New Zealand. Participants were consecutive adult patients attending the practice during a 2-week period, as well as all practice staff. Patients completed eCHAT, doctors accessed the summarized reports. Outcome measures were patients' responses to eCHAT, and patients' written and staff recorded interview feedback. RESULTS: Of the 233 invited patients, 196 (84%) completed eCHAT and received feedback. Domains where patients wanted immediate help were anxiety (9%), depression (7%), physical activity (6%), and smoking (5%), which was not overwhelming for physicians to address. Most patients found the iPad easy to use, and the questions easy to understand and appropriate; they did not object to questions. Feedback from 7 doctors, 2 practice managers, 4 nurses, and 5 receptionists was generally positive. Practices continue to use eCHAT regularly since the research was completed. CONCLUSIONS: eCHAT is an acceptable and feasible means of systemic screening patients for unhealthy behaviors and negative mood states and is easily integrated into the primary care electronic health record.

Gender differences in cardiovascular disease risk management for Pacific Islanders in primary care.

OBJECTIVES: To assess gender differences in cardiovascular disease risk (CVR) assessment and management for Pacific people in New Zealand. METHODS: New Zealand guidelines indicate CVR assessment from age 35 years for Pacific men and from age 45 years for Pacific women. Using general practice electronic medical records from 16 practices in New Zealand, the rate of CVR screening, treatment patterns and physiological measures for high-CVR (≥15% five-year) patients were assessed for Pacific patients ≥20 years of age by gender. RESULTS: Records for 10 863 Pacific patients showed a higher proportion of indicated women screened for CVR (65 vs 56%), but a lower proportion of assessed women with high CVR (28% for Pacific women vs 40% for Pacific men). Many of these high-CVR patients had physiological measures well above desirable levels based on their most recent readings. In the high-CVR group, women had similar CVR levels to men, but higher systolic blood pressure and HbA1c level, and a higher proportion of women were treated with antihypertensive and oral antidiabetic medication. There were substantial levels of poor medication adherence, particularly for cholesterol-lowering medication. Women and men were equally likely to adhere to treatment. Those adhering to relevant medications had lower blood pressure, total-to-HDL cholesterol ratio and HbA1c than non-adherers. CONCLUSIONS: Pacific men were less likely than Pacific women to have their CVR assessed when indicated, more likely once assessed to have high CVR and equally likely to adhere to treatment. Medication adherence was associated with better control of risk factors and should be further promoted in this population.

The Impact of Patient Preference on Attendance and Completion Rates at Centre-Based and mHealth Pulmonary Rehabilitation: A Non-Inferiority Pragmatic Clinical Trial.

Purpose: Pulmonary rehabilitation (PR) is vital in the management of chronic respiratory disorders (CRDs) although uptake, attendance and completion are poor. Differing models of delivering PR are emerging in an attempt to increase the uptake and completion of this intervention. This study aimed to evaluate participant rate of attendance and completion of PR when given a preference regarding model of delivery (centre-based and mPR). Secondary aims were to evaluate the factors affecting patient preference for model of delivery and determine whether mPR is non-inferior to centre-based PR in health outcomes. Methods: A multi-centre non-inferiority preference based clinical trial in Auckland, New Zealand. Participants with a CRD referred for PR were offered the choice of centre-based or mHealth PR (mPR). The primary outcome was completion rate of chosen intervention. Results: A total of 105 participants were recruited to the study with 67 (64%) preferring centre-based and 38 (36%) mPR. The odds of completing the PR programme were higher in the centre-based group compared to mPR (odds ratio 1.90 95% CI [0.83-4.35]). Participants opting for mPR were significantly younger (p = 0.002) and significantly more likely to be working (p = 0.0001). Results showed that mPR was not inferior to centre-based regarding changes in symptom scores (CAT) or time spent in sedentary behaviour (SBQ). When services were forced to transition to telehealth services during COVID-19 restrictions, the attendance and completion rates were higher with telephone calls and video conferencing compared to mPR - suggesting that synchronous interpersonal interactions with clinicians may facilitate the best attendance and completion rates. Conclusion: When offered the choice of PR delivery method, the majority of participants preferred centre-based PR and this facilitated the best completion rates. mPR was the preferred choice for younger, working participants suggesting that mPR may offer a viable alternative to centre-based PR for some participants, especially younger, employed participants.

Electronic referrals: what matters to the users.

BACKGROUND: Between September 2010 and May 2011 we evaluated three implementations of electronic referral (eReferral) systems at Hutt Valley, Northland and Canterbury District Health Boards in New Zealand. METHODS: Qualitative and quantitative data were gathered through project documentation, database records and stakeholder interviews. This paper reports on the user perspectives based on interviews with 78 clinical, management and operational stakeholders in the three regions. Themes that emerge across the regions are compared and synthesised. Interviews focused on pre-planned domains including quality of referral, ease of use and patient safety, but agendas were adapted progressively to elaborate and triangulate on themes emerging from earlier interviews and to clarify indications from analysis of database records. RESULTS AND DISCUSSION: The eReferral users, including general practitioners, specialists and administrative staff, report benefits in the areas of: (1) availability and transparency of referral-related data; (2) work transformation; (3) improved data quality and (4) the convenience of auto-population from the practice management system into the referral forms. eReferral provides enhanced visibility of referral data and status within the limits of the implementation (which only goes to the hospital door in some cases). Users in all projects indicated the desire to further exploit IT to enhance two-way communication between community and hospital. Reduced administrative handling is a clear work transformation benefit with mixed feedback regarding clinical workload impact. Innovations such as GP eReferral triaging teams illustrate the further potential for workflow transformation. Consistent structure in eReferrals, as well as simple legibility, enhances data quality. Efficiency and completeness is provided by auto-population of forms from system data, but opens issues around data accuracy. All three projects highlight the importance of user involvement in design, implementation and refinement. In keeping with this, Canterbury utilises a systematic pathway definition process that brings together GPs and specialist to debate and agree on the local management of a condition. User feedback exposes many opportunities for improving usability. STUDY LIMITATIONS: The findings are based on individual experiences accounted by participating stakeholders; the risk of bias is mitigated, however, by triangulation across three distinct implementations of eReferrals. Quantitative follow-up on key outstanding issues, notably impact of structured eReferral forms on GP time to write a referral, is warranted. CONCLUSION: Key eReferral users include clinicians on both ends of the referral process as well as the administrative staff. User experience in three eReferral projects has shown that they particularly appreciate improvement of referral visibility, as well as information quality; promising workflow transformations have been achieved in some places. Auto-population of forms leads to opportunities, and issues, that are prompting further attention to data quality. While the importance of user feedback should be obvious, it is not universal to seek it or to provide resources to effectively follow up with improvements driven by such feedback. To maximise benefits, innovative health IT projects must take an iterative approach guided by ongoing user experience.

Using the general practice EMR for improving blood pressure medication adherence.

PURPOSE: Analysis of practice electronic medical records (EMRs) demonstrated widespread antihypertensive medication adherence problems in a Pacific-led general practice serving a predominantly Pacific (majority Samoan) caseload in suburban New Zealand. Adherence was quantified in terms of medication possession ratio (MPR, percent of days covered by medication supply) from the practice's prescribing data. We studied the effectiveness of general practice staff follow-up guided by EMR data to improve medication adherence. METHODS: A framework for identification of suboptimal long-term condition management from routinely-collected EMR data, the ChronoMedIt (Chronological Medical Audit) tool, was applied to data of two Pacific-led general practices to identify patients with low MPR. One practice undertook intervention, the other provided usual care. A cohort was based on MPR<80% for antihypertensive medication in a baseline 6-month period. At the intervention practice a team was established to provide reminders and motivation for these patients and discuss their specific needs for assistance to improve adherence for 12 months. MPR and systolic blood pressure (SBP) was collected at baseline and for last six months of intervention based on practice EMRs; national claims data provided assessment of MPR based on dispensing. Nursing notes were analysed, and patient and provider focus groups were conducted. RESULTS: Of the 252 intervention patients with MPR<80% initially, MPR improved 12.0% (p=0.0002) and systolic blood pressure was 3.5mmHg lower (p=0.07) as compared to the control cohort. MPR from national claims data improved by 11.5% (p=0.0001) as compared to the control. Patients welcomed the approach as caring and useful. Providers felt the approach worthy of wider deployment but that it required dedicated staffing. DISCUSSION AND CONCLUSIONS: Systematic follow-up of patients with demonstrated poor medication possession appears effective in the context of a Pacific-led general practice serving a largely Pacific caseload. It was possible to exploit the EMR database to identify patients with low antihypertensive medication possession and to raise their level of medication possession significantly. The measured effect on systolic BP was only marginally significant, leaving open the question of the precise value of the intervention in terms of morbidity and mortality. The intervention was found to be feasible and was met with good acceptance from the intervention patients, who appreciated the concern reflected in the follow-up effort. The intervention practice is continuing use of ChronoMedIt to guide long-term condition management with extension to cholesterol and blood sugar.

Human Versus Machine: How Do We Know Who Is Winning? ROC Analysis for Comparing Human and Machine Performance under Varying Cost-Prevalence Assumptions.

BACKGROUND: Receiver operating characteristic (ROC) analysis is commonly used for comparing models and humans; however, the exact analytical techniques vary and some are flawed. OBJECTIVES: The aim of the study is to identify common flaws in ROC analysis for human versus model performance, and address them. METHODS: We review current use and identify common errors. We also review the ROC analysis literature for more appropriate techniques. RESULTS: We identify concerns in three techniques: (1) using mean human sensitivity and specificity; (2) assuming humans can be approximated by ROCs; and (3) matching sensitivity and specificity. We identify a technique from Provost et al using dominance tables and cost-prevalence gradients that can be adapted to address these concerns. CONCLUSION: Dominance tables and cost-prevalence gradients provide far greater detail when comparing performances of models and humans, and address common failings in other approaches. This should be the standard method for such analyses moving forward.

Multivariate Sequential Analytics for Cardiovascular Disease Event Prediction.

BACKGROUND: Automated clinical decision support for risk assessment is a powerful tool in combating cardiovascular disease (CVD), enabling targeted early intervention that could avoid issues of overtreatment or undertreatment. However, current CVD risk prediction models use observations at baseline without explicitly representing patient history as a time series. OBJECTIVE: The aim of this study is to examine whether by explicitly modelling the temporal dimension of patient history event prediction may be improved. METHODS: This study investigates methods for multivariate sequential modelling with a particular emphasis on long short-term memory (LSTM) recurrent neural networks. Data from a CVD decision support tool is linked to routinely collected national datasets including pharmaceutical dispensing, hospitalization, laboratory test results, and deaths. The study uses a 2-year observation and a 5-year prediction window. Selected methods are applied to the linked dataset. The experiments performed focus on CVD event prediction. CVD death or hospitalization in a 5-year interval was predicted for patients with history of lipid-lowering therapy. RESULTS: The results of the experiments showed temporal models are valuable for CVD event prediction over a 5-year interval. This is especially the case for LSTM, which produced the best predictive performance among all models compared achieving AUROC of 0.801 and average precision of 0.425. The non-temporal model comparator ridge classifier (RC) trained using all quarterly data or by aggregating quarterly data (averaging time-varying features) was highly competitive achieving AUROC of 0.799 and average precision of 0.420 and AUROC of 0.800 and average precision of 0.421, respectively. CONCLUSION: This study provides evidence that the use of deep temporal models particularly LSTM in clinical decision support for chronic disease would be advantageous with LSTM significantly improving on commonly used regression models such as logistic regression and Cox proportional hazards on the task of CVD event prediction.