How can we discover the most valuable types of big data and artificial intelligence-based solutions? A methodology for the efficient development of the underlying analytics that improve care.

BACKGROUND: Much has been invested in big data and artificial intelligence-based solutions for healthcare. However, few applications have been implemented in clinical practice. Early economic evaluations can help to improve decision-making by developers of analytics underlying these solutions aiming to increase the likelihood of successful implementation, but recommendations about their use are lacking. The aim of this study was to develop and apply a framework that positions best practice methods for economic evaluations alongside development of analytics, thereby enabling developers to identify barriers to success and to select analytics worth further investments. METHODS: The framework was developed using literature, recommendations for economic evaluations and by applying the framework to use cases (chronic lymphocytic leukaemia (CLL), intensive care, diabetes). First, the feasibility of developing clinically relevant analytics was assessed and critical barriers to successful development and implementation identified. Economic evaluations were then used to determine critical thresholds and guide investment decisions. RESULTS: When using the framework to assist decision-making of developers of analytics, continuing development was not always feasible or worthwhile. Developing analytics for progressive CLL and diabetes was clinically relevant but not feasible with the data available. Alternatively, developing analytics for newly diagnosed CLL patients was feasible but continuing development was not considered worthwhile because the high drug costs made it economically unattractive for potential users. Alternatively, in the intensive care unit, analytics reduced mortality and per-patient costs when used to identify infections (- 0.5%, - €886) and to improve patient-ventilator interaction (- 3%, - €264). Both analytics have the potential to save money but the potential benefits of analytics that identify infections strongly depend on infection rate; a higher rate implies greater cost-savings. CONCLUSIONS: We present a framework that stimulates efficiency of development of analytics for big data and artificial intelligence-based solutions by selecting those applications of analytics for which development is feasible and worthwhile. For these applications, results from early economic evaluations can be used to guide investment decisions and identify critical requirements.

Optimization and Validation of a Classification Algorithm for Assessment of Physical Activity in Hospitalized Patients.

Low amounts of physical activity (PA) and prolonged periods of sedentary activity are common in hospitalized patients. Objective PA monitoring is needed to prevent the negative effects of inactivity, but a suitable algorithm is lacking. The aim of this study is to optimize and validate a classification algorithm that discriminates between sedentary, standing, and dynamic activities, and records postural transitions in hospitalized patients under free-living conditions. Optimization and validation in comparison to video analysis were performed in orthopedic and acutely hospitalized elderly patients with an accelerometer worn on the upper leg. Data segmentation window size (WS), amount of PA threshold (PA Th) and sensor orientation threshold (SO Th) were optimized in 25 patients, validation was performed in another 25. Sensitivity, specificity, accuracy, and (absolute) percentage error were used to assess the algorithm's performance. Optimization resulted in the best performance with parameter settings: WS 4 s, PA Th 4.3 counts per second, SO Th 0.8 g. Validation showed that all activities were classified within acceptable limits (>80% sensitivity, specificity and accuracy, ±10% error), except for the classification of standing activity. As patients need to increase their PA and interrupt sedentary behavior, the algorithm is suitable for classifying PA in hospitalized patients.

The potential of real-time analytics to improve care for mechanically ventilated patients in the intensive care unit: an early economic evaluation.

BACKGROUND: Mechanical ventilation services are an important driver of the high costs of intensive care. An optimal interaction between a patient and a ventilator is therefore paramount. Suboptimal interaction is present when patients repeatedly demand, but do not receive, breathing support from a mechanical ventilator (> 30 times in 3 min), also known as an ineffective effort event (IEEV). IEEVs are associated with increased hospital mortality prolonged intensive care stay, and prolonged time on ventilation and thus development of real-time analytics that identify IEEVs is essential. To assist decision-making about further development we estimate the potential cost-effectiveness of real-time analytics that identify ineffective effort events. METHODS: We developed a cost-effectiveness model combining a decision tree and Markov model for long-term outcomes with data on current care from a Greek hospital and literature. A lifetime horizon and a healthcare payer perspective were used. Uncertainty about the results was assessed using sensitivity and scenario analyses to examine the impact of varying parameters like the intensive care costs per day and the effectiveness of treatment of IEEVs. RESULTS: Use of the analytics could lead to reduced mortality (3% absolute reduction), increased quality adjusted life years (0.21 per patient) and cost-savings (€264 per patient) compared to current care. Moreover, cost-savings for hospitals and health improvements can be incurred even if the treatment's effectiveness is reduced from 30 to 10%. The estimated savings increase to €1,155 per patient in countries where costs of an intensive care day are high (e.g. the Netherlands). There is considerable headroom for development and the analytics generate savings when the price of the analytics per bed per year is below €7,307. Furthermore, even when the treatment's effectiveness is 10%, the probability that the analytics are cost-effective exceeds 90%. CONCLUSIONS: Implementing real-time analytics to identify ineffective effort events can lead to health and financial benefits. Therefore, it will be worthwhile to continue assessment of the effectiveness of the analytics in clinical practice and validate our findings. Eventually, their adoption in settings where costs of an intensive care day are high and ineffective efforts are frequent could yield a high return on investment.

The Validation of a Pocket Worn Activity Tracker for Step Count and Physical Behavior in Older Adults during Simulated Activities of Daily Living.

PURPOSE: The purpose of this study was to validate optimized algorithm parameter settings for step count and physical behavior for a pocket worn activity tracker in older adults during ADL. Secondly, for a more relevant interpretation of the results, the performance of the optimized algorithm was compared to three reference applications. METHODS: In a cross-sectional validation study, 20 older adults performed an activity protocol based on ADL with MOXMissActivity versus MOXAnnegarn, activPAL, and Fitbit. The protocol was video recorded and analyzed for step count and dynamic, standing, and sedentary time. Validity was assessed by percentage error (PE), absolute percentage error (APE), Bland-Altman plots and correlation coefficients. RESULTS: For step count, the optimized algorithm had a mean APE of 9.3% and a correlation coefficient of 0.88. The mean APE values of dynamic, standing, and sedentary time were 15.9%, 19.9%, and 9.6%, respectively. The correlation coefficients were 0.55, 0.91, and 0.92, respectively. Three reference applications showed higher errors and lower correlations for all outcome variables. CONCLUSION: This study showed that the optimized algorithm parameter settings can more validly estimate step count and physical behavior in older adults wearing an activity tracker in the trouser pocket during ADL compared to reference applications.

Economic evaluations of big data analytics for clinical decision-making: a scoping review.

OBJECTIVE: Much has been invested in big data analytics to improve health and reduce costs. However, it is unknown whether these investments have achieved the desired goals. We performed a scoping review to determine the health and economic impact of big data analytics for clinical decision-making. MATERIALS AND METHODS: We searched Medline, Embase, Web of Science and the National Health Services Economic Evaluations Database for relevant articles. We included peer-reviewed papers that report the health economic impact of analytics that assist clinical decision-making. We extracted the economic methods and estimated impact and also assessed the quality of the methods used. In addition, we estimated how many studies assessed "big data analytics" based on a broad definition of this term. RESULTS: The search yielded 12 133 papers but only 71 studies fulfilled all eligibility criteria. Only a few papers were full economic evaluations; many were performed during development. Papers frequently reported savings for healthcare payers but only 20% also included costs of analytics. Twenty studies examined "big data analytics" and only 7 reported both cost-savings and better outcomes. DISCUSSION: The promised potential of big data is not yet reflected in the literature, partly since only a few full and properly performed economic evaluations have been published. This and the lack of a clear definition of "big data" limit policy makers and healthcare professionals from determining which big data initiatives are worth implementing.

Optimization and Validation of an Adjustable Activity Classification Algorithm for Assessment of Physical Behavior in Elderly.

Due to a lack of transparency in both algorithm and validation methodology, it is difficult for researchers and clinicians to select the appropriate tracker for their application. The aim of this work is to transparently present an adjustable physical activity classification algorithm that discriminates between dynamic, standing, and sedentary behavior. By means of easily adjustable parameters, the algorithm performance can be optimized for applications using different target populations and locations for tracker wear. Concerning an elderly target population with a tracker worn on the upper leg, the algorithm is optimized and validated under simulated free-living conditions. The fixed activity protocol (FAP) is performed by 20 participants; the simulated free-living protocol (SFP) involves another 20. Data segmentation window size and amount of physical activity threshold are optimized. The sensor orientation threshold does not vary. The validation of the algorithm is performed on 10 participants who perform the FAP and on 10 participants who perform the SFP. Percentage error (PE) and absolute percentage error (APE) are used to assess the algorithm performance. Standing and sedentary behavior are classified within acceptable limits (±10% error) both under fixed and simulated free-living conditions. Dynamic behavior is within acceptable limits under fixed conditions but has some limitations under simulated free-living conditions. We propose that this approach should be adopted by developers of activity trackers to facilitate the activity tracker selection process for researchers and clinicians. Furthermore, we are convinced that the adjustable algorithm potentially could contribute to the fast realization of new applications.

Monitoring Parkinson's disease symptoms during daily life: a feasibility study.

Parkinson's disease symptoms are most often charted using the MDS-UPDRS. Limitations of this approach include the subjective character of the assessments and a discrepant performance in the clinic compared to the home situation. Continuous monitoring using wearable devices is believed to eventually replace this golden standard, but measurements often lack a parallel ground truth or are only tested in lab settings. To overcome these limitations, this study explores the feasibility of a newly developed Parkinson's disease monitoring system, which aims to measure Parkinson's disease symptoms during daily life by combining wearable sensors with an experience sampling method application. Twenty patients with idiopathic Parkinson's disease participated in this study. During a period of two consecutive weeks, participants had to wear three wearable sensors and had to complete questionnaires at seven semi-random moments per day on their mobile phone. Wearable sensors collected objective movement data, and the questionnaires containing questions about amongst others Parkinson's disease symptoms served as parallel ground truth. Results showed that participants wore the wearable sensors during 94% of the instructed timeframe and even beyond. Furthermore, questionnaire completion rates were high (79,1%) and participants evaluated the monitoring system positively. A preliminary analysis showed that sensor data could reliably predict subjectively reported OFF moments. These results show that our Parkinson's disease monitoring system is a feasible method to use in a diverse Parkinson's disease population for at least a period of two weeks. For longer use, the monitoring system may be too intense and wearing comfort needs to be optimized.

Clinical reasoning in the context of active decision support during medication prescribing.

OBJECTIVE: Describe and analyze reasoning patterns of clinicians responding to drug-drug interaction alerts in order to understand the role of patient-specific information in the decision-making process about the risks and benefits of medication therapy. Insights could be used to inform the design of decision-support interventions. METHODS: Thirty-two clinicians working with five EHRs in two countries completed sets of six medication orders each and responded to high- and low-severity drug-drug interaction alerts while verbalizing their thoughts in a standard think-aloud protocol. Tasks were recorded and analyzed to describe reasoning patterns about patient-risk assessment and strategies to avoid or mitigate it. RESULTS: We observed a total of 171 prescribing decisions. Clinicians actively sought to reduce risk when responding to high-severity alerts, mostly by monitoring patients and making dose adjustments (52 alerts, 40%). In contrast, they routinely left prescriptions unchanged after low-severity alerts when they felt confident that patients would tolerate the drug combination and that treatment benefits outweighed the risks (30 alerts, 71%). Clinicians used similar reasoning patterns regardless of the EHR used and differences in alert design. DISCUSSION: Clinicians conceptualized risk as a complex set of interdependent tradeoffs specific to individual patients and had a tendency not to follow advice they considered of low clinical value. Omission of patient-specific data, which was not shown in alerts or included in trigger logic, may have contributed to the constancy of reasoning and to similarities in risk-control strategies we observed despite significant differences in interface design and system function. CONCLUSION: Declining an alert suggestion was preceded by sometimes brief but often complex reasoning, prioritizing different aspects of care quality and safety, especially when the perceived risk was higher. Clinicians believed that the risk indicated in drug-drug interaction alerts needs to be interpreted as one factor in the broader context of care, specific to a patient.

On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop.

BACKGROUND: Clinical decision support (CDS) for electronic prescribing systems (computerized physician order entry) should help prescribers in the safe and rational use of medicines. However, the best ways to alert users to unsafe or irrational prescribing are uncertain. Specifically, CDS systems may generate too many alerts, producing unwelcome distractions for prescribers, or too few alerts running the risk of overlooking possible harms. Obtaining the right balance of alerting to adequately improve patient safety should be a priority. METHODS: A workshop funded through the European Regional Development Fund was convened by the University Hospitals Birmingham NHS Foundation Trust to assess current knowledge on alerts in CDS and to reach a consensus on a future research agenda on this topic. Leading European researchers in CDS and alerts in electronic prescribing systems were invited to the workshop. RESULTS: We identified important knowledge gaps and suggest research priorities including (1) the need to determine the optimal sensitivity and specificity of alerts; (2) whether adaptation to the environment or characteristics of the user may improve alerts; and (3) whether modifying the timing and number of alerts will lead to improvements. We have also discussed the challenges and benefits of using naturalistic or experimental studies in the evaluation of alerts and suggested appropriate outcome measures. CONCLUSIONS: We have identified critical problems in CDS, which should help to guide priorities in research to evaluate alerts. It is hoped that this will spark the next generation of novel research from which practical steps can be taken to implement changes to CDS systems that will ultimately reduce alert fatigue and improve the design of future systems.

Context sensitive health informatics: concepts, methods and tools.

Context is a key consideration when designing and evaluating health information technology (HIT) and cannot be overstated. Unintended consequences are common post HIT implementation and even well designed technology may not achieve desired outcomes because of contextual issues. While context should be considered in the design and evaluation of health information systems (HISs) there is a shortcoming of empirical research on contextual aspects of HIT. This conference integrates the sociotechnical and Human-Centered-Design (HCD) approaches and showcases current research on context sensitive health informatics. The papers and presentations outlines theories and models for studying contextual issues and insights on how we can better design HIT to accommodate different healthcare contexts.

Patient-centered care requires a patient-oriented workflow model.

Effective design of health information technology (HIT) for patient-centered care requires consideration of workflow from the patient's perspective, termed 'patient-oriented workflow.' This approach organizes the building blocks of work around the patients who are moving through the care system. Patient-oriented workflow complements the more familiar clinician-oriented workflow approaches, and offers several advantages, including the ability to capture simultaneous, cooperative work, which is essential in care delivery. Patient-oriented workflow models can also provide an understanding of healthcare work taking place in various formal and informal health settings in an integrated manner. We present two cases demonstrating the potential value of patient-oriented workflow models. Significant theoretical, methodological, and practical challenges must be met to ensure adoption of patient-oriented workflow models. Patient-oriented workflow models define meaningful system boundaries and can lead to HIT implementations that are more consistent with cooperative work and its emergent features.

The social act of electronic medication prescribing.

Prescribing medication is embedded in social norms and cultures. In modern Western health care professionals and policy makers have attempted to rationalize medicine by addressing cost-effectiveness of diagnostic and therapeutic treatments and the development of guidelines and protocols based on the outcomes of clinical studies. These notions of cost-effectiveness and evidence-based medicine have also been embedded in technology such as electronic prescribing systems. Such constraining systems may clash with the reality of clinical practice, where formal boundaries of responsibility and authorization are often blurred. Such systems may therefore even impede patient care. Medication is seen as the essence of medical practice. Prescribing is a social act. In a hospital medications may be aimed at treating a patient for a specific condition, in primary care the professional often meets the patient with her or his social and cultural notions of a health problem. The author argues that the design and implementation of electronic prescribing systems should address the social and cultural context of prescribing. Especially in primary care, where health problems are often ill defined and evidence-based medicine guidelines do not always work as intended, studies need to take into account the sociotechnical character of electronic prescribing systems.

A comparative review of patient safety initiatives for national health information technology.

OBJECTIVE: To collect and critically review patient safety initiatives for health information technology (HIT). METHOD: Publicly promulgated set of advisories, recommendations, guidelines, or standards potentially addressing safe system design, build, implementation or use were identified by searching the websites of regional and national agencies and programmes in a non-exhaustive set of exemplar countries including England, Denmark, the Netherlands, the USA, Canada and Australia. Initiatives were categorised by type and software systems covered. RESULTS: We found 27 patient safety initiatives for HIT predominantly dealing with software systems for health professionals. Three initiatives addressed consumer systems. Seven of the initiatives specifically dealt with software for diagnosis and treatment, which are regulated as medical devices in England, Denmark and Canada. Four initiatives dealt with blood bank and image management software which is regulated in the USA. Of the 16 initiatives directed at unregulated software, 11 were aimed at increasing standardisation using guidelines and standards for safe system design, build, implementation and use. Three initiatives for unregulated software were aimed at certification in the USA, Canada and Australia. Safety is addressed alongside interoperability in the Australian certification programme but it is not explicitly addressed in the US and Canadian programmes, though conformance with specific functionality, interoperability, security and privacy requirements may lead to safer systems. England appears to have the most comprehensive safety management programme for unregulated software, incorporating safety assurance at a local healthcare organisation level based on standards for risk management and user interface design, with national incident monitoring and a response function. CONCLUSIONS: There are significant gaps in the safety initiatives for HIT systems. Current initiatives are largely focussed on software. With the exception of diagnostic, prognostic, monitoring and treatment software, which are subject to medical device regulations in some countries, the safety of the most common types of HIT systems such as EHRs and CPOE without decision support is not being explicitly addressed in most nations. Appropriate mechanisms for safety assurance are required for the full range of HIT systems for health professionals and consumers including all software and hardware throughout the system lifecycle. In addition to greater standardisation and oversight to ensure safe system design and build, appropriate implementation and use of HIT is critical to ensure patient safety.

The dangerous decade.

Over the next 10 years, more information and communication technology (ICT) will be deployed in the health system than in its entire previous history. Systems will be larger in scope, more complex, and move from regional to national and supranational scale. Yet we are at roughly the same place the aviation industry was in the 1950s with respect to system safety. Even if ICT harm rates do not increase, increased ICT use will increase the absolute number of ICT related harms. Factors that could diminish ICT harm include adoption of common standards, technology maturity, better system development, testing, implementation and end user training. Factors that will increase harm rates include complexity and heterogeneity of systems and their interfaces, rapid implementation and poor training of users. Mitigating these harms will not be easy, as organizational inertia is likely to generate a hysteresis-like lag, where the paths to increase and decrease harm are not identical.

Clinical observational gait analysis to evaluate improvement of balance during gait with vibrotactile biofeedback.

BACKGROUND AND PURPOSE: This study explores the effect of vibrotactile biofeedback on gait in 20 patients with bilateral vestibular areflexia using observational gait analysis to score individual balance. METHODS: A tilt sensor mounted on the head or trunk is used to detect head or body tilt and activates, via a microprocessor, 12 equally distributed vibrators placed around the waist. Two positions of the tilt sensor were evaluated besides no biofeedback in three different gait velocity tasks (slow/fast tandem gait, normal gait on foam) resulting in nine different randomized conditions. Biofeedback activated versus inactivated was compared. Twenty patients (10 males, 10 females, age 39-77 years) with a bilateral vestibular areflexia or severe bilateral vestibular hyporeflexia, severe balance problems and frequent falls participated in this study. RESULTS: Significant improvements in balance during gait were shown in our patients using biofeedback and sensor on the trunk. Only two patients showed a significant individual gait improvement with the biofeedback system, but in the majority of our patients, it increased confidence and a feeling of balance. CONCLUSION: This study indicates the feasibility of vibrotactile biofeedback for vestibular rehabilitation and to improve balance during gait.

The shift in workarounds upon implementation of computerized physician order entry.

Workarounds are working processes deviating from formal rules or intended work methods to smooth workflow and circumvent problems without eliminating them. Former research focused on workarounds in the medication use process after implementation of computerized physician order entry (CPOE). This study on 2 wards of a general hospital shows that workarounds exist in both paper-based and electronic prescribing systems. After CPOE implementation workarounds present in the paper-based system had disappeared or remained existent, and new ones had emerged.

What makes an information system more preferable for clinicians? a qualitative comparison of two systems.

Two different information systems with respect to their ability to afford clinicians' needs in the chemotherapy medication process were implemented in a large Dutch academic hospital. A commercially available Computerized Physician Order Entry (CPOE) system was not appreciated because clinicians believed that it could not support complex chemotherapy process. Later, a home-grown IT system with the capability of prescribing chemotherapy medications based on standard care protocols was appreciated and fully used by clinicians. We evaluated both systems from their users' perspective to find the sources of clinicians' preference and to trace them back to their Systems Development Life Cycle (SDLC).