How can we discover the most valuable types of big data and artificial intelligence-based solutions? A methodology for the efficient development of the underlying analytics that improve care.

BACKGROUND: Much has been invested in big data and artificial intelligence-based solutions for healthcare. However, few applications have been implemented in clinical practice. Early economic evaluations can help to improve decision-making by developers of analytics underlying these solutions aiming to increase the likelihood of successful implementation, but recommendations about their use are lacking. The aim of this study was to develop and apply a framework that positions best practice methods for economic evaluations alongside development of analytics, thereby enabling developers to identify barriers to success and to select analytics worth further investments. METHODS: The framework was developed using literature, recommendations for economic evaluations and by applying the framework to use cases (chronic lymphocytic leukaemia (CLL), intensive care, diabetes). First, the feasibility of developing clinically relevant analytics was assessed and critical barriers to successful development and implementation identified. Economic evaluations were then used to determine critical thresholds and guide investment decisions. RESULTS: When using the framework to assist decision-making of developers of analytics, continuing development was not always feasible or worthwhile. Developing analytics for progressive CLL and diabetes was clinically relevant but not feasible with the data available. Alternatively, developing analytics for newly diagnosed CLL patients was feasible but continuing development was not considered worthwhile because the high drug costs made it economically unattractive for potential users. Alternatively, in the intensive care unit, analytics reduced mortality and per-patient costs when used to identify infections (- 0.5%, - €886) and to improve patient-ventilator interaction (- 3%, - €264). Both analytics have the potential to save money but the potential benefits of analytics that identify infections strongly depend on infection rate; a higher rate implies greater cost-savings. CONCLUSIONS: We present a framework that stimulates efficiency of development of analytics for big data and artificial intelligence-based solutions by selecting those applications of analytics for which development is feasible and worthwhile. For these applications, results from early economic evaluations can be used to guide investment decisions and identify critical requirements.

Optimization and Validation of a Classification Algorithm for Assessment of Physical Activity in Hospitalized Patients.

Low amounts of physical activity (PA) and prolonged periods of sedentary activity are common in hospitalized patients. Objective PA monitoring is needed to prevent the negative effects of inactivity, but a suitable algorithm is lacking. The aim of this study is to optimize and validate a classification algorithm that discriminates between sedentary, standing, and dynamic activities, and records postural transitions in hospitalized patients under free-living conditions. Optimization and validation in comparison to video analysis were performed in orthopedic and acutely hospitalized elderly patients with an accelerometer worn on the upper leg. Data segmentation window size (WS), amount of PA threshold (PA Th) and sensor orientation threshold (SO Th) were optimized in 25 patients, validation was performed in another 25. Sensitivity, specificity, accuracy, and (absolute) percentage error were used to assess the algorithm's performance. Optimization resulted in the best performance with parameter settings: WS 4 s, PA Th 4.3 counts per second, SO Th 0.8 g. Validation showed that all activities were classified within acceptable limits (>80% sensitivity, specificity and accuracy, ±10% error), except for the classification of standing activity. As patients need to increase their PA and interrupt sedentary behavior, the algorithm is suitable for classifying PA in hospitalized patients.

The potential of real-time analytics to improve care for mechanically ventilated patients in the intensive care unit: an early economic evaluation.

BACKGROUND: Mechanical ventilation services are an important driver of the high costs of intensive care. An optimal interaction between a patient and a ventilator is therefore paramount. Suboptimal interaction is present when patients repeatedly demand, but do not receive, breathing support from a mechanical ventilator (> 30 times in 3 min), also known as an ineffective effort event (IEEV). IEEVs are associated with increased hospital mortality prolonged intensive care stay, and prolonged time on ventilation and thus development of real-time analytics that identify IEEVs is essential. To assist decision-making about further development we estimate the potential cost-effectiveness of real-time analytics that identify ineffective effort events. METHODS: We developed a cost-effectiveness model combining a decision tree and Markov model for long-term outcomes with data on current care from a Greek hospital and literature. A lifetime horizon and a healthcare payer perspective were used. Uncertainty about the results was assessed using sensitivity and scenario analyses to examine the impact of varying parameters like the intensive care costs per day and the effectiveness of treatment of IEEVs. RESULTS: Use of the analytics could lead to reduced mortality (3% absolute reduction), increased quality adjusted life years (0.21 per patient) and cost-savings (€264 per patient) compared to current care. Moreover, cost-savings for hospitals and health improvements can be incurred even if the treatment's effectiveness is reduced from 30 to 10%. The estimated savings increase to €1,155 per patient in countries where costs of an intensive care day are high (e.g. the Netherlands). There is considerable headroom for development and the analytics generate savings when the price of the analytics per bed per year is below €7,307. Furthermore, even when the treatment's effectiveness is 10%, the probability that the analytics are cost-effective exceeds 90%. CONCLUSIONS: Implementing real-time analytics to identify ineffective effort events can lead to health and financial benefits. Therefore, it will be worthwhile to continue assessment of the effectiveness of the analytics in clinical practice and validate our findings. Eventually, their adoption in settings where costs of an intensive care day are high and ineffective efforts are frequent could yield a high return on investment.

Optimization and Validation of an Adjustable Activity Classification Algorithm for Assessment of Physical Behavior in Elderly.

Due to a lack of transparency in both algorithm and validation methodology, it is difficult for researchers and clinicians to select the appropriate tracker for their application. The aim of this work is to transparently present an adjustable physical activity classification algorithm that discriminates between dynamic, standing, and sedentary behavior. By means of easily adjustable parameters, the algorithm performance can be optimized for applications using different target populations and locations for tracker wear. Concerning an elderly target population with a tracker worn on the upper leg, the algorithm is optimized and validated under simulated free-living conditions. The fixed activity protocol (FAP) is performed by 20 participants; the simulated free-living protocol (SFP) involves another 20. Data segmentation window size and amount of physical activity threshold are optimized. The sensor orientation threshold does not vary. The validation of the algorithm is performed on 10 participants who perform the FAP and on 10 participants who perform the SFP. Percentage error (PE) and absolute percentage error (APE) are used to assess the algorithm performance. Standing and sedentary behavior are classified within acceptable limits (±10% error) both under fixed and simulated free-living conditions. Dynamic behavior is within acceptable limits under fixed conditions but has some limitations under simulated free-living conditions. We propose that this approach should be adopted by developers of activity trackers to facilitate the activity tracker selection process for researchers and clinicians. Furthermore, we are convinced that the adjustable algorithm potentially could contribute to the fast realization of new applications.

On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop.

BACKGROUND: Clinical decision support (CDS) for electronic prescribing systems (computerized physician order entry) should help prescribers in the safe and rational use of medicines. However, the best ways to alert users to unsafe or irrational prescribing are uncertain. Specifically, CDS systems may generate too many alerts, producing unwelcome distractions for prescribers, or too few alerts running the risk of overlooking possible harms. Obtaining the right balance of alerting to adequately improve patient safety should be a priority. METHODS: A workshop funded through the European Regional Development Fund was convened by the University Hospitals Birmingham NHS Foundation Trust to assess current knowledge on alerts in CDS and to reach a consensus on a future research agenda on this topic. Leading European researchers in CDS and alerts in electronic prescribing systems were invited to the workshop. RESULTS: We identified important knowledge gaps and suggest research priorities including (1) the need to determine the optimal sensitivity and specificity of alerts; (2) whether adaptation to the environment or characteristics of the user may improve alerts; and (3) whether modifying the timing and number of alerts will lead to improvements. We have also discussed the challenges and benefits of using naturalistic or experimental studies in the evaluation of alerts and suggested appropriate outcome measures. CONCLUSIONS: We have identified critical problems in CDS, which should help to guide priorities in research to evaluate alerts. It is hoped that this will spark the next generation of novel research from which practical steps can be taken to implement changes to CDS systems that will ultimately reduce alert fatigue and improve the design of future systems.

The social act of electronic medication prescribing.

Prescribing medication is embedded in social norms and cultures. In modern Western health care professionals and policy makers have attempted to rationalize medicine by addressing cost-effectiveness of diagnostic and therapeutic treatments and the development of guidelines and protocols based on the outcomes of clinical studies. These notions of cost-effectiveness and evidence-based medicine have also been embedded in technology such as electronic prescribing systems. Such constraining systems may clash with the reality of clinical practice, where formal boundaries of responsibility and authorization are often blurred. Such systems may therefore even impede patient care. Medication is seen as the essence of medical practice. Prescribing is a social act. In a hospital medications may be aimed at treating a patient for a specific condition, in primary care the professional often meets the patient with her or his social and cultural notions of a health problem. The author argues that the design and implementation of electronic prescribing systems should address the social and cultural context of prescribing. Especially in primary care, where health problems are often ill defined and evidence-based medicine guidelines do not always work as intended, studies need to take into account the sociotechnical character of electronic prescribing systems.

Does electronic clinical microbiology results reporting influence medical decision making: a pre- and post-interview study of medical specialists.

BACKGROUND: Clinicians view the accuracy of test results and the turnaround time as the two most important service aspects of the clinical microbiology laboratory. Because of the time needed for the culturing of infectious agents, final hardcopy culture results will often be available too late to have a significant impact on early antimicrobial therapy decisions, vital in infectious disease management. The clinical microbiologist therefore reports to the clinician clinically relevant preliminary results at any moment during the diagnostic process, mostly by telephone. Telephone reporting is error prone, however. Electronic reporting of culture results instead of reporting on paper may shorten the turnaround time and may ensure correct communication of results. The purpose of this study was to assess the impact of the implementation of electronic reporting of final microbiology results on medical decision making. METHODS: In a pre- and post-interview study using a semi-structured design we asked medical specialists in our hospital about their use and appreciation of clinical microbiology results reporting before and after the implementation of an electronic reporting system. RESULTS: Electronic reporting was highly appreciated by all interviewed clinicians. Major advantages were reduction of hardcopy handling and the possibility to review results in relation to other patient data. Use and meaning of microbiology reports differ significantly between medical specialties. Most clinicians need preliminary results for therapy decisions quickly. Therefore, after the implementation of electronic reporting, telephone consultation between clinician and microbiologist remained the key means of communication. CONCLUSIONS: Overall, electronic reporting increased the workflow efficiency of the medical specialists, but did not have an impact on their decision-making.

The future of electronic prescribing.

Implementing electronic prescribing in health care has been a slow process. Health authorities are now requiring mandatory electronic prescribing because of patient safety concerns. Electronic prescribing is not yet a mature technology, and may therefore pose a risk if especially organizational conditions are not taken into account. The paper offers some thoughts on the future of electronic prescribing in practice. It is especially important to extend electronic prescribing to the continuum of care in order avoid that medication safety falls in the cracks of fragmented health care organizations.

The shift in workarounds upon implementation of computerized physician order entry.

Workarounds are working processes deviating from formal rules or intended work methods to smooth workflow and circumvent problems without eliminating them. Former research focused on workarounds in the medication use process after implementation of computerized physician order entry (CPOE). This study on 2 wards of a general hospital shows that workarounds exist in both paper-based and electronic prescribing systems. After CPOE implementation workarounds present in the paper-based system had disappeared or remained existent, and new ones had emerged.

What makes an information system more preferable for clinicians? a qualitative comparison of two systems.

Two different information systems with respect to their ability to afford clinicians' needs in the chemotherapy medication process were implemented in a large Dutch academic hospital. A commercially available Computerized Physician Order Entry (CPOE) system was not appreciated because clinicians believed that it could not support complex chemotherapy process. Later, a home-grown IT system with the capability of prescribing chemotherapy medications based on standard care protocols was appreciated and fully used by clinicians. We evaluated both systems from their users' perspective to find the sources of clinicians' preference and to trace them back to their Systems Development Life Cycle (SDLC).

Reporting qualitative research in health informatics: REQ-HI recommendations.

To develop a set of recommendations for authors of qualitative studies in the field of health informatics, we conducted an extensive literature search and also manually checked major journals in the field of biomedical informatics and qualitative research looking for papers, checklists, and guidelines pertaining to assessing and reporting of qualitative studies. We synthesized the found criteria to develop an initial set of reporting recommendations that are particularly relevant to qualitative studies of health information technology systems. This paper presents a preliminary version of these recommendations. We are planning to refine and revise this version using comments and suggestions of experts in evaluation of health informatics applications and publish a detailed set of recommendations.

From safe systems to patient safety.

In this introduction we summarize the contributions to the Fourth International Conference Information Technology in Health Care: Socio-technical Approaches. We argue that putting to use information systems in health care is a difficult and winding road. Systems may even compromise instead of improving patient safety. Designing and implementing systems requires a thorough understanding of the context in which technology is being used. But this understanding should also lead to better design and implementation methods. Therefore this conference wishes to address the challenges of safe systems and patient safety, ten years after the publication of the landmark reports on human error and quality of health care by the Institute of Medicine.

Economic evaluations of big data analytics for clinical decision-making: a scoping review.

OBJECTIVE: Much has been invested in big data analytics to improve health and reduce costs. However, it is unknown whether these investments have achieved the desired goals. We performed a scoping review to determine the health and economic impact of big data analytics for clinical decision-making. MATERIALS AND METHODS: We searched Medline, Embase, Web of Science and the National Health Services Economic Evaluations Database for relevant articles. We included peer-reviewed papers that report the health economic impact of analytics that assist clinical decision-making. We extracted the economic methods and estimated impact and also assessed the quality of the methods used. In addition, we estimated how many studies assessed "big data analytics" based on a broad definition of this term. RESULTS: The search yielded 12 133 papers but only 71 studies fulfilled all eligibility criteria. Only a few papers were full economic evaluations; many were performed during development. Papers frequently reported savings for healthcare payers but only 20% also included costs of analytics. Twenty studies examined "big data analytics" and only 7 reported both cost-savings and better outcomes. DISCUSSION: The promised potential of big data is not yet reflected in the literature, partly since only a few full and properly performed economic evaluations have been published. This and the lack of a clear definition of "big data" limit policy makers and healthcare professionals from determining which big data initiatives are worth implementing.

The Validation of a Pocket Worn Activity Tracker for Step Count and Physical Behavior in Older Adults during Simulated Activities of Daily Living.

PURPOSE: The purpose of this study was to validate optimized algorithm parameter settings for step count and physical behavior for a pocket worn activity tracker in older adults during ADL. Secondly, for a more relevant interpretation of the results, the performance of the optimized algorithm was compared to three reference applications. METHODS: In a cross-sectional validation study, 20 older adults performed an activity protocol based on ADL with MOXMissActivity versus MOXAnnegarn, activPAL, and Fitbit. The protocol was video recorded and analyzed for step count and dynamic, standing, and sedentary time. Validity was assessed by percentage error (PE), absolute percentage error (APE), Bland-Altman plots and correlation coefficients. RESULTS: For step count, the optimized algorithm had a mean APE of 9.3% and a correlation coefficient of 0.88. The mean APE values of dynamic, standing, and sedentary time were 15.9%, 19.9%, and 9.6%, respectively. The correlation coefficients were 0.55, 0.91, and 0.92, respectively. Three reference applications showed higher errors and lower correlations for all outcome variables. CONCLUSION: This study showed that the optimized algorithm parameter settings can more validly estimate step count and physical behavior in older adults wearing an activity tracker in the trouser pocket during ADL compared to reference applications.

The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review.

Previous studies have shown the importance of workflow issues in the implementation of CPOE systems and patient safety practices. To understand the impact of CPOE on clinical workflow, we developed a conceptual framework and conducted a literature search for CPOE evaluations between 1990 and June 2007. Fifty-one publications were identified that disclosed mixed effects of CPOE systems. Among the frequently reported workflow advantages were the legible orders, remote accessibility of the systems, and the shorter order turnaround times. Among the frequently reported disadvantages were the time-consuming and problematic user-system interactions, and the enforcement of a predefined relationship between clinical tasks and between providers. Regarding the diversity of findings in the literature, we conclude that more multi-method research is needed to explore CPOE's multidimensional and collective impact on especially collaborative workflow.

Unintended consequences of reducing QT-alert overload in a computerized physician order entry system.

PURPOSE: After complaints of too many low-specificity drug-drug interaction (DDI) alerts on QT prolongation, the rules for QT alerting in the Dutch national drug database were restricted in 2007 to obviously QT-prolonging drugs. The aim of this virtual study was to investigate whether this adjustment would improve the identification of patients at risk of developing Torsades de Pointes (TdP) due to QT-prolonging drug combinations in a computerized physician order entry system (CPOE) and whether these new rules should be implemented. METHODS: During a half-year study period, inpatients with overridden DDI alerts regarding QT prolongation and with an electrocardiogram recorded before and within 1 month of the alert override were included if they did not have a ventricular pacemaker and did not use the low-risk combination cotrimoxazole and tacrolimus. QT-interval prolongation and the risk of developing TdP were calculated for all patients and related to the number of patients for whom a QT-alert would be generated in the new situation with the restricted database. RESULTS: Forty-nine patients (13%) met the inclusion criteria. In this study population, knowledge base-adjustment would reduce the number of alerts by 53%. However, the positive predictive value of QT alerts would not change (31% before and 30% after) and only 47% of the patients at risk of developing TdP would be identified in CPOEs using the adjusted knowledge base. CONCLUSION: The new rules for QT alerting would result in a poorer identification of patients at risk of developing TdP than the old rules. This is caused by the many non-drug-related risk factors for QT prolongation not being incorporated in CPOE alert generation. The partial contribution of all risk factors should be studied and used to create clinical rules for QT alerting with an acceptable positive predictive value.

Drug safety alert generation and overriding in a large Dutch university medical centre.

PURPOSE: To evaluate numbers and types of drug safety alerts generated and overridden in a large Dutch university medical centre. METHODS: A disguised observation study lasting 25 days on two internal medicine wards evaluating alert generation and handling of alerts. A retrospective analysis was also performed of all drug safety alerts overridden in the hospital using pharmacy log files over 24 months. RESULTS: In the disguised observation study 34% of the orders generated a drug safety alert of which 91% were overridden. The majority of alerts generated (56%) concerned drug-drug interactions (DDIs) and these were overridden more often (98%) than overdoses (89%) or duplicate orders (80%). All drug safety alerts concerning admission medicines were overridden.Retrospective analysis of pharmacy log files for all wards revealed one override per five prescriptions. Of all overrides, DDIs accounted for 59%, overdoses 24% and duplicate orders 17%. DDI alerts of medium-level seriousness were overridden more often (55%) than low-level (22%) or high-level DDIs (19%). In 36% of DDI overrides, it would have been possible to monitor effects by measuring serum levels. The top 20 of overridden DDIs accounted for 76% of all DDI overrides. CONCLUSIONS: Drug safety alerts were generated in one third of orders and were frequently overridden. Duplicate order alerts more often resulted in order cancellation (20%) than did alerts for overdose (11%) or DDIs (2%). DDIs were most frequently overridden. Only a small number of DDIs caused these overrides. Studies on improvement of alert handling should focus on these frequently-overridden DDIs.

CPOE, alerts and workflow: taking stock of ten years research at Erasmus MC.

Ten years ago research of the impact of health information technology (HIT) on medical work practices started at Erasmus MC. The research is characterized by practice driven field research. From the beginning computerized physician order entry systems (CPOE) were a major topic. Research questions were how implementation of CPOE could be understood, how physicians were responding to reminders and alerts and how CPOE impacted professional workflow and collaboration. Studies of CPOE implementation aimed to understand why the adoption rate is so low and riddled with difficulties. Studies of reminders and alerts addressed the problem of alert fatigue. Finally, studies of workflow explored how CPOE influenced clinical workflow and how simplistic and linear models underlying CPOE may lead to poor designed systems and even compromise patient safety. Findings include the need for a shared understanding of medical challenges when implementing CPOE, conceptual models to understand alert fatigue and medical workflow and the impossibility of agreeing which alerts to suppress hospital-wide. The underlying research principle is the sociotechnical approach, which stipulates that technology, people and organizations should be studied from a single theoretical framework. This paper summarizes the results of ten years of research.

Monitoring Parkinson's disease symptoms during daily life: a feasibility study.

Parkinson's disease symptoms are most often charted using the MDS-UPDRS. Limitations of this approach include the subjective character of the assessments and a discrepant performance in the clinic compared to the home situation. Continuous monitoring using wearable devices is believed to eventually replace this golden standard, but measurements often lack a parallel ground truth or are only tested in lab settings. To overcome these limitations, this study explores the feasibility of a newly developed Parkinson's disease monitoring system, which aims to measure Parkinson's disease symptoms during daily life by combining wearable sensors with an experience sampling method application. Twenty patients with idiopathic Parkinson's disease participated in this study. During a period of two consecutive weeks, participants had to wear three wearable sensors and had to complete questionnaires at seven semi-random moments per day on their mobile phone. Wearable sensors collected objective movement data, and the questionnaires containing questions about amongst others Parkinson's disease symptoms served as parallel ground truth. Results showed that participants wore the wearable sensors during 94% of the instructed timeframe and even beyond. Furthermore, questionnaire completion rates were high (79,1%) and participants evaluated the monitoring system positively. A preliminary analysis showed that sensor data could reliably predict subjectively reported OFF moments. These results show that our Parkinson's disease monitoring system is a feasible method to use in a diverse Parkinson's disease population for at least a period of two weeks. For longer use, the monitoring system may be too intense and wearing comfort needs to be optimized.

Turning off frequently overridden drug alerts: limited opportunities for doing it safely.

OBJECTIVES: This study sought to identify opportunities to safely turn off frequently overridden drug-drug interaction alerts (DDIs) in computerized physician order entry (CPOE). DESIGN: Quantitative retrospective analysis of drug safety alerts overridden during 1 month and qualitative interviews with 24 respondents (18 physicians and 6 pharmacists) about turning off frequently overridden DDI alerts, based on the Dutch drug database, in a hospital setting. Screen shots and complete texts of frequently overridden DDIs were presented to physicians of internal medicine, cardiology, and surgery and to hospital pharmacists who were asked whether these could be turned off hospital-wide without impairing patient safety, and the reasons for their recommendations. RESULTS: Data on the frequency of alerts overridden in 1 month identified 3,089 overrides, of which 1,963 were DDIs. The category DDIs showed 86 different alerts, of which 24 frequently overridden alerts, accounting for 72% of all DDI overrides, were selected for further evaluation. The 24 respondents together made 576 assessments. Upon investigation, differences in the reasons for turning off alerts were found across medical specialties and among respondents within a specialty. Frequently mentioned reasons for turning off were "alert well known," "alert not serious," or "alert not needing (additional) action," or that the effects of the combination were monitored or intended. For none of the alerts did all respondents agree that it could be safely turned off hospital-wide. The highest agreement was 13 of 24 respondents (54%). A positive correlation was found between the number of alerts overridden and the number of clinicians recommending to turn them off. CONCLUSION: Although the Dutch drug database is already a selected reduction from all DDIs mentioned in literature, the majority of respondents wanted to turn off DDI alerts to reduce alert overload. Turning off DDI alerts hospital-wide appeared to be problematic because of differences among physicians regarding drug-related knowledge and of differences across the hospital in routine drug monitoring practices. Furthermore, several reasons for suppression of alerts could be questioned from a safety perspective. Further research should investigate when each of the following might help: changes in alert texts; new differential alert triggers based on clinician knowledge or specialty; and nonintrusive alert presentation so long as serum levels and patient parameters are measured and stay within limits.