On the evaluation of synthetic longitudinal electronic health records.

BACKGROUND: Synthetic Electronic Health Records (EHRs) are becoming increasingly popular as a privacy enhancing technology. However, for longitudinal EHRs specifically, little research has been done into how to properly evaluate synthetically generated samples. In this article, we provide a discussion on existing methods and recommendations when evaluating the quality of synthetic longitudinal EHRs. METHODS: We recommend to assess synthetic EHR quality through similarity to real EHRs in low-dimensional projections, accuracy of a classifier discriminating synthetic from real samples, performance of synthetic versus real trained algorithms in clinical tasks, and privacy risk through risk of attribute inference. For each metric we discuss strengths and weaknesses, next to showing how it can be applied on a longitudinal dataset. RESULTS: To support the discussion on evaluation metrics, we apply discussed metrics on a dataset of synthetic EHRs generated from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) repository. CONCLUSIONS: The discussion on evaluation metrics provide guidance for researchers on how to use and interpret different metrics when evaluating the quality of synthetic longitudinal EHRs.

Evaluating sentence representations for biomedical text: Methods and experimental results.

Text representations ar one of the main inputs to various Natural Language Processing (NLP) methods. Given the fast developmental pace of new sentence embedding methods, we argue that there is a need for a unified methodology to assess these different techniques in the biomedical domain. This work introduces a comprehensive evaluation of novel methods across ten medical classification tasks. The tasks cover a variety of BioNLP problems such as semantic similarity, question answering, citation sentiment analysis and others with binary and multi-class datasets. Our goal is to assess the transferability of different sentence representation schemes to the medical and clinical domain. Our analysis shows that embeddings based on Language Models which account for the context-dependent nature of words, usually outperform others in terms of performance. Nonetheless, there is no single embedding model that perfectly represents biomedical and clinical texts with consistent performance across all tasks. This illustrates the need for a more suitable bio-encoder. Our MedSentEval source code, pre-trained embeddings and examples have been made available on GitHub.

Efficiency of Clinical Decision Support Systems Improves with Experience.

Efficiency, or the resources spent while performing a specific task, is widely regarded as one the determinants of usability. In this study, the authors hypothesize that having a group of users perform a similar task over a prolonged period of time will lead to improvements in efficiency of that task. This study was performed in the domain of decision-supported medication reviews. Data was gathered during a randomized controlled trial. Three expert teams consisting of an independent physician and an independent pharmacist conducted 150 computerized medication reviews on patients in 13 general practices located in Amsterdam, the Netherlands. Results were analyzed with a linear mixed model. A fixed effects test on the linear mixed model showed a significant difference in the time required to conduct medication reviews over time; F(31.145) = 14.043, p < .001. The average time in minutes required to conduct medication reviews up to the first quartile was M = 20.42 (SD = 9.00), while the time from the third quartile up was M = 9.81 (SD = 6.13). This leads the authors to conclude that the amount of time users needed to perform similar tasks decreased significantly as they gained experience over time.

Frequency and Acceptance of Clinical Decision Support System-Generated STOPP/START Signals for Hospitalised Older Patients with Polypharmacy and Multimorbidity.

BACKGROUND: The Screening Tool of Older Persons' Prescriptions (STOPP)/Screening Tool to Alert to Right Treatment (START) instrument is used to evaluate the appropriateness of medication in older people. STOPP/START criteria have been converted into software algorithms and implemented in a clinical decision support system (CDSS) to facilitate their use in clinical practice. OBJECTIVE: Our objective was to determine the frequency of CDSS-generated STOPP/START signals and their subsequent acceptance by a pharmacotherapy team in a hospital setting. DESIGN AND METHODS: Hospitalised older patients with polypharmacy and multimorbidity allocated to the intervention arm of the OPERAM (OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly) trial underwent a CDSS-assisted structured medication review in four European hospitals. We evaluated the frequency of CDSS-generated STOPP/START signals and the subsequent acceptance of these signals by a trained pharmacotherapy team consisting of a physician and pharmacist after evaluation of clinical applicability to the individual patient, prior to discussing pharmacotherapy optimisation recommendations with the patient and attending physicians. Multivariate linear regression analysis was used to investigate potential patient-related (e.g. age, number of co-morbidities and medications) and setting-related (e.g. ward type, country of inclusion) determinants for acceptance of STOPP and START signals. RESULTS: In 819/826 (99%) of the patients, at least one STOPP/START signal was generated using a set of 110 algorithms based on STOPP/START v2 criteria. Overall, 39% of the 5080 signals were accepted by the pharmacotherapy team. There was a high variability in the frequency and the subsequent acceptance of the individual STOPP/START criteria. The acceptance ranged from 2.5 to 75.8% for the top ten most frequently generated STOPP and START signals. The signal to stop a drug without a clinical indication was most frequently generated (28%), with more than half of the signals accepted (54%). No difference in mean acceptance of STOPP versus START signals was found. In multivariate analysis, most patient-related determinants did not predict acceptance, although the acceptance of START signals increased in patients with one or more hospital admissions (+ 7.9; 95% confidence interval [CI] 1.6-14.1) or one or more falls in the previous year (+ 7.1; 95% CI 0.7-13.4). A higher number of co-morbidities was associated with lower acceptance of STOPP (- 11.8%; 95% CI - 19.2 to - 4.5) and START (- 11.0%; 95% CI - 19.4 to - 2.6) signals for patients with more than nine and between seven and nine co-morbidities, respectively. For setting-related determinants, the acceptance differed significantly between the participating trial sites. Compared with Switzerland, the acceptance was higher in Ireland (STOPP: + 26.8%; 95% CI 16.8-36.7; START: + 31.1%; 95% CI 18.2-44.0) and in the Netherlands (STOPP: + 14.7%; 95% CI 7.8-21.7). Admission to a surgical ward was positively associated with acceptance of STOPP signals (+ 10.3%; 95% CI 3.8-16.8). CONCLUSION: The involvement of an expert team in translating population-based CDSS signals to individual patients is essential, as more than half of the signals for potential overuse, underuse, and misuse were not deemed clinically appropriate in a hospital setting. Patient-related potential determinants were poor predictors of acceptance. Future research investigating factors that affect patients' and physicians' agreement with medication changes recommended by expert teams may provide further insight for implementation in clinical practice. REGISTRATION: ClinicalTrials.gov Identifier: NCT02986425.

A data mining-based cross-industry process for predicting major bleeding in mechanical circulatory support.

Aims: Over a third of patients, treated with mechanical circulatory support (MCS) for end-stage heart failure, experience major bleeding. Currently, the prediction of a major bleeding in the near future is difficult because of many contributing factors. Predictive analytics using data mining could help calculating the risk of bleeding; however, its application is generally reserved for experienced researchers on this subject. We propose an easily applicable data mining tool to predict major bleeding in MCS patients. Methods and results: All data of electronic health records of MCS patients in the University Medical Centre Utrecht were included. Based on the cross-industry standard process for data mining (CRISP-DM) methodology, an application named Auto-Crisp was developed. Auto-Crisp was used to evaluate the predictive models for a major bleeding in the next 3, 7, and 30 days after the first 30 days post-operatively following MCS implantation. The performance of the predictive models is investigated by the area under the curve (AUC) evaluation measure. In 25.6% of 273 patients, a total of 142 major bleedings occurred during a median follow-up period of 542 [interquartile range (IQR) 205-1044] days. The best predictive models assessed by Auto-Crisp had AUC values of 0.792, 0.788, and 0.776 for bleedings in the next 3, 7, and 30 days, respectively. Conclusion: The Auto-Crisp-based predictive model created in this study had an acceptable performance to predict major bleeding in MCS patients in the near future. However, further validation of the application is needed to evaluate Auto-Crisp in other research projects.

The SNPcurator: literature mining of enriched SNP-disease associations.

Database URL: http://snpcurator.science.uu.nl/.

Improving Comprehension Efficiency of High Content Screening Data Through Interactive Visualizations.

In this study, an experiment is conducted to measure the performance in speed and accuracy of interactive visualizations. A platform for interactive data visualizations was implemented using Django, D3, and Angular. Using this platform, a questionnaire was designed to measure a difference in performance between interactive and noninteractive data visualizations. In this questionnaire consisting of 12 questions, participants were given tasks in which they had to identify trends or patterns. Other tasks were directed at comparing and selecting algorithms with a certain outcome based on visualizations. All tasks were performed on high content screening data sets with the help of visualizations. The difference in time to carry out tasks and accuracy of performance was measured between a group viewing interactive visualizations and a group viewing noninteractive visualizations. The study shows a significant advantage in time and accuracy in the group that used interactive visualizations over the group that used noninteractive visualizations. In tasks comparing results of different algorithms, a significant decrease in time was observed in using interactive visualizations over noninteractive visualizations.

Computerized Decision Support Improves Medication Review Effectiveness: An Experiment Evaluating the STRIP Assistant's Usability.

BACKGROUND: Polypharmacy poses threats to patients' health. The Systematic Tool to Reduce Inappropriate Prescribing (STRIP) is a drug optimization process for conducting medication reviews in primary care. To effectively and efficiently incorporate this method into daily practice, the STRIP Assistant--a decision support system that aims to assist physicians with the pharmacotherapeutic analysis of patients' medical records--has been developed. It generates context-specific advice based on clinical guidelines. OBJECTIVE: The aim of this study was to validate the STRIP Assistant's usability as a tool for physicians to optimize medical records for polypharmacy patients. METHODS: In an online experiment, 42 physicians were asked to optimize medical records for two comparable polypharmacy patients, one in their usual manner and one using the STRIP Assistant. Changes in effectiveness were measured by comparing respondents' optimized medicine prescriptions with medication prepared by an expert panel of two geriatrician-pharmacologists. Efficiency was operationalized by recording the time the respondents took to optimize the two cases. User satisfaction was measured with the System Usability Scale (SUS). Independent and paired t tests were used for analysis. RESULTS: Medication optimization significantly improved with the STRIP Assistant. Appropriate decisions increased from 58% without the STRIP Assistant to 76% with it (p < 0.0001). Inappropriate decisions decreased from 42% without the STRIP Assistant to 24% with it (p < 0.0001). Participants spent significantly more time optimizing medication with the STRIP Assistant (24 min) than without it (13 min; p < 0.0001). They assigned it a below-average SUS score of 63.25. CONCLUSION: The STRIP Assistant improves the effectiveness of medication reviews for polypharmacy patients.