Prospective study validating a multidimensional treatment decision score predicting the 24-month outcome in untreated patients with clinically isolated syndrome and early relapsing-remitting multiple sclerosis, the ProVal-MS study.

INTRODUCTION: In Multiple Sclerosis (MS), patients´ characteristics and (bio)markers that reliably predict the individual disease prognosis at disease onset are lacking. Cohort studies allow a close follow-up of MS histories and a thorough phenotyping of patients. Therefore, a multicenter cohort study was initiated to implement a wide spectrum of data and (bio)markers in newly diagnosed patients. METHODS: ProVal-MS (Prospective study to validate a multidimensional decision score that predicts treatment outcome at 24 months in untreated patients with clinically isolated syndrome or early Relapsing-Remitting-MS) is a prospective cohort study in patients with clinically isolated syndrome (CIS) or Relapsing-Remitting (RR)-MS (McDonald 2017 criteria), diagnosed within the last two years, conducted at five academic centers in Southern Germany. The collection of clinical, laboratory, imaging, and paraclinical data as well as biosamples is harmonized across centers. The primary goal is to validate (discrimination and calibration) the previously published DIFUTURE MS-Treatment Decision score (MS-TDS). The score supports clinical decision-making regarding the options of early (within 6 months after study baseline) platform medication (Interferon beta, glatiramer acetate, dimethyl/diroximel fumarate, teriflunomide), or no immediate treatment (> 6 months after baseline) of patients with early RR-MS and CIS by predicting the probability of new or enlarging lesions in cerebral magnetic resonance images (MRIs) between 6 and 24 months. Further objectives are refining the MS-TDS score and providing data to identify new markers reflecting disease course and severity. The project also provides a technical evaluation of the ProVal-MS cohort within the IT-infrastructure of the DIFUTURE consortium (Data Integration for Future Medicine) and assesses the efficacy of the data sharing techniques developed. PERSPECTIVE: Clinical cohorts provide the infrastructure to discover and to validate relevant disease-specific findings. A successful validation of the MS-TDS will add a new clinical decision tool to the armamentarium of practicing MS neurologists from which newly diagnosed MS patients may take advantage. Trial registration ProVal-MS has been registered in the German Clinical Trials Register, `Deutsches Register Klinischer Studien` (DRKS)-ID: DRKS00014034, date of registration: 21 December 2018; https://drks.de/search/en/trial/DRKS00014034.

MISM: A Medical Image Segmentation Metric for Evaluation of Weak Labeled Data.

Performance measures are an important tool for assessing and comparing different medical image segmentation algorithms. Unfortunately, the current measures have their weaknesses when it comes to assessing certain edge cases. These limitations arise when images with a very small region of interest or without a region of interest at all are assessed. As a solution to these limitations, we propose a new medical image segmentation metric: MISm. This metric is a composition of the Dice similarity coefficient and the weighted specificity. MISm was investigated for definition gaps, an appropriate scoring gradient, and different weighting coefficients used to propose a constant value. Furthermore, an evaluation was performed by comparing the popular metrics in the medical image segmentation and MISm using images of magnet resonance tomography from several fictitious prediction scenarios. Our analysis shows that MISm can be applied in a general way and thus also covers the mentioned edge cases, which are not covered by other metrics, in a reasonable way. In order to allow easy access to MISm and therefore widespread application in the community, as well as reproducibility of experimental results, we included MISm in the publicly available evaluation framework MISeval.

Towards a guideline for evaluation metrics in medical image segmentation.

In the last decade, research on artificial intelligence has seen rapid growth with deep learning models, especially in the field of medical image segmentation. Various studies demonstrated that these models have powerful prediction capabilities and achieved similar results as clinicians. However, recent studies revealed that the evaluation in image segmentation studies lacks reliable model performance assessment and showed statistical bias by incorrect metric implementation or usage. Thus, this work provides an overview and interpretation guide on the following metrics for medical image segmentation evaluation in binary as well as multi-class problems: Dice similarity coefficient, Jaccard, Sensitivity, Specificity, Rand index, ROC curves, Cohen's Kappa, and Hausdorff distance. Furthermore, common issues like class imbalance and statistical as well as interpretation biases in evaluation are discussed. As a summary, we propose a guideline for standardized medical image segmentation evaluation to improve evaluation quality, reproducibility, and comparability in the research field.

MISeval: A Metric Library for Medical Image Segmentation Evaluation.

Correct performance assessment is crucial for evaluating modern artificial intelligence algorithms in medicine like deep-learning based medical image segmentation models. However, there is no universal metric library in Python for standardized and reproducible evaluation. Thus, we propose our open-source publicly available Python package MISeval: a metric library for Medical Image Segmentation Evaluation. The implemented metrics can be intuitively used and easily integrated into any performance assessment pipeline. The package utilizes modern DevOps strategies to ensure functionality and stability. MISeval is available from PyPI (miseval) and GitHub: https://github.com/frankkramer-lab/miseval.

DrNote: An open medical annotation service.

In the context of clinical trials and medical research medical text mining can provide broader insights for various research scenarios by tapping additional text data sources and extracting relevant information that is often exclusively present in unstructured fashion. Although various works for data like electronic health reports are available for English texts, only limited work on tools for non-English text resources has been published that offers immediate practicality in terms of flexibility and initial setup. We introduce DrNote, an open source text annotation service for medical text processing. Our work provides an entire annotation pipeline with its focus on a fast yet effective and easy to use software implementation. Further, the software allows its users to define a custom annotation scope by filtering only for relevant entities that should be included in its knowledge base. The approach is based on OpenTapioca and combines the publicly available datasets from WikiData and Wikipedia, and thus, performs entity linking tasks. In contrast to other related work our service can easily be built upon any language-specific Wikipedia dataset in order to be trained on a specific target language. We provide a public demo instance of our DrNote annotation service at https://drnote.misit-augsburg.de/.

Multi-Disease Detection in Retinal Imaging Based on Ensembling Heterogeneous Deep Learning Models.

Preventable or undiagnosed visual impairment and blindness affect billion of people worldwide. Automated multi-disease detection models offer great potential to address this problem via clinical decision support in diagnosis. In this work, we proposed an innovative multi-disease detection pipeline for retinal imaging which utilizes ensemble learning to combine the predictive capabilities of several heterogeneous deep convolutional neural network models. Our pipeline includes state-of-the-art strategies like transfer learning, class weighting, real-time image augmentation and Focal loss utilization. Furthermore, we integrated ensemble learning techniques like heterogeneous deep learning models, bagging via 5-fold cross-validation and stacked logistic regression models. Through internal and external evaluation, we were able to validate and demonstrate high accuracy and reliability of our pipeline, as well as the comparability with other state-of-the-art pipelines for retinal disease prediction.

Robust chest CT image segmentation of COVID-19 lung infection based on limited data.

BACKGROUND: The coronavirus disease 2019 (COVID-19) affects billions of lives around the world and has a significant impact on public healthcare. For quantitative assessment and disease monitoring medical imaging like computed tomography offers great potential as alternative to RT-PCR methods. For this reason, automated image segmentation is highly desired as clinical decision support. However, publicly available COVID-19 imaging data is limited which leads to overfitting of traditional approaches. METHODS: To address this problem, we propose an innovative automated segmentation pipeline for COVID-19 infected regions, which is able to handle small datasets by utilization as variant databases. Our method focuses on on-the-fly generation of unique and random image patches for training by performing several preprocessing methods and exploiting extensive data augmentation. For further reduction of the overfitting risk, we implemented a standard 3D U-Net architecture instead of new or computational complex neural network architectures. RESULTS: Through a k-fold cross-validation on 20 CT scans as training and validation of COVID-19, we were able to develop a highly accurate as well as robust segmentation model for lungs and COVID-19 infected regions without overfitting on limited data. We performed an in-detail analysis and discussion on the robustness of our pipeline through a sensitivity analysis based on the cross-validation and impact on model generalizability of applied preprocessing techniques. Our method achieved Dice similarity coefficients for COVID-19 infection between predicted and annotated segmentation from radiologists of 0.804 on validation and 0.661 on a separate testing set consisting of 100 patients. CONCLUSIONS: We demonstrated that the proposed method outperforms related approaches, advances the state-of-the-art for COVID-19 segmentation and improves robust medical image analysis based on limited data.

COVID-19 Image Segmentation Based on Deep Learning and Ensemble Learning.

Medical imaging offers great potential for COVID-19 diagnosis and monitoring. Our work introduces an automated pipeline to segment areas of COVID-19 infection in CT scans using deep convolutional neural networks. Furthermore, we evaluate the performance impact of ensemble learning techniques (Bagging and Augmenting). Our models showed highly accurate segmentation results, in which Bagging achieved the highest dice similarity coefficient.

Electronic Image Documentation of Patient Reported Outcomes Using Mobile Technologies.

Patient Reported Outcomes (PROs) provide essential clinical data for the diagnosis and treatment of patients. Mobile technologies enable rapid and structured collection of PROs with a high usability. MoPat is an electronic PRO system developed at the Münster University that enables patients to complete PROs in multiple languages. This research reports the further development of MoPat and the inclusion of features to document images electronically that will be evaluated in a multi-site clinical research.

Reviving 30 Year Old Technology: Lessons Learned from Transferring Patient Data Using Data Matrix Codes.

In patient care and medical research patient data often has to be transferred between different electronic systems. These systems can be very heterogeneous, sometimes even legacy systems, and thus, often do not support standardized interfaces for data transfer. Since nowadays barcode scanners are commonly used in clinical routine and smartphones are accessible to most patients, we implemented different interfaces based on Data Matrix codes to transfer patient data between several medical applications. Objective of this work is to show different use cases in which Data Matrix codes have been successfully applied and discuss the lessons we have learned during the process of implementation and practical usage.

Interoperability Improvement of Mobile Patient Survey (MoPat) Implementing Fast Health Interoperability Resources (FHIR).

Despite the advances in health information technology and the increasing usage of electronic systems, syntactic and semantic interoperability between different health information systems remains challenging. An emerging standard to tackle interoperability issues is HL7 FHIR, which uses modern web technologies for communication like Representational State Transfer. The electronic patient reported outcome system Mobile Patient Survey (MoPat) was adapted to support metadata import and clinical data export using HL7 FHIR. Thereby, the data models of HL7 FHIR and MoPat were compared and the existing import and export functions of MoPat were extended to support HL7 FHIR. A test protocol including eight test datasets to proof functioning of the new features was successfully conducted. In the near future, a real time searching toolbar of FHIR metadata resources will be integrated within MoPat. MoPat FHIR import and export functions are ready to be used in a clinical setting in combination with a FHIR compliant clinical data server.

A Smart Device System to Identify New Phenotypical Characteristics in Movement Disorders.

Parkinson's disease and Essential Tremor are two of the most common movement disorders and are still associated with high rates of misdiagnosis. Collected data by technology-based objective measures (TOMs) has the potential to provide new promising and highly accurate movement data for a better understanding of phenotypical characteristics and diagnostic support. A technology-based system called Smart Device System (SDS) is going to be implemented for multi-modal high-resolution acceleration measurement of patients with PD or ET within a clinical setting. The 2-year prospective observational study is conducted to identify new phenotypical biomarkers and train an Artificial Intelligence System. The SDS is going to be integrated and tested within a 20-min assessment including smartphone-based questionnaires, two smartwatches at both wrists and tablet-based Archimedean spirals drawing for deeper tremor-analyses. The electronic questionnaires will cover data on medication, family history and non-motor symptoms. In this paper, we describe the steps for this novel technology-utilizing examination, the principal steps for data analyses and the targeted performances of the system. Future work considers integration with Deep Brain Stimulation, dissemination into further sites and patient's home setting as well as integration with further data sources as neuroimaging and biobanks. Study Registration ID on ClinicalTrials.gov: NCT03638479.

Humanistic burden of chronic pruritus in patients with inflammatory dermatoses: Results of the European Academy of Dermatology and Venereology Network on Assessment of Severity and Burden of Pruritus (PruNet) cross-sectional trial.

BACKGROUND: Chronic pruritus is a multifactorial, challenging symptom of global relevance. OBJECTIVE: The European Academy of Dermatology and Venereology Network on Assessment of Severity and Burden of Pruritus (PruNet) investigation aimed to analyze the severity and humanistic burden of chronic pruritus in patients suffering from inflammatory dermatoses across Europe. METHODS: Prospectively collected routine data on 552 patients (with atopic dermatitis, contact dermatitis, prurigo nodularis, psoriasis vulgaris, lichen planus, or mycosis fungoides [pruritus numeric rating scale score ≥3]) from 9 European centers (in Austria, France, Germany, Italy, Poland, Russia, Spain, Switzerland, and Turkey) were analyzed by univariate and multivariate variance analyses of various itch characteristics and quality of life (as measured by the Dermatology Life Quality Index and the ItchyQoL). RESULTS: Duration, frequency, and intensity of pruritus (according to a numeric rating scale and visual analog scale) and related impairment of quality of life differed between European centers and dermatologic diagnoses (P < .05). The country in which the center was located had a greater impact on how patients evaluated pruritus intensity and quality of life than diagnosis did (P < .001). LIMITATIONS: One center per country was included. CONCLUSION: The humanistic burden of chronic pruritus in patients with inflammatory dermatoses is high. European cross-cultural factors may have a stronger influence than a specific dermatologic diagnosis on how patients rate intensity of pruritus and quality of life.

Conducting a Multilingual Study Researching Traumatised Refugees Utilizing a Patient-Reported Outcome System.

Approximately 300,000 asylum-seeking children arrived in Europe in 2015. The chance of experiencing a traumatic event is very high for fleeing children. Since the origin of the refugees is widespread, the languages spoken are diverse. Multilingual electronic patient-reported outcome systems (ePROs) can be used to gather medical data in a foreign language and display the results in the health professionals' language, which helps overcoming the language barrier. Utilizing such a system, a two-phase study aiming to screen refugee minors for potential mental health issues has started. Potential eligible participants are examined using questionnaires with good psychometric properties and cross-cultural applicability. To date, 75 minors and 21 of their relatives participated in the study, being German and Arabic the most desired languages for the electronic survey. Developing a system that provides multilingual questionnaires entails several drawbacks like a cumbersome translation process and dealing with writing directions. The proposed translation process and the ePRO can be re-used in similar studies.

Standardising the Development of ODM Converters: The ODMToolBox.

CDISC's Operational Data Model (ODM) is a flexible standard for exchanging and archiving metadata and subject clinical data in clinical trials. The Portal of Medical Data Models (MDM-Portal) uses ODM to store more than 15000 medical forms. As not every electronic health system accepts ODM as input format, there is a need for conversion between ODM and other data standards and formats. This research proposes a standardised template-based process to develop ODM converters. So far, ten converters have been developed and integrated in the MDM-Portal following this process and new ones should be included soon. The template, programming utilities and an ODM test suite have been made online available and can be used to easily develop new converters.

Validation of Pruritus Measures Gathered with the Electronic Patient-reported Outcome System MoPat.

In order to improve diagnosis and treatment, physicians require information about the social context and quality of life of their patients. The Center for Chronic Pruritus at the University Hospital Münster achieves this goal using the electronic patient-reported outcome system "Mobile Patient Survey", which assesses pruritus and quality of life measures. The aim of this study is to evaluate the consistency and reliability of such measures. A total of 42 patients, age range 19-82 years, participated in the study and were asked to assess the measures at baseline via a paper questionnaire, and to use the "Mobile Patient Survey" at baseline and after 1 h in order to test reliability. Statistical analysis was performed using coefficient rc for metric variables and weighted kappa κw for categorical variables. The internal consistency of all measures was unaffected. It was shown that 6 out of 7 measures can be assessed without loss of reliability. It is recommended that questionnaires for electronic usage are assessed for validity and reliability.

Automated Transformation of CDISC ODM to OpenClinica.

Due to the increasing use of electronic data capture systems for clinical research, the interest in saving resources by automatically generating and reusing case report forms in clinical studies is growing. OpenClinica, an open-source electronic data capture system enables the reuse of metadata in its own Excel import template, hampering the reuse of metadata defined in other standard formats. One of these standard formats is the Operational Data Model for metadata, administrative and clinical data in clinical studies. This work suggests a mapping from Operational Data Model to OpenClinica and describes the implementation of a converter to automatically generate OpenClinica conform case report forms based upon metadata in the Operational Data Model.

Operational Data Model Conversion to ResearchKit.

The increasing use of electronic health information systems brings up an unresolved issue: the lack of interoperability between these systems. The Clinical Data Interchange Standards Consortium's Operational Data Model (ODM) is an xml based standard for the exchange of clinical data and metadata. The University of Münster has been using ODM to store medical forms in a web based metadata registry called Portal of Medical Data Models, which includes a complete set of tools to transform ODM forms into other formats. One kind of medical form is the Patient Reported Outcome, a trending type due to its easy integration with mobile data capture systems. ResearchKit is a development framework that allows the easy creation of these for iOS devices; unfortunately its current interoperability is limited. This research proposes a mapping between ODM and ResearchKit and presents the successful implementation of a converter for ODM into JSON based ResearchKit readable files.

The Portal of Medical Data Models: Where Have We Been and Where Are We Going?

To address current key problems of medical documentation: lack of transparency, overwhelming amount of medical contents to be documented and missing interoperability, the Portal of Medical Data Models (http://medical-data-models.org/) was established in 2012. Constantly evolving, four years later, the portal displays more than 8900 medical data models with more than 250000 items, of which 84 % have been semantically annotated with UMLS codes to support interoperability. Giving an update on new functions and contents of the portal, two additional export formats have been implemented, allowing the reuse of forms such as HL7's framework Fast Health Interoperability Resources (FHIR) Questionnaires, as well as the OpenDataKit format. Future projects include the implementation of an ODMtoOpenClinica converter, as well as supporting the reuse of forms with Apple's ResearchKit and Android's ResearchStack.

Implementation of an ODM and HL7 Compliant Electronic Patient-Reported Outcome System.

Interoperability is one of the biggest issues in health informatics despite of the huge effort invested to solve it. Clinical Data Interchange Standards Consortium (CDISC) and Health Level 7 (HL7) are two of the most recognized institutions working on this field. Several systems are becoming compliant with their standards; however, the process to accomplish it is not always straightforward. In this manuscript, we present the successful implementation of the CDISC ODM and HL7 import and export functions for "MoPat", a web-based multi-language electronic patient-reported outcomes system. The system has been evaluated and tested and is currently being used for clinical study and routine data collection, including more than 10.000 patient encounters.