Interpretability of Causal Discovery in Tracking Deterioration in a Highly Dynamic Process.

In a dynamic production processes, mechanical degradation poses a significant challenge, impacting product quality and process efficiency. This paper explores a novel approach for monitoring degradation in the context of viscose fiber production, a highly dynamic manufacturing process. Using causal discovery techniques, our method allows domain experts to incorporate background knowledge into the creation of causal graphs. Further, it enhances the interpretability and increases the ability to identify potential problems via changes in causal relations over time. The case study employs a comprehensive analysis of the viscose fiber production process within a prominent textile industry, emphasizing the advantages of causal discovery for monitoring degradation. The results are compared with state-of-the-art methods, which are not considered to be interpretable, specifically LSTM-based autoencoder, UnSupervised Anomaly Detection on Multivariate Time Series (USAD), and Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data (TranAD), showcasing the alignment and validation of our approach. This paper provides valuable information on degradation monitoring strategies, demonstrating the efficacy of causal discovery in dynamic manufacturing environments. The findings contribute to the evolving landscape of process optimization and quality control.

Image capturing, segmentation and data analysis of shredded refuse streams.

Refuse sorting is an important cornerstone of the recycling industry, but ever-changing refuse compositions and the desire to increase recycling rates still pose many unsolved challenges. The digitalisation of refuse sorting plants promises to overcome these challenges by optimising and automatically adapting the sorting process. This publication describes a system for image capturing, segmentation-based refuse recognition and data analysis of shredded refuse streams. The image capturing collects multispectral 2D and 3D images of the refuse streams on conveyor belts. The image recognition performs a semantic segmentation of the images to determine the refuse composition from the 2D images, whereas the 3D images approximate the volumes on the conveyor belts. The semantic segmentation is done by a combined convolutional neural network model, consisting of a foreground-background and a refuse class segmentation. Both models rely on synthetic training data to reduce the necessary amount of manually labelled training data, whereas the final segmentation performance reaches an Intersection over Union of up to 75%. The results of the semantic segmentation and volume estimation are combined with data of the shredding machinery by transforming it into a unified representation. This combined dataset is the basis for estimating the processed refuse masses from the semantic segmentation and volume estimation.

A historical perspective of biomedical explainable AI research.

The black-box nature of most artificial intelligence (AI) models encourages the development of explainability methods to engender trust into the AI decision-making process. Such methods can be broadly categorized into two main types: post hoc explanations and inherently interpretable algorithms. We aimed at analyzing the possible associations between COVID-19 and the push of explainable AI (XAI) to the forefront of biomedical research. We automatically extracted from the PubMed database biomedical XAI studies related to concepts of causality or explainability and manually labeled 1,603 papers with respect to XAI categories. To compare the trends pre- and post-COVID-19, we fit a change point detection model and evaluated significant changes in publication rates. We show that the advent of COVID-19 in the beginning of 2020 could be the driving factor behind an increased focus concerning XAI, playing a crucial role in accelerating an already evolving trend. Finally, we present a discussion with future societal use and impact of XAI technologies and potential future directions for those who pursue fostering clinical trust with interpretable machine learning models.

Information extraction from German radiological reports for general clinical text and language understanding.

Recent advances in deep learning and natural language processing (NLP) have opened many new opportunities for automatic text understanding and text processing in the medical field. This is of great benefit as many clinical downstream tasks rely on information from unstructured clinical documents. However, for low-resource languages like German, the use of modern text processing applications that require a large amount of training data proves to be difficult, as only few data sets are available mainly due to legal restrictions. In this study, we present an information extraction framework that was initially pre-trained on real-world computed tomographic (CT) reports of head examinations, followed by domain adaptive fine-tuning on reports from different imaging examinations. We show that in the pre-training phase, the semantic and contextual meaning of one clinical reporting domain can be captured and effectively transferred to foreign clinical imaging examinations. Moreover, we introduce an active learning approach with an intrinsic strategic sampling method to generate highly informative training data with low human annotation cost. We see that the model performance can be significantly improved by an appropriate selection of the data to be annotated, without the need to train the model on a specific downstream task. With a general annotation scheme that can be used not only in the radiology field but also in a broader clinical setting, we contribute to a more consistent labeling and annotation process that also facilitates the verification and evaluation of language models in the German clinical setting.

ManEx: The Visual Analysis of Measurements for the Assessment of Errors in Electrical Engines.

Electrical engines are a key technology all automotive manufacturers must master to stay competitive. Engineers need to analyze an overwhelming number of engine measurements to improve the manufacturing for this technology. They are hindered in the task of analyzing large numbers of engines, however, by the following challenges: 1) Engines comprise a complex hierarchical structure of subcomponents. 2) Locating the cause of errors along manufacturing processes is a difficult procedure. 3) Large numbers of heterogeneous measurements impair the ability to explain errors in engines. We address these challenges in a design study with automotive engineers and by developing the visual analytics system Manufacturing Explorer (ManEx), which provides interactive interfaces to analyze measurements of engines across the manufacturing process. ManEx was validated by five experts. Our results suggest high usability and usefulness scores and the improvement of a real-world manufacturing process. Specifically, with ManEx, experts reduced scraped parts by over 3%.

A Causality-Inspired Approach for Anomaly Detection in a Water Treatment Testbed.

Critical infrastructure, such as water treatment facilities, largely relies on the effective functioning of industrial control systems (ICSs). Due to the wide adoption of high-speed network and digital infrastructure technologies, these systems are now highly interconnected not only to corporate networks but also to the public Internet, mostly for remote control and monitoring purposes. Sophisticated cyber-attacks may take advantage the increased interconnectedness or other security gaps of an ICS and infiltrate the system with devastating consequences to the economy, national security, and even human life. Due to the paramount importance of detecting and isolating these attacks, we propose an unsupervised anomaly detection approach that employs causal inference to construct a robust anomaly score in two phases. First, minimal domain knowledge via causal models helps identify critical interdependencies in the system, while univariate models contribute to individually learn the normal behavior of the system's components. In the final phase, we employ the extreme studentized deviate (ESD) on the computed score to detect attacks and to exclude any irrelevant sensor signals. Our approach is validated on the widely used Secure Water Treatment (SWaT) benchmark, and it exhibits the highest F1 score with zero false alarms, which is extremely important for real-world deployment.

Deep learning-a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact.

Deep learning belongs to the field of artificial intelligence, where machines perform tasks that typically require some kind of human intelligence. Deep learning tries to achieve this by drawing inspiration from the learning of a human brain. Similar to the basic structure of a brain, which consists of (billions of) neurons and connections between them, a deep learning algorithm consists of an artificial neural network, which resembles the biological brain structure. Mimicking the learning process of humans with their senses, deep learning networks are fed with (sensory) data, like texts, images, videos or sounds. These networks outperform the state-of-the-art methods in different tasks and, because of this, the whole field saw an exponential growth during the last years. This growth resulted in way over 10,000 publications per year in the last years. For example, the search engine PubMed alone, which covers only a sub-set of all publications in the medical field, provides already over 11,000 results in Q3 2020 for the search term 'deep learning', and around 90% of these results are from the last three years. Consequently, a complete overview over the field of deep learning is already impossible to obtain and, in the near future, it will potentially become difficult to obtain an overview over a subfield. However, there are several review articles about deep learning, which are focused on specific scientific fields or applications, for example deep learning advances in computer vision or in specific tasks like object detection. With these surveys as a foundation, the aim of this contribution is to provide a first high-level, categorized meta-survey of selected reviews on deep learning across different scientific disciplines and outline the research impact that they already have during a short period of time. The categories (computer vision, language processing, medical informatics and additional works) have been chosen according to the underlying data sources (image, language, medical, mixed). In addition, we review the common architectures, methods, pros, cons, evaluations, challenges and future directions for every sub-category.

Should We Embed in Chemistry? A Comparison of Unsupervised Transfer Learning with PCA, UMAP, and VAE on Molecular Fingerprints.

Methods for dimensionality reduction are showing significant contributions to knowledge generation in high-dimensional modeling scenarios throughout many disciplines. By achieving a lower dimensional representation (also called embedding), fewer computing resources are needed in downstream machine learning tasks, thus leading to a faster training time, lower complexity, and statistical flexibility. In this work, we investigate the utility of three prominent unsupervised embedding techniques (principal component analysis-PCA, uniform manifold approximation and projection-UMAP, and variational autoencoders-VAEs) for solving classification tasks in the domain of toxicology. To this end, we compare these embedding techniques against a set of molecular fingerprint-based models that do not utilize additional pre-preprocessing of features. Inspired by the success of transfer learning in several fields, we further study the performance of embedders when trained on an external dataset of chemical compounds. To gain a better understanding of their characteristics, we evaluate the embedders with different embedding dimensionalities, and with different sizes of the external dataset. Our findings show that the recently popularized UMAP approach can be utilized alongside known techniques such as PCA and VAE as a pre-compression technique in the toxicology domain. Nevertheless, the generative model of VAE shows an advantage in pre-compressing the data with respect to classification accuracy.

Treatment outcome clustering patterns correspond to discrete asthma phenotypes in children.

Despite widely and regularly used therapy asthma in children is not fully controlled. Recognizing the complexity of asthma phenotypes and endotypes imposed the concept of precision medicine in asthma treatment. By applying machine learning algorithms assessed with respect to their accuracy in predicting treatment outcome, we have successfully identified 4 distinct clusters in a pediatric asthma cohort with specific treatment outcome patterns according to changes in lung function (FEV1 and MEF50), airway inflammation (FENO) and disease control likely affected by discrete phenotypes at initial disease presentation, differing in the type and level of inflammation, age of onset, comorbidities, certain genetic and other physiologic traits. The smallest and the largest of the 4 clusters- 1 (N = 58) and 3 (N = 138) had better treatment outcomes compared to clusters 2 and 4 and were characterized by more prominent atopic markers and a predominant allelic (A allele) effect for rs37973 in the GLCCI1 gene previously associated with positive treatment outcomes in asthmatics. These patients also had a relatively later onset of disease (6 + yrs). Clusters 2 (N = 87) and 4 (N = 64) had poorer treatment success, but varied in the type of inflammation (predominantly neutrophilic for cluster 4 and likely mixed-type for cluster 2), comorbidities (obesity for cluster 2), level of systemic inflammation (highest hsCRP for cluster 2) and platelet count (lowest for cluster 4). The results of this study emphasize the issues in asthma management due to the overgeneralized approach to the disease, not taking into account specific disease phenotypes.

Predicting Treatment Outcomes Using Explainable Machine Learning in Children with Asthma.

Asthma in children is a heterogeneous disease manifested by various phenotypes and endotypes. The level of disease control, as well as the effectiveness of anti-inflammatory treatment, is variable and inadequate in a significant portion of patients. By applying machine learning algorithms, we aimed to predict the treatment success in a pediatric asthma cohort and to identify the key variables for understanding the underlying mechanisms. We predicted the treatment outcomes in children with mild to severe asthma (N = 365), according to changes in asthma control, lung function (FEV1 and MEF50) and FENO values after 6 months of controller medication use, using Random Forest and AdaBoost classifiers. The highest prediction power is achieved for control- and, to a lower extent, for FENO-related treatment outcomes, especially in younger children. The most predictive variables for asthma control are related to asthma severity and the total IgE, which were also predictive for FENO-based outcomes. MEF50-related treatment outcomes were better predicted than the FEV1-based response, and one of the best predictive variables for this response was hsCRP, emphasizing the involvement of the distal airways in childhood asthma. Our results suggest that asthma control- and FENO-based outcomes can be more accurately predicted using machine learning than the outcomes according to FEV1 and MEF50. This supports the symptom control-based asthma management approach and its complementary FENO-guided tool in children. T2-high asthma seemed to respond best to the anti-inflammatory treatment. The results of this study in predicting the treatment success will help to enable treatment optimization and to implement the concept of precision medicine in pediatric asthma treatment.

Predictive Capability of QSAR Models Based on the CompTox Zebrafish Embryo Assays: An Imbalanced Classification Problem.

The CompTox Chemistry Dashboard (ToxCast) contains one of the largest public databases on Zebrafish (Danio rerio) developmental toxicity. The data consists of 19 toxicological endpoints on unique 1018 compounds measured in relatively low concentration ranges. The endpoints are related to developmental effects occurring in dechorionated zebrafish embryos for 120 hours post fertilization and monitored via gross malformations and mortality. We report the predictive capability of 209 quantitative structure-activity relationship (QSAR) models developed by machine learning methods using penalization techniques and diverse model quality metrics to cope with the imbalanced endpoints. All these QSAR models were generated to test how the imbalanced classification (toxic or non-toxic) endpoints could be predicted regardless which of three algorithms is used: logistic regression, multi-layer perceptron, or random forests. Additionally, QSAR toxicity models are developed starting from sets of classical molecular descriptors, structural fingerprints and their combinations. Only 8 out of 209 models passed the 0.20 Matthew's correlation coefficient value defined a priori as a threshold for acceptable model quality on the test sets. The best models were obtained for endpoints mortality (MORT), ActivityScore and JAW (deformation). The low predictability of the QSAR model developed from the zebrafish embryotoxicity data in the database is mainly due to a higher sensitivity of 19 measurements of endpoints carried out on dechorionated embryos at low concentrations.

Understanding the true effects of the COVID-19 lockdown on air pollution by means of machine learning.

During March 2020, most European countries implemented lockdowns to restrict the transmission of SARS-CoV-2, the virus which causes COVID-19 through their populations. These restrictions had positive impacts for air quality due to a dramatic reduction of economic activity and atmospheric emissions. In this work, a machine learning approach was designed and implemented to analyze local air quality improvements during the COVID-19 lockdown in Graz, Austria. The machine learning approach was used as a robust alternative to simple, historical measurement comparisons for various individual pollutants. Concentrations of NO2 (nitrogen dioxide), PM10 (particulate matter), O3 (ozone) and Ox (total oxidant) were selected from five measurement sites in Graz and were set as target variables for random forest regression models to predict their expected values during the city's lockdown period. The true vs. expected difference is presented here as an indicator of true pollution during the lockdown. The machine learning models showed a high level of generalization for predicting the concentrations. Therefore, the approach was suitable for analyzing reductions in pollution concentrations. The analysis indicated that the city's average concentration reductions for the lockdown period were: -36.9 to -41.6%, and -6.6 to -14.2% for NO2 and PM10, respectively. However, an increase of 11.6-33.8% for O3 was estimated. The reduction in pollutant concentration, especially NO2 can be explained by significant drops in traffic-flows during the lockdown period (-51.6 to -43.9%). The results presented give a real-world example of what pollutant concentration reductions can be achieved by reducing traffic-flows and other economic activities.

PySpark and RDKit: Moving towards Big Data in Cheminformatics.

The authors present an implementation of the cheminformatics toolkit RDKit in a distributed computing environment, Apache Hadoop. Together with the Apache Spark analytics engine, wrapped by PySpark, resources from commodity scalable hardware can be employed for cheminformatic calculations and query operations with basic knowledge in Python programming and understanding of the resilient distributed datasets (RDD). Three use cases of cheminfomatical computing in Spark on the Hadoop cluster are presented; querying substructures, calculating fingerprint similarity and calculating molecular descriptors. The source code for the PySpark-RDKit implementation is provided. The use cases showed that Spark provides a reasonable scalability depending on the use case and can be a suitable choice for datasets too big to be processed with current low-end workstations.

Authorship identification of documents with high content similarity.

The goal of our work is inspired by the task of associating segments of text to their real authors. In this work, we focus on analyzing the way humans judge different writing styles. This analysis can help to better understand this process and to thus simulate/ mimic such behavior accordingly. Unlike the majority of the work done in this field (i.e. authorship attribution, plagiarism detection, etc.) which uses content features, we focus only on the stylometric, i.e. content-agnostic, characteristics of authors. Therefore, we conducted two pilot studies to determine, if humans can identify authorship among documents with high content similarity. The first was a quantitative experiment involving crowd-sourcing, while the second was a qualitative one executed by the authors of this paper. Both studies confirmed that this task is quite challenging. To gain a better understanding of how humans tackle such a problem, we conducted an exploratory data analysis on the results of the studies. In the first experiment, we compared the decisions against content features and stylometric features. While in the second, the evaluators described the process and the features on which their judgment was based. The findings of our detailed analysis could (1) help to improve algorithms such as automatic authorship attribution as well as plagiarism detection, (2) assist forensic experts or linguists to create profiles of writers, (3) support intelligence applications to analyze aggressive and threatening messages and (4) help editor conformity by adhering to, for instance, journal specific writing style.

Understanding why users tag: A survey of tagging motivation literature and results from an empirical study.

While recent progress has been achieved in understanding the structure and dynamics of social tagging systems, we know little about the underlying user motivations for tagging, and how they influence resulting folksonomies and tags. This paper addresses three issues related to this question. (1) What distinctions of user motivations are identified by previous research, and in what ways are the motivations of users amenable to quantitative analysis? (2) To what extent does tagging motivation vary across different social tagging systems? (3) How does variability in user motivation influence resulting tags and folksonomies? In this paper, we present measures to detect whether a tagger is primarily motivated by categorizing or describing resources, and apply these measures to datasets from seven different tagging systems. Our results show that (a) users' motivation for tagging varies not only across, but also within tagging systems, and that (b) tag agreement among users who are motivated by categorizing resources is significantly lower than among users who are motivated by describing resources. Our findings are relevant for (1) the development of tag-based user interfaces, (2) the analysis of tag semantics and (3) the design of search algorithms for social tagging systems.