MMIR: an open-source software for the registration of multimodal histological images.

BACKGROUND: Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality. RESULTS: We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system. CONCLUSION: Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

Siamese Networks for Clinically Relevant Bacteria Classification Based on Raman Spectroscopy.

Identifying bacterial strains is essential in microbiology for various practical applications, such as disease diagnosis and quality monitoring of food and water. Classical machine learning algorithms have been utilized to identify bacteria based on their Raman spectra. However, convolutional neural networks (CNNs) offer higher classification accuracy, but they require extensive training sets and retraining of previous untrained class targets can be costly and time-consuming. Siamese networks have emerged as a promising solution. They are composed of two CNNs with the same structure and a final network that acts as a distance metric, converting the classification problem into a similarity problem. Classical machine learning approaches, shallow and deep CNNs, and two Siamese network variants were tailored and tested on Raman spectral datasets of bacteria. The methods were evaluated based on mean sensitivity, training time, prediction time, and the number of parameters. In this comparison, Siamese-model2 achieved the highest mean sensitivity of 83.61 ± 4.73 and demonstrated remarkable performance in handling unbalanced and limited data scenarios, achieving a prediction accuracy of 73%. Therefore, the choice of model depends on the specific trade-off between accuracy, (prediction/training) time, and resources for the particular application. Classical machine learning models and shallow CNN models may be more suitable if time and computational resources are a concern. Siamese networks are a good choice for small datasets and CNN for extensive data.

Illuminating the Tiny World: A Navigation Guide for Proper Raman Studies on Microorganisms.

Raman spectroscopy is an emerging method for the identification of bacteria. Nevertheless, a lot of different parameters need to be considered to establish a reliable database capable of identifying real-world samples such as medical or environmental probes. In this review, the establishment of such reliable databases with the proper design in microbiological Raman studies is demonstrated, shining a light into all the parts that require attention. Aspects such as the strain selection, sample preparation and isolation requirements, the phenotypic influence, measurement strategies, as well as the statistical approaches for discrimination of bacteria, are presented. Furthermore, the influence of these aspects on spectra quality, result accuracy, and read-out are discussed. The aim of this review is to serve as a guide for the design of microbiological Raman studies that can support the establishment of this method in different fields.

Correction for Extrinsic Background in Raman Hyperspectral Images.

Raman hyperspectral microscopy is a valuable tool in biological and biomedical imaging. Because Raman scattering is often weak in comparison to other phenomena, prevalent spectral fluctuations and contaminations have brought advancements in analytical and chemometric methods for Raman spectra. These chemometric advances have been key contributors to the applicability of Raman imaging to biological systems. As studies increase in scale, spectral contamination from extrinsic background, intensity from sources such as the optical components that are extrinsic to the sample of interest, has become an emerging issue. Although existing baseline correction schemes often reduce intrinsic background such as autofluorescence originating from the sample of interest, extrinsic background is not explicitly considered, and these methods often fail to reduce its effects. Here, we show that extrinsic background can significantly affect a classification model using Raman images, yielding misleadingly high accuracies in the distinction of benign and malignant samples of follicular thyroid cell lines. To mitigate its effects, we develop extrinsic background correction (EBC) and demonstrate its use in combination with existing methods on Raman hyperspectral images. EBC isolates regions containing the smallest amounts of sample materials that retain extrinsic contributions that are specific to the device or environment. We perform classification both with and without the use of EBC, and we find that EBC retains biological characteristics in the spectra while significantly reducing extrinsic background. As the methodology used in EBC is not specific to Raman spectra, correction of extrinsic effects in other types of hyperspectral and grayscale images is also possible.

Identification of bacteria in mixed infection from urinary tract of patient's samples using Raman analysis of dried droplets.

Urinary tract infections (UTI) are among the most frequent nosocomial infections. A fast identification of the pathogen and assignment of Gram type could help to prescribe most suitable treatments. Raman spectroscopy holds high potential for fast and reliable bacterial pathogens identification. While most studies so far have focused on individual pathogens or artificial mixtures, this contribution aims to translate the analysis to primary urine samples from patients with suspected UTIs. For this, we have included 59 primary urine samples out of which 29 were diagnosed as mixed infections. For Raman analysis, we first trained two classification models based on principal component analysis - linear discriminant analysis (PCA-LDA) with more than 3500 Raman spectra of 85 clinical isolates from 23 species in order to (1) identify the Gram type of the bacteria and (2) assign family membership to one of the six most abundant bacterial families in urinary tract infections (Enterobacteriaceae, Morganellaceae, Pseudomonadaceae, Enterococcaceae, Staphylococcaceae and Streptococcaceae). The classification models were applied to artificial mixtures of Gram positive and Gram negative bacteria to correctly predict mixed infections with an accuracy of 75%. Raman scans of dried droplets did not yet yield optimal classification results on family level. When translating the method to primary urine samples, we observed a strong bias towards Gram negative bacteria, on family level towards Morganellaceae, which reduced prediction accuracy. Spectral differences were observed between isolates grown on standard growth medium and bacteria of the same strain when characterized directly from the patient. Thus, improvement of the classification accuracy is expected with a larger data base containing also bacteria measured directly from the urine sample.

Differentiability of cell types enhanced by detrending a non-homogeneous pattern in a line-illumination Raman microscope.

A line illumination Raman microscope extracts the underlying spatial and spectral information of a sample, typically a few hundred times faster than raster scanning. This makes it possible to measure a wide range of biological samples such as cells and tissues - that only allow modest intensity illumination to prevent potential damage - within feasible time frame. However, a non-uniform intensity distribution of laser line illumination may induce some artifacts in the data and lower the accuracy of machine learning models trained to predict sample class membership. Here, using cancerous and normal human thyroid follicular epithelial cell lines, FTC-133 and Nthy-ori 3-1 lines, whose Raman spectral difference is not so large, we show that the standard pre-processing of spectral analyses widely used for raster scanning microscopes introduced some artifacts. To address this issue, we proposed a detrending scheme based on random forest regression, a nonparametric model-free machine learning algorithm, combined with a position-dependent wavenumber calibration scheme along the illumination line. It was shown that the detrending scheme minimizes the artifactual biases arising from non-uniform laser sources and significantly enhances the differentiability of the sample states, i.e., cancerous or normal epithelial cells, compared to the standard pre-processing scheme.

Artificial intelligence for the classification of focal liver lesions in ultrasound - a systematic review.

Focal liver lesions are detected in about 15% of abdominal ultrasound examinations. The diagnosis of frequent benign lesions can be determined reliably based on the characteristic B-mode appearance of cysts, hemangiomas, or typical focal fatty changes. In the case of focal liver lesions which remain unclear on B-mode ultrasound, contrast-enhanced ultrasound (CEUS) increases diagnostic accuracy for the distinction between benign and malignant liver lesions. Artificial intelligence describes applications that try to emulate human intelligence, at least in subfields such as the classification of images. Since ultrasound is considered to be a particularly examiner-dependent technique, the application of artificial intelligence could be an interesting approach for an objective and accurate diagnosis. In this systematic review we analyzed how artificial intelligence can be used to classify the benign or malignant nature and entity of focal liver lesions on the basis of B-mode or CEUS data. In a structured search on Scopus, Web of Science, PubMed, and IEEE, we found 52 studies that met the inclusion criteria. Studies showed good diagnostic performance for both the classification as benign or malignant and the differentiation of individual tumor entities. The results could be improved by inclusion of clinical parameters and were comparable to those of experienced investigators in terms of diagnostic accuracy. However, due to the limited spectrum of lesions included in the studies and a lack of independent validation cohorts, the transfer of the results into clinical practice is limited.

Exploring the Steps of Infrared (IR) Spectral Analysis: Pre-Processing, (Classical) Data Modelling, and Deep Learning.

Infrared (IR) spectroscopy has greatly improved the ability to study biomedical samples because IR spectroscopy measures how molecules interact with infrared light, providing a measurement of the vibrational states of the molecules. Therefore, the resulting IR spectrum provides a unique vibrational fingerprint of the sample. This characteristic makes IR spectroscopy an invaluable and versatile technology for detecting a wide variety of chemicals and is widely used in biological, chemical, and medical scenarios. These include, but are not limited to, micro-organism identification, clinical diagnosis, and explosive detection. However, IR spectroscopy is susceptible to various interfering factors such as scattering, reflection, and interference, which manifest themselves as baseline, band distortion, and intensity changes in the measured IR spectra. Combined with the absorption information of the molecules of interest, these interferences prevent direct data interpretation based on the Beer-Lambert law. Instead, more advanced data analysis approaches, particularly artificial intelligence (AI)-based algorithms, are required to remove the interfering contributions and, more importantly, to translate the spectral signals into high-level biological/chemical information. This leads to the tasks of spectral pre-processing and data modeling, the main topics of this review. In particular, we will discuss recent developments in both tasks from the perspectives of classical machine learning and deep learning.

Detection of multi-resistant clinical strains of E. coli with Raman spectroscopy.

In recent years, we have seen a steady rise in the prevalence of antibiotic-resistant bacteria. This creates many challenges in treating patients who carry these infections, as well as stopping and preventing outbreaks. Identifying these resistant bacteria is critical for treatment decisions and epidemiological studies. However, current methods for identification of resistance either require long cultivation steps or expensive reagents. Raman spectroscopy has been shown in the past to enable the rapid identification of bacterial strains from single cells and cultures. In this study, Raman spectroscopy was applied for the differentiation of resistant and sensitive strains of Escherichia coli. Our focus was on clinical multi-resistant (extended-spectrum ß-lactam and carbapenem-resistant) bacteria from hospital patients. The spectra were collected using both UV resonance Raman spectroscopy in bulk and single-cell Raman microspectroscopy, without exposure to antibiotics. We found resistant strains have a higher nucleic acid/protein ratio, and used the spectra to train a machine learning model that differentiates resistant and sensitive strains. In addition, we applied a majority of voting system to both improve the accuracy of our models and make them more applicable for a clinical setting. This method could allow rapid and accurate identification of antibiotic resistant bacteria, and thus improve public health.

CARS Imaging Advances Early Diagnosis of Cardiac Manifestation of Fabry Disease.

Vibrational spectroscopy can detect characteristic biomolecular signatures and thus has the potential to support diagnostics. Fabry disease (FD) is a lipid disorder disease that leads to accumulations of globotriaosylceramide in different organs, including the heart, which is particularly critical for the patient's prognosis. Effective treatment options are available if initiated at early disease stages, but many patients are late- or under-diagnosed. Since Coherent anti-Stokes Raman (CARS) imaging has a high sensitivity for lipid/protein shifts, we applied CARS as a diagnostic tool to assess cardiac FD manifestation in an FD mouse model. CARS measurements combined with multivariate data analysis, including image preprocessing followed by image clustering and data-driven modeling, allowed for differentiation between FD and control groups. Indeed, CARS identified shifts of lipid/protein content between the two groups in cardiac tissue visually and by subsequent automated bioinformatic discrimination with a mean sensitivity of 90-96%. Of note, this genotype differentiation was successful at a very early time point during disease development when only kidneys are visibly affected by globotriaosylceramide depositions. Altogether, the sensitivity of CARS combined with multivariate analysis allows reliable diagnostic support of early FD organ manifestation and may thus improve diagnosis, prognosis, and possibly therapeutic monitoring of FD.

A Review on Data Fusion of Multidimensional Medical and Biomedical Data.

Data fusion aims to provide a more accurate description of a sample than any one source of data alone. At the same time, data fusion minimizes the uncertainty of the results by combining data from multiple sources. Both aim to improve the characterization of samples and might improve clinical diagnosis and prognosis. In this paper, we present an overview of the advances achieved over the last decades in data fusion approaches in the context of the medical and biomedical fields. We collected approaches for interpreting multiple sources of data in different combinations: image to image, image to biomarker, spectra to image, spectra to spectra, spectra to biomarker, and others. We found that the most prevalent combination is the image-to-image fusion and that most data fusion approaches were applied together with deep learning or machine learning methods.

Monitoring Deuterium Uptake in Single Bacterial Cells via Two-Dimensional Raman Correlation Spectroscopy.

Raman-stable isotope labeling using heavy water (Raman-D2O) is attracting great interest as a fast technique with various applications ranging from the identification of pathogens in medical samples to the determination of microbial activity in the environment. Despite its widespread applications, little is known about the fundamental processes of hydrogen-deuterium (H/D) exchange, which are crucial for understanding molecular interactions in microorganisms. By combining two-dimensional (2D) correlation spectroscopy and Raman deuterium labeling, we have investigated H/D exchange in bacterial cells under time dependence. Most C-H stretching signals decreased in intensity over time, prior to the formation of the C-D stretching vibration signals. The intensity of the C-D signal gradually increased over time, and the shape of the C-D signal was more uniform after longer incubation times. Deuterium uptake showed high variability between the bacterial genera and mainly led to an observable labeling of methylene and methyl groups. Thus, the C-D signal encompassed a combination of symmetric and antisymmetric CD2 and CD3 stretching vibrations, depending on the bacterial genera. The present study allowed for the determination of the sequential order of deuterium incorporation into the functional groups of proteins, lipids, and nucleic acids and hence understanding the process of biomolecule synthesis and the growth strategies of different bacterial taxa. We present the combination of Raman-D2O labeling and 2D correlation spectroscopy as a promising approach to gain a fundamental understanding of molecular interactions in biological systems.

Nondestructive molecular imaging by Raman spectroscopy vs. marker detection by MALDI IMS for an early diagnosis of HCC.

Hepatocellular carcinoma (HCC) is a leading cause of cancer-related deaths worldwide with a steadily increasing mortality rate. Fast diagnosis at early stages of HCC is of key importance for the improvement of patient survival rates. In this regard, we combined two imaging techniques with high potential for HCC diagnosis in order to improve the prediction of liver cancer. In detail, Raman spectroscopic imaging and matrix-assisted laser desorption ionization imaging mass spectrometry (MALDI IMS) were applied for the diagnosis of 36 HCC tissue samples. The data were analyzed using multivariate methods, and the results revealed that Raman spectroscopy alone showed a good capability for HCC tumor identification (sensitivity of 88% and specificity of 80%), which could not be improved by combining the Raman data with MALDI IMS. In addition, it could be shown that the two methods in combination can differentiate between well-, moderately- and poorly-differentiated HCC using a linear classification model. MALDI IMS not only classified the HCC grades with a sensitivity of 100% and a specificity of 80%, but also showed significant differences in the expression of glycerophospholipids and fatty acyls during HCC differentiation. Furthermore, important differences in the protein, lipid and collagen compositions of differentiated HCC were detected using the model coefficients of a Raman based classification model. Both Raman and MALDI IMS, as well as their combination showed high potential for resolving concrete questions in liver cancer diagnosis.

Comparison of functional and discrete data analysis regimes for Raman spectra.

Raman spectral data are best described by mathematical functions; however, due to the spectroscopic measurement setup, only discrete points of these functions are measured. Therefore, we investigated the Raman spectral data for the first time in the functional framework. First, we approximated the Raman spectra by using B-spline basis functions. Afterwards, we applied the functional principal component analysis followed by the linear discriminant analysis (FPCA-LDA) and compared the results with those of the classical principal component analysis followed by the linear discriminant analysis (PCA-LDA). In this context, simulation and experimental Raman spectra were used. In the simulated Raman spectra, normal and abnormal spectra were used for a classification model, where the abnormal spectra were built by shifting one peak position. We showed that the mean sensitivities of the FPCA-LDA method were higher than the mean sensitivities of the PCA-LDA method, especially when the signal-to-noise ratio is low and the shift of the peak position is small. However, for a higher signal-to-noise ratio, both methods performed equally. Additionally, a slight improvement of the mean sensitivity could be shown if the FPCA-LDA method was applied to experimental Raman data.

Vibrational Spectroscopic Investigation of Blood Plasma and Serum by Drop Coating Deposition for Clinical Application.

In recent decades, vibrational spectroscopic methods such as Raman and FT-IR spectroscopy are widely applied to investigate plasma and serum samples. These methods are combined with drop coating deposition techniques to pre-concentrate the biomolecules in the dried droplet to improve the detected vibrational signal. However, most often encountered challenge is the inhomogeneous redistribution of biomolecules due to the coffee-ring effect. In this study, the variation in biomolecule distribution within the dried-sample droplet has been investigated using Raman and FT-IR spectroscopy and fluorescence lifetime imaging method. The plasma-sample from healthy donors were investigated to show the spectral differences between the inner and outer-ring region of the dried-sample droplet. Further, the preferred location of deposition of the most abundant protein albumin in the blood during the drying process of the plasma has been illustrated by using deuterated albumin. Subsequently, two patients with different cardiac-related diseases were investigated exemplarily to illustrate the variation in the pattern of plasma and serum biomolecule distribution during the drying process and its impact on patient-stratification. The study shows that a uniform sampling position of the droplet, both at the inner and the outer ring, is necessary for thorough clinical characterization of the patient's plasma and serum sample using vibrational spectroscopy.

Biochemical Analysis of Leukocytes after In Vitro and In Vivo Activation with Bacterial and Fungal Pathogens Using Raman Spectroscopy.

Biochemical information from activated leukocytes provide valuable diagnostic information. In this study, Raman spectroscopy was applied as a label-free analytical technique to characterize the activation pattern of leukocyte subpopulations in an in vitro infection model. Neutrophils, monocytes, and lymphocytes were isolated from healthy volunteers and stimulated with heat-inactivated clinical isolates of Candida albicans, Staphylococcus aureus, and Klebsiella pneumoniae. Binary classification models could identify the presence of infection for monocytes and lymphocytes, classify the type of infection as bacterial or fungal for neutrophils, monocytes, and lymphocytes and distinguish the cause of infection as Gram-negative or Gram-positive bacteria in the monocyte subpopulation. Changes in single-cell Raman spectra, upon leukocyte stimulation, can be explained with biochemical changes due to the leukocyte's specific reaction to each type of pathogen. Raman spectra of leukocytes from the in vitro infection model were compared with spectra from leukocytes of patients with infection (DRKS-ID: DRKS00006265) with the same pathogen groups, and a good agreement was revealed. Our study elucidates the potential of Raman spectroscopy-based single-cell analysis for the differentiation of circulating leukocyte subtypes and identification of the infection by probing the molecular phenotype of those cells.

Towards an Interpretable Classifier for Characterization of Endoscopic Mayo Scores in Ulcerative Colitis Using Raman Spectroscopy.

Ulcerative colitis (UC) is one of the main types of chronic inflammatory diseases that affect the bowel, but its pathogenesis is yet to be completely defined. Assessing the disease activity of UC is vital for developing a personalized treatment. Conventionally, the assessment of UC is performed by colonoscopy and histopathology. However, conventional methods fail to retain biomolecular information associated to the severity of UC and are solely based on morphological characteristics of the inflamed colon. Furthermore, assessing endoscopic disease severity is limited by the requirement for experienced human reviewers. Therefore, this work presents a nondestructive biospectroscopic technique, for example, Raman spectroscopy, for assessing endoscopic disease severity according to the four-level Mayo subscore. This contribution utilizes multidimensional Raman spectroscopic data to generate a predictive model for identifying colonic inflammation. The predictive modeling of the Raman spectroscopic data is performed using a one-dimensional deep convolutional neural network (1D-CNN). The classification results of 1D-CNN achieved a mean sensitivity of 78% and a mean specificity of 93% for the four Mayo endoscopic scores. Furthermore, the results of the 1D-CNN are interpreted by a first-order Taylor expansion, which extracts the Raman bands important for classification. Additionally, a regression model of the 1D-CNN model is constructed to study the extent of misclassification and border-line patients. The overall results of Raman spectroscopy with 1D-CNN as a classification and regression model show a good performance, and such a method can serve as a complementary method for UC analysis.

Spatiotemporal Organization of Biofilm Matrix Revealed by Confocal Raman Mapping Integrated with Non-negative Matrix Factorization Analysis.

Biofilms are microbial aggregates of microorganisms surrounded by a hydrogel-like matrix formed by extracellular polymeric substances (EPS). The formation of biofilms is intrinsically complex, from the attachment of microbial cells to the dispersion of the biofilm. Meanwhile, the three-dimensional framework built up by EPS changes with time and protects the microorganisms against environmental stress. Simultaneously acquiring chemical and structural information within the biofilm matrix is vital for the cognition and regulation of biofilms, yet it remains a great challenge due to the sample complexity and the limited approaches. In this study, confocal Raman microscopy and non-negative matrix factorization (NMF) analysis were combined to investigate spatiotemporal organization of Escherichia coli biofilms during development at molecular-level detail. The alternating non-negative least-squares (ANLS) approach was incorporated with the sequential coordinate-wise descent (SCD) algorithm to realize the NMF analysis for the large-scale hyperspectral data set. As a result, three components, including bacteria, protein, and polyhydroxybutyrate (PHB), were successfully resolved from the spectra of E. coli biofilm. Furthermore, the structural changes of biofilms could be visualized and quantified by their abundances derived from the NMF analysis, which might be related to the nutrient and oxygen gradient and physiological functions. This methodology provides a comprehensive understanding of the chemical constituents and their spatiotemporal distribution within the biofilm matrix. Furthermore, it also shows great potential for the analysis of unknown and complex biological samples with 3D Raman mapping.

Influence of Carbon Sources on Quantification of Deuterium Incorporation in Heterotrophic Bacteria: A Raman-Stable Isotope Labeling Approach.

A rapid and reliable method for the differentiation between active and inactive bacteria at single cell level is urgently needed in many fields including clinical diagnosis and environmental microbiology, to understand the contribution of metabolically active bacteria in fundamental processes triggering environmental and public health risks. Here, using heavy water (D2O) with Raman-stable isotope labeling (Raman-D2O), we evaluated the reliability of the quantification of deuterium uptake, a well-known indicator for the general metabolic activity of bacteria. For this purpose, we based our study on the quantification of deuterium assimilation from heavy water into single bacterial cells to check the influence of carbon source and bacterial identity on the deuterium uptake. We show that compared to complex carbon substrates, the deuterium assimilation is higher in the presence of simpler substrates such as sugars but differs significantly among bacterial isolates. Despite this variability, the developed classification models could differentiate deuterium labeled and nonlabeled single cells with high sensitivity and specificity. Highlighting the variability between single bacterial cells, the study emphasizes the challenges in establishing a threshold in terms of deuterium uptake to distinguish deuterium labeled and nonlabeled cells. Overall, we show that the Raman-D2O approach, when coupled with chemometrics, constitutes a powerful approach for monitoring single bacterial cells.

Deep learning as phase retrieval tool for CARS spectra.

Finding efficient and reliable methods for the extraction of the phase in optical measurements is challenging and has been widely investigated. Although sophisticated optical settings, e.g. holography, measure directly the phase, the use of algorithmic methods has gained attention due to its efficiency, fast calculation and easy setup requirements. We investigated three phase retrieval methods: the maximum entropy technique (MEM), the Kramers-Kronig relation (KK), and for the first time deep learning using the Long Short-Term Memory network (LSTM). LSTM shows superior results for the phase retrieval problem of coherent anti-Stokes Raman spectra in comparison to MEM and KK.