MMIR: an open-source software for the registration of multimodal histological images.

BACKGROUND: Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality. RESULTS: We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system. CONCLUSION: Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

A Data Augmentation Methodology to Reduce the Class Imbalance in Histopathology Images.

Deep learning techniques have recently yielded remarkable results across various fields. However, the quality of these results depends heavily on the quality and quantity of data used during the training phase. One common issue in multi-class and multi-label classification is class imbalance, where one or several classes make up a substantial portion of the total instances. This imbalance causes the neural network to prioritize features of the majority classes during training, as their detection leads to higher scores. In the context of object detection, two types of imbalance can be identified: (1) an imbalance between the space occupied by the foreground and background and (2) an imbalance in the number of instances for each class. This paper aims to address the second type of imbalance without exacerbating the first. To achieve this, we propose a modification of the copy-paste data augmentation technique, combined with weight-balancing methods in the loss function. This strategy was specifically tailored to improve the performance in datasets with a high instance density, where instance overlap could be detrimental. To validate our methodology, we applied it to a highly unbalanced dataset focused on nuclei detection. The results show that this hybrid approach improves the classification of minority classes without significantly compromising the performance of majority classes.

Siamese Networks for Clinically Relevant Bacteria Classification Based on Raman Spectroscopy.

Identifying bacterial strains is essential in microbiology for various practical applications, such as disease diagnosis and quality monitoring of food and water. Classical machine learning algorithms have been utilized to identify bacteria based on their Raman spectra. However, convolutional neural networks (CNNs) offer higher classification accuracy, but they require extensive training sets and retraining of previous untrained class targets can be costly and time-consuming. Siamese networks have emerged as a promising solution. They are composed of two CNNs with the same structure and a final network that acts as a distance metric, converting the classification problem into a similarity problem. Classical machine learning approaches, shallow and deep CNNs, and two Siamese network variants were tailored and tested on Raman spectral datasets of bacteria. The methods were evaluated based on mean sensitivity, training time, prediction time, and the number of parameters. In this comparison, Siamese-model2 achieved the highest mean sensitivity of 83.61 ± 4.73 and demonstrated remarkable performance in handling unbalanced and limited data scenarios, achieving a prediction accuracy of 73%. Therefore, the choice of model depends on the specific trade-off between accuracy, (prediction/training) time, and resources for the particular application. Classical machine learning models and shallow CNN models may be more suitable if time and computational resources are a concern. Siamese networks are a good choice for small datasets and CNN for extensive data.

Illuminating the Tiny World: A Navigation Guide for Proper Raman Studies on Microorganisms.

Raman spectroscopy is an emerging method for the identification of bacteria. Nevertheless, a lot of different parameters need to be considered to establish a reliable database capable of identifying real-world samples such as medical or environmental probes. In this review, the establishment of such reliable databases with the proper design in microbiological Raman studies is demonstrated, shining a light into all the parts that require attention. Aspects such as the strain selection, sample preparation and isolation requirements, the phenotypic influence, measurement strategies, as well as the statistical approaches for discrimination of bacteria, are presented. Furthermore, the influence of these aspects on spectra quality, result accuracy, and read-out are discussed. The aim of this review is to serve as a guide for the design of microbiological Raman studies that can support the establishment of this method in different fields.

Exploring the Steps of Infrared (IR) Spectral Analysis: Pre-Processing, (Classical) Data Modelling, and Deep Learning.

Infrared (IR) spectroscopy has greatly improved the ability to study biomedical samples because IR spectroscopy measures how molecules interact with infrared light, providing a measurement of the vibrational states of the molecules. Therefore, the resulting IR spectrum provides a unique vibrational fingerprint of the sample. This characteristic makes IR spectroscopy an invaluable and versatile technology for detecting a wide variety of chemicals and is widely used in biological, chemical, and medical scenarios. These include, but are not limited to, micro-organism identification, clinical diagnosis, and explosive detection. However, IR spectroscopy is susceptible to various interfering factors such as scattering, reflection, and interference, which manifest themselves as baseline, band distortion, and intensity changes in the measured IR spectra. Combined with the absorption information of the molecules of interest, these interferences prevent direct data interpretation based on the Beer-Lambert law. Instead, more advanced data analysis approaches, particularly artificial intelligence (AI)-based algorithms, are required to remove the interfering contributions and, more importantly, to translate the spectral signals into high-level biological/chemical information. This leads to the tasks of spectral pre-processing and data modeling, the main topics of this review. In particular, we will discuss recent developments in both tasks from the perspectives of classical machine learning and deep learning.

Impact of thermal annealing and laser treatment on the morphology and optical responses of mono- and bi-metallic plasmonic honeycomb lattice.

Plasmonic nanoparticle arrays with a specific lattice arrangement can support surface lattice resonances (SLRs). SLR exhibits a sharp spectral peak and finds many applications including optical sensing and plasmonic lasers. To optimize SLR for application, a robust method that allows the mass production of plasmonic nanoparticle arrays with refined particle morphology and well-defined lattice arrangement is required. In this work, we combine nanosphere lithography (NSL) with thermal annealing or nanosecond-pulsed laser treatment to refine plasmonic nanoparticles in a honeycomb lattice. We comparatively study the effects of the two treatment methods on the particle morphology and lattice arrangement of mono (Ag and Pd) and bi-metallic (Ag-Pd) nanoparticle lattices. In general, thermal annealing preserves the lattice arrangement but fairly changes the particle roundness, while laser treatment produces particles with varying morphologies and spatial distribution. We also theoretically and experimentally investigate the optical responses of Ag nanoparticle lattices produced by different treatment methods. The observed difference in spectra can be attributed to the varying particle morphology, which shifts the localized surface plasmon resonance differently, resulting in a significant change in SLR. These findings provide valuable insights for optimizing plasmonic nanoparticle arrays for various applications.

Feasibility studies of multimodal nonlinear endoscopy using multicore fiber bundles for remote scanning from tissue sections to bulk organs.

Here, we report on the development and application of a compact multi-core fiber optical probe for multimodal non-linear imaging, combining the label-free modalities of Coherent Anti-Stokes Raman Scattering, Second Harmonic Generation, and Two-Photon Excited Fluorescence. Probes of this multi-core fiber design avoid moving and voltage-carrying parts at the distal end, thus providing promising improved compatibility with clinical requirements over competing implementations. The performance characteristics of the probe are established using thin cryo-sections and artificial targets before the applicability to clinically relevant samples is evaluated using ex vivo bulk human and porcine intestine tissues. After image reconstruction to counteract the data's inherently pixelated nature, the recorded images show high image quality and morpho-chemical conformity on the tissue level compared to multimodal non-linear images obtained with a laser-scanning microscope using a standard microscope objective. Furthermore, a simple yet effective reconstruction procedure is presented and demonstrated to yield satisfactory results. Finally, a clear pathway for further developments to facilitate a translation of the multimodal fiber probe into real-world clinical evaluation and application is outlined.

Correction for Extrinsic Background in Raman Hyperspectral Images.

Raman hyperspectral microscopy is a valuable tool in biological and biomedical imaging. Because Raman scattering is often weak in comparison to other phenomena, prevalent spectral fluctuations and contaminations have brought advancements in analytical and chemometric methods for Raman spectra. These chemometric advances have been key contributors to the applicability of Raman imaging to biological systems. As studies increase in scale, spectral contamination from extrinsic background, intensity from sources such as the optical components that are extrinsic to the sample of interest, has become an emerging issue. Although existing baseline correction schemes often reduce intrinsic background such as autofluorescence originating from the sample of interest, extrinsic background is not explicitly considered, and these methods often fail to reduce its effects. Here, we show that extrinsic background can significantly affect a classification model using Raman images, yielding misleadingly high accuracies in the distinction of benign and malignant samples of follicular thyroid cell lines. To mitigate its effects, we develop extrinsic background correction (EBC) and demonstrate its use in combination with existing methods on Raman hyperspectral images. EBC isolates regions containing the smallest amounts of sample materials that retain extrinsic contributions that are specific to the device or environment. We perform classification both with and without the use of EBC, and we find that EBC retains biological characteristics in the spectra while significantly reducing extrinsic background. As the methodology used in EBC is not specific to Raman spectra, correction of extrinsic effects in other types of hyperspectral and grayscale images is also possible.

Differentiability of cell types enhanced by detrending a non-homogeneous pattern in a line-illumination Raman microscope.

A line illumination Raman microscope extracts the underlying spatial and spectral information of a sample, typically a few hundred times faster than raster scanning. This makes it possible to measure a wide range of biological samples such as cells and tissues - that only allow modest intensity illumination to prevent potential damage - within feasible time frame. However, a non-uniform intensity distribution of laser line illumination may induce some artifacts in the data and lower the accuracy of machine learning models trained to predict sample class membership. Here, using cancerous and normal human thyroid follicular epithelial cell lines, FTC-133 and Nthy-ori 3-1 lines, whose Raman spectral difference is not so large, we show that the standard pre-processing of spectral analyses widely used for raster scanning microscopes introduced some artifacts. To address this issue, we proposed a detrending scheme based on random forest regression, a nonparametric model-free machine learning algorithm, combined with a position-dependent wavenumber calibration scheme along the illumination line. It was shown that the detrending scheme minimizes the artifactual biases arising from non-uniform laser sources and significantly enhances the differentiability of the sample states, i.e., cancerous or normal epithelial cells, compared to the standard pre-processing scheme.

Comparison of denoising tools for the reconstruction of nonlinear multimodal images.

Biophotonic multimodal imaging techniques provide deep insights into biological samples such as cells or tissues. However, the measurement time increases dramatically when high-resolution multimodal images (MM) are required. To address this challenge, mathematical methods can be used to shorten the acquisition time for such high-quality images. In this research, we compared standard methods, e.g., the median filter method and the phase retrieval method via the Gerchberg-Saxton algorithm with artificial intelligence (AI) based methods using MM images of head and neck tissues. The AI methods include two approaches: the first one is a transfer learning-based technique that uses the pre-trained network DnCNN. The second approach is the training of networks using augmented head and neck MM images. In this manner, we compared the Noise2Noise network, the MIRNet network, and our deep learning network namely incSRCNN, which is derived from the super-resolution convolutional neural network and inspired by the inception network. These methods reconstruct improved images using measured low-quality (LQ) images, which were measured in approximately 2 seconds. The evaluation was performed on artificial LQ images generated by degrading high-quality (HQ) images measured in 8 seconds using Poisson noise. The results showed the potential of using deep learning on these multimodal images to improve the data quality and reduce the acquisition time. Our proposed network has the advantage of having a simple architecture compared with similar-performing but highly parametrized networks DnCNN, MIRNet, and Noise2Noise.

Investigating the effect of different pre-treatment methods on Raman spectra recorded with different excitation wavelengths.

Raman reference libraries can be used for identification of components in unknown samples as Raman spectroscopy offers fingerprint information of the measured samples. Since Raman libraries often contain many different and/or highly similar spectra, it is important that the spectra are a reliable fingerprint for each compound. However, Raman spectra are highly sensitive to the experimental conditions, and the Raman spectra will change in different conditions even though the same sample is measured. Raman data pre-treatment minimizes the differences between Raman spectra arising from different experimental conditions. In this study, different combinations of pre-treatment methods are used to quantify the effect of each pre-treatment step in minimizing the differences between Raman spectra of the same sample in different experimental conditions, e.g., different excitation wavelengths. These different pre-treatment processes are evaluated for six solvents. The spectra differences between spectra recorded with three excitation wavelengths (532 nm, 633 nm, and 830 nm) are evaluated by angular difference index and the influence on a classification model is tested. The angular difference index of each spectrum after every data pre-treatment step shows a decreasing behavior. It could be demonstrated that wavenumber calibration has the largest effect on the differences between the Raman spectra. However, ω4 correction doesn't have a significate effect in this dataset. The classification results show that the prediction accuracy is improving by doing data pre-treatment. In the dataset obtained in 633 nm a lower amount of pre-treatment steps is needed but in the dataset 830 nm more pre-treatment steps are needed for a high accuracy. The result shows that the choice of an optimal pre-treatment method or combination of methods strongly influences the analysis results, but is far from straightforward, since it depends on the characteristics of the data set and the goal of data analysis.

Identification of bacteria in mixed infection from urinary tract of patient's samples using Raman analysis of dried droplets.

Urinary tract infections (UTI) are among the most frequent nosocomial infections. A fast identification of the pathogen and assignment of Gram type could help to prescribe most suitable treatments. Raman spectroscopy holds high potential for fast and reliable bacterial pathogens identification. While most studies so far have focused on individual pathogens or artificial mixtures, this contribution aims to translate the analysis to primary urine samples from patients with suspected UTIs. For this, we have included 59 primary urine samples out of which 29 were diagnosed as mixed infections. For Raman analysis, we first trained two classification models based on principal component analysis - linear discriminant analysis (PCA-LDA) with more than 3500 Raman spectra of 85 clinical isolates from 23 species in order to (1) identify the Gram type of the bacteria and (2) assign family membership to one of the six most abundant bacterial families in urinary tract infections (Enterobacteriaceae, Morganellaceae, Pseudomonadaceae, Enterococcaceae, Staphylococcaceae and Streptococcaceae). The classification models were applied to artificial mixtures of Gram positive and Gram negative bacteria to correctly predict mixed infections with an accuracy of 75%. Raman scans of dried droplets did not yet yield optimal classification results on family level. When translating the method to primary urine samples, we observed a strong bias towards Gram negative bacteria, on family level towards Morganellaceae, which reduced prediction accuracy. Spectral differences were observed between isolates grown on standard growth medium and bacteria of the same strain when characterized directly from the patient. Thus, improvement of the classification accuracy is expected with a larger data base containing also bacteria measured directly from the urine sample.

Simple, Fast and Convenient Magnetic Bead-Based Sample Preparation for Detecting Viruses via Raman-Spectroscopy.

We introduce a magnetic bead-based sample preparation scheme for enabling the Raman spectroscopic differentiation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2)-positive and -negative samples. The beads were functionalized with the angiotensin-converting enzyme 2 (ACE2) receptor protein, which is used as a recognition element to selectively enrich SARS-CoV-2 on the surface of the magnetic beads. The subsequent Raman measurements directly enable discriminating SARS-CoV-2-positive and -negative samples. The proposed approach is also applicable for other virus species when the specific recognition element is exchanged. A series of Raman spectra were measured on three types of samples, namely SARS-CoV-2, Influenza A H1N1 virus and a negative control. For each sample type, eight independent replicates were considered. All of the spectra are dominated by the magnetic bead substrate and no obvious differences between the sample types are apparent. In order to address the subtle differences in the spectra, we calculated different correlation coefficients, namely the Pearson coefficient and the Normalized cross correlation coefficient. By comparing the correlation with the negative control, differentiating between SARS-CoV-2 and Influenza A virus is possible. This study provides a first step towards the detection and potential classification of different viruses with the use of conventional Raman spectroscopy.

Artificial intelligence for the classification of focal liver lesions in ultrasound - a systematic review.

Focal liver lesions are detected in about 15% of abdominal ultrasound examinations. The diagnosis of frequent benign lesions can be determined reliably based on the characteristic B-mode appearance of cysts, hemangiomas, or typical focal fatty changes. In the case of focal liver lesions which remain unclear on B-mode ultrasound, contrast-enhanced ultrasound (CEUS) increases diagnostic accuracy for the distinction between benign and malignant liver lesions. Artificial intelligence describes applications that try to emulate human intelligence, at least in subfields such as the classification of images. Since ultrasound is considered to be a particularly examiner-dependent technique, the application of artificial intelligence could be an interesting approach for an objective and accurate diagnosis. In this systematic review we analyzed how artificial intelligence can be used to classify the benign or malignant nature and entity of focal liver lesions on the basis of B-mode or CEUS data. In a structured search on Scopus, Web of Science, PubMed, and IEEE, we found 52 studies that met the inclusion criteria. Studies showed good diagnostic performance for both the classification as benign or malignant and the differentiation of individual tumor entities. The results could be improved by inclusion of clinical parameters and were comparable to those of experienced investigators in terms of diagnostic accuracy. However, due to the limited spectrum of lesions included in the studies and a lack of independent validation cohorts, the transfer of the results into clinical practice is limited.

Use of polymers as wavenumber calibration standards in deep-UVRR.

Deep-UV resonance Raman spectroscopy (UVRR) allows the classification of bacterial species with high accuracy and is a promising tool to be developed for clinical application. For this attempt, the optimization of the wavenumber calibration is required to correct the overtime changes of the Raman setup. In the present study, different polymers were investigated as potential calibration agents. The ones with many sharp bands within the spectral range 400-1900 cm-1 were selected and used for wavenumber calibration of bacterial spectra. Classification models were built using a training cross-validation dataset that was then evaluated with an independent test dataset obtained after 4 months. Without calibration, the training cross-validation dataset provided an accuracy for differentiation above 99 % that dropped to 51.2 % after test evaluation. Applying the test evaluation with PET and Teflon calibration allowed correct assignment of all spectra of Gram-positive isolates. Calibration with PS and PEI leads to misclassifications that could be overcome with majority voting. Concerning the very closely related and similar in genome and cell biochemistry Enterobacteriaceae species, all spectra of the training cross-validation dataset were correctly classified but were misclassified in test evaluation. These results show the importance of selecting the most suitable calibration agent in the classification of bacterial species and help in the optimization of the deep-UVRR technique.

A Review on Data Fusion of Multidimensional Medical and Biomedical Data.

Data fusion aims to provide a more accurate description of a sample than any one source of data alone. At the same time, data fusion minimizes the uncertainty of the results by combining data from multiple sources. Both aim to improve the characterization of samples and might improve clinical diagnosis and prognosis. In this paper, we present an overview of the advances achieved over the last decades in data fusion approaches in the context of the medical and biomedical fields. We collected approaches for interpreting multiple sources of data in different combinations: image to image, image to biomarker, spectra to image, spectra to spectra, spectra to biomarker, and others. We found that the most prevalent combination is the image-to-image fusion and that most data fusion approaches were applied together with deep learning or machine learning methods.

Software tools and platforms in Digital Pathology: a review for clinicians and computer scientists.

At the end of the twentieth century, a new technology was developed that allowed an entire tissue section to be scanned on an objective slide. Originally called virtual microscopy, this technology is now known as Whole Slide Imaging (WSI). WSI presents new challenges for reading, visualization, storage, and analysis. For this reason, several technologies have been developed to facilitate the handling of these images. In this paper, we analyze the most widely used technologies in the field of digital pathology, ranging from specialized libraries for the reading of these images to complete platforms that allow reading, visualization, and analysis. Our aim is to provide the reader, whether a pathologist or a computational scientist, with the knowledge to choose the technologies to use for new studies, development, or research.

Raman Spectroscopic Differentiation of Streptococcus pneumoniae From Other Streptococci Using Laboratory Strains and Clinical Isolates.

Streptococcus pneumoniae, commonly referred to as pneumococci, can cause severe and invasive infections, which are major causes of communicable disease morbidity and mortality in Europe and globally. The differentiation of S. pneumoniae from other Streptococcus species, especially from other oral streptococci, has proved to be particularly difficult and tedious. In this work, we evaluate if Raman spectroscopy holds potential for a reliable differentiation of S. pneumoniae from other streptococci. Raman spectra of eight different S. pneumoniae strains and four other Streptococcus species (S. sanguinis, S. thermophilus, S. dysgalactiae, S. pyogenes) were recorded and their spectral features analyzed. Together with Raman spectra of 59 Streptococcus patient isolates, they were used to train and optimize binary classification models (PLS-DA). The effect of normalization on the model accuracy was compared, as one example for optimization potential for future modelling. Optimized models were used to identify S. pneumoniae from other streptococci in an independent, previously unknown data set of 28 patient isolates. For this small data set balanced accuracy of around 70% could be achieved. Improvement of the classification rate is expected with optimized model parameters and algorithms as well as with a larger spectral data base for training.

Comparison of Different Label-Free Raman Spectroscopy Approaches for the Discrimination of Clinical MRSA and MSSA Isolates.

Methicillin-resistant Staphylococcus aureus (MRSA) is classified as one of the priority pathogens that threaten human health. Resistance detection with conventional microbiological methods takes several days, forcing physicians to administer empirical antimicrobial treatment that is not always appropriate. A need exists for a rapid, accurate, and cost-effective method that allows targeted antimicrobial therapy in limited time. In this pilot study, we investigate the efficacy of three different label-free Raman spectroscopic approaches to differentiate methicillin-resistant and -susceptible clinical isolates of S. aureus (MSSA). Single-cell analysis using 532 nm excitation was shown to be the most suitable approach since it captures information on the overall biochemical composition of the bacteria, predicting 87.5% of the strains correctly. UV resonance Raman microspectroscopy provided a balanced accuracy of 62.5% and was not sensitive enough in discriminating MRSA from MSSA. Excitation of 785 nm directly on the petri dish provided a balanced accuracy of 87.5%. However, the difference between the strains was derived from the dominant staphyloxanthin bands in the MRSA, a cell component not associated with the presence of methicillin resistance. This is the first step toward the development of label-free Raman spectroscopy for the discrimination of MRSA and MSSA using single-cell analysis with 532 nm excitation. IMPORTANCE Label-free Raman spectra capture the high chemical complexity of bacterial cells. Many different Raman approaches have been developed using different excitation wavelength and cell analysis methods. This study highlights the major importance of selecting the most suitable Raman approach, capable of providing spectral features that can be associated with the cell mechanism under investigation. It is shown that the approach of choice for differentiating MRSA from MSSA should be single-cell analysis with 532 nm excitation since it captures the difference in the overall biochemical composition. These results should be taken into consideration in future studies aiming for the development of label-free Raman spectroscopy as a clinical analytical tool for antimicrobial resistance determination.