A Data Augmentation Methodology to Reduce the Class Imbalance in Histopathology Images.

Deep learning techniques have recently yielded remarkable results across various fields. However, the quality of these results depends heavily on the quality and quantity of data used during the training phase. One common issue in multi-class and multi-label classification is class imbalance, where one or several classes make up a substantial portion of the total instances. This imbalance causes the neural network to prioritize features of the majority classes during training, as their detection leads to higher scores. In the context of object detection, two types of imbalance can be identified: (1) an imbalance between the space occupied by the foreground and background and (2) an imbalance in the number of instances for each class. This paper aims to address the second type of imbalance without exacerbating the first. To achieve this, we propose a modification of the copy-paste data augmentation technique, combined with weight-balancing methods in the loss function. This strategy was specifically tailored to improve the performance in datasets with a high instance density, where instance overlap could be detrimental. To validate our methodology, we applied it to a highly unbalanced dataset focused on nuclei detection. The results show that this hybrid approach improves the classification of minority classes without significantly compromising the performance of majority classes.

MMIR: an open-source software for the registration of multimodal histological images.

BACKGROUND: Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality. RESULTS: We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system. CONCLUSION: Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

Label-Free SERS and MD Analysis of Biomarkers for Rapid Point-of-Care Sensors Detecting Head and Neck Cancer and Infections.

For the progress of point-of-care medicine, where individual health status can be easily and quickly monitored using a handheld sensor, saliva serves as one of the best-suited body fluids thanks to its availability and abundance of physiological indicators. Salivary biomarkers, combined with rapid and highly sensitive detection tools, may pave the way to new real-time health monitoring and personalized preventative therapy branches using saliva as a target matrix. Saliva is increasing in importance in liquid biopsy, a non-invasive approach that helps physicians diagnose and characterize specific diseases in patients. Here, we propose a proof-of-concept study combining the unique specificity in biomolecular recognition provided by surface-enhanced Raman spectroscopy (SERS) in combination with molecular dynamics (MD) simulations, which give leave to explore the biomolecular absorption mechanism on nanoparticle surfaces, in order to verify the traceability of two validated salivary indicators, i.e., interleukin-8 (IL-8) and lysozyme (LYZ), implicated in oropharyngeal squamous cell carcinoma (OSCC) and oral infection. This strategy simultaneously assures the detection and interpretation of protein biomarkers in saliva, ultimately opening a new route for the evolution of fast and accurate point-of-care SERS-based sensors of interest in precision medicine diagnostics.

Feasibility studies of multimodal nonlinear endoscopy using multicore fiber bundles for remote scanning from tissue sections to bulk organs.

Here, we report on the development and application of a compact multi-core fiber optical probe for multimodal non-linear imaging, combining the label-free modalities of Coherent Anti-Stokes Raman Scattering, Second Harmonic Generation, and Two-Photon Excited Fluorescence. Probes of this multi-core fiber design avoid moving and voltage-carrying parts at the distal end, thus providing promising improved compatibility with clinical requirements over competing implementations. The performance characteristics of the probe are established using thin cryo-sections and artificial targets before the applicability to clinically relevant samples is evaluated using ex vivo bulk human and porcine intestine tissues. After image reconstruction to counteract the data's inherently pixelated nature, the recorded images show high image quality and morpho-chemical conformity on the tissue level compared to multimodal non-linear images obtained with a laser-scanning microscope using a standard microscope objective. Furthermore, a simple yet effective reconstruction procedure is presented and demonstrated to yield satisfactory results. Finally, a clear pathway for further developments to facilitate a translation of the multimodal fiber probe into real-world clinical evaluation and application is outlined.

Raman Spectroscopy Profiling of Splenic T-Cells in Sepsis and Endotoxemia in Mice.

Sepsis is a life-threatening condition that results from an overwhelming and disproportionate host response to an infection. Currently, the quality and extent of the immune response are evaluated based on clinical symptoms and the concentration of inflammatory biomarkers released or expressed by the immune cells. However, the host response toward sepsis is heterogeneous, and the roles of the individual immune cell types have not been fully conceptualized. During sepsis, the spleen plays a vital role in pathogen clearance, such as bacteria by an antibody response, macrophage bactericidal capacity, and bacterial endotoxin detoxification. This study uses Raman spectroscopy to understand the splenic T-lymphocyte compartment profile changes during bona fide bacterial sepsis versus hyperinflammatory endotoxemia. The Raman spectral analysis showed marked changes in splenocytes of mice subjected to septic peritonitis principally in the DNA region, with minor changes in the amino acids and lipoprotein areas, indicating significant transcriptomic activity during sepsis. Furthermore, splenocytes from mice exposed to endotoxic shock by injection of a high dose of lipopolysaccharide showed significant changes in the protein and lipid profiles, albeit with interindividual variations in inflammation severity. In summary, this study provided experimental evidence for the applicability and informative value of Raman spectroscopy for profiling the immune response in a complex, systemic infection scenario. Importantly, changes within the acute phase of inflammation onset (24 h) were reliably detected, lending support to the concept of early treatment and severity control by extracorporeal Raman profiling of immunocyte signatures.

Towards therapeutic drug monitoring of antibiotic levels - analyzing the pharmacokinetics of levofloxacin using DUV-resonance Raman spectroscopy.

Therapeutic drug monitoring (TDM) plays an important role in clinical practice. Here, pharmacokinetics has a decisive influence on the effective antibiotic concentration during treatment. Moreover, different kinetics exist for different administration forms. Accordingly, adjusting the correct concentration depends, in addition to gender, age, weight, clinical picture, etc., on the dosage form of the antibiotic. This study investigates the capability of deep UV resonance Raman spectroscopy (DUV-RRS) to simulate the pharmacokinetics of fluoroquinolone levofloxacin after two different administration forms (intravenous and oral). Three different pre-processing methods were applied, and the best agreement with the simulation was achieved using the extended multiplicative scatter correction. The resulting spectra were used for partial least squares (PLS) regression and ordinary least squares (OLS) regression. The kinetic parameters were compared with the simulated data, with PLS showing the best performance for intravenous administration and a comparable result to OLS for oral administration, while the errors are smaller. The acquired results show the potential of DUV-RRS in combination with PLS regression as a promising supportive method for TDM.

Ultrafast Spectral Tuning of a Fiber Laser for Time-Encoded Multiplex Coherent Raman Scattering Microscopy.

Coherent Raman scattering microscopy utilizing bioorthogonal tagging approaches like isotope or alkyne labeling allows for a targeted monitoring of spatial distribution and dynamics of small molecules of interest in cells, tissues, and other complex biological matrices. To fully exploit this approach in terms of real-time monitoring of several Raman tags, e.g., to study drug uptake dynamics, extremely fast tunable lasers are needed. Here, we present a laser concept without moving parts and fully electronically controlled for the quasi-simultaneous acquisition of coherent anti-Stokes Raman scattering images at multiple Raman resonances. The laser concept is based on the combination of a low noise and spectrally narrow Fourier domain mode-locked laser seeding a compact four wave mixing-based high-power fiber-based optical parametric amplifier.

Software tools and platforms in Digital Pathology: a review for clinicians and computer scientists.

At the end of the twentieth century, a new technology was developed that allowed an entire tissue section to be scanned on an objective slide. Originally called virtual microscopy, this technology is now known as Whole Slide Imaging (WSI). WSI presents new challenges for reading, visualization, storage, and analysis. For this reason, several technologies have been developed to facilitate the handling of these images. In this paper, we analyze the most widely used technologies in the field of digital pathology, ranging from specialized libraries for the reading of these images to complete platforms that allow reading, visualization, and analysis. Our aim is to provide the reader, whether a pathologist or a computational scientist, with the knowledge to choose the technologies to use for new studies, development, or research.

[Multimodal spectroscopic imaging : A new, powerful tool for intraoperative tumor diagnostics]. / Multimodale spektroskopische Bildgebung : Ein neues leistungsstarkes Werkzeug für die intraoperative Tumordiagnostik.

BACKGROUND: The increasing number of cancer cases requires new imaging approaches for intraoperative tumor characterization. OBJECTIVE: Utilization of new optical/photonic methods in combination with artificial intelligence (AI) approaches to address urgent challenges in clinical pathology in terms of intraoperative computational spectral histopathology. METHODS: Multimodal nonlinear imaging by combining the spectroscopic methods coherent anti-Stokes Raman scattering (CARS), two-photon excited autofluorescence (TPEF), fluorescence lifetime imaging microscopy (FLIM), and second harmonic generation (SHG). RESULTS: By using multimodal spectroscopic imaging, tissue morphochemistry, i.e., its morphology and molecular structure can be visualized in a label-free manner. The multimodal images can be automatically analyzed using AI-based image analysis approaches. For clinical application in terms of frozen section diagnostics or in vivo use, the presented multimodal imaging approach can be translated into a compact microscope or endoscopic probe concepts. CONCLUSIONS: The synergistic combination of spectroscopic imaging modalities in combination with automated data analysis has great potential for fast and precise tumor diagnostics e.g., in terms of precise surgical guidance in laser or robotic surgery. Overall, intraoperative multimodal spectroscopic imaging may represent an innovative advancement for tumor diagnostics in the future, directly leading to improved patient care and significant cost savings.

Identification of inflammatory markers in eosinophilic cells of the immune system: fluorescence, Raman and CARS imaging can recognize markers but differently.

Eosinophils (Eos) play an important role in the immune system's response releasing several inflammatory factors and contributing to allergic rhinitis, asthma, or atopic dermatitis. Since Eos have a relatively short lifetime after isolation from blood, usually eosinophilic cell line (EoL-1) is used to study mechanisms of their activation and to test therapies. In particular, EoL-1 cells are examined in terms of signalling pathways of the inflammatory response manifested by the presence of lipid bodies (LBs). Here we examined the differences in response to inflammation modelled by various factors, between isolated human eosinophils and EoL-1 cells, as manifested in the number and chemical composition of LBs. The analysis was performed using fluorescence, Raman, and coherent anti-Stokes Raman scattering (CARS) microscopy, which recognised the inflammatory process in the cells, but it is manifested slightly differently depending on the method used. We showed that unstimulated EoL-1 cells, compared to isolated eosinophils, contained more LBs, displayed different nucleus morphology and did not have eosinophilic peroxidase (EPO). In EoL-1 cells stimulated with various proinflammatory agents, including butyric acid (BA), liposaccharide (LPS), or cytokines (IL-1ß, TNF-α), an increased production of LBs with a various degree of lipid unsaturation was observed in spontaneous Raman spectra. Furthermore, stimulation of EoL-1 cells resulted in alterations of the LBs morphology. In conclusion, a level of lipid unsaturation and eosinophilic peroxidase as well as LBs distribution among cell population mainly accounted for the biochemistry of eosinophils upon inflammation.

Multimodal nonlinear endomicroscopic imaging probe using a double-core double-clad fiber and focus-combining micro-optical concept.

Multimodal non-linear microscopy combining coherent anti-Stokes Raman scattering, second harmonic generation, and two-photon excited fluorescence has proved to be a versatile and powerful tool enabling the label-free investigation of tissue structure, molecular composition, and correlation with function and disease status. For a routine medical application, the implementation of this approach into an in vivo imaging endoscope is required. However, this is a difficult task due to the requirements of a multicolour ultrashort laser delivery from a compact and robust laser source through a fiber with low losses and temporal synchronization, the efficient signal collection in epi-direction, the need for small-diameter but highly corrected endomicroobjectives of high numerical aperture and compact scanners. Here, we introduce an ultra-compact fiber-scanning endoscope platform for multimodal non-linear endomicroscopy in combination with a compact four-wave mixing based fiber laser. The heart of this fiber-scanning endoscope is an in-house custom-designed, single mode, double clad, double core pure silica fiber in combination with a 2.4 mm diameter NIR-dual-waveband corrected endomicroscopic objective of 0.55 numerical aperture and 180 µm field of view for non-linear imaging, allowing a background free, low-loss, high peak power laser delivery, and an efficient signal collection in backward direction. A linear diffractive optical grating overlays pump and Stokes laser foci across the full field of view, such that diffraction-limited performance is demonstrated for tissue imaging at one frame per second with sub-micron spatial resolution and at a high transmission of 65% from the laser to the specimen using a distal resonant fiber scanner.

Hydrogen and C2-C6 Alkane Sensing in Complex Fuel Gas Mixtures with Fiber-Enhanced Raman Spectroscopy.

Power-to-gas is a heavily discussed option to store surplus electricity from renewable sources. Part of the generated hydrogen could be fed into the gas grid and lead to fluctuations in the composition of the fuel gas. Consequently, both operators of transmission networks and end users would need to frequently monitor the gas to ensure safety as well as optimal and stable operation. Currently, gas chromatography-based analysis methods are the state of the art. However, these methods have several downsides for time-resolved and distributed application and Raman gas spectroscopy is favorable for future point-of-use monitoring. Here, we demonstrate that fiber-enhanced Raman gas spectroscopy (FERS) enables the simultaneous detection of all relevant gases, from major (methane, CH4; hydrogen, H2) to minor (C2-C6 alkanes) fuel gas components. The characteristic peaks of H2 (585 cm-1), CH4 (2917 cm-1), isopentane (765 cm-1), i-butane (798 cm-1), n-butane (830 cm-1), n-pentane (840 cm-1), propane (869 cm-1), ethane (993 cm-1), and n-hexane (1038 cm-1) are well resolved in the broadband spectra acquired with a compact spectrometer. The fiber enhancement achieved in a hollow-core antiresonant fiber enables highly sensitive measurements with limits of detection between 90 and 180 ppm for different hydrocarbons. Both methane and hydrogen were quantified with high accuracy with average relative errors of 1.1% for CH4 and 1.5% for H2 over a wide concentration range. These results show that FERS is ideally suited for comprehensive fuel gas analysis in a future, where regenerative sources lead to fluctuations in the composition of gas.

Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning.

Hematoxylin and Eosin (H&E) staining is the 'gold-standard' method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for 'real-time' disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author's best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.

In vivo coherent anti-Stokes Raman scattering microscopy reveals vitamin A distribution in the liver.

Here we present a microscope setup for coherent anti-Stokes Raman scattering (CARS) imaging, devised to specifically address the challenges of in vivo experiments. We exemplify its capabilities by demonstrating how CARS microscopy can be used to identify vitamin A (VA) accumulations in the liver of a living mouse, marking the positions of hepatic stellate cells (HSCs). HSCs are the main source of extracellular matrix protein after hepatic injury and are therefore the main target of novel nanomedical strategies in the development of a treatment for liver fibrosis. Their role in the VA metabolism makes them an ideal target for a CARS-based approach as they store most of the body's VA, a class of compounds sharing a retinyl group as a structural motive, a moiety that is well known for its exceptionally high Raman cross section of the C═C stretching vibration of the conjugated backbone.

Nondestructive molecular imaging by Raman spectroscopy vs. marker detection by MALDI IMS for an early diagnosis of HCC.

Hepatocellular carcinoma (HCC) is a leading cause of cancer-related deaths worldwide with a steadily increasing mortality rate. Fast diagnosis at early stages of HCC is of key importance for the improvement of patient survival rates. In this regard, we combined two imaging techniques with high potential for HCC diagnosis in order to improve the prediction of liver cancer. In detail, Raman spectroscopic imaging and matrix-assisted laser desorption ionization imaging mass spectrometry (MALDI IMS) were applied for the diagnosis of 36 HCC tissue samples. The data were analyzed using multivariate methods, and the results revealed that Raman spectroscopy alone showed a good capability for HCC tumor identification (sensitivity of 88% and specificity of 80%), which could not be improved by combining the Raman data with MALDI IMS. In addition, it could be shown that the two methods in combination can differentiate between well-, moderately- and poorly-differentiated HCC using a linear classification model. MALDI IMS not only classified the HCC grades with a sensitivity of 100% and a specificity of 80%, but also showed significant differences in the expression of glycerophospholipids and fatty acyls during HCC differentiation. Furthermore, important differences in the protein, lipid and collagen compositions of differentiated HCC were detected using the model coefficients of a Raman based classification model. Both Raman and MALDI IMS, as well as their combination showed high potential for resolving concrete questions in liver cancer diagnosis.

Towards an Interpretable Classifier for Characterization of Endoscopic Mayo Scores in Ulcerative Colitis Using Raman Spectroscopy.

Ulcerative colitis (UC) is one of the main types of chronic inflammatory diseases that affect the bowel, but its pathogenesis is yet to be completely defined. Assessing the disease activity of UC is vital for developing a personalized treatment. Conventionally, the assessment of UC is performed by colonoscopy and histopathology. However, conventional methods fail to retain biomolecular information associated to the severity of UC and are solely based on morphological characteristics of the inflamed colon. Furthermore, assessing endoscopic disease severity is limited by the requirement for experienced human reviewers. Therefore, this work presents a nondestructive biospectroscopic technique, for example, Raman spectroscopy, for assessing endoscopic disease severity according to the four-level Mayo subscore. This contribution utilizes multidimensional Raman spectroscopic data to generate a predictive model for identifying colonic inflammation. The predictive modeling of the Raman spectroscopic data is performed using a one-dimensional deep convolutional neural network (1D-CNN). The classification results of 1D-CNN achieved a mean sensitivity of 78% and a mean specificity of 93% for the four Mayo endoscopic scores. Furthermore, the results of the 1D-CNN are interpreted by a first-order Taylor expansion, which extracts the Raman bands important for classification. Additionally, a regression model of the 1D-CNN model is constructed to study the extent of misclassification and border-line patients. The overall results of Raman spectroscopy with 1D-CNN as a classification and regression model show a good performance, and such a method can serve as a complementary method for UC analysis.

Fiber-Enhanced Raman Gas Spectroscopy for the Study of Microbial Methanogenesis.

Microbial methanogenesis is a key biogeochemical process in the carbon cycle that is responsible for 70% of global emissions of the potent greenhouse gas methane (CH4). Further knowledge about microbial methanogenesis is crucial to mitigate emissions, increase climate model accuracy, or advance methanogenic biogas production. The current understanding of the substrate use of methanogenic microbes is limited, especially regarding the methylotrophic pathway. Here, we present fiber-enhanced Raman spectroscopy (FERS) of headspace gases as an alternate tool to study methanogenesis and substrate use in particular. The optical technique is nondestructive and sensitive to CH4, hydrogen (H2), and carbon dioxide with a large dynamic range from trace levels (demonstrated LoDs: CH4, 3 ppm; H2, 49 ppm) to pure gases. In addition, the portable FERS system can provide quantitative information about methanol concentration in the liquid phase of microbial cultures through headspace gas sampling (LoD 25 ppm). We demonstrate how FERS gas sensing could enable us to track substrate and product levels of microbial methanogenesis with just one instrument. The versatility of Raman gas spectroscopy could moreover help us to elucidate links between nitrogen and carbon cycle in microbial communities in the near future.

Deep learning as phase retrieval tool for CARS spectra.

Finding efficient and reliable methods for the extraction of the phase in optical measurements is challenging and has been widely investigated. Although sophisticated optical settings, e.g. holography, measure directly the phase, the use of algorithmic methods has gained attention due to its efficiency, fast calculation and easy setup requirements. We investigated three phase retrieval methods: the maximum entropy technique (MEM), the Kramers-Kronig relation (KK), and for the first time deep learning using the Long Short-Term Memory network (LSTM). LSTM shows superior results for the phase retrieval problem of coherent anti-Stokes Raman spectra in comparison to MEM and KK.