Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color-near-infrared sensors.

Significance: Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. Aim: We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. Approach: A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Results: Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. Conclusions: We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.

Influence of signal-to-noise ratio on DoLP and AoP measurements during reflectance-mode division-of-focal plane Stokes polarimetry of biological tissues.

Stokes polarimeter based endoscopes are emerging as an area of technology where polarization imaging can greatly impact clinical care by improving diagnostic tools without the use of exogenous contrast. Image acquisition in minimally invasive surgical settings is often beset by inherently limited illumination. A comprehensive analysis of how signal-to-noise (SNR) propagates through Stokes polarimetric outcomes such as degree of linear polarization (DoLP) and angle of polarization (AoP) in low light is important for future interpretation of data acquired in low-light conditions. A previously developed theoretical model of quantitative polarized light imaging (QPLI) analysis described SNR as a function of both incident light intensity and DoLP. When polarized light interacts with biological tissues, the resultant DoLP of exiting light is dependent on the underlying tissue microstructure. Therefore, in this study we explore how low light impacts SNR of QPLI outcomes of DoLP and AoP differently in tissue phantoms of varying microstructures. Data are compared to theoretical solutions of SNR of DoLP and AoP. Tissues were additionally loaded to varying magnitudes of strain to investigate how variable SNR affects the ability to discern dynamic realignment in biological tissues. We observed a high degree of congruency between experimental and theoretical data, with SNR depending on both light intensity and DoLP. Additionally, we found that AoP may have a greater resilience to noise overall than DoLP and, as such, may be particularly useful in conditions where light is inherently limited.

Learning a global underwater geolocalization model with sectoral transformer.

Polarization-based underwater geolocalization presents an innovative method for positioning unmanned autonomous devices beneath the water surface, in environments where GPS signals are ineffective. While the state-of-the-art deep neural network (DNN) method achieves high-precision geolocalization based on sun polarization patterns in same-site tasks, its learning-based nature limits its generalizability to unseen sites and subsequently impairs its performance on cross-site tasks, where an unavoidable domain gap between training and test data exists. In this paper, we present an advanced Deep Neural Network (DNN) methodology, which includes a neural network built on a Transformer architecture, similar to the core of large language models such as ChatGPT, and integrates an unscented Kalman filter (UKF) for estimating underwater geolocation using polarization-based images. This combination effectively simulates the sun's daily trajectory, yielding enhanced performance across different locations and quicker inference speeds compared to current benchmarks. Following thorough analysis of over 10 million polarization images from four global locations, we conclude that our proposed technique significantly boosts cross-site geolocalization accuracy by around 28% when contrasted with traditional DNN methods.

Biomimetic-Membrane-Protected Plasmonic Nanostructures as Dual-Modality Contrast Agents for Correlated Surface-Enhanced Raman Scattering and Photoacoustic Detection of Hidden Tumor Lesions.

Optical imaging and spectroscopic modalities are of considerable current interest for in vivo cancer detection and image-guided surgery, but the turbid or scattering nature of biomedical tissues has severely limited their abilities to detect buried or occluded tumor lesions. Here we report the development of a dual-modality plasmonic nanostructure based on colloidal gold nanostars (AuNSs) for simultaneous surface-enhanced Raman scattering (SERS) and photoacoustic (PA) detection of tumor phantoms embedded (hidden) in ex vivo animal tissues. By using red blood cell membranes as a naturally derived biomimetic coating, we show that this class of dual-modality contrast agents can provide both Raman spectroscopic and PA signals for the detection and differentiation of hidden solid tumors with greatly improved depths of tissue penetration. Compared to previous polymer-coated AuNSs, the biomimetic coatings are also able to minimize protein adsorption and cellular uptake when exposed to human plasma without compromising their SERS or PA signals. We further show that tumor-targeting peptides (such as cyclic RGD) can be noncovalently inserted for targeting the ανß3-integrin receptors expressed on metastatic cancer cells and tracked via both SERS and PA imaging (PAI). Finally, we demonstrate image-guided resections of tumor-mimicking phantoms comprising metastatic tumor cells buried under layers of skin and fat tissues (6 mm in thickness). Specifically, PAI was used to determine the precise tumor location, while SERS spectroscopic signals were used for tumor identification and differentiation. This work opens the possibility of using these biomimetic dual-modality nanoparticles with superior signal and biological stability for intraoperative cancer detection and resection.