Learning a global underwater geolocalization model with sectoral transformer.

Polarization-based underwater geolocalization presents an innovative method for positioning unmanned autonomous devices beneath the water surface, in environments where GPS signals are ineffective. While the state-of-the-art deep neural network (DNN) method achieves high-precision geolocalization based on sun polarization patterns in same-site tasks, its learning-based nature limits its generalizability to unseen sites and subsequently impairs its performance on cross-site tasks, where an unavoidable domain gap between training and test data exists. In this paper, we present an advanced Deep Neural Network (DNN) methodology, which includes a neural network built on a Transformer architecture, similar to the core of large language models such as ChatGPT, and integrates an unscented Kalman filter (UKF) for estimating underwater geolocation using polarization-based images. This combination effectively simulates the sun's daily trajectory, yielding enhanced performance across different locations and quicker inference speeds compared to current benchmarks. Following thorough analysis of over 10 million polarization images from four global locations, we conclude that our proposed technique significantly boosts cross-site geolocalization accuracy by around 28% when contrasted with traditional DNN methods.

Angle of polarization calibration for omnidirectional polarization cameras.

Polarization cameras quantify one of the fundamental properties of light and capture intrinsic properties of the imaged environment that are otherwise omitted by color sensors. Many polarization applications, such as underwater geolocalization and sky-based polarization compass, require simultaneous imaging of the entire radial optical field with omnidirectional lenses. However, the reconstructed angle of polarization captured with omnidirectional lenses has a radial offset due to redirection of the light rays within these lenses. In this paper, we describe a calibration method for correcting angle of polarization images captured with omnidirectional lenses. Our calibration method reduces the variance of reconstructed angle of polarization from 76.2 ∘ to 4.1 ∘. Example images collected both on an optical bench and in nature, demonstrate the improved accuracy of the reconstructed angle of polarization with our calibration method. The improved accuracy in the angle of polarization images will aid the development of polarization-based applications with omnidirectional lenses.

Interpolant-based demosaicing routines for dual-mode visible/near-infrared imaging systems.

Dual-mode visible/near-infrared imaging systems, including a bioinspired six-channel design and more conventional four-channel implementations, have transitioned from a niche in surveillance to general use in machine vision. However, the demosaicing routines that transform the raw images from these sensors into processed images that can be consumed by humans or computers rely on assumptions that may not be appropriate when the two portions of the spectrum contribute different information about a scene. A solution can be found in a family of demosaicing routines that utilize interpolating polynomials and splines of different dimensionalities and orders to process images with minimal assumptions.

Thresholds of polarization vision in octopuses.

Polarization vision is widespread in nature, mainly among invertebrates, and is used for a range of tasks including navigation, habitat localization and communication. In marine environments, some species such as those from the Crustacea and Cephalopoda that are principally monochromatic, have evolved to use this adaptation to discriminate objects across the whole visual field, an ability similar to our own use of colour vision. The performance of these polarization vision systems varies, and the few cephalopod species tested so far have notably acute thresholds of discrimination. However, most studies to date have used artificial sources of polarized light that produce levels of polarization much higher than found in nature. In this study, the ability of octopuses to detect polarization contrasts varying in angle of polarization (AoP) was investigated over a range of different degrees of linear polarization (DoLP) to better judge their visual ability in more ecologically relevant conditions. The 'just-noticeable-differences' (JND) of AoP contrasts varied consistently with DoLP. These JND thresholds could be largely explained by their 'polarization distance', a neurophysical model that effectively calculates the level of activity in opposing horizontally and vertically oriented polarization channels in the cephalopod visual system. Imaging polarimetry from the animals' natural environment was then used to illustrate the functional advantage that these polarization thresholds may confer in behaviourally relevant contexts.

Implementation of a logarithmic division-of-focal-plane polarimeter to quantify changes in collagen alignment at varying levels of illumination.

We examine the impact of illumination, aperture, and sample thickness on two division-of-focal-plane (DoFP) polarimeters, one created using a standard 3 T pixel and the other with a forward-biased, logarithmic pixel. Across all measured metrics the logarithmic DoFP polarimeter was better able to track real-time changes in collagen alignment than the standard DoFP polarimeter.

Polarisation signals: a new currency for communication.

Most polarisation vision studies reveal elegant examples of how animals, mainly the invertebrates, use polarised light cues for navigation, course-control or habitat selection. Within the past two decades it has been recognised that polarised light, reflected, blocked or transmitted by some animal and plant tissues, may also provide signals that are received or sent between or within species. Much as animals use colour and colour signalling in behaviour and survival, other species additionally make use of polarisation signalling, or indeed may rely on polarisation-based signals instead. It is possible that the degree (or percentage) of polarisation provides a more reliable currency of information than the angle or orientation of the polarised light electric vector (e-vector). Alternatively, signals with specific e-vector angles may be important for some behaviours. Mixed messages, making use of polarisation and colour signals, also exist. While our knowledge of the physics of polarised reflections and sensory systems has increased, the observational and behavioural biology side of the story needs more (and more careful) attention. This Review aims to critically examine recent ideas and findings, and suggests ways forward to reveal the use of light that we cannot see.

Optical See-Through Cancer Vision Goggles Enable Direct Patient Visualization and Real-Time Fluorescence-Guided Oncologic Surgery.

BACKGROUND: The inability to visualize the patient and surgical site directly, limits the use of current near infrared fluorescence-guided surgery systems for real-time sentinel lymph node biopsy and tumor margin assessment. METHODS: We evaluated an optical see-through goggle augmented imaging and navigation system (GAINS) for near-infrared, fluorescence-guided surgery. Tumor-bearing mice injected with a near infrared cancer-targeting agent underwent fluorescence-guided, tumor resection. Female Yorkshire pigs received hind leg intradermal indocyanine green injection and underwent fluorescence-guided, popliteal lymph node resection. Four breast cancer patients received 99mTc-sulfur colloid and indocyanine green retroareolarly before undergoing sentinel lymph node biopsy using radioactive tracking and fluorescence imaging. Three other breast cancer patients received indocyanine green retroareolarly before undergoing standard-of-care partial mastectomy, followed by fluorescence imaging of resected tumor and tumor cavity for margin assessment. RESULTS: Using near-infrared fluorescence from the dyes, the optical see-through GAINS accurately identified all mouse tumors, pig lymphatics, and four pig popliteal lymph nodes with high signal-to-background ratio. In 4 human breast cancer patients, 11 sentinel lymph nodes were identified with a detection sensitivity of 86.67 ± 0.27% for radioactive tracking and 100% for GAINS. Tumor margin status was accurately predicted by GAINS in all three patients, including clear margins in patients 1 and 2 and positive margins in patient 3 as confirmed by paraffin-embedded section histopathology. CONCLUSIONS: The optical see-through GAINS prototype enhances near infrared fluorescence-guided surgery for sentinel lymph node biopsy and tumor margin assessment in breast cancer patients without disrupting the surgical workflow in the operating room.

Optical characterization of rigid endoscopes and polarization calibration methods.

Polarization imaging can reveal orthogonal information with respect to color about the structural composition of biological tissue, and with the advance of superior polarimeters its use for biomedical applications has proliferated in the last decade. Polarimetry can be used in pre-clinical and clinical settings for the early detection of cancerous tissue. Polarization-based endoscopy with the complementary near-infrared fluorescence imaging modality improves the early diagnosis of flat cancerous lesions in colorectal tumor models. With the development of new polarization sensors the need to use standard laboratory optics to create custom imaging systems increases. These additional optics can behave as polarization filters effectively degrading and modifying the original tissue's polarization signatures leading to erroneous judgments. Here, we present a framework to characterize the spectral and polarization properties of rigid endoscopes for polarization-based endoscopic imaging. We describe and evaluate two calibration schemes based on Mueller calculus to reconstruct the original polarization information. Optical limitations of the endoscopes and minimum polarimeter requirements are discussed that may be of interest to other researchers working with custom polarization-based imaging systems.

Residual interpolation for division of focal plane polarization image sensors.

Division of focal plane (DoFP) polarization image sensors capture polarization properties of light at every imaging frame. However, these imaging sensors capture only partial polarization information, resulting in reduced spatial resolution output and a varying instantaneous field of overview (IFoV). Interpolation methods are used to reduce the drawbacks and recover the missing polarization information. In this paper, we propose residual interpolation as an alternative to normal interpolation for division of focal plane polarization image sensors, where the residual is the difference between an observed and a tentatively estimated pixel value. Our results validate that our proposed algorithm using residual interpolation can give state-of-the-art performance over several previously published interpolation methods, namely bilinear, bicubic, spline and gradient-based interpolation. Visual image evaluation as well as mean square error analysis is applied to test images. For an outdoor polarized image of a car, residual interpolation has less mean square error and better visual evaluation results.

Polarization signaling in swordtails alters female mate preference.

Polarization of light, and visual sensitivity to it, is pervasive across aquatic and terrestrial environments. Documentation of invertebrate use of polarized light is widespread from navigation and foraging to species recognition. However, studies demonstrating that polarization body patterning serves as a communication signal (e.g., with evidence of changes in receiver behavior) are rare among invertebrate taxa and conspicuously absent among vertebrates. Here, we investigate polarization-mediated communication by northern swordtails, Xiphophorus nigrensis, using a custom-built videopolarimeter to measure polarization signals and an experimental paradigm that manipulates polarization signals without modifying their brightness or color. We conducted mate choice trials in an experimental tank that illuminates a pair of males with light passed through a polarization filter and a diffusion filter. By alternating the order of these filters between males, we presented females with live males that differed in polarization reflectance by >200% but with intensity and color differences below detection thresholds (∼5%). Combining videopolarimetry and polarization-manipulated mate choice trials, we found sexually dimorphic polarized reflectance and polarization-dependent female mate choice behavior with no polarization-dependent courtship behavior by males. Male swordtails exhibit greater within-body and body-to-background polarization contrast than females, and females preferentially associate with high-polarization-reflecting males. We also found limited support that males increase polarization contrast in social conditions over asocial conditions. Polarization cues in mate choice contexts may provide aquatic vertebrates with enhanced detection of specific display features (e.g., movements, angular information), as well as a signaling mechanism that may enhance detection by intended viewers while minimizing detection by others.

260 frames-per-second 648x488 resolution division-of-focal-plane polarimeter with structural dynamics and tracking applications.

We have designed an image sensor that can capture the first three Stokes parameters at 648 by 488 spatial resolution at 260 frames per second. The sensor consists of a CCD image sensor monolithically integrated with pixel pitch-matched aluminum nanowire polarization filters. The sensor demonstrates a Malus law response over all pixels, and has a relatively uniform diattenuation over the visible spectrum. We demonstrate two potential applications for the sensor. The first uses circular polarization in transmission mode to observe high-speed stress failure in polycarbonate. The second uses polarization in reflected mode to track high speed automobile traffic.

Surface normal reconstruction using circularly polarized light.

The polarization properties of reflected light capture important information about the object's inherent properties: material composition, i.e. index of refraction and scattering properties, and shape of the object, i.e. surface normal. Polarization information therefore has been used for surface reconstruction using a single-view camera with unpolarized incident light. However, this surface normal reconstruction technique suffers from a zenith angle ambiguity. In this paper, we have utilized circularly polarized light to solve for the zenith ambiguity by developing a detailed model using Mueller matrix formulism and division of focal plane polarization imaging technology. Experiment results validate our model for accurate surface reconstruction.

Image interpolation and denoising for division of focal plane sensors using Gaussian processes.

Image interpolation and denoising are important techniques in image processing. These methods are inherent to digital image acquisition as most digital cameras are composed of a 2D grid of heterogeneous imaging sensors. Current polarization imaging employ four different pixelated polarization filters, commonly referred to as division of focal plane polarization sensors. The sensors capture only partial information of the true scene, leading to a loss of spatial resolution as well as inaccuracy of the captured polarization information. Interpolation is a standard technique to recover the missing information and increase the accuracy of the captured polarization information. Here we focus specifically on Gaussian process regression as a way to perform a statistical image interpolation, where estimates of sensor noise are used to improve the accuracy of the estimated pixel information. We further exploit the inherent grid structure of this data to create a fast exact algorithm that operates in ����(N(3/2)) (vs. the naive ���� (N³)), thus making the Gaussian process method computationally tractable for image data. This modeling advance and the enabling computational advance combine to produce significant improvements over previously published interpolation methods for polarimeters, which is most pronounced in cases of low signal-to-noise ratio (SNR). We provide the comprehensive mathematical model as well as experimental results of the GP interpolation performance for division of focal plane polarimeter.

Dual-mode optical imaging system for fluorescence image-guided surgery.

In this Letter, we present a novel imaging concept that a single imaging system can image different spectral bands with different aperture sizes. It is achieved by using a filter with different transmitted spectral bands in different annular rings as the aperture stop. This concept will enable more efficient system configurations and practical clinical applications. We have demonstrated this concept with a dual-mode near-infrared fluorescence image guided surgical system.

Bioinspired Polarization Imaging Sensors: From Circuits and Optics to Signal Processing Algorithms and Biomedical Applications: Analysis at the focal plane emulates nature's method in sensors to image and diagnose with polarized light.

In this paper, we present recent work on bioinspired polarization imaging sensors and their applications in biomedicine. In particular, we focus on three different aspects of these sensors. First, we describe the electro-optical challenges in realizing a bioinspired polarization imager, and in particular, we provide a detailed description of a recent low-power complementary metal-oxide-semiconductor (CMOS) polarization imager. Second, we focus on signal processing algorithms tailored for this new class of bioinspired polarization imaging sensors, such as calibration and interpolation. Third, the emergence of these sensors has enabled rapid progress in characterizing polarization signals and environmental parameters in nature, as well as several biomedical areas, such as label-free optical neural recording, dynamic tissue strength analysis, and early diagnosis of flat cancerous lesions in a murine colorectal tumor model. We highlight results obtained from these three areas and discuss future applications for these sensors.

Influence of signal-to-noise ratio on DoLP and AoP measurements during reflectance-mode division-of-focal plane Stokes polarimetry of biological tissues.

Stokes polarimeter based endoscopes are emerging as an area of technology where polarization imaging can greatly impact clinical care by improving diagnostic tools without the use of exogenous contrast. Image acquisition in minimally invasive surgical settings is often beset by inherently limited illumination. A comprehensive analysis of how signal-to-noise (SNR) propagates through Stokes polarimetric outcomes such as degree of linear polarization (DoLP) and angle of polarization (AoP) in low light is important for future interpretation of data acquired in low-light conditions. A previously developed theoretical model of quantitative polarized light imaging (QPLI) analysis described SNR as a function of both incident light intensity and DoLP. When polarized light interacts with biological tissues, the resultant DoLP of exiting light is dependent on the underlying tissue microstructure. Therefore, in this study we explore how low light impacts SNR of QPLI outcomes of DoLP and AoP differently in tissue phantoms of varying microstructures. Data are compared to theoretical solutions of SNR of DoLP and AoP. Tissues were additionally loaded to varying magnitudes of strain to investigate how variable SNR affects the ability to discern dynamic realignment in biological tissues. We observed a high degree of congruency between experimental and theoretical data, with SNR depending on both light intensity and DoLP. Additionally, we found that AoP may have a greater resilience to noise overall than DoLP and, as such, may be particularly useful in conditions where light is inherently limited.

Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color-near-infrared sensors.

Significance: Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. Aim: We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. Approach: A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Results: Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. Conclusions: We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.

Biomimetic-Membrane-Protected Plasmonic Nanostructures as Dual-Modality Contrast Agents for Correlated Surface-Enhanced Raman Scattering and Photoacoustic Detection of Hidden Tumor Lesions.

Optical imaging and spectroscopic modalities are of considerable current interest for in vivo cancer detection and image-guided surgery, but the turbid or scattering nature of biomedical tissues has severely limited their abilities to detect buried or occluded tumor lesions. Here we report the development of a dual-modality plasmonic nanostructure based on colloidal gold nanostars (AuNSs) for simultaneous surface-enhanced Raman scattering (SERS) and photoacoustic (PA) detection of tumor phantoms embedded (hidden) in ex vivo animal tissues. By using red blood cell membranes as a naturally derived biomimetic coating, we show that this class of dual-modality contrast agents can provide both Raman spectroscopic and PA signals for the detection and differentiation of hidden solid tumors with greatly improved depths of tissue penetration. Compared to previous polymer-coated AuNSs, the biomimetic coatings are also able to minimize protein adsorption and cellular uptake when exposed to human plasma without compromising their SERS or PA signals. We further show that tumor-targeting peptides (such as cyclic RGD) can be noncovalently inserted for targeting the ανß3-integrin receptors expressed on metastatic cancer cells and tracked via both SERS and PA imaging (PAI). Finally, we demonstrate image-guided resections of tumor-mimicking phantoms comprising metastatic tumor cells buried under layers of skin and fat tissues (6 mm in thickness). Specifically, PAI was used to determine the precise tumor location, while SERS spectroscopic signals were used for tumor identification and differentiation. This work opens the possibility of using these biomimetic dual-modality nanoparticles with superior signal and biological stability for intraoperative cancer detection and resection.

Gradient-based interpolation method for division-of-focal-plane polarimeters.

Recent advancements in nanotechnology and nanofabrication have allowed for the emergence of the division-of-focal-plane (DoFP) polarization imaging sensors. These sensors capture polarization properties of the optical field at every imaging frame. However, the DoFP polarization imaging sensors suffer from large registration error as well as reduced spatial-resolution output. These drawbacks can be improved by applying proper image interpolation methods for the reconstruction of the polarization results. In this paper, we present a new gradient-based interpolation method for DoFP polarimeters. The performance of the proposed interpolation method is evaluated against several previously published interpolation methods by using visual examples and root mean square error (RMSE) comparison. We found that the proposed gradient-based interpolation method can achieve better visual results while maintaining a lower RMSE than other interpolation methods under various dynamic ranges of a scene ranging from dim to bright conditions.

Calibration methods for division-of-focal-plane polarimeters.

Division-of-focal plane (DoFP) imaging polarimeters are useful instruments for measuring polarization information for a variety of applications. Recent advances in nanofabrication have enabled the practical manufacture of DoFP sensors for the visible spectrum. These sensors are made by integrating nanowire polarization filters directly with an imaging array, and size variations of the nanowires due to fabrication can cause the optical properties of the filters to vary up to 20% across the imaging array. If left unchecked, these variations introduce significant errors when reconstructing the polarization image. Calibration methods offer a means to correct these errors. This work evaluates a scalar and matrix calibration derived from a mathematical model of the polarimeter behavior. The methods are evaluated quantitatively with an existing DoFP polarimeter under varying illumination intensity and angle of linear polarization.