Computational Hyperspectral Microflow Cytometry.

Miniaturized flow cytometry has significant potential for portable applications, such as cell-based diagnostics and the monitoring of therapeutic cell manufacturing, however, the performance of current techniques is often limited by the inability to resolve spectrally-overlapping fluorescence labels. Here, the study presents a computational hyperspectral microflow cytometer (CHC) that enables accurate discrimination of spectrally-overlapping fluorophores labeling single cells. CHC employs a dispersive optical element and an optimization algorithm to detect the full fluorescence emission spectrum from flowing cells, with a high spectral resolution of ≈3 nm in the range from 450 to 650 nm. CHC also includes a dedicated microfluidic device that ensures in-focus imaging through viscoelastic sheathless focusing, thereby enhancing the accuracy and reliability of microflow cytometry analysis. The potential of CHC for analyzing T lymphocyte subpopulations and monitoring changes in cell composition during T cell expansion is demonstrated. Overall, CHC represents a major breakthrough in microflow cytometry and can facilitate its use for immune cell monitoring.

Spectral Reconstruction from RGB Imagery: A Potential Option for Infinite Spectral Data?

Spectral imaging has revolutionisedvarious fields by capturing detailed spatial and spectral information. However, its high cost and complexity limit the acquisition of a large amount of data to generalise processes and methods, thus limiting widespread adoption. To overcome this issue, a body of the literature investigates how to reconstruct spectral information from RGB images, with recent methods reaching a fairly low error of reconstruction, as demonstrated in the recent literature. This article explores the modification of information in the case of RGB-to-spectral reconstruction beyond reconstruction metrics, with a focus on assessing the accuracy of the reconstruction process and its ability to replicate full spectral information. In addition to this, we conduct a colorimetric relighting analysis based on the reconstructed spectra. We investigate the information representation by principal component analysis and demonstrate that, while the reconstruction error of the state-of-the-art reconstruction method is low, the nature of the reconstructed information is different. While it appears that the use in colour imaging comes with very good performance to handle illumination, the distribution of information difference between the measured and estimated spectra suggests that caution should be exercised before generalising the use of this approach.

A Triangular-Matrix-Based Spectral Encoding Method for Broadband Filtering and Reconstruction-Based Spectral Measurement.

Broadband filtering and reconstruction-based spectral measurement represent a hot technical route for miniaturized spectral measurement; the measurement encoding scheme has a great effect on the spectral reconstruction fidelity. The existing spectral encoding schemes are usually complex and hard to implement; thus, the applications are severely limited. Considering this, here, a simple spectral encoding method based on a triangular matrix is designed. The condition number of the proposed spectral encoding system is estimated and demonstrated to be relatively low theoretically; then, verification experiments are carried out, and the results show that the proposed encoding can work well under precise or unprecise encoding and measurement conditions; therefore, the proposed scheme is demonstrated to be an effective trade-off of the spectral encoding efficiency and implementation cost.

Charge-Coupled Frequency Response Multispectral Inversion Network-Based Detection Method of Oil Contamination on Airport Runway.

Aircraft failures can result in the leakage of fuel, hydraulic oil, or other lubricants onto the runway during landing or taxiing. Damage to fuel tanks or oil lines during hard landings or accidents can also contribute to these spills. Further, improper maintenance or operational errors may leave oil traces on the runway before take-off or after landing. Identifying oil spills in airport runway videos is crucial to flight safety and accident investigation. Advanced image processing techniques can overcome the limitations of conventional RGB-based detection, which struggles to differentiate between oil spills and sewage due to similar coloration; given that oil and sewage have distinct spectral absorption patterns, precise detection can be performed based on multispectral images. In this study, we developed a method for spectrally enhancing RGB images of oil spills on airport runways to generate HSI images, facilitating oil spill detection in conventional RGB imagery. To this end, we employed the MST++ spectral reconstruction network model to effectively reconstruct RGB images into multispectral images, yielding improved accuracy in oil detection compared with other models. Additionally, we utilized the Fast R-CNN oil spill detection model, resulting in a 5% increase in Intersection over Union (IOU) for HSI images. Moreover, compared with RGB images, this approach significantly enhanced detection accuracy and completeness by 25.3% and 26.5%, respectively. These findings clearly demonstrate the superior precision and accuracy of HSI images based on spectral reconstruction in oil spill detection compared with traditional RGB images. With the spectral reconstruction technique, we can effectively make use of the spectral information inherent in oil spills, thereby enhancing detection accuracy. Future research could delve deeper into optimization techniques and conduct extensive validation in real airport environments. In conclusion, this spectral reconstruction-based technique for detecting oil spills on airport runways offers a novel and efficient approach that upholds both efficacy and accuracy. Its wide-scale implementation in airport operations holds great potential for improving aviation safety and environmental protection.

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images.

Recently, many deep neural networks (DNN) have been proposed to solve the spectral reconstruction (SR) problem: recovering spectra from RGB measurements. Most DNNs seek to learn the relationship between an RGB viewed in a given spatial context and its corresponding spectra. Significantly, it is argued that the same RGB can map to different spectra depending on the context with respect to which it is seen and, more generally, that accounting for spatial context leads to improved SR. However, as it stands, DNN performance is only slightly better than the much simpler pixel-based methods where spatial context is not used. In this paper, we present a new pixel-based algorithm called A++ (an extension of the A+ sparse coding algorithm). In A+, RGBs are clustered, and within each cluster, a designated linear SR map is trained to recover spectra. In A++, we cluster the spectra instead in an attempt to ensure neighboring spectra (i.e., spectra in the same cluster) are recovered by the same SR map. A polynomial regression framework is developed to estimate the spectral neighborhoods given only the RGB values in testing, which in turn determines which mapping should be used to map each testing RGB to its reconstructed spectrum. Compared to the leading DNNs, not only does A++ deliver the best results, it is parameterized by orders of magnitude fewer parameters and has a significantly faster implementation. Moreover, in contradistinction to some DNN methods, A++ uses pixel-based processing, which is robust to image manipulations that alter the spatial context (e.g., blurring and rotations). Our demonstration on the scene relighting application also shows that, while SR methods, in general, provide more accurate relighting results compared to the traditional diagonal matrix correction, A++ provides superior color accuracy and robustness compared to the top DNN methods.

Improving Generalizability of Spectral Reflectance Reconstruction Using L1-Norm Penalization.

Spectral reflectance reconstruction for multispectral images (such as Weiner estimation) may perform sub-optimally when the object being measured has a texture that is not in the training set. The accuracy of the reconstruction is significantly lower without training samples. We propose an improved reflectance reconstruction method based on L1-norm penalization to solve this issue. Using L1-norm, our method can provide the transformation matrix with the favorable sparse property, which can help to achieve better results when measuring the unseen samples. We verify the proposed method by reconstructing spectral reflection for four types of materials (cotton, paper, polyester, and nylon) captured by a multispectral imaging system. Each of the materials has its texture and there are 204 samples in each of the materials/textures in the experiments. The experimental results show that when the texture is not included in the training dataset, L1-norm can achieve better results compared with existing methods using colorimetric measure (i.e., color difference) and shows consistent accuracy across four kinds of materials.

SpectralMAE: Spectral Masked Autoencoder for Hyperspectral Remote Sensing Image Reconstruction.

Accurate hyperspectral remote sensing information is essential for feature identification and detection. Nevertheless, the hyperspectral imaging mechanism poses challenges in balancing the trade-off between spatial and spectral resolution. Hardware improvements are cost-intensive and depend on strict environmental conditions and extra equipment. Recent spectral imaging methods have attempted to directly reconstruct hyperspectral information from widely available multispectral images. However, fixed mapping approaches used in previous spectral reconstruction models limit their reconstruction quality and generalizability, especially dealing with missing or contaminated bands. Moreover, data-hungry issues plague increasingly complex data-driven spectral reconstruction methods. This paper proposes SpectralMAE, a novel spectral reconstruction model that can take arbitrary combinations of bands as input and improve the utilization of data sources. In contrast to previous spectral reconstruction techniques, SpectralMAE explores the application of a self-supervised learning paradigm and proposes a masked autoencoder architecture for spectral dimensions. To further enhance the performance for specific sensor inputs, we propose a training strategy by combining random masking pre-training and fixed masking fine-tuning. Empirical evaluations on five remote sensing datasets demonstrate that SpectralMAE outperforms state-of-the-art methods in both qualitative and quantitative metrics.

Performance-Enhanced Static Modulated Fourier Transform Spectrometer with a Spectral Reconstruction.

A static modulated Fourier transform spectrometer has been noted to be a compact and fast evaluation tool for spectroscopic inspection, and many novel structures have been reported to support its performance. However, it still suffers from poor spectral resolution due to the limited sampling data points, which marks its intrinsic drawback. In this paper, we outline the enhanced performance of a static modulated Fourier transform spectrometer with a spectral reconstruction method that can compensate for the insufficient data points. An enhanced spectrum can be reconstructed by applying a linear regression method to a measured interferogram. We obtain the transfer function of a spectrometer by analyzing what interferogram can be detected with different values of parameters such as focal length of the Fourier lens, mirror displacement, and wavenumber range, instead of direct measurement of the transfer function. Additionally, the optimal experimental conditions for the narrowest spectral width are investigated. Application of the spectral reconstruction method achieves an improved spectral resolution from 74 cm-1 when spectral reconstruction is not applied to 8.9 cm-1, and a narrowed spectral width from 414 cm-1 to 371 cm-1, which are close to the values of the spectral reference. In conclusion, the spectral reconstruction method in a compact static modulated Fourier transform spectrometer effectively enhances its performance without any additional optic in the structure.

A Snapshot Imaging Spectrometer Based on Uniformly Distributed-Slit Array (UDA).

Herein, we propose a system for a snapshot video hyperspectral imaging method based on a uniformly distributed-slit array (UDA) coding plate that not only effectively improves the scanning speed of spectrometers but also achieves a high spectral fidelity of snapshot videos. A mathematical model and optical link simulation of the new system are established. The analysis results show that the proposed method can more efficiently collect information and restore the spectral data cube, and the spectral smile of the system is less than 4.86 µm. The results of the spectral performance and external imaging tests of the system show that the system has the ability to collect spatial spectrum video information with a frame rate of 10 Hz and identify dynamic targets, laying a foundation for the design of a system with a higher frame rate and resolution.

Measurement of Secondary Structure Changes in Poly-L-lysine and Lysozyme during Acoustically Levitated Single Droplet Drying Experiments by In Situ Raman Spectroscopy.

Drying processes such as spray drying, as commonly used in the pharmaceutical industry to convert protein-based drugs into their particulate form, can lead to an irreversible loss of protein activity caused by protein secondary structure changes. Due to the nature of these processes (high droplet number, short drying time), an in situ investigation of the structural changes occurring during a real drying process is hardly possible. Therefore, an approach for the in situ investigation of the expected secondary structural changes during single droplet protein drying in an acoustic levitator by time-resolved Raman spectroscopy was developed and is demonstrated in this paper. For that purpose, a self-developed NIR-Raman sensor generates and detects the Raman signal from the levitated solution droplet. A mathematical spectral reconstruction by multiple Voigt functions is used to quantify the relative secondary structure changes occurring during the drying process. With the developed setup, it was possible to detect and quantify the relative secondary structure changes occurring during single droplet drying experiments for the two chosen model substances: poly-L-lysine, a homopolypeptide widely used as a protein mimic, and lysozyme. Throughout drying, an increase in the ß-sheet structure and a decrease in the other two structural elements, α-helix, and random coil, could be identified. In addition, it was observed that the degree of structural changes increased with increasing temperature.

A simple approach for reconstruction of non-uniformly sampled pseudo-3D NMR data for accurate measurement of spin relaxation parameters.

We explain how to conduct a pseudo-3D relaxation series NUS measurement so that it can be reconstructed by existing 3D NUS reconstruction methods to give accurate relaxation values. We demonstrate using reconstruction algorithms IST and SMILE that this 3D approach allows lower sampling densities than for independent 2D reconstructions. This is in keeping with the common finding that higher dimensionality increases signal sparsity, enabling lower sampling density. The approach treats the relaxation series as ordinary 3D time-domain data whose imaginary part in the pseudo-dimension is zero, and applies any suitably linear 3D NUS reconstruction method accordingly. Best results on measured and simulated data were achieved using acquisitions with 9 to 12 planes and exponential spacing in the pseudo-dimension out to ~ 2 times the inverse decay time. Given these criteria, in typical cases where 2D reconstructions require 50% sampling, the new 3D approach generates spectra reliably at sampling densities of 25%.

FID-Net: A versatile deep neural network architecture for NMR spectral reconstruction and virtual decoupling.

In recent years, the transformative potential of deep neural networks (DNNs) for analysing and interpreting NMR data has clearly been recognised. However, most applications of DNNs in NMR to date either struggle to outperform existing methodologies or are limited in scope to a narrow range of data that closely resemble the data that the network was trained on. These limitations have prevented a widescale uptake of DNNs in NMR. Addressing this, we introduce FID-Net, a deep neural network architecture inspired by WaveNet, for performing analyses on time domain NMR data. We first demonstrate the effectiveness of this architecture in reconstructing non-uniformly sampled (NUS) biomolecular NMR spectra. It is shown that a single network is able to reconstruct a diverse range of 2D NUS spectra that have been obtained with arbitrary sampling schedules, with a range of sweep widths, and a variety of other acquisition parameters. The performance of the trained FID-Net in this case exceeds or matches existing methods currently used for the reconstruction of NUS NMR spectra. Secondly, we present a network based on the FID-Net architecture that can efficiently virtually decouple 13Cα-13Cß couplings in HNCA protein NMR spectra in a single shot analysis, while at the same time leaving glycine residues unmodulated. The ability for these DNNs to work effectively in a wide range of scenarios, without retraining, paves the way for their widespread usage in analysing NMR data.

Spectral Reconstruction Using an Iteratively Reweighted Regulated Model from Two Illumination Camera Responses.

An improved spectral reflectance estimation method was developed to transform captured RGB images to spectral reflectance. The novelty of our method is an iteratively reweighted regulated model that combines polynomial expansion signals, which was developed for spectral reflectance estimation, and a cross-polarized imaging system, which is used to eliminate glare and specular highlights. Two RGB images are captured under two illumination conditions. The method was tested using ColorChecker charts. The results demonstrate that the proposed method could make a significant improvement of the accuracy in both spectral and colorimetric: it can achieve 23.8% improved accuracy in mean CIEDE2000 color difference, while it achieves 24.6% improved accuracy in RMS error compared with classic regularized least squares (RLS) method. The proposed method is sufficiently accurate in predicting the spectral properties and their performance within an acceptable range, i.e., typical customer tolerance of less than 3 DE units in the graphic arts industry.

On the Optimization of Regression-Based Spectral Reconstruction.

Spectral reconstruction (SR) algorithms attempt to recover hyperspectral information from RGB camera responses. Recently, the most common metric for evaluating the performance of SR algorithms is the Mean Relative Absolute Error (MRAE)-an ℓ1 relative error (also known as percentage error). Unsurprisingly, the leading algorithms based on Deep Neural Networks (DNN) are trained and tested using the MRAE metric. In contrast, the much simpler regression-based methods (which actually can work tolerably well) are trained to optimize a generic Root Mean Square Error (RMSE) and then tested in MRAE. Another issue with the regression methods is-because in SR the linear systems are large and ill-posed-that they are necessarily solved using regularization. However, hitherto the regularization has been applied at a spectrum level, whereas in MRAE the errors are measured per wavelength (i.e., per spectral channel) and then averaged. The two aims of this paper are, first, to reformulate the simple regressions so that they minimize a relative error metric in training-we formulate both ℓ2 and ℓ1 relative error variants where the latter is MRAE-and, second, we adopt a per-channel regularization strategy. Together, our modifications to how the regressions are formulated and solved leads to up to a 14% increment in mean performance and up to 17% in worst-case performance (measured with MRAE). Importantly, our best result narrows the gap between the regression approaches and the leading DNN model to around 8% in mean accuracy.

Physically Plausible Spectral Reconstruction.

Spectral reconstruction algorithms recover spectra from RGB sensor responses. Recent methods-with the very best algorithms using deep learning-can already solve this problem with good spectral accuracy. However, the recovered spectra are physically incorrect in that they do not induce the RGBs from which they are recovered. Moreover, if the exposure of the RGB image changes then the recovery performance often degrades significantly-i.e., most contemporary methods only work for a fixed exposure. In this paper, we develop a physically accurate recovery method: the spectra we recover provably induce the same RGBs. Key to our approach is the idea that the set of spectra that integrate to the same RGB can be expressed as the sum of a unique fundamental metamer (spanned by the camera's spectral sensitivities and linearly related to the RGB) and a linear combination of a vector space of metameric blacks (orthogonal to the spectral sensitivities). Physically plausible spectral recovery resorts to finding a spectrum that adheres to the fundamental metamer plus metameric black decomposition. To further ensure spectral recovery that is robust to changes in exposure, we incorporate exposure changes in the training stage of the developed method. In experiments we evaluate how well the methods recover spectra and predict the actual RGBs and RGBs under different viewing conditions (changing illuminations and/or cameras). The results show that our method generally improves the state-of-the-art spectral recovery (with more stabilized performance when exposure varies) and provides zero colorimetric error. Moreover, our method significantly improves the color fidelity under different viewing conditions, with up to a 60% reduction in some cases.

Brightness Invariant Deep Spectral Super-Resolution.

Spectral reconstruction from RGB or spectral super-resolution (SSR) offers a cheap alternative to otherwise costly and more complex spectral imaging devices. In recent years, deep learning based methods consistently achieved the best reconstruction quality in terms of spectral error metrics. However, there are important properties that are not maintained by deep neural networks. This work is primarily dedicated to scale invariance, also known as brightness invariance or exposure invariance. When RGB signals only differ in their absolute scale, they should lead to identical spectral reconstructions apart from the scaling factor. Scale invariance is an essential property that signal processing must guarantee for a wide range of practical applications. At the moment, scale invariance can only be achieved by relying on a diverse database during network training that covers all possibly occurring signal intensities. In contrast, we propose and evaluate a fundamental approach for deep learning based SSR that holds the property of scale invariance by design and is independent of the training data. The approach is independent of concrete network architectures and instead focuses on reevaluating what neural networks should actually predict. The key insight is that signal magnitudes are irrelevant for acquiring spectral reconstructions from camera signals and are only useful for a potential signal denoising.

Using Deep Neural Networks to Reconstruct Non-uniformly Sampled NMR Spectra.

Non-uniform and sparse sampling of multi-dimensional NMR spectra has over the last decade become an important tool to allow for fast acquisition of multi-dimensional NMR spectra with high resolution. The success of non-uniform sampling NMR hinge on both the development of algorithms to accurately reconstruct the sparsely sampled spectra and the design of sampling schedules that maximise the information contained in the sampled data. Traditionally, the reconstruction tools and algorithms have aimed at reconstructing the full spectrum and thus 'fill out the missing points' in the time-domain spectrum, although other techniques are based on multi-dimensional decomposition and extraction of multi-dimensional shapes. Also over the last decade, machine learning, deep neural networks, and artificial intelligence have seen new applications in an enormous range of sciences, including analysis of MRI spectra. As a proof-of-principle, it is shown here that simple deep neural networks can be trained to reconstruct sparsely sampled NMR spectra. For the reconstruction of two-dimensional NMR spectra, reconstruction using a deep neural network performs as well, if not better than, the currently and widely used techniques. It is therefore anticipated that deep neural networks provide a very valuable tool for the reconstruction of sparsely sampled NMR spectra in the future to come.

Optimized Multi-Spectral Filter Arrays for Spectral Reconstruction.

Multispectral filter array (MSFA)-based imaging is a compact, practical technique for snapshot spectral image capturing and reconstruction. The imaging and reconstruction quality is highly influenced by the spectral sensitivities and spatial arrangement of channels on MSFAs, and the used reconstruction method. In order to design a MSFA with high imaging capacity, we propose a sparse representation based approach to optimize spectral sensitivities and spatial arrangement of MSFAs. The proposed approach first overall models the various errors associated with spectral reconstruction, and then uses a global heuristic searching method to optimize MSFAs via minimizing the estimated error of MSFAs. Our MSFA optimization method can select filters from off-the-shelf candidate filter sets while assigning the selected filters to the designed MSFA. Experimental results on three datasets show that the proposed method is more efficient, flexible, and can design MSFAs with lower spectral construction errors when compared with existing state-of-the-art methods. The MSFAs designed by our method show better performance than others even using different spectral reconstruction methods.

A Spectral Reconstruction Algorithm of Miniature Spectrometer Based on Sparse Optimization and Dictionary Learning.

The miniaturization of spectrometer can broaden the application area of spectrometry, which has huge academic and industrial value. Among various miniaturization approaches, filter-based miniaturization is a promising implementation by utilizing broadband filters with distinct transmission functions. Mathematically, filter-based spectral reconstruction can be modeled as solving a system of linear equations. In this paper, we propose an algorithm of spectral reconstruction based on sparse optimization and dictionary learning. To verify the feasibility of the reconstruction algorithm, we design and implement a simple prototype of a filter-based miniature spectrometer. The experimental results demonstrate that sparse optimization is well applicable to spectral reconstruction whether the spectra are directly sparse or not. As for the non-directly sparse spectra, their sparsity can be enhanced by dictionary learning. In conclusion, the proposed approach has a bright application prospect in fabricating a practical miniature spectrometer.

Filter Selection for Optimizing the Spectral Sensitivity of Broadband Multispectral Cameras Based on Maximum Linear Independence.

Previous research has shown that the effectiveness of selecting filter sets from among a large set of commercial broadband filters by a vector analysis method based on maximum linear independence (MLI). However, the traditional MLI approach is suboptimal due to the need to predefine the first filter of the selected filter set to be the maximum ℓ2 norm among all available filters. An exhaustive imaging simulation with every single filter serving as the first filter is conducted to investigate the features of the most competent filter set. From the simulation, the characteristics of the most competent filter set are discovered. Besides minimization of the condition number, the geometric features of the best-performed filter set comprise a distinct transmittance peak along the wavelength axis of the first filter, a generally uniform distribution for the peaks of the filters and substantial overlaps of the transmittance curves of the adjacent filters. Therefore, the best-performed filter sets can be recognized intuitively by simple vector analysis and just a few experimental verifications. A practical two-step framework for selecting optimal filter set is recommended, which guarantees a significant enhancement of the performance of the systems. This work should be useful for optimizing the spectral sensitivity of broadband multispectral imaging sensors.