A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception.

Humans recognize and manipulate objects relying on the multidimensional force features captured by the tactile sense of skin during the manipulation. Since the current sensors integrated in robots cannot support the robots to sense the multiple interaction states between manipulator and objects, achieving human-like perception and analytical capabilities remains a major challenge for service robots. Prompted by the tactile perception involved in robots performing complex tasks, a multimodal tactile sensory system is presented to provide in situ simultaneous sensing for robots when approaching, touching, and manipulating objects. The system comprises a capacitive sensor owning the high sensitivity of 1.11E-2 pF mm-1, a triboelectricity nanogenerator with the fast response speed of 30 ms, and a pressure sensor array capable of 3D force detection. By Combining transfer learning models, which fuses multimodal tactile information to achieve high-precision (up to 95%) recognition of the multi-featured targets such as random hardness and texture information under random sampling conditions, including random grasp force and velocity. This sensory system is expected to enhance the intelligent recognition and behavior-planning capabilities of autonomous robots when performing complex tasks in undefined surrounding environments.

Polar optical flow network: extracting overlapped holograms of dynamic particles.

Digital holography with lensless in-line setup has been extensively used in particle field measurements. As particle concentration increases, the holograms of dynamic particles locating at different depths tend to superpose with each other with incoherent overlap, hampering effective measurement of individual particles with incomplete information. Drawing inspiration from suborbicular nature of the in-line holographic fringes, in this study, we propose an optical flow method in polar coordinates to mitigate the overlap issue. The approach employs a radial transformer-enhanced network that leverages both the radial and angular characteristics of the polar hologram. Through ablation tests and experimental results, we have demonstrated the effectiveness and superiority of our proposed method.

Diffraction-Net: a robust single-shot holography for multi-distance lensless imaging.

Digital holography based on lensless imaging is a developing method adopted in microscopy and micro-scale measurement. To retrieve complex-amplitude on the sample surface, multiple images are required for common reconstruction methods. A promising single-shot approach points to deep learning, which has been used in lensless imaging but suffering from the unsatisfied generalization ability and stability. Here, we propose and construct a diffraction network (Diff-Net) to connect diffraction images at different distances, which breaks through the limitations of physical devices. The Diff-Net based single-shot holography is robust as there is no practical errors between the multiple images. An iterative complex-amplitude retrieval approach based on light transfer function through the Diff-Net generated multiple images is used for complex-amplitude recovery. This process indicates a hybrid-driven method including both physical model and deep learning, and the experimental results demonstrate that the Diff-Net possesses qualified generalization ability for samples with significantly different morphologies.

Quantitative phase imaging based on model transfer learning.

Convolutional neural networks have been widely used in optical information processing and the generalization ability of the network depends greatly on the scale and diversity of the datasets, however, the acquisition of mass datasets and later annotation have become a common problem that hinders its further progress. In this study, a model transfer-based quantitative phase imaging (QPI) method is proposed, which fine-tunes the network parameters through loading pre-training base model and transfer learning, enable the network with good generalization ability. Most importantly, a feature fusion method based on moment reconstruction is proposed for training dataset generation, which can construct rich enough datasets that can cover most situations and accurately annotated, it fundamentally solves the problem from the scale and representational ability of the datasets. Besides, a feature distribution distance scoring (FDDS) rule is proposed to evaluate the rationality of the constructed datasets. The experimental results show that this method is suitable for different types of samples to achieve fast and high-accuracy phase imaging, which greatly relieves the pressure of data, tagging and generalization ability in the data-driven method.

Quantitative phase imaging in dual-wavelength interferometry using a single wavelength illumination and deep learning.

In this manuscript, we propose a quantitative phase imaging method based on deep learning, using a single wavelength illumination to realize dual-wavelength phase-shifting phase recovery. By using the conditional generative adversarial network (CGAN), from one interferogram recorded at a single wavelength, we obtain interferograms at other wavelengths, the corresponding wrapped phases and then the phases at synthetic wavelengths. The feasibility of the proposed method is verified by simulation and experiments. The results demonstrate that the measurement range of single-wavelength interferometry (SWI) is improved by keeping a simple setup, avoiding the difficulty caused by using two wavelengths simultaneously. This will provide an effective solution for the problem of phase unwrapping and the measurement range limitation in phase-shifting interferometry.

Image definition assessment based on Tchebichef moments for micro-imaging.

This paper proposes a Tchebichef moment (TM)-based image definition assessment (IDA) method that employs the difference in the logarithmic spectra (DLS). To avoid the influence of the original image, the essential element point spread function (PSF) is extracted from the DLS to characterize the IDA function uniquely. The amplification of the PSF spot radius to the defocus amount in the micro-imaging system enhances the featural differences among the DLSs, thereby improving the sensitivity to the defocus amount. The DLS with an obvious geometric feature variation is described by a TM with a low order, which improves the anti-noise performance. The performed simulation and experiment verified the superiority of the proposed method.

Real-time measurement of the liquid-crystal optic-axis angle and effective refractive index distribution based on a common-path interferometer.

A common-path interferometer for the real-time measurement of the liquid-crystal (LC) optic-axis angle and effective refractive index distribution is proposed. This method involves adding a polarizer and polarization camera to a general optical microscope. This requires only single-exposure imaging without changing any optical elements, and greatly simplifies the measurement process and system. In addition, the measurement results are unaffected by light-source power fluctuations or a non-uniform spatial distribution. Therefore, this method is suitable for measuring the LC optic-axis angle and effective refractive index of electrically controlled LC devices. Finally, the feasibility and validity of the proposed method are verified by simulation and experimentation.

Dynamic Imaging of Transferrin Receptor Molecules on Single Live Cell with Bridge Gaps-Enhanced Raman Tags.

A metal nanoparticles-based surface-enhanced Raman scattering (SERS) technique has been developed for biosensing and bioimaging due to its advantages in ultra-narrow line width for multiplexing, ultra-high sensitivity and excellent photostability. However, the "hotspots" effect between nanoparticles usually leads to unstable and nonuniform Raman enhancement, and this will greatly reduce the quality of SERS imaging. In this study, we employ the bridge gaps-enhanced Raman tags (BGERTs) to perform SERS imaging, in which BGERTs can not only reduce the influence of the "hotspots" effect between nanoparticles on Raman signal intensity but provide a great Raman enhancement when the Gold (Au) shell is thick enough. Based on BGERTs and its conjugation with the thiol-terminated polyethylene glycol (PEG) and transferrin, we construct a targeted Transferrin (TF)-PEG-BGERTs SERS nanoprobe and achieve the dynamic imaging of transferrin receptor (TfR) molecules on a single live cell, in which the role of transferrin-conjugated PEG-BGERT is for targeting TfR molecules located in cellular membrane surface. Importantly, this BGERTs-based SERS imaging could potentially provide a useful tool for studying the precise mechanism during the receptor-mediated nanoparticles endocytosis or cell proliferation, apoptosis, and other complicated molecular events.

Dynamic refractive index distribution measurement of dynamic process by combining dual-channel simultaneous phase-shifting interferometry and total internal reflection.

We propose and demonstrate a novel method to measure dynamic refractive index distributions using a combination of total internal reflection (TIR) and dual-channel simultaneous phase-shifting interferometry (DCSPSI). First, a right-angle prism is introduced into the DCSPSI system, the reflection phase variation induced by TIR, which contains the refractive index information of tested sample, can be achieved by the spatial carrier-frequency phase-shifting algorithm from a pair of interferograms with the phase shifts of π/2 captured by DCSPSI system. Second, based on the relationship between the reflection phase variation and the refractive index, the 2D refractive index distribution can be calculated easily. Importantly, the proposed TIR-DCSPSI method will supply a useful tool for dynamic refractive index distribution measurement of dynamic process, such as the droplet evaporation, mutual solubilization and diffusion of different droplets, cell culture, colloid curing, etc.

Dual-channel simultaneous spatial and temporal polarization phase-shifting interferometry.

Without the limitations of fringe number and fringe shape, a dual-channel simultaneous spatial and temporal polarization (DC-SSTP) phase-shifting interferometry system is proposed to achieve rapid and accurate phase retrieval through only one-time phase-shifting procedure with unknown phase shifts. First, an arbitrary phase shifts is simultaneously introduced into two channels of DC-SSTP system by a spatial light modulator (SLM). Second, by performing the subtraction operation between each pair of phase-shifting interferograms captured in the same channel, the background deduction of interferogram can be achieved easily, so the accurate phase can be retrieved rapidly. Especially, it is found even if the fringe number in interferogram is less than one, the proposed DC-SSTP method still reveals high accuracy of phase retrieval. Both the simulation and experimental results demonstrate the outstanding performance of proposed DC-SSTP method in phase measurement.

Self-calibration phase-shifting algorithm with interferograms containing very few fringes based on Fourier domain estimation.

Based on Fourier domain estimation, a novel self-calibration phase-shifting algorithm, named Mid-Band Spatial Spectrum Matching (MSSM), is proposed to achieve phase retrieval from a small amount of phase-shifting interferograms containing very few fringes (defined as ultra-sparse fringe pattern(USFP)), which is still a difficult problem for optical interferometry. Both simulation and experimental results demonstrate that the proposed MSSM algorithm can accurately and rapidly achieve the phase distribution encoded in USFP while other current self-calibration algorithms fail, and this will supply a powerful tool to extend the application of phase-shifting interferometry.

Spatial dual-orthogonal (SDO) phase-shifting algorithm by pre-recomposing the interference fringe.

In the case that the phase distribution of interferogram is nonuniform and the background/modulation amplitude change rapidly, the current self-calibration algorithms with better performance like principal components analysis (PCA) and advanced iterative algorithm (AIA) cannot work well. In this study, from three or more phase-shifting interferograms with unknown phase-shifts, we propose a spatial dual-orthogonal (SDO) phase-shifting algorithm with high accuracy through using the spatial orthogonal property of interference fringe, in which a new sequence of fringe patterns with uniform phase distribution can be constructed by pre-recomposing original interferograms to determine their corresponding optimum combination coefficients, which are directly related with the phase shifts. Both simulation and experimental results show that using the proposed SDO algorithm, we can achieve accurate phase from the phase-shifting interferograms with nonuniform phase distribution, non-constant background and arbitrary phase shifts. Specially, it is found that the accuracy of phase retrieval with the proposed SDO algorithm is insensitive to the variation of fringe pattern, and this will supply a guarantee for high accuracy phase measurement and application.

Quantitative refractive index distribution of single cell by combining phase-shifting interferometry and AFM imaging.

Cell refractive index, an intrinsic optical parameter, is closely correlated with the intracellular mass and concentration. By combining optical phase-shifting interferometry (PSI) and atomic force microscope (AFM) imaging, we constructed a label free, non-invasive and quantitative refractive index of single cell measurement system, in which the accurate phase map of single cell was retrieved with PSI technique and the cell morphology with nanoscale resolution was achieved with AFM imaging. Based on the proposed AFM/PSI system, we achieved quantitative refractive index distributions of single red blood cell and Jurkat cell, respectively. Further, the quantitative change of refractive index distribution during Daunorubicin (DNR)-induced Jurkat cell apoptosis was presented, and then the content changes of intracellular biochemical components were achieved. Importantly, these results were consistent with Raman spectral analysis, indicating that the proposed PSI/AFM based refractive index system is likely to become a useful tool for intracellular biochemical components analysis measurement, and this will facilitate its application for revealing cell structure and pathological state from a new perspective.

General spatial phase-shifting interferometry by optimizing the signal retrieving function.

A general spatial phase-shifting (GSPS) interferometry method is proposed to achieve phase retrieval from one-frame spatial carrier frequency interferogram. By optimizing the internal signal retrieving function of the spatial phase-shifting (SPS) method, the accuracy, anti-noise ability and speed of phase retrieval can be significantly improved, meanwhile the corresponding local calculation property is reserved. Especially, in the case that the ratio of the spatial carrier to the phase variation rate are small, the proposed method reveals obvious advantage in the accuracy improvement relative to the conventional SPS methods, so the more details of measured sample can be effectively reserved through introducing smaller spatial carrier frequency, and this will facilitate its application in interference microscopy. The principle analysis, numerical simulation and experimental result are employed to verify the performance of the proposed GSPS method.

Improved phase retrieval method of dual-wavelength interferometry based on a shorter synthetic-wavelength.

In dual-wavelength interferometry (DWI), by combing the advantage of the shorter synthetic-wavelength and the immune algorithm of phase ambiguity, we propose an improved phase retrieval method with both high accuracy and large measurement range, which is a pair of contradiction in the reported DWI method. First, we calculate the height of measured object at longer synthetic-wavelength through using the wrapped phases of two single-wavelengths. Second, by combining the immune algorithm of phase ambiguity and the height of measured object at longer synthetic-wavelength, we can perform the phase unwrapping of the larger one of the two single-wavelengths, then achieve accurate height at single-wavelength named as the transition height. Finally, we perform phase unwrapping of shorter synthetic-wavelength through using the immune algorithm of phase ambiguity and the transition height, and then the height at shorter synthetic-wavelength can be achieved. Compared with the reported method, in addition to maintaining the advantage of high accuracy, the proposed method does not need the additional wavelength, so the corresponding measurement procedures is greatly simplified. Simulation and experimental results demonstrate the performance of proposed method.

An advanced phase retrieval algorithm in N-step phase-shifting interferometry with unknown phase shifts.

In phase-shifting interferometry with unknown phase shifts, a normalization and orthogonalization phase-shifting algorithm (NOPSA) is proposed to achieve phase retrieval. The background of interferogram is eliminated through using the orthogonality of complex sinusoidal function; and the influence of phase shifts deviation on accuracy of phase retrieval is avoided through both normalization and orthogonalization processing. Compared with the current algorithms with unknown phase shifts, the proposed algorithm reveals significantly faster computation speed, higher accuracy, better stability and non-sensitivity of phase shifts deviation.

Raman spectrum reveals Mesenchymal stem cells inhibiting HL60 cells growth.

Though some research results reveals that Mesenchymal stem cells (MSCs) have the ability of inhibiting tumor cells proliferation, it remains controversial about the precise interaction mechanism during MSCs and tumor cells co-culture. In this study, combing Raman spectroscopic data and principle component analysis (PCA), the biochemical changes of MSCs or Human promyelocytic leukemia (HL60) cells during their co-culture were presented. The obtained results showed that some main Raman peaks of HL60 assigned to nucleic acids or proteins were greatly higher in intensity in the late stage of co-culture than those in the early stage of co-culture while they were still lower relative to the control group, implicating that the effect of MSCs inhibiting HL60 proliferation appeared in the early stage but gradually lost the inhibiting ability in the late stage of co-culture. Moreover, some other peaks of HL60 assigned to proteins were decreased in intensity in the early stage of co-culture relative to the control group but rebounded to the level similar to the control group in the late stage, showing that the content and structure changes of these proteins might be generated in the early stage but returned to the original state in the late stage of co-culture. As a result, in the early stage of MSCs-HL60 co-culture, along with the level of Akt phosphorylation of HL60 was lowered relative to its control group, the proliferation rate of HL60 cells was decreased. And in the late stage of co-culture, along with the level of Akt phosphorylation was rebounded, the reverse transfer of Raman peaks within 875-880cm-1 appeared, thus MSCs lost the ability to inhibit HL60 growth and HL60 proliferation was increased. In addition, it was observed that the peak at 811cm-1, which is a marker of RNA, was higher in intensity in the late stage than that in the control group, indicating that MSCs might be differentiated into myofibroblast-like MSCs. In addition, PCA results also exhibited that the physiological state of MSCs can be separated by the first two main components of PC1 or PC2 easily, and the effect of MSCs inhibiting HL60 growth was greatly associated with the time of co-culture.

Phase retrieval based on temporal and spatial hybrid matching in simultaneous phase-shifting dual-wavelength interferometry.

In simultaneous phase-shifting dual-wavelength interferometry, by matching both the phase-shifting period number and the fringe number in interferogram of two wavelengths to the integers, the phase with high accuracy can be retrieved through combining the principle component analysis (PCA) and least-squares iterative algorithm (LSIA). First, by using the approximate ratio of two wavelengths, we can match both the temporal phase-shifting period number and the spatial fringe number in interferogram of two wavelengths to the integers. Second, using above temporal and spatial hybrid matching condition, we can achieve accurate phase shifts of single-wavelength of phase-shifting interferograms through using PCA algorithm. Third, using above phase shifts to perform the iterative calculation with the LSIA method, the wrapped phases of single-wavelength can be determined. Both simulation calculation and experimental research demonstrate that by using the temporal and spatial hybrid matching condition, the PCA + LSIA based phase retrieval method possesses significant advantages in accuracy, stability and processing time.

Dynamic phase measurement based on spatial carrier-frequency phase-shifting method.

Combining spatial carrier-frequency phase-shifting (SCPS) technique and Fourier transform method, from one-frame spatial carrier-frequency interferogram (SCFI), a novel phase retrieval method is proposed and applied to dynamic phase measurement. First, using the SCPS technique, four-frame phase-shifting sub-interferograms can be constructed from one-frame SCFI. Second, using Fourier transform method, the accurate phase-shifts of four sub-interferograms can be extracted rapidly, so there is no requirement of calibration for the carrier-frequency in advance compared to most existing SCPS methods. Third, the wrapped phase can be retrieved with the least-squares algorithm through using the above phase-shifts. Finally, the phase variations of a water droplet evaporation and a Jurkat cell apoptosis induced by a drug are presented with the proposed method. Both the simulation and experimental results demonstrate that in addition to maintaining high accuracy of the SCPS method, the proposed method reveals more rapid processing speed of phase retrieval, and this will greatly facilitate its application in dynamic phase measurement.

Dual-wavelength interferometry based on the spatial carrier-frequency phase-shifting method.

From a single-frame dual-wavelength spatial carrier-frequency interferogram (SCFI), we propose a novel phase retrieval method of dual-wavelength interferometry (DWI). First, by continuously moving the intercepted area pixel-by-pixel in a single-frame SCFI along the horizontal and vertical directions, we construct a sequence of phase-shifting sub-interferograms. Second, the wrapped phases of each single wavelength can be retrieved from those phase-shifting sub-interferograms via the least-squares iteration algorithm. Third, the phase of synthetic wavelength can be obtained by subtraction between the wrapped phases of single wavelengths. Both the numerical simulation and the experimental result demonstrate that the proposed method reveals greater accuracy and convenience. Furthermore, because only single-frame SCFI can perform the phase retrieval of DWI, the proposed method offers better ability in resisting external vibration and disturbance, which will greatly facilitate the application of DWI in the dynamic phase measurement.