Random access vision: an imaging method to observe arbitrary and multiple gaze directions in frame-by-frame manner.

In robot vision, it is often desired to measure an area larger than the field of view of the camera, so the camera tends to be mounted on a mechanical pan/tilt platform. However, such mechanisms have a non-negligible response time compared to the frame rate of the camera. In this paper, we describe what we believe to be a new method that allows arbitrary and multiple gaze directions to be observed in a frame-by-frame manner based on a resonant mirror and a lock-in pixel image sensor. In the proposed method, while the gaze direction oscillates due to the resonance mirror, the image sensor makes an exposure of several hundreds of nanoseconds every time the gaze passes through the direction to be captured, and accumulates the captured signals. A prototype system was developed using a lock-in pixel image sensor with four image storage units called taps and a resonant mirror with a resonant frequency of 12 kHz. The system achieved both arbitrary control of the gaze direction in a frame-by-frame manner, and simultaneous capturing of four images with arbitrary gaze directions also in a frame-by-frame manner.

Depth Quality Improvement with a 607 MHz Time-Compressive Computational Pseudo-dToF CMOS Image Sensor.

In this paper, we present a prototype pseudo-direct time-of-flight (ToF) CMOS image sensor, achieving high distance accuracy, precision, and robustness to multipath interference. An indirect ToF (iToF)-based image sensor, which enables high spatial resolution, is used to acquire temporal compressed signals in the charge domain. Whole received light waveforms, like those acquired with conventional direct ToF (dToF) image sensors, can be obtained after image reconstruction based on compressive sensing. Therefore, this method has the advantages of both dToF and iToF depth image sensors, such as high resolution, high accuracy, immunity to multipath interference, and the absence of motion artifacts. Additionally, two approaches to refine the depth resolution are explained: (1) the introduction of a sub-time window; and (2) oversampling in image reconstruction and quadratic fitting in the depth calculation. Experimental results show the separation of two reflections 40 cm apart under multipath interference conditions and a significant improvement in distance precision down to around 1 cm. Point cloud map videos demonstrate the improvements in depth resolution and accuracy. These results suggest that the proposed method could be a promising approach for virtually implementing dToF imaging suitable for challenging environments with multipath interference.

A Time-of-Flight Image Sensor Using 8-Tap P-N Junction Demodulator Pixels.

This paper presents a time-of-flight image sensor based on 8-Tap P-N junction demodulator (PND) pixels, which is designed for hybrid-type short-pulse (SP)-based ToF measurements under strong ambient light. The 8-tap demodulator implemented with multiple p-n junctions used for modulating the electric potential to transfer photoelectrons to eight charge-sensing nodes and charge drains has an advantage of high-speed demodulation in large photosensitive areas. The ToF image sensor implemented using 0.11 µm CIS technology, consisting of an 120 (H) × 60 (V) image array of the 8-tap PND pixels, successfully works with eight consecutive time-gating windows with the gating width of 10 ns and demonstrates for the first time that long-range (>10 m) ToF measurements under high ambient light are realized using single-frame signals only, which is essential for motion-artifact-free ToF measurements. This paper also presents an improved depth-adaptive time-gating-number assignment (DATA) technique for extending the depth range while having ambient-light canceling capability and a nonlinearity error correction technique. By applying these techniques to the implemented image sensor chip, hybrid-type single-frame ToF measurements with depth precision of maximally 16.4 cm (1.4% of the maximum range) and the maximum non-linearity error of 0.6% for the full-scale depth range of 1.0-11.5 m and operations under direct-sunlight-level ambient light (80 klux) have been realized. The depth linearity achieved in this work is 2.5 times better than that of the state-of-the-art 4-tap hybrid-type ToF image sensor.

Development of a Current Injection-Type Impedance Measurement System for Monitoring Soil Water Content and Ion Concentration.

This study was conducted with the aim of developing a circuit system that enables the measurement of the moisture content and ion concentration with a simple circuit configuration. Our previous studies have shown that soil can be represented by an equivalent circuit of a parallel circuit of resistors and capacitors. We designed a circuit that can convert the voltage transient characteristics of the soil when a current is applied to it into a square wave and output frequency information and developed an algorithm to analyze the two types of square waves and calculate R and C. Normal operation was confirmed in the range of 10 kΩ-1 MΩ for the designed circuit, and the calculation algorithm matched within a maximum error of 5%, thus confirming the validity of the program. These successfully confirmed the changes in the water content and ionic concentration. The soil moisture content measurement succeeded in measuring a maximum error of about 10%, except at one point, and the soil ion concentration measurement succeeded in measuring a maximum error of 6.6%. A new, simple, noise-resistant moisture content and ion concentration measurement circuit system with square wave output has been realized.

A Dual-Mode 303-Megaframes-per-Second Charge-Domain Time-Compressive Computational CMOS Image Sensor.

An ultra-high-speed computational CMOS image sensor with a burst frame rate of 303 megaframes per second, which is the fastest among the solid-state image sensors, to our knowledge, is demonstrated. This image sensor is compatible with ordinary single-aperture lenses and can operate in dual modes, such as single-event filming mode or multi-exposure imaging mode, by reconfiguring the number of exposure cycles. To realize this frame rate, the charge modulator drivers were adequately designed to suppress the peak driving current taking advantage of the operational constraint of the multi-tap charge modulator. The pixel array is composed of macropixels with 2 × 2 4-tap subpixels. Because temporal compressive sensing is performed in the charge domain without any analog circuit, ultrafast frame rates, small pixel size, low noise, and low power consumption are achieved. In the experiments, single-event imaging of plasma emission in laser processing and multi-exposure transient imaging of light reflections to extend the depth range and to decompose multiple reflections for time-of-flight (TOF) depth imaging with a compression ratio of 8× were demonstrated. Time-resolved images similar to those obtained by the direct-type TOF were reproduced in a single shot, while the charge modulator for the indirect TOF was utilized.

Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor.

Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.

Range-precision improvement of a time-of-flight range sensor using dual reference plane sampling.

This study aimed to achieve high range precision in the sub-100 µm order with time-of-flight (TOF) range imaging for 3-D scanners. The precision of a TOF range imager was improved using dual reference plane sampling (DRPS). DRPS using two short-pulse lasers reduces driver jitter, which limits the range precision below sub-100 µm. A proof-of-concept measurement system implemented using a TOF range imager demonstrated the reduction in driver jitters, resulting in reduced column-to-column variation in range precision. The developed system also achieved a high precision of 52 µm using a single frame and 27 µm using a 10-frame average.

High-Linearity High-Resolution Time-of-Flight Linear-Array Digital Image Sensor Using Time-Domain Feedback.

This paper presents a high-linearity high-resolution time-of-flight (ToF) linear-array digital image sensor using a time-domain negative feedback technique. A coarse ToF measurement loop uses a 5-bit digital-to-time converter (DTC) and a delayed gating-pulse generator for time-domain feedback to find the zero of the difference between ToF and the digital estimate of the gating-pulse delay while maintaining a constant operating point of the analog readout circuits. A fine ToF measurement uses a delta-sigma modulation (DSM) loop using the time-domain feedback with a bit-stream signal form. Because of the self-contained property of the DSM for low distortion and noise exploited by the oversampling signal processing, the proposed technique provides high-linearity and high-range resolution in the fine ToF measurement. A prototype ToF sensor of 16.8 × 16.8 µm2 two-tap pixels and fabricated in a 0.11 µm (1P4M) CMOS image sensors (CIS) process achieves +0.9%/-0.47% maximum nonlinearity error and a resolution of 0.24 mm (median) for the measurement range of 0-1.05 m. The ToF sensor produces an 11-bit fully digital output with a ToF measurement time of 22.4 ms.

Quasi-simultaneous multi-focus imaging using a lock-in pixel image sensor and TAG lens.

In this paper, a quasi-simultaneous multi-focus imaging technique named simulfocus imaging is reported. This technique was developed for measuring an entire object distributed in the depth direction beyond the depth of field (DOF) with high resolution in a single shot. Simulfocus imaging can acquire multiple focal planes in one shot by synchronizing a tunable acoustic gradient index (TAG) lens and a lock-in pixel image sensor. The TAG lens is a tunable-focus lens whose focal position can be changed at a high speed of several tens to several hundreds of kilohertz. The lock-in pixel image sensor is a special image sensor that can execute multiple exposures at an arbitrary timing during a single shooting. The sensor includes a number of photoelectron storage units in each pixel, and the units where the photoelectrons generated by each exposure are stored can be freely selected. Since an image can be acquired for a single storage unit, and the lock-in pixel image sensor has a number of storage units, the lock-in pixel image sensor can acquire multiple images in one shot. By assigning a specific exposure timing to each unit and synchronizing the exposure timing with the focus fluctuation of the TAG lens, it is possible to simultaneously acquire images in different focal planes. To evaluate the system, we conducted experiments to show the effectiveness of simulfocus imaging in microscope and telescope configurations. From the experimental results, it was confirmed that simulfocus was effective in both configurations.

Fluoroimmunoassay of influenza virus using sulfur-doped graphitic carbon nitride quantum dots coupled with Ag2S nanocrystals.

Novel sulfur-doped graphitic carbon nitride quantum dots (S-gCNQDs) are synthesized using a single-source precursor in a one-step solvothermal process. The S-gCNQDs with a size of ~ 5-nm displayed a strong green intrinsic fluorescence at 512 nm when excited at 400 nm, with a quantum yield of ~ 33% in aqueous solution. The prepared S-gCNQDs and Ag2S nanocrystals were applied as innovative functional materials to fabricate a biosensor for virus detection based on the conjugation of specific anti-human influenza A monoclonal antibody to the S-gCNQDs and Ag2S NCs, respectively. In the presence of the influenza A virus, an interaction between the S-gCNQDs/Ag2S-labeled antibody resulted in the formation of a nanosandwich structure, which is accompanied by the fluorescence enhancement of the S-gCNQDs. The change in fluorescence intensity linearly correlats with the concentration of the influenza A virus (H1N1) in the 10 fg/mL to 1.0 ng/mL range, with a limit of detection of 5.5 fg/mL. The assay was applied to the assay of clinically isolated influenza A virus (H3N2/Yokohama) mixed with human serum. The obtained limit of detection was 100 PFU/mL within the detection range of 102- 5 × 104 PFU/mL for the H3N2 virus. Graphical abstract.

An 8-Tap CMOS Lock-In Pixel Image Sensor for Short-Pulse Time-of-Flight Measurements.

An 8-tap CMOS lock-in pixel image sensor that has seven carrier-capturing and a draining time window was developed for short-pulse time-of-flight (TOF) measurements. The proposed pixel for the short-pulse TOF measurements has seven consecutive time-gating windows, each of which has the width of 6 ns, which is advantageous for high-resolution range imaging, particularly for relatively longer distances (>5 m) and under high ambient light operations. In order to enhance the depth resolution, a technique for the depth-adaptive time-gating-number assignment (DATA) for the short-pulse TOF measurement is proposed. A prototype of the 8-tap CMOS lock-in pixel image sensor is implemented with a 1POLY 4METAL 0.11-µm CIS process. The maximum non-linearity error of 1.56%FS for the range of 1-6.4 m and the depth resolution of 6.4 mm was obtained at 6.2 m using the DATA technique.

Multi-exposure laser speckle contrast imaging using a video-rate multi-tap charge modulation image sensor.

Multi-exposure laser speckle contrast imaging (MELSCI) systems based on high frame rate cameras are suitable for wide-field quantitative measurement of blood flow. However, high-speed camera-based MELSCI requires high power consumption, large memory, and high processing capability, which may lead to relatively large and expensive hardware. To realize a compact and cost-efficient MELSCI system, we discuss an application of the multi-tap CMOS image sensor originally designed for time-of-flight range imaging. This image sensor operated in the global shutter mode and every pixel was provided with multiple charge-storage diodes. Multiple images for different exposures were acquired simultaneously because exposure patterns were programmable to implement an arbitrary exposure duration for each tap. The frame rate was close to video frame rates (30 frames per second (fps)) regardless of the exposure pattern. The feasibility of the proposed method was verified by simulations that were performed with real speckle images captured by a high-speed camera at 40 kfps. Experiments with a four-tap CMOS image sensor demonstrated that a flow speed map was obtained at a moderate frame rate such as 35 fps for a moving ground glass plate and 45 fps for flowing Intralipose, which were linearly moved at speeds of 1-5 mm/s.

A Time-of-Flight Range Sensor Using Four-Tap Lock-In Pixels with High near Infrared Sensitivity for LiDAR Applications.

In this paper, a back-illuminated (BSI) time-of-flight (TOF) sensor using 0.2 µm silicon-on-insulator (SOI) complementary metal oxide semiconductor (CMOS) technology is developed for long-range laser imaging detection and ranging (LiDAR) application. A 200 µm-thick bulk silicon in the SOI substrate is fully depleted by applying high negative voltage at the backside for higher quantum efficiency (QE) in a near-infrared (NIR) region. The proposed SOI-based four-tap charge modulator achieves a high-speed charge modulation and high modulation contrast of 71% in a NIR region. In addition, in-pixel drain function is used for short-pulse TOF measurements. A distance measurement up to 27 m is carried out with +1.8~-3.0% linearity error and range resolution of 4.5 cm in outdoor conditions. The measured QE of 55% is attained at 940 nm which is suitable for outdoor use due to the reduced spectral components of solar radiation.

Plasmonic Color Filter Array with High Color Purity for CMOS Image Sensors.

We demonstrate the multiband color filtering of a standard RGB color and a complementary CMY color by a plasmonic color filter, composed of concentric corrugated metallic thin film rings. The surface plasmon resonance is excited by the periodic corrugation, and the coupled light is transmitted through the central subwavelength aperture. Color selectivity is achieved not only in the visible but also in the near-infrared (NIR) region. Therefore, simultaneous imaging with visible and NIR can be realized by the integration of plasmonic color filters with sensors. We investigate the angle of incidence dependence of the transmission color selectivity and the color purity of the fabricated plasmonic color filter array.

A Wide Dynamic Range CMOS Image Sensor with a Charge Splitting Gate and Two Storage Diodes.

In this paper, a wide dynamic range (WDR) CMOS image sensor (CIS) with a charge splitting gate (SG) and two storage diodes (SDs) is presented. By using single-gate on/off control with the SG, photocurrent path to the first (SD1) or second storage diodes (SD2) is switched alternatively and periodically during exposure and signal electrons generated in a photodiode (PD) are transferred to and accumulated in the SD1 or SD2. By setting a large ratio of the off-time to on-time of the SG, two different sensitivity signals, which are originated by the same photodiode, are generated and a WDR image signal is obtained. This technique has a distinct advantage on mitigating the problem of motion artifact in WDR imaging with high and low sensitivity signals and flexible dynamic control of the dynamic range. An experimental WDR CMOS image sensor with 280 (H) × 406 (V)-pixel array consisting of 14 sub-arrays, each of which have 20 (H) × 406 (V) pixels, was implemented and tested. For the SG on/off-time ratio of 30 and 279, the DR of 93 dB and 104 dB, respectively, was demonstrated. The effect of the proposed WDR imaging operation on the reduced motion artifact was experimentally confirmed.

Multi-band plasmonic color filters for visible-to-near-infrared image sensors.

We propose a plasmonic color filter consisting of a single aperture surrounded by concentric periodic corrugations for simultaneous imaging of a spectral range from the visible to the near-infrared. The incident light coupled with surface plasmons propagates through the sub-wavelength aperture as beaming light. The beaming light transmission is able to suppress the spatial color cross-talk between the pixels in an image sensor. We analyzed the transmission characteristics of a plasmonic color filter with periodic corrugations in a silver thin film by using the finite-difference time-domain algorithm. We demonstrated a multi-band transmission wavelength selectivity, of about 100 nm, for the spectral bandwidth ranging from visible to near-infrared. The simultaneous discrimination of visible and near-infrared light with a high color purity by the plasmonic color filter achieves both improved image recognition and smaller system-size compared with conventional systems, which is particularly important for applications such as vehicle-mounted cameras, security, and biological tissue engineering.

The Dynamic Photometric Stereo Method Using a Multi-Tap CMOS Image Sensor.

The photometric stereo method enables estimation of surface normals from images that have been captured using different but known lighting directions. The classical photometric stereo method requires at least three images to determine the normals in a given scene. However, this method cannot be applied to dynamic scenes because it is assumed that the scene remains static while the required images are captured. In this work, we present a dynamic photometric stereo method for estimation of the surface normals in a dynamic scene. We use a multi-tap complementary metal-oxide-semiconductor (CMOS) image sensor to capture the input images required for the proposed photometric stereo method. This image sensor can divide the electrons from the photodiode from a single pixel into the different taps of the exposures and can thus capture multiple images under different lighting conditions with almost identical timing. We implemented a camera lighting system and created a software application to enable estimation of the normal map in real time. We also evaluated the accuracy of the estimated surface normals and demonstrated that our proposed method can estimate the surface normals of dynamic scenes.

Multi-Aperture-Based Probabilistic Noise Reduction of Random Telegraph Signal Noise and Photon Shot Noise in Semi-Photon-Counting Complementary-Metal-Oxide-Semiconductor Image Sensor.

A probabilistic method to remove the random telegraph signal (RTS) noise and to increase the signal level is proposed, and was verified by simulation based on measured real sensor noise. Although semi-photon-counting-level (SPCL) ultra-low noise complementary-metal-oxide-semiconductor (CMOS) image sensors (CISs) with high conversion gain pixels have emerged, they still suffer from huge RTS noise, which is inherent to the CISs. The proposed method utilizes a multi-aperture (MA) camera that is composed of multiple sets of an SPCL CIS and a moderately fast and compact imaging lens to emulate a very fast single lens. Due to the redundancy of the MA camera, the RTS noise is removed by the maximum likelihood estimation where noise characteristics are modeled by the probability density distribution. In the proposed method, the photon shot noise is also relatively reduced because of the averaging effect, where the pixel values of all the multiple apertures are considered. An extremely low-light condition that the maximum number of electrons per aperture was the only 2 e - was simulated. PSNRs of a test image for simple averaging, selective averaging (our previous method), and the proposed method were 11.92 dB, 11.61 dB, and 13.14 dB, respectively. The selective averaging, which can remove RTS noise, was worse than the simple averaging because it ignores the pixels with RTS noise and photon shot noise was less improved. The simulation results showed that the proposed method provided the best noise reduction performance.

A Silicon-on-Insulator-Based Dual-Gain Charge-Sensitive Pixel Detector for Low-Noise X-ray Imaging for Future Astronomical Satellite Missions.

In this paper, we report on the development of a monolithic active pixel sensor for X-ray imaging using 0.2 µm fully depleted silicon-on-insulator (SOI)-based technology to support next generation astronomical satellite missions. Detail regarding low-noise dual-gain SOI based pixels with a charge sensitive amplifier and pinned depleted diode sensor structure is presented. The proposed multi-well sensor structure underneath the fully-depleted SOI allows the design of a detector with low node capacitance and high charge collection efficiency. Configurations for achieving very high charge-to-voltage conversion gain of 52 µV/e- and 187 µV/e- are demonstrated. Furthermore, in-pixel dual gain selection is used for low-noise and wide dynamic range X-ray energy detection. A technique to improve the noise performance by removing correlated system noise leads to an improvement in the spectroscopic performance of the measured X-ray energy. Taken together, the implemented chip has low dark current (44.8 pA/cm² at -30 °C), improved noise performance (8.5 e- rms for high gain and 11.7 e- rms for low gain), and better energy resolution of 2.89% (171 eV FWHM) at 5.9 keV using 55Fe and 1.67% (234 eV FWHM) at 13.95 keV using 241Am.

Label-Free Biomedical Imaging Using High-Speed Lock-In Pixel Sensor for Stimulated Raman Scattering.

Raman imaging eliminates the need for staining procedures, providing label-free imaging to study biological samples. Recent developments in stimulated Raman scattering (SRS) have achieved fast acquisition speed and hyperspectral imaging. However, there has been a problem of lack of detectors suitable for MHz modulation rate parallel detection, detecting multiple small SRS signals while eliminating extremely strong offset due to direct laser light. In this paper, we present a complementary metal-oxide semiconductor (CMOS) image sensor using high-speed lock-in pixels for stimulated Raman scattering that is capable of obtaining the difference of Stokes-on and Stokes-off signal at modulation frequency of 20 MHz in the pixel before reading out. The generated small SRS signal is extracted and amplified in a pixel using a high-speed and large area lateral electric field charge modulator (LEFM) employing two-step ion implantation and an in-pixel pair of low-pass filter, a sample and hold circuit and a switched capacitor integrator using a fully differential amplifier. A prototype chip is fabricated using 0.11 µm CMOS image sensor technology process. SRS spectra and images of stearic acid and 3T3-L1 samples are successfully obtained. The outcomes suggest that hyperspectral and multi-focus SRS imaging at video rate is viable after slight modifications to the pixel architecture and the acquisition system.