High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep.

Active vision systems (AVSs) have been widely used to obtain high-resolution images of objects of interest. However, tracking small objects in high-magnification scenes is challenging due to shallow depth of field (DoF) and narrow field of view (FoV). To address this, we introduce a novel high-speed AVS with a continuous autofocus (C-AF) approach based on dynamic-range focal sweep and a high-frame-rate (HFR) frame-by-frame tracking pipeline. Our AVS leverages an ultra-fast pan-tilt mechanism based on a Galvano mirror, enabling high-frequency view direction adjustment. Specifically, the proposed C-AF approach uses a 500 fps high-speed camera and a focus-tunable liquid lens operating at a sine wave, providing a 50 Hz focal sweep around the object's optimal focus. During each focal sweep, 10 images with varying focuses are captured, and the one with the highest focus value is selected, resulting in a stable output of well-focused images at 50 fps. Simultaneously, the object's depth is measured using the depth-from-focus (DFF) technique, allowing dynamic adjustment of the focal sweep range. Importantly, because the remaining images are only slightly less focused, all 500 fps images can be utilized for object tracking. The proposed tracking pipeline combines deep-learning-based object detection, K-means color clustering, and HFR tracking based on color filtering, achieving 500 fps frame-by-frame tracking. Experimental results demonstrate the effectiveness of the proposed C-AF approach and the advanced capabilities of the high-speed AVS for magnified object tracking.

A Novel Dynamic Light-Section 3D Reconstruction Method for Wide-Range Sensing.

Existing galvanometer-based laser-scanning systems are challenging to apply in multi-scale 3D reconstruction because of the difficulty in achieving a balance between a high reconstruction accuracy and a wide reconstruction range. This paper presents a novel method that synchronizes laser scanning by switching the field-of-view (FOV) of a camera using multi-galvanometers. Beyond the advanced hardware setup, we establish a comprehensive geometric model of the system by modeling dynamic camera, dynamic laser, and their combined interaction. Furthermore, since existing calibration methods mainly focus on either dynamic lasers or dynamic cameras and have certain limitations, we propose a novel high-precision and flexible calibration method by constructing an error model and minimizing the objective function. The performance of the proposed method was evaluated by scanning standard components. The results show that the proposed 3D reconstruction system achieves an accuracy of 0.3 mm when the measurement range is extended to 1100 mm × 1300 mm × 650 mm. This demonstrates that for meter-scale reconstruction ranges, a sub-millimeter measurement accuracy is achieved, indicating that the proposed method realizes multi-scale 3D reconstruction and simultaneously allows for high-precision and wide-range 3D reconstruction in industrial applications.

An Active Multi-Object Ultrafast Tracking System with CNN-Based Hybrid Object Detection.

This study proposes a visual tracking system that can detect and track multiple fast-moving appearance-varying targets simultaneously with 500 fps image processing. The system comprises a high-speed camera and a pan-tilt galvanometer system, which can rapidly generate large-scale high-definition images of the wide monitored area. We developed a CNN-based hybrid tracking algorithm that can robustly track multiple high-speed moving objects simultaneously. Experimental results demonstrate that our system can track up to three moving objects with velocities lower than 30 m per second simultaneously within an 8-m range. The effectiveness of our system was demonstrated through several experiments conducted on simultaneous zoom shooting of multiple moving objects (persons and bottles) in a natural outdoor scene. Moreover, our system demonstrates high robustness to target loss and crossing situations.

HFR-Video-Based Stereo Correspondence Using High Synchronous Short-Term Velocities.

This study focuses on solving the correspondence problem of multiple moving objects with similar appearances in stereoscopic videos. Specifically, we address the multi-camera correspondence problem by taking into account the pixel-level and feature-level stereo correspondences, and object-level cross-camera multiple object correspondence. Most correspondence algorithms rely on texture and color information of the stereo images, making it challenging to distinguish between similar-looking objects, such as ballet dancers and corporate employees wearing similar dresses, or farm animals such as chickens, ducks, and cows. However, by leveraging the low latency and high synchronization of high-speed cameras, we can perceive the phase and frequency differences between the movements of similar-looking objects. In this study, we propose using short-term velocities (STVs) of objects as motion features to determine the correspondence of multiple objects by calculating the similarity of STVs. To validate our approach, we conducted stereo correspondence experiments using markers attached to a metronome and natural hand movements to simulate simple and complex motion scenes. The experimental results demonstrate that our method achieved good performance in stereo correspondence.

Experimental studies on droplet characteristics in a microfluidic flow focusing droplet generator: effect of continuous phase on droplet encapsulation.

The efficacy of droplet-based microfluidic assays depends on droplet size, pattern, generation rate, etc. The size of the droplet is affected by numerous variables as flow rate ratio, viscosity ratio, microchannel geometry, surfactants, nature of fluids and other dimensionless numbers. This work reports rigorous analysis and optimization of the behavior of droplets with change in flow rate ratio and viscosity ratio in a flow-focusing device. Droplets were produced for different flow rate ratios maintaining a constant aqueous phase and varying the continuous phase, to have capillary numbers ranging from 0.01 to 0.1. It was observed that the droplet size decreased with the increase in flow rate ratio, and vice versa. It was noted that as the viscosity ratio was increased, the dispersed phase elongated before the complete breakup and long droplets were formed in the microchannel. Smaller droplets were formed for lower viscosity ratios with a combination of higher flow rate ratios. An empirical relation has been developed to predict the droplet length in terms of capillary number and flow rate ratio for different viscosity ratios. In addition, microparticle encapsulation in individual droplets was attempted to realize the effect of flow rate of the continuous phase for various flow rate ratios on encapsulation efficiency.

Single-Camera-Based Bridge Structural Displacement Measurement with Traffic Counting.

Vision-based structural displacement methods allow convenient monitoring of civil structures such as bridges, though they are often limited due to the small number of measurement points, constrained spatial resolution, and inability to identify the acting forces of the measured displacement. To increase the number of measurement points in vision-based bridge displacement measurement, this study introduces a front-view tandem marker motion capture system with side-view traffic counting to identify the force-inducing passing vehicles on the bridge's deck. The proposed system was able to measure structural displacement at submillimeter resolution on eight measurement points at once at a distance of 40.8-64.2 m from a front-view camera. The traffic counting system with a side-view camera recorded the passing vehicles from two opposing lanes. We conducted a 35-min experiment for a 25 m-span steel road bridge with hundreds of cars passing over it and confirmed dynamic displacement distributions with amplitudes of several millimeters when large vehicles passed.

HFR Projector Camera Based Visible Light Communication System for Real-Time Video Streaming.

This study develops a projector-camera-based visible light communication (VLC) system for real-time broadband video streaming, in which a high frame rate (HFR) projector can encode and project a color input video sequence into binary image patterns modulated at thousands of frames per second and an HFR vision system can capture and decode these binary patterns into the input color video sequence with real-time video processing. For maximum utilization of the high-throughput transmission ability of the HFR projector, we introduce a projector-camera VLC protocol, wherein a multi-level color video sequence is binary-modulated with a gray code for encoding and decoding instead of pure-code-based binary modulation. Gray code encoding is introduced to address the ambiguity with mismatched pixel alignments along the gradients between the projector and vision system. Our proposed VLC system consists of an HFR projector, which can project 590 × 1060 binary images at 1041 fps via HDMI streaming and a monochrome HFR camera system, which can capture and process 12-bit 512 × 512 images in real time at 3125 fps; it can simultaneously decode and reconstruct 24-bit RGB video sequences at 31 fps, including an error correction process. The effectiveness of the proposed VLC system was verified via several experiments by streaming offline and live video sequences.

An FPGA-Based Ultra-High-Speed Object Detection Algorithm with Multi-Frame Information Fusion.

An ultra-high-speed algorithm based on Histogram of Oriented Gradient (HOG) and Support Vector Machine (SVM) for hardware implementation at 10,000 frames per second (FPS) under complex backgrounds is proposed for object detection. The algorithm is implemented on the field-programmable gate array (FPGA) in the high-speed-vision platform, in which 64 pixels are input per clock cycle. The high pixel parallelism of the vision platform limits its performance, as it is difficult to reduce the strides between detection windows below 16 pixels, thus introduce non-negligible deviation of object detection. In addition, limited by the transmission bandwidth, only one frame in every four frames can be transmitted to PC for post-processing, that is, 75% image information is wasted. To overcome the mentioned problem, a multi-frame information fusion model is proposed in this paper. Image data and synchronization signals are first regenerated according to image frame numbers. The maximum HOG feature value and corresponding coordinates of each frame are stored in the bottom of the image with that of adjacent frames'. The compensated ones will be obtained through information fusion with the confidence of continuous frames. Several experiments are conducted to demonstrate the performance of the proposed algorithm. As the evaluation result shows, the deviation is reduced with our proposed method compared with the existing one.

Development of an Active High-Speed 3-D Vision System.

High-speed recognition of the shape of a target object is indispensable for robots to perform various kinds of dexterous tasks in real time. In this paper, we propose a high-speed 3-D sensing system with active target-tracking. The system consists of a high-speed camera head and a high-speed projector, which are mounted on a two-axis active vision system. By measuring a projected coded pattern, 3-D measurement at a rate of 500 fps was achieved. The measurement range was increased as a result of the active tracking, and the shape of the target was accurately observed even when it moved quickly. In addition, to obtain the position and orientation of the target, 500 fps real-time model matching was achieved.

Motion-Blur-Free High-Speed Video Shooting Using a Resonant Mirror.

This study proposes a novel concept of actuator-driven frame-by-frame intermittent tracking for motion-blur-free video shooting of fast-moving objects. The camera frame and shutter timings are controlled for motion blur reduction in synchronization with a free-vibration-type actuator vibrating with a large amplitude at hundreds of hertz so that motion blur can be significantly reduced in free-viewpoint high-frame-rate video shooting for fast-moving objects by deriving the maximum performance of the actuator. We develop a prototype of a motion-blur-free video shooting system by implementing our frame-by-frame intermittent tracking algorithm on a high-speed video camera system with a resonant mirror vibrating at 750 Hz. It can capture 1024 × 1024 images of fast-moving objects at 750 fps with an exposure time of 0.33 ms without motion blur. Several experimental results for fast-moving objects verify that our proposed method can reduce image degradation from motion blur without decreasing the camera exposure time.

Pixel-Level and Robust Vibration Source Sensing in High-Frame-Rate Video Analysis.

We investigate the effect of appearance variations on the detectability of vibration feature extraction with pixel-level digital filters for high-frame-rate videos. In particular, we consider robust vibrating object tracking, which is clearly different from conventional appearance-based object tracking with spatial pattern recognition in a high-quality image region of a certain size. For 512 × 512 videos of a rotating fan located at different positions and orientations and captured at 2000 frames per second with different lens settings, we verify how many pixels are extracted as vibrating regions with pixel-level digital filters. The effectiveness of dynamics-based vibration features is demonstrated by examining the robustness against changes in aperture size and the focal condition of the camera lens, the apparent size and orientation of the object being tracked, and its rotational frequency, as well as complexities and movements of background scenes. Tracking experiments for a flying multicopter with rotating propellers are also described to verify the robustness of localization under complex imaging conditions in outside scenarios.

Droplet-based lab-on-chip platform integrated with laser ablated graphene heaters to synthesize gold nanoparticles for electrochemical sensing and fuel cell applications.

Controlled, stable and uniform temperature environment with quick response are crucial needs for many lab-on-chip (LOC) applications requiring thermal management. Laser Induced Graphene (LIG) heater is one such mechanism capable of maintaining a wide range of steady state temperature. LIG heaters are thin, flexible, and inexpensive and can be fabricated easily in different geometric configurations. In this perspective, herein, the electro-thermal performance of the LIG heater has been examined for different laser power values and scanning speeds. The experimented laser ablated patterns exhibited varying electrical conductivity corresponding to different combinations of power and speed of the laser. The conductivity of the pattern can be tailored by tuning the parameters which exhibit, a wide range of temperatures making them suitable for diverse lab-on-chip applications. A maximum temperature of 589 °C was observed for a combination of 15% laser power and 5.5% scanning speed. A LOC platform was realized by integrating the developed LIG heaters with a droplet-based microfluidic device. The performance of this LOC platform was analyzed for effective use of LIG heaters to synthesize Gold nanoparticles (GNP). Finally, the functionality of the synthesized GNPs was validated by utilizing them as catalyst in enzymatic glucose biofuel cell and in electrochemical applications.

Real-time laryngoscopic measurements of vocal-fold vibrations.

We have developed a high-frame-rate laryngoscope that can measure the vibration distribution of a human vocal fold in real time at hundreds of hertz. Our laryngoscope can extract a vocal-fold contour at 4000 fps as 20 pairs of its left and right border points from 256 × 512-pixel images to quantify left-right asymmetry of vocal-fold vibrations. Experiments on artificial vocal-fold-like vibrations of a silicon rubber membrane were performed to confirm the laryngoscope's effectiveness, and the vocal folds of human subjects, including patients with laryngeal diseases, were examined under clinical conditions.

Development of broad-view camera unit for laparoscopic surgery.

A disadvantage of laparoscopic surgery is the narrow operative field provided by the endoscope camera. This paper describes a newly developed broad-view camera unit for use with the Broad-View Camera System, which is capable of providing a wider view of the internal organs during laparoscopic surgery. The developed camera unit is composed of a miniature color CMOS camera, an indwelling needle, and an extra-thin connector. The specific design of the camera unit and the method for positioning it are shown. The performance of the camera unit has been confirmed through basic and animal experiments.

A sensitive gait parameter for quantification of arthritis in rats.

Quantification of arthritis is helpful for investigating pain mechanisms of arthritis and for developing new drugs. We assessed and identified a feasible parameter for quantification of rat arthritis using a novel gait analyzing system. Knee-joint injection with small doses of lambda-carrageenan decreased swing time ratio (STR, swing time of the non-treated hindlimb/swing time of the lambda-carrageenan-injected hindlimb) in a dose-dependent manner. Intraperitoneal treatment with indomethacin restored the decreased STR dose-dependently. The arthritis could not be accurately quantified by swing time and swelling, common indices of arthritis. These results show that STR is a sensitive, reliable parameter for quantification of arthritis.