Synthetic Data Enhancement and Network Compression Technology of Monocular Depth Estimation for Real-Time Autonomous Driving System.

Accurate 3D image recognition, critical for autonomous driving safety, is shifting from the LIDAR-based point cloud to camera-based depth estimation technologies driven by cost considerations and the point cloud's limitations in detecting distant small objects. This research aims to enhance MDE (Monocular Depth Estimation) using a single camera, offering extreme cost-effectiveness in acquiring 3D environmental data. In particular, this paper focuses on novel data augmentation methods designed to enhance the accuracy of MDE. Our research addresses the challenge of limited MDE data quantities by proposing the use of synthetic-based augmentation techniques: Mask, Mask-Scale, and CutFlip. The implementation of these synthetic-based data augmentation strategies has demonstrably enhanced the accuracy of MDE models by 4.0% compared to the original dataset. Furthermore, this study introduces the RMS (Real-time Monocular Depth Estimation configuration considering Resolution, Efficiency, and Latency) algorithm, designed for the optimization of neural networks to augment the performance of contemporary monocular depth estimation technologies through a three-step process. Initially, it selects a model based on minimum latency and REL criteria, followed by refining the model's accuracy using various data augmentation techniques and loss functions. Finally, the refined model is compressed using quantization and pruning techniques to minimize its size for efficient on-device real-time applications. Experimental results from implementing the RMS algorithm indicated that, within the required latency and size constraints, the IEBins model exhibited the most accurate REL (absolute RELative error) performance, achieving a 0.0480 REL. Furthermore, the data augmentation combination of the original dataset with Flip, Mask, and CutFlip, alongside the SigLoss loss function, displayed the best REL performance, with a score of 0.0461. The network compression technique using FP16 was analyzed as the most effective, reducing the model size by 83.4% compared to the original while maintaining the least impact on REL performance and latency. Finally, the performance of the RMS algorithm was validated on the on-device autonomous driving platform, NVIDIA Jetson AGX Orin, through which optimal deployment strategies were derived for various applications and scenarios requiring autonomous driving technologies.

Optimal Configuration of Multi-Task Learning for Autonomous Driving.

For autonomous driving, it is imperative to perform various high-computation image recognition tasks with high accuracy, utilizing diverse sensors to perceive the surrounding environment. Specifically, cameras are used to perform lane detection, object detection, and segmentation, and, in the absence of lidar, tasks extend to inferring 3D information through depth estimation, 3D object detection, 3D reconstruction, and SLAM. However, accurately processing all these image recognition operations in real-time for autonomous driving under constrained hardware conditions is practically unfeasible. In this study, considering the characteristics of image recognition tasks performed by these sensors and the given hardware conditions, we investigated MTL (multi-task learning), which enables parallel execution of various image recognition tasks to maximize their processing speed, accuracy, and memory efficiency. Particularly, this study analyzes the combinations of image recognition tasks for autonomous driving and proposes the MDO (multi-task decision and optimization) algorithm, consisting of three steps, as a means for optimization. In the initial step, a MTS (multi-task set) is selected to minimize overall latency while meeting minimum accuracy requirements. Subsequently, additional training of the shared backbone and individual subnets is conducted to enhance accuracy with the predefined MTS. Finally, both the shared backbone and each subnet undergo compression while maintaining the already secured accuracy and latency performance. The experimental results indicate that integrated accuracy performance is critically important in the configuration and optimization of MTL, and this integrated accuracy is determined by the ITC (inter-task correlation). The MDO algorithm was designed to consider these characteristics and construct multi-task sets with tasks that exhibit high ITC. Furthermore, the implementation of the proposed MDO algorithm, coupled with additional SSL (semi-supervised learning) based training, resulted in a significant performance enhancement. This advancement manifested as approximately a 12% increase in object detection mAP performance, a 15% improvement in lane detection accuracy, and a 27% reduction in latency, surpassing the results of previous three-task learning techniques like YOLOP and HybridNet.

Improved Feature-Based Gaze Estimation Using Self-Attention Module and Synthetic Eye Images.

Gaze is an excellent indicator and has utility in that it can express interest or intention and the condition of an object. Recent deep-learning methods are mainly appearance-based methods that estimate gaze based on a simple regression from entire face and eye images. However, sometimes, this method does not give satisfactory results for gaze estimations in low-resolution and noisy images obtained in unconstrained real-world settings (e.g., places with severe lighting changes). In this study, we propose a method that estimates gaze by detecting eye region landmarks through a single eye image; and this approach is shown to be competitive with recent appearance-based methods. Our approach acquires rich information by extracting more landmarks and including iris and eye edges, similar to the existing feature-based methods. To acquire strong features even at low resolutions, we used the HRNet backbone network to learn representations of images at various resolutions. Furthermore, we used the self-attention module CBAM to obtain a refined feature map with better spatial information, which enhanced the robustness to noisy inputs, thereby yielding a performance of a 3.18% landmark localization error, a 4% improvement over the existing error and A large number of landmarks were acquired and used as inputs for a lightweight neural network to estimate the gaze. We conducted a within-datasets evaluation on the MPIIGaze, which was obtained in a natural environment and achieved a state-of-the-art performance of 4.32 degrees, a 6% improvement over the existing performance.

A Novel Method Based on GAN Using a Segmentation Module for Oligodendroglioma Pathological Image Generation.

Digital pathology analysis using deep learning has been the subject of several studies. As with other medical data, pathological data are not easily obtained. Because deep learning-based image analysis requires large amounts of data, augmentation techniques are used to increase the size of pathological datasets. This study proposes a novel method for synthesizing brain tumor pathology data using a generative model. For image synthesis, we used embedding features extracted from a segmentation module in a general generative model. We also introduce a simple solution for training a segmentation model in an environment in which the masked label of the training dataset is not supplied. As a result of this experiment, the proposed method did not make great progress in quantitative metrics but showed improved results in the confusion rate of more than 70 subjects and the quality of the visual output.

Numerical Approach to Facial Palsy Using a Novel Registration Method with 3D Facial Landmark.

Treatment of facial palsy is essential because neglecting this disorder can lead to serious sequelae and further damage. For an objective evaluation and consistent rehabilitation training program of facial palsy patients, a clinician's evaluation must be simultaneously performed alongside quantitative evaluation. Recent research has evaluated facial palsy using 68 facial landmarks as features. However, facial palsy has numerous features, whereas existing studies use relatively few landmarks; moreover, they do not confirm the degree of improvement in the patient. In addition, as the face of a normal person is not perfectly symmetrical, it must be compared with previous images taken at a different time. Therefore, we introduce three methods to numerically approach measuring the degree of facial palsy after extracting 478 3D facial landmarks from 2D RGB images taken at different times. The proposed numerical approach performs registration to compare the same facial palsy patients at different times. We scale landmarks by performing scale matching before global registration. After scale matching, coarse registration is performed with global registration. Point-to-plane ICP is performed using the transformation matrix obtained from global registration as the initial matrix. After registration, the distance symmetry, angular symmetry, and amount of landmark movement are calculated for the left and right sides of the face. The degree of facial palsy at a certain point in time can be approached numerically and can be compared with the degree of palsy at other times. For the same facial expressions, the degree of facial palsy at different times can be measured through distance and angle symmetry. For different facial expressions, the simultaneous degree of facial palsy in the left and right sides can be compared through the amount of landmark movement. Through experiments, the proposed method was tested using the facial palsy patient database at different times. The experiments involved clinicians and confirmed that using the proposed numerical approach can help assess the progression of facial palsy.

Robust Extrinsic Calibration of Multiple RGB-D Cameras with Body Tracking and Feature Matching.

RGB-D cameras have been commercialized, and many applications using them have been proposed. In this paper, we propose a robust registration method of multiple RGB-D cameras. We use a human body tracking system provided by Azure Kinect SDK to estimate a coarse global registration between cameras. As this coarse global registration has some error, we refine it using feature matching. However, the matched feature pairs include mismatches, hindering good performance. Therefore, we propose a registration refinement procedure that removes these mismatches and uses the global registration. In an experiment, the ratio of inliers among the matched features is greater than 95% for all tested feature matchers. Thus, we experimentally confirm that mismatches can be eliminated via the proposed method even in difficult situations and that a more precise global registration of RGB-D cameras can be obtained.

Efficient Single-Shot Multi-Object Tracking for Vehicles in Traffic Scenarios.

Multi-object tracking is a significant field in computer vision since it provides essential information for video surveillance and analysis. Several different deep learning-based approaches have been developed to improve the performance of multi-object tracking by applying the most accurate and efficient combinations of object detection models and appearance embedding extraction models. However, two-stage methods show a low inference speed since the embedding extraction can only be performed at the end of the object detection. To alleviate this problem, single-shot methods, which simultaneously perform object detection and embedding extraction, have been developed and have drastically improved the inference speed. However, there is a trade-off between accuracy and efficiency. Therefore, this study proposes an enhanced single-shot multi-object tracking system that displays improved accuracy while maintaining a high inference speed. With a strong feature extraction and fusion, the object detection of our model achieves an AP score of 69.93% on the UA-DETRAC dataset and outperforms previous state-of-the-art methods, such as FairMOT and JDE. Based on the improved object detection performance, our multi-object tracking system achieves a MOTA score of 68.5% and a PR-MOTA score of 24.5% on the same dataset, also surpassing the previous state-of-the-art trackers.

Development of Real-Time Hand Gesture Recognition for Tabletop Holographic Display Interaction Using Azure Kinect.

The use of human gesturing to interact with devices such as computers or smartphones has presented several problems. This form of interaction relies on gesture interaction technology such as Leap Motion from Leap Motion, Inc, which enables humans to use hand gestures to interact with a computer. The technology has excellent hand detection performance, and even allows simple games to be played using gestures. Another example is the contactless use of a smartphone to take a photograph by simply folding and opening the palm. Research on interaction with other devices via hand gestures is in progress. Similarly, studies on the creation of a hologram display from objects that actually exist are also underway. We propose a hand gesture recognition system that can control the Tabletop holographic display based on an actual object. The depth image obtained using the latest Time-of-Flight based depth camera Azure Kinect is used to obtain information about the hand and hand joints by using the deep-learning model CrossInfoNet. Using this information, we developed a real time system that defines and recognizes gestures indicating left, right, up, and down basic rotation, and zoom in, zoom out, and continuous rotation to the left and right.

Real-Time Vehicle-Detection Method in Bird-View Unmanned-Aerial-Vehicle Imagery.

Vehicle detection is an important research area that provides background information for the diversity of unmanned-aerial-vehicle (UAV) applications. In this paper, we propose a vehicle-detection method using a convolutional-neural-network (CNN)-based object detector. We design our method, DRFBNet300, with a Deeper Receptive Field Block (DRFB) module that enhances the expressiveness of feature maps to detect small objects in the UAV imagery. We also propose the UAV-cars dataset that includes the composition and angular distortion of vehicles in UAV imagery to train our DRFBNet300. Lastly, we propose a Split Image Processing (SIP) method to improve the accuracy of the detection model. Our DRFBNet300 achieves 21 mAP with 45 FPS in the MS COCO metric, which is the highest score compared to other lightweight single-stage methods running in real time. In addition, DRFBNet300, trained on the UAV-cars dataset, obtains the highest AP score at altitudes of 20-50 m. The gap of accuracy improvement by applying the SIP method became larger when the altitude increases. The DRFBNet300 trained on the UAV-cars dataset with SIP method operates at 33 FPS, enabling real-time vehicle detection.

360-degree color hologram generation for real 3D objects.

Recently, holographic display and computer-generated holograms calculated from real existing objects have been more actively investigated to support holographic video applications. In this paper, we proposed a method of generating 360-degree color holograms of real 3D objects in an efficient manner. 360-degree 3D images are generated using the actual 3D image acquisition system consisting of a depth camera and a turntable and intermediate view generation. Then, 360-degree color holograms are calculated using a viewing-window-based computer-generated hologram. We confirmed that floating 3D objects are faithfully reconstructed around a 360-degree direction using our 360-degree tabletop color holographic display.

Scalable hologram video coding for adaptive transmitting service.

This paper discusses processing techniques for an adaptive digital holographic video service in various reconstruction environments, and proposes two new scalable coding schemes. The proposed schemes are constructed according to the hologram generation or acquisition schemes: hologram-based resolution-scalable coding (HRS) and light source-based signal-to-noise ratio scalable coding (LSS). HRS is applied for holograms that are already acquired or generated, while LSS is applied to the light sources before generating digital holograms. In the LSS scheme, the light source information is lossless coded because it is too important to lose, while the HRS scheme adopts a lossy coding method. In an experiment, we provide eight stages of an HRS scheme whose data compression ratios range from 1:1 to 100:1 for each layered data. For LSS, four layers and 16 layers of scalable coding schemes are provided. We experimentally show that the proposed techniques make it possible to service a digital hologram video adaptively to the various displays with different resolutions, computation capabilities of the receiver side, or bandwidths of the network.

Hardware architecture of high-performance digital hologram generator on the basis of a pixel-by-pixel calculation scheme.

In this paper we propose a hardware architecture for high-speed computer-generated hologram generation that significantly reduces the number of memory access times to avoid the bottleneck in the memory access operation. For this, we use three main schemes. The first is pixel-by-pixel calculation, rather than light source-by-source calculation. The second is a parallel calculation scheme extracted by modifying the previous recursive calculation scheme. The last scheme is a fully pipelined calculation scheme and exactly structured timing scheduling, achieved by adjusting the hardware. The proposed hardware is structured to calculate a row of a computer-generated hologram in parallel and each hologram pixel in a row is calculated independently. It consists of and input interface, an initial parameter calculator, hologram pixel calculators, a line buffer, and a memory controller. The implemented hardware to calculate a row of a 1920×1080 computer-generated hologram in parallel uses 168,960 lookup tables, 153,944 registers, and 19,212 digital signal processing blocks in an Altera field programmable gate array environment. It can stably operate at 198 MHz. Because of three schemes, external memory bandwidth is reduced to approximately 1/20,000 of the previous ones at the same calculation speed.

Cell-based hardware architecture for full-parallel generation algorithm of digital holograms.

This paper proposes a new hardware architecture to speed-up the digital hologram calculation by parallel computation. To realize it, we modify the computer-generated hologram (CGH) equation and propose a cell-based very large scale integrated circuit architecture. We induce a new equation to calculate the horizontal or vertical hologram pixel values in parallel, after finding the calculation regularity in the horizontal or vertical direction from the basic CGH equation. We also propose the architecture of the computer-generated hologram cell consisting of an initial parameter calculator and update-phase calculators based on the equation, and then implement them in hardware. Modifying the equation could simplify the hardware, and approximating the cosine function could optimize the hardware. In addition, we show the hardware architecture to parallelize the calculation in the horizontal direction by extending computer-generated holograms. In the experiments, we analyze hardware resource usage and the performance-capability characteristics of the look-up table used in the computer-generated hologram cell. These analyses make it possible to select the amount of hardware to the precision of the results. Here, we used the platform from our previous work for the computer-generated hologram kernel and the structure of the processor.

CD44-Mediated Methotrexate Delivery by Hyaluronan-Coated Nanoparticles Composed of a Branched Cell-Penetrating Peptide.

Branched polymers as drug delivery carriers have been widely attempted due to their outstanding drug loading capability and complex stability like branched polyethyleneimine (B-PEI). However, branched polymers without biodegradability may cause toxicity as they can accumulate in the body. Herein, we report branched modified nona-arginine (B-mR9) composed of redox-cleavable disulfide bonds to form stable complexes with methotrexate (MTX) as an anticancer agent, which is further coated with hyaluronic acid (HA). The HA-coated nanoparticles provide targetability for the CD44 cell surface receptor. The B-mR9-MTX/HA can effectively aid in intracellular MTX delivery to CD44 overexpressing cancer cells being degradable by the reducing environments of the cancer cells. The B-mR9-MTX/HA exhibits not only a glutathione-triggered degradability but also an outstanding CD44-mediated MTX delivery efficacy. In addition, its superior tumor inhibition capability was confirmed through an in vivo study. The results suggest that the HA-coated B-mR9 nanoparticle can be used as a drug delivery platform.

An Extended Method for Saccadic Eye Movement Measurements Using a Head-Mounted Display.

Saccadic eye movementis an important ability in our daily life and is especially important in driving and sports. Traditionally, the Developmental Eye Movement (DEM) test and the King-Devick (K-D) test have been used to measure saccadic eye movement, but these only involve measurements with "adjusted time". Therefore, a different approach is required to obtain the eye movement speed and reaction rate in detail, as some are rapid eye movements, while others are slow actions, and vice versa. This study proposed a extended method that can acquire the "rest time" and "transfer time", as well as the "adjusted time", by implementing a virtual reality-based DEM test, using a FOVE virtual reality (VR) head-mounted display, equipped with an eye-tracking module. This approach was tested in 30 subjects with normal vision and no ophthalmologic disease by using a 2-diopter (50-cm) distance. This allowed for measurement of the "adjusted time" and the "rest time" for focusing on each target number character, the "transfer time" for moving to the next target number character, and recording of the gaze-tracking log. The results of this experiment showed that it was possible to analyze more parameters of the saccadic eye movement with the proposed method than with the traditional methods.

A Helical Polypeptide-Based Potassium Ionophore Induces Endoplasmic Reticulum Stress-Mediated Apoptosis by Perturbing Ion Homeostasis.

Perturbation of potassium homeostasis can affect various cell functions and lead to the onset of programmed cell death. Although ionophores have been intensively used as an ion homeostasis disturber, the mechanisms of cell death are unclear and the bioapplicability is limited. In this study, helical polypeptide-based potassium ionophores are developed to induce endoplasmic reticulum (ER) stress-mediated apoptosis. The polypeptide-based potassium ionophores disturb ion homeostasis and then induce prolonged ER stress in the cells. The ER stress results in oxidative environments that accelerate the activation of mitochondria-dependent apoptosis. Moreover, ER stress-mediated apoptosis is triggered in a tumor-bearing mouse model that suppresses tumor proliferation. This study provides the first evidence showing that helical polypeptide-based potassium ionophores trigger ER stress-mediated apoptosis by perturbation of potassium homeostasis.

Curcumin as a Novel Nanocarrier System for Doxorubicin Delivery to MDR Cancer Cells: In Vitro and In Vivo Evaluation.

Curcumin (CRC) has been widely used as a therapeutic agent for various drug delivery applications. In this work, we focused on the applicability of CRC as a nanodrug delivery agent for doxorubicin hydrochloride (DOX) (commercially known as Adriamycin) coated with poly(ethylene glycol) (PEG) as an effective therapeutic strategy against multidrug-resistant cancer cells. The developed PEG-coated CRC/DOX nanoparticles (NPs) (PEG-CRC/DOX NPs) were well localized within the resistant cancer cells inducing apoptosis confirmed by flow cytometry and DNA fragmentation assays. The PEG-CRC/DOX NPs suppressed the major efflux proteins in DOX-resistant cancer cells. The in vivo biodistribution studies on HCT-8/DOX-resistant tumor xenograft showed improved bioavailability of the PEG-CRC/DOX NPs, and thereby suppressed tumor growth significantly compared to the other samples. This study clearly shows that curcumin nanoparticles could deliver DOX efficiently into the multidrug-resistant cancer cells to have potential therapeutic benefits.

Author Correction: Evaluation of cell penetrating peptide coated Mn:ZnS nanoparticles for paclitaxel delivery to cancer cells.

A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has not been fixed in the paper.

Evaluation of cell penetrating peptide coated Mn:ZnS nanoparticles for paclitaxel delivery to cancer cells.

This work aimed at formulating paclitaxel (PTX) loaded cell penetrating peptide (CPP) coated Mn doped ZnS nanoparticles (Mn:ZnS NPs) for improved anti-cancer efficacy in vitro and in vivo. The developed PTX loaded Mn:ZnS NPs with different CPPs (PEN, pVEC and R9) showed enhanced anti-cancer effect compared to bare PTX, which has been validated by MTT assay followed by apoptosis assay and DNA fragmentation analysis. The in vivo bio-distribution and anti-cancer efficacy was studied on breast cancer xenograft model showing maximum tumor localization and enhanced therapeutic efficacy with R9 coated Mn:ZnS NPs (R9:Mn:ZnS NPs) and was confirmed by H/E staining. Thus, R9:Mn:ZnS NPs could be an ideal theranostic nano-carrier for PTX with enhanced  the rapeutic efficacy toward cancer cells, where penetration and sustainability of therapeutics are essential.

Protease-activatable cell-penetrating peptide possessing ROS-triggered phase transition for enhanced cancer therapy.

Reactive oxygen species (ROS)- or protease-responsive materials have been utilized as carriers in cancer therapies because ROS and specific proteases are overproduced in cancer cells. Methionine-based polypeptides containing a thioether group are promising candidates due to their ROS-responsiveness which provides a phase transition. Herein, we developed protease-activatable cell-penetrating peptide containing a ROS-responsive methionine, a cell permeable lysine chain, and a matrix metalloproteinase (MMP)-cleavable linker. We designed a poly(l-methionine-block-l-lysine)-PLGLAG-PEG (MLMP) and doxorubicin (DOX) was loaded into the micelle core. The MLMP exhibited MMP-sensitive cleavage and ROS-induced DOX release. Moreover, we confirmed efficient DOX delivery into cancer cells and induction of the apoptotic capability in vitro. In a bio-distribution study, IR-780 dye encapsulated MLMP showed superior tumor targetability with long retention. Furthermore, MLMP (DOX) exhibited outstanding tumor inhibition capability with non-toxicity compared to free DOX, indicating that dual stimuli-MLMP has great potential as an anticancer drug delivery platform.