Communication Delay Outlier Detection and Compensation for Teleoperation Using Stochastic State Estimation.

There have been numerous studies attempting to overcome the limitations of current autonomous driving technologies. However, there is no doubt that it is challenging to promise integrity of safety regarding urban driving scenarios and dynamic driving environments. Among the reported countermeasures to supplement the uncertain behavior of autonomous vehicles, teleoperation of the vehicle has been introduced to deal with the disengagement of autonomous driving. However, teleoperation can lead the vehicle to unforeseen and hazardous situations from the viewpoint of wireless communication stability. In particular, communication delay outliers that severely deviate from the passive communication delay should be highlighted because they could hamper the cognition of the circumstances monitored by the teleoperator, or the control signal could be contaminated regardless of the teleoperator's intention. In this study, communication delay outliers were detected and classified based on the stochastic approach (passive delays and outliers were estimated as 98.67% and 1.33%, respectively). Results indicate that communication delay outliers can be automatically detected, independently of the real-time quality of wireless communication stability. Moreover, the proposed framework demonstrates resilience against outliers, thereby mitigating potential performance degradation.

Predictive Path-Tracking Control of an Autonomous Electric Vehicle with Various Multi-Actuation Topologies.

This paper presents the development of path-tracking control strategies for an over-actuated autonomous electric vehicle. The vehicle platform is equipped with four-wheel steering (4WS) as well as torque vectoring (TV) capabilities, which enable the control of vehicle dynamics to be enhanced. A nonlinear model predictive controller is proposed taking into account the nonlinearities in vehicle dynamics at the limits of handling as well as the crucial actuator constraints. Controllers with different actuation formulations are presented and compared to study the path-tracking performance of the vehicle with different levels of actuation. The controllers are implemented in a high-fidelity simulation environment considering scenarios of vehicle handling limits. According to the simulation results, the vehicle achieves the best overall path-tracking performance with combined 4WS and TV, which illustrates that the over-actuation topology can enhance the path-tracking performance during conditions under the limits of handling. In addition, the performance of the over-actuation controller is further assessed with different sampling times as well as prediction horizons in order to investigate the effect of such parameters on the control performance, and its capability for real-time execution. In the end, the over-actuation control strategy is implemented on a target machine for real-time validation. The control formulation proposed in this paper is proven to be compatible with different levels of actuation, and it is also demonstrated in this work that it is possible to include the particular over-actuation formulation and specific nonlinear vehicle dynamics in real-time operation, with the sampling time and prediction time providing a compromise between path-tracking performance and computational time.

Visual Detection of Road Cracks for Autonomous Vehicles Based on Deep Learning.

Detecting road cracks is essential for inspecting and assessing the integrity of concrete pavement structures. Traditional image-based methods often require complex preprocessing to extract crack features, making them challenging when dealing with noisy concrete surfaces in diverse real-world scenarios, such as autonomous vehicle road detection. This study introduces an image-based crack detection approach that combines a Random Forest machine learning classifier with a deep convolutional neural network (CNN) to address these challenges. Three state-of-the-art models, namely MobileNet, InceptionV3, and Xception, were employed and trained using a dataset of 30,000 images to build an effective CNN. A systematic comparison of validation accuracy across various base learning rates identified a base learning rate of 0.001 as optimal, achieving a maximum validation accuracy of 99.97%. This optimal learning rate was then applied in the subsequent testing phase. The robustness and flexibility of the trained models were evaluated using 6,000 test photos, each with a resolution of 224 × 224 pixels, which were not part of the training or validation sets. The outstanding results, boasting a remarkable 99.95% accuracy, 99.95% precision, 99.94% recall, and a matching 99.94% F1 Score, unequivocally affirm the efficacy of the proposed technique in precisely identifying road fractures in photographs taken on real concrete surfaces.

A Survey of Autonomous Vehicle Behaviors: Trajectory Planning Algorithms, Sensed Collision Risks, and User Expectations.

Autonomous vehicles are rapidly advancing and have the potential to revolutionize transportation in the future. This paper primarily focuses on vehicle motion trajectory planning algorithms, examining the methods for estimating collision risks based on sensed environmental information and approaches for achieving user-aligned trajectory planning results. It investigates the different categories of planning algorithms within the scope of local trajectory planning applications for autonomous driving, discussing and differentiating their properties in detail through a review of the recent studies. The risk estimation methods are classified and introduced based on their descriptions of the sensed collision risks in traffic environments and their integration with trajectory planning algorithms. Additionally, various user experience-oriented methods, which utilize human data to enhance the trajectory planning performance and generate human-like trajectories, are explored. The paper provides comparative analyses of these algorithms and methods from different perspectives, revealing the interconnections between these topics. The current challenges and future prospects of the trajectory planning tasks in autonomous vehicles are also discussed.

Persona-PhysioSync AV: Personalized Interaction through Personality and Physiology Monitoring in Autonomous Vehicles.

The emergence of autonomous vehicles (AVs) marks a transformative leap in transportation technology. Central to the success of AVs is ensuring user safety, but this endeavor is accompanied by the challenge of establishing trust and acceptance of this novel technology. The traditional "one size fits all" approach to AVs may limit their broader societal, economic, and cultural impact. Here, we introduce the Persona-PhysioSync AV (PPS-AV). It adopts a comprehensive approach by combining personality traits with physiological and emotional indicators to personalize the AV experience to enhance trust and comfort. A significant aspect of the PPS-AV framework is its real-time monitoring of passenger engagement and comfort levels within AVs. It considers a passenger's personality traits and their interaction with physiological and emotional responses. The framework can alert passengers when their engagement drops to critical levels or when they exhibit low situational awareness, ensuring they regain attentiveness promptly, especially during Take-Over Request (TOR) events. This approach fosters a heightened sense of Human-Vehicle Interaction (HVI), thereby building trust in AV technology. While the PPS-AV framework currently provides a foundational level of state diagnosis, future developments are expected to include interaction protocols that utilize interfaces like haptic alerts, visual cues, and auditory signals. In summary, the PPS-AV framework is a pivotal tool for the future of autonomous transportation. By prioritizing safety, comfort, and trust, it aims to make AVs not just a mode of transport but a personalized and trusted experience for passengers, accelerating the adoption and societal integration of autonomous vehicles.

MRD-YOLO: A Multispectral Object Detection Algorithm for Complex Road Scenes.

Object detection is one of the core technologies for autonomous driving. Current road object detection mainly relies on visible light, which is prone to missed detections and false alarms in rainy, night-time, and foggy scenes. Multispectral object detection based on the fusion of RGB and infrared images can effectively address the challenges of complex and changing road scenes, improving the detection performance of current algorithms in complex scenarios. However, previous multispectral detection algorithms suffer from issues such as poor fusion of dual-mode information, poor detection performance for multi-scale objects, and inadequate utilization of semantic information. To address these challenges and enhance the detection performance in complex road scenes, this paper proposes a novel multispectral object detection algorithm called MRD-YOLO. In MRD-YOLO, we utilize interaction-based feature extraction to effectively fuse information and introduce the BIC-Fusion module with attention guidance to fuse different modal information. We also incorporate the SAConv module to improve the model's detection performance for multi-scale objects and utilize the AIFI structure to enhance the utilization of semantic information. Finally, we conduct experiments on two major public datasets, FLIR_Aligned and M3FD. The experimental results demonstrate that compared to other algorithms, the proposed algorithm achieves superior detection performance in complex road scenes.

Multi-Tracking Sensor Architectures for Reconstructing Autonomous Vehicle Crashes: An Exploratory Study.

With the continuous development of new sensor features and tracking algorithms for object tracking, researchers have opportunities to experiment using different combinations. However, there is no standard or agreed method for selecting an appropriate architecture for autonomous vehicle (AV) crash reconstruction using multi-sensor-based sensor fusion. This study proposes a novel simulation method for tracking performance evaluation (SMTPE) to solve this problem. The SMTPE helps select the best tracking architecture for AV crash reconstruction. This study reveals that a radar-camera-based centralized tracking architecture of multi-sensor fusion performed the best among three different architectures tested with varying sensor setups, sampling rates, and vehicle crash scenarios. We provide a brief guideline for the best practices in selecting appropriate sensor fusion and tracking architecture arrangements, which can be helpful for future vehicle crash reconstruction and other AV improvement research.

INSANet: INtra-INter Spectral Attention Network for Effective Feature Fusion of Multispectral Pedestrian Detection.

Pedestrian detection is a critical task for safety-critical systems, but detecting pedestrians is challenging in low-light and adverse weather conditions. Thermal images can be used to improve robustness by providing complementary information to RGB images. Previous studies have shown that multi-modal feature fusion using convolution operation can be effective, but such methods rely solely on local feature correlations, which can degrade the performance capabilities. To address this issue, we propose an attention-based novel fusion network, referred to as INSANet (INtra-INter Spectral Attention Network), that captures global intra- and inter-information. It consists of intra- and inter-spectral attention blocks that allow the model to learn mutual spectral relationships. Additionally, we identified an imbalance in the multispectral dataset caused by several factors and designed an augmentation strategy that mitigates concentrated distributions and enables the model to learn the diverse locations of pedestrians. Extensive experiments demonstrate the effectiveness of the proposed methods, which achieve state-of-the-art performance on the KAIST dataset and LLVIP dataset. Finally, we conduct a regional performance evaluation to demonstrate the effectiveness of our proposed network in various regions.

A Novel Machine Learning-Based ANFIS Calibrated RISS/GNSS Integration for Improved Navigation in Urban Environments.

Autonomous vehicles (AVs) require accurate navigation, but the reliability of Global Navigation Satellite Systems (GNSS) can be degraded by signal blockage and multipath interference in urban areas. Therefore, a navigation system that integrates a calibrated Reduced Inertial Sensors System (RISS) with GNSS is proposed. The system employs a machine-learning-based Adaptive Neuro-Fuzzy Inference System (ANFIS) as a novel calibration technique to improve the accuracy and reliability of the RISS. The ANFIS-based RISS/GNSS integration provides a more precise navigation solution in such environments. The effectiveness of the proposed integration scheme was validated by conducting tests using real road trajectory and simulated GNSS outages ranging from 50 to 150 s. The results demonstrate a significant improvement in 2D position Root Mean Square Error (RMSE) of 43.8% and 28% compared to the traditional RISS/GNSS and the frequency modulated continuous wave (FMCW) Radar (Rad)/RISS/GNSS integrated navigation systems, respectively. Moreover, an improvement of 47.5% and 23.4% in 2D position maximum errors is achieved compared to the RISS/GNSS and the Rad/RISS/GNSS integrated navigation systems, respectively. These results reveal significant improvements in positioning accuracy, which is essential for safe and efficient navigation. The long-term stability of the proposed system makes it suitable for various navigation applications, particularly those requiring continuous and precise positioning information. The ANFIS-based approach used in the proposed system is extendable to other low-end IMUs, making it an attractive option for a wide range of applications.

Investigation of Automotive LiDAR Vision in Rain from Material and Optical Perspectives.

With the emergence of autonomous functions in road vehicles, there has been increased use of Advanced Driver Assistance Systems comprising various sensors to perform automated tasks. Light Detection and Ranging (LiDAR) is one of the most important types of optical sensor, detecting the positions of obstacles by representing them as clusters of points in three-dimensional space. LiDAR performance degrades significantly when a vehicle is driving in the rain as raindrops adhere to the outer surface of the sensor assembly. Performance degradation behaviors include missing points and reduced reflectivity of the points. It was found that the extent of degradation is highly dependent on the interface material properties. This subsequently affects the shapes of the adherent droplets, causing different perturbations to the optical rays. A fundamental investigation is performed on the protective polycarbonate cover of a LiDAR assembly coated with four classes of material-hydrophilic, almost-hydrophobic, hydrophobic, and superhydrophobic. Water droplets are controllably dispensed onto the cover to quantify the signal alteration due to the different droplets of various sizes and shapes. To further understand the effects of droplet motion on LiDAR signals, sliding droplet conditions are simulated using numerical analysis. The results are validated with physical optical tests, using a 905 nm laser source and receiver to mimic the LiDAR detection mechanism. Comprehensive explanations of LiDAR performance degradation in rain are presented from both material and optical perspectives. These can aid component selection and the development of signal-enhancing strategies for the integration of LiDARs into vehicle designs to minimize the impact of rain.

Optimizing Lane Departure Warning System towards AI-Centered Autonomous Vehicles.

The operational efficacy of lane departure warning systems (LDWS) in autonomous vehicles is critically influenced by the retro-reflectivity of road markings, which varies with environmental wear and weather conditions. This study investigated how changes in road marking retro-reflectivity, due to factors such as weather and physical wear, impact the performance of LDWS. The study was conducted at the Yeoncheon SOC Demonstration Research Center, where various weather scenarios, including rainfall and transitions between day and night lighting, were simulated. We applied controlled wear to white, yellow, and blue road markings and measured their retro-reflectivity at multiple stages of degradation. Our methods included rigorous testing of the LDWS's recognition rates under these diverse environmental conditions. Our results showed that higher retro-reflectivity levels significantly improve the detection capability of LDWS, particularly in adverse weather conditions. Additionally, the study led to the development of a simulation framework for analyzing the cost-effectiveness of road marking maintenance strategies. This framework aims to align maintenance costs with the safety requirements of autonomous vehicles. The findings highlight the need for revising current road marking guidelines to accommodate the advanced sensor-based needs of autonomous driving systems. By enhancing retro-reflectivity standards, the study suggests a path towards optimizing road safety in the age of autonomous vehicles.

Adversarial Attacks on Intrusion Detection Systems in In-Vehicle Networks of Connected and Autonomous Vehicles.

Rapid advancements in connected and autonomous vehicles (CAVs) are fueled by breakthroughs in machine learning, yet they encounter significant risks from adversarial attacks. This study explores the vulnerabilities of machine learning-based intrusion detection systems (IDSs) within in-vehicle networks (IVNs) to adversarial attacks, shifting focus from the common research on manipulating CAV perception models. Considering the relatively simple nature of IVN data, we assess the susceptibility of IVN-based IDSs to manipulation-a crucial examination, as adversarial attacks typically exploit complexity. We propose an adversarial attack method using a substitute IDS trained with data from the onboard diagnostic port. In conducting these attacks under black-box conditions while adhering to realistic IVN traffic constraints, our method seeks to deceive the IDS into misclassifying both normal-to-malicious and malicious-to-normal cases. Evaluations on two IDS models-a baseline IDS and a state-of-the-art model, MTH-IDS-demonstrated substantial vulnerability, decreasing the F1 scores from 95% to 38% and from 97% to 79%, respectively. Notably, inducing false alarms proved particularly effective as an adversarial strategy, undermining user trust in the defense mechanism. Despite the simplicity of IVN-based IDSs, our findings reveal critical vulnerabilities that could threaten vehicle safety and necessitate careful consideration in the development of IVN-based IDSs and in formulating responses to the IDSs' alarms.

Research on Autonomous Vehicle Path Planning Algorithm Based on Improved RRT* Algorithm and Artificial Potential Field Method.

For the RRT* algorithm, there are problems such as greater randomness, longer time consumption, more redundant nodes, and inability to perform local obstacle avoidance when encountering unknown obstacles in the path planning process of autonomous vehicles. And the artificial potential field method (APF) applied to autonomous vehicles is prone to problems such as local optimality, unreachable targets, and inapplicability to global scenarios. A fusion algorithm combining the improved RRT* algorithm and the improved artificial potential field method is proposed. First of all, for the RRT* algorithm, the concept of the artificial potential field and probability sampling optimization strategy are introduced, and the adaptive step size is designed according to the road curvature. The path post-processing of the planned global path is carried out to reduce the redundant nodes of the generated path, enhance the purpose of sampling, solve the problem where oscillation may occur when expanding near the target point, reduce the randomness of RRT* node sampling, and improve the efficiency of path generation. Secondly, for the artificial potential field method, by designing obstacle avoidance constraints, adding a road boundary repulsion potential field, and optimizing the repulsion function and safety ellipse, the problem of unreachable targets can be solved, unnecessary steering in the path can be reduced, and the safety of the planned path can be improved. In the face of U-shaped obstacles, virtual gravity points are generated to solve the local minimum problem and improve the passing performance of the obstacles. Finally, the fusion algorithm, which combines the improved RRT* algorithm and the improved artificial potential field method, is designed. The former first plans the global path, extracts the path node as the temporary target point of the latter, guides the vehicle to drive, and avoids local obstacles through the improved artificial potential field method when encountered with unknown obstacles, and then smooths the path planned by the fusion algorithm, making the path satisfy the vehicle kinematic constraints. The simulation results in the different road scenes show that the method proposed in this paper can quickly plan a smooth path that is more stable, more accurate, and suitable for vehicle driving.

Network Latency in Teleoperation of Connected and Autonomous Vehicles: A Review of Trends, Challenges, and Mitigation Strategies.

With remarkable advancements in the development of connected and autonomous vehicles (CAVs), the integration of teleoperation has become crucial for improving safety and operational efficiency. However, teleoperation faces substantial challenges, with network latency being a critical factor influencing its performance. This survey paper explores the impact of network latency along with state-of-the-art mitigation/compensation approaches. It examines cascading effects on teleoperation communication links (i.e., uplink and downlink) and how delays in data transmission affect the real-time perception and decision-making of operators. By elucidating the challenges and available mitigation strategies, the paper offers valuable insights for researchers, engineers, and practitioners working towards the seamless integration of teleoperation in the evolving landscape of CAVs.

Research on Lane-Changing Decision Making and Planning of Autonomous Vehicles Based on GCN and Multi-Segment Polynomial Curve Optimization.

This paper considers the interactive effects between the ego vehicle and other vehicles in a dynamic driving environment and proposes an autonomous vehicle lane-changing behavior decision-making and trajectory planning method based on graph convolutional networks (GCNs) and multi-segment polynomial curve optimization. Firstly, hierarchical modeling is applied to the dynamic driving environment, aggregating the dynamic interaction information of driving scenes in the form of graph-structured data. Graph convolutional neural networks are employed to process interaction information and generate ego vehicle's driving behavior decision commands. Subsequently, collision-free drivable areas are constructed based on the dynamic driving scene information. An optimization-based multi-segment polynomial curve trajectory planning method is employed to solve the optimization model, obtaining collision-free motion trajectories satisfying dynamic constraints and efficiently completing the lane-changing behavior of the vehicle. Finally, simulation and on-road vehicle experiments are conducted for the proposed method. The experimental results demonstrate that the proposed method outperforms traditional decision-making and planning methods, exhibiting good robustness, real-time performance, and strong scenario generalization capabilities.

Point Cloud Densification Algorithm for Multiple Cameras and Lidars Data Fusion.

Fusing data from many sources helps to achieve improved analysis and results. In this work, we present a new algorithm to fuse data from multiple cameras with data from multiple lidars. This algorithm was developed to increase the sensitivity and specificity of autonomous vehicle perception systems, where the most accurate sensors measuring the vehicle's surroundings are cameras and lidar devices. Perception systems based on data from one type of sensor do not use complete information and have lower quality. The camera provides two-dimensional images; lidar produces three-dimensional point clouds. We developed a method for matching pixels on a pair of stereoscopic images using dynamic programming inspired by an algorithm to match sequences of amino acids used in bioinformatics. We improve the quality of the basic algorithm using additional data from edge detectors. Furthermore, we also improve the algorithm performance by reducing the size of matched pixels determined by available car speeds. We perform point cloud densification in the final step of our method, fusing lidar output data with stereo vision output. We implemented our algorithm in C++ with Python API, and we provided the open-source library named Stereo PCD. This library very efficiently fuses data from multiple cameras and multiple lidars. In the article, we present the results of our approach to benchmark databases in terms of quality and performance. We compare our algorithm with other popular methods.

Improving Mobile Robot Maneuver Performance Using Fractional-Order Controller.

In this paper, the low-level velocity controller of an autonomous vehicle is studied. The performance of the traditional controller used in this kind of system, a PID, is analyzed. This kind of controller cannot follow ramp references without error, so when the reference implies a change in the speed, the vehicle cannot follow the proposed reference, and there is a significant difference between the actual and desired vehicle behaviors. A fractional controller is proposed which changes the ordinary dynamics allowing faster responses for small times, at the cost of slower responses for large times. The idea is to take advantage of this fact to follow fast setpoint changes with a smaller error than that obtained with a classic non-fractional PI controller. Using this controller, the vehicle can follow variable speed references with zero stationary error, significantly reducing the difference between reference and actual vehicle behavior. The paper presents the fractional controller, studies its stability in function of the fractional parameters, designs the controller, and tests its stability. The designed controller is tested on a real prototype, and its behavior is compared to a standard PID controller. The designed fractional PID controller overcomes the results of the standard PID controller.

Joint Calibration of a Multimodal Sensor System for Autonomous Vehicles.

Multimodal sensor systems require precise calibration if they are to be used in the field. Due to the difficulty of obtaining the corresponding features from different modalities, the calibration of such systems is an open problem. We present a systematic approach for calibrating a set of cameras with different modalities (RGB, thermal, polarization, and dual-spectrum near infrared) with regard to a LiDAR sensor using a planar calibration target. Firstly, a method for calibrating a single camera with regard to the LiDAR sensor is proposed. The method is usable with any modality, as long as the calibration pattern is detected. A methodology for establishing a parallax-aware pixel mapping between different camera modalities is then presented. Such a mapping can then be used to transfer annotations, features, and results between highly differing camera modalities to facilitate feature extraction and deep detection and segmentation methods.

Creating an Autonomous Hovercraft for Bathymetric Surveying in Extremely Shallow Water (<1 m).

Coastal shallow water environments (<5 m) are extremely biodiverse and dynamic yet are often mapped too infrequently or at too low resolutions to capture the important processes occurring in these regions. Common forms of coastal surveying can leave gaps in data in the shallow water zone due to optical instrument capabilities and a vessel's ability to navigate in this region. One solution to these issues is an autonomous hovercraft that can fly over land and water and begin surveying at sub-meter water depths, bridging the gap between common optical and acoustic surveying methods. The craft's autonomy is tested via four autonomous flight paths, or missions, and the desired path is compared to both the observed heading and direction of motion. Although the accuracy for each track in the mission varies, most headings and directions of motion of the hovercraft are within 50 degrees of the desired direction. A single-beam echo sounder was used to map the bathymetry of the study site, showing a gently sloping beach.

Communication between Autonomous Vehicles and Pedestrians: An Experimental Study Using Virtual Reality.

One of the major challenges of autonomous vehicles (AV) is their interaction with pedestrians. Unofficial interactions such as gestures, eye contact, waving, and flashing lights are very common behavioral patterns for drivers to express their intent to give priority. In our research we composed a virtual reality experiment for a pedestrian crossing in an urban environment in order to test pedestrians' reactions on an LED light display mounted on a virtual AV. Our main research interest was to investigate whether communication patterns influence the decision making of pedestrians when crossing the road. In a VR environment, four scenarios were created with a vehicle approaching a pedestrian crossing with different speeds and displaying a special red/green sign to pedestrians. Here, 51 persons participating in the experiment had to decide when crossing is safe. Results show that the majority of people indicated they would cross in the time windows when it was actually safe to cross. Male subjects made their decision to cross slightly faster but no significant differences were found in the decision making by gender. It was found that age is not an influencing factor, either. Overall, a quick learning process was experienced proving that explicit communication patterns are self-explaining.