Parallel Driving with Big Models and Foundation Intelligence in Cyber-Physical-Social Spaces.

Recent years have witnessed numerous technical breakthroughs in connected and autonomous vehicles (CAVs). On the one hand, these breakthroughs have significantly advanced the development of intelligent transportation systems (ITSs); on the other hand, these new traffic participants introduce more complex and uncertain elements to ITSs from the social space. Digital twins (DTs) provide real-time, data-driven, precise modeling for constructing the digital mapping of physical-world ITSs. Meanwhile, the metaverse integrates emerging technologies such as virtual reality/mixed reality, artificial intelligence, and DTs to model and explore how to realize improved sustainability, increased efficiency, and enhanced safety. More recently, as a leading effort toward general artificial intelligence, the concept of foundation model was proposed and has achieved significant success, showing great potential to lay the cornerstone for diverse artificial intelligence applications across different domains. In this article, we explore the big models embodied foundation intelligence for parallel driving in cyber-physical-social spaces, which integrate metaverse and DTs to construct a parallel training space for CAVs, and present a comprehensive elucidation of the crucial characteristics and operational mechanisms. Beyond providing the infrastructure and foundation intelligence of big models for parallel driving, this article also discusses future trends and potential research directions, and the "6S" goals of parallel driving.

A New Parallel Intelligence Based Light Field Dataset for Depth Refinement and Scene Flow Estimation.

Computer vision tasks, such as motion estimation, depth estimation, object detection, etc., are better suited to light field images with more structural information than traditional 2D monocular images. However, since costly data acquisition instruments are difficult to calibrate, it is always hard to obtain real-world scene light field images. The majority of the datasets for static light field images now available are modest in size and cannot be used in methods such as transformer to fully leverage local and global correlations. Additionally, studies on dynamic situations, such as object tracking and motion estimates based on 4D light field images, have been rare, and we anticipate a superior performance. In this paper, we firstly propose a new static light field dataset that contains up to 50 scenes and takes 8 to 10 perspectives for each scene, with the ground truth including disparities, depths, surface normals, segmentations, and object poses. This dataset is larger scaled compared to current mainstream datasets for depth estimation refinement, and we focus on indoor and some outdoor scenarios. Second, to generate additional optical flow ground truth that indicates 3D motion of objects in addition to the ground truth obtained in static scenes in order to calculate more precise pixel level motion estimation, we released a light field scene flow dataset with dense 3D motion ground truth of pixels, and each scene has 150 frames. Thirdly, by utilizing the DistgDisp and DistgASR, which decouple the angular and spatial domain of the light field, we perform disparity estimation and angular super-resolution to evaluate the performance of our light field dataset. The performance and potential of our dataset in disparity estimation and angular super-resolution have been demonstrated by experimental results.

Parallel Radars: From Digital Twins to Digital Intelligence for Smart Radar Systems.

Radar is widely employed in many applications, especially in autonomous driving. At present, radars are only designed as simple data collectors, and they are unable to meet new requirements for real-time and intelligent information processing as environmental complexity increases. It is inevitable that smart radar systems will need to be developed to deal with these challenges and digital twins in cyber-physical systems (CPS) have proven to be effective tools in many aspects. However, human involvement is closely related to radar technology and plays an important role in the operation and management of radars; thus, digital twins' radars in CPS are insufficient to realize smart radar systems due to the inadequate consideration of human factors. ACP-based parallel intelligence in cyber-physical-social systems (CPSS) is used to construct a novel framework for smart radars, called Parallel Radars. A Parallel Radar consists of three main parts: a Descriptive Radar for constructing artificial radar systems in cyberspace, a Predictive Radar for conducting computational experiments with artificial systems, and a Prescriptive Radar for providing prescriptive control to both physical and artificial radars to complete parallel execution. To connect silos of data and protect data privacy, federated radars are proposed. Additionally, taking mines as an example, the application of Parallel Radars in autonomous driving is discussed in detail, and various experiments have been conducted to demonstrate the effectiveness of Parallel Radars.

Learning Lightweight Dynamic Kernels With Attention Inside via Local-Global Context Fusion.

Traditional convolutional neural networks (CNNs) share their kernels among all positions of the input, which may constrain the representation ability in feature extraction. Dynamic convolution proposes to generate different kernels for different inputs to improve the model capacity. However, the total parameters of the dynamic network can be significantly huge. In this article, we propose a lightweight dynamic convolution method to strengthen traditional CNNs with an affordable increase of total parameters and multiply-adds. Instead of generating the whole kernels directly or combining several static kernels, we choose to "look inside", learning the attention within convolutional kernels. An extra network is used to adjust the weights of kernels for every feature aggregation operation. By combining local and global contexts, the proposed approach can capture the variance among different samples, the variance in different positions of the feature maps, and the variance in different positions inside sliding windows. With a minor increase in the number of model parameters, remarkable improvements in image classification on CIFAR and ImageNet with multiple backbones have been obtained. Experiments on object detection also verify the effectiveness of the proposed method.

Mask SSD: An Effective Single-Stage Approach to Object Instance Segmentation.

We propose Mask SSD, an efficient and effective approach to address the challenging instance segmentation task. Based on a single-shot detector, Mask SSD detects all instances in an image and marks the pixels that belong to each instance. It consists of a detection subnetwork that predicts object categories and bounding box locations, and an instance-level segmentation subnetwork that generates the foreground mask for each instance. In the detection subnetwork, multi-scale and feedback features from different layers are used to better represent objects of various sizes and provide high-level semantic information. Then, we adopt an assistant classification network to guide per-class score prediction, which consists of objectness prior and category likelihood. The instance-level segmentation subnetwork outputs pixel-wise segmentation for each detection while providing the multi-scale and feedback features from different layers as input. These two subnetworks are jointly optimized by a multi-task loss function, which renders Mask SSD direct prediction on detection and segmentation results. We conduct extensive experiments on PASCAL VOC, SBD, and MS COCO datasets to evaluate the performance of Mask SSD. Experimental results verify that as compared with state-of-the-art approaches, our proposed method has a comparable precision with less speed overhead.