End-to-End Autonomous Driving: Challenges and Frontiers.

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework.

Towards Scalable Multi-View Reconstruction of Geometry and Materials.

In this paper, we propose a novel method for joint recovery of camera pose, object geometry and spatially-varying Bidirectional Reflectance Distribution Function (svBRDF) of 3D scenes that exceed object-scale and hence cannot be captured with stationary light stages. The input are high-resolution RGB-D images captured by a mobile, hand-held capture system with point lights for active illumination. Compared to previous works that jointly estimate geometry and materials from a hand-held scanner, we formulate this problem using a single objective function that can be minimized using off-the-shelf gradient-based solvers. To facilitate scalability to large numbers of observation views and optimization variables, we introduce a distributed optimization algorithm that reconstructs 2.5D keyframe-based representations of the scene. A novel multi-view consistency regularizer effectively synchronizes neighboring keyframes such that the local optimization results allow for seamless integration into a globally consistent 3D model. We provide a study on the importance of each component in our formulation and show that our method compares favorably to baselines. We further demonstrate that our method accurately reconstructs various objects and materials and allows for expansion to spatially larger scenes. We believe that this work represents a significant step towards making geometry and material estimation from hand-held scanners scalable.

Unifying Flow, Stereo and Depth Estimation.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields.

Neural fields have revolutionized the area of 3D reconstruction and novel view synthesis of rigid scenes. A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space. We propose a new articulation module for neural fields, Fast-SNARF, which finds accurate correspondences between canonical space and posed space via iterative root finding. Fast-SNARF is a drop-in replacement in functionality to our previous work, SNARF, while significantly improving its computational efficiency. We contribute several algorithmic and implementation improvements over SNARF, yielding a speed-up of 150×. These improvements include voxel-based correspondence search, pre-computing the linear blend skinning function, and an efficient software implementation with CUDA kernels. Fast-SNARF enables efficient and simultaneous optimization of shape and skinning weights given deformed observations without correspondences (e.g. 3D meshes). Because learning of deformation maps is a crucial component in many 3D human avatar methods and since Fast-SNARF provides a computationally efficient solution, we believe that this work represents a significant step towards the practical creation of 3D virtual humans.

NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields.

Visually exploring in a real-world 4D spatiotemporal space freely in VR has been a long-term quest. The task is especially appealing when only a few or even single RGB cameras are used for capturing the dynamic scene. To this end, we present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. First, we propose to decompose the 4D spatiotemporal space according to temporal characteristics. Points in the 4D space are associated with probabilities of belonging to three categories: static, deforming, and new areas. Each area is represented and regularized by a separate neural field. Second, we propose a hybrid representations based feature streaming scheme for efficiently modeling the neural fields. Our approach, coined NeRFPlayer, is evaluated on dynamic scenes captured by single hand-held cameras and multi-camera arrays, achieving comparable or superior rendering performance in terms of quality and speed comparable to recent state-of-the-art methods, achieving reconstruction in 10 seconds per frame and interactive rendering. Project website: https://bit.ly/nerfplayer.

Periodic Photobleaching with Structured Illumination for Diffusion Imaging.

The use of periodically structured illumination coupled with spatial Fourier-transform fluorescence recovery after photobleaching (FT-FRAP) was shown to support diffusivity mapping within segmented domains of arbitrary shape. Periodic "comb-bleach" patterning of the excitation beam during photobleaching encoded spatial maps of diffusion onto harmonic peaks in the spatial Fourier transform. Diffusion manifests as a simple exponential decay of a given harmonic, improving the signal to noise ratio and simplifying mathematical analysis. Image segmentation prior to Fourier transformation was shown to support pooling for signal to noise enhancement for regions of arbitrary shape expected to exhibit similar diffusivity within a domain. Following proof-of-concept analyses based on simulations with known ground-truth maps, diffusion imaging by FT-FRAP was used to map spatially-resolved diffusion differences within phase-separated domains of model amorphous solid dispersion spin-cast thin films. Notably, multi-harmonic analysis by FT-FRAP was able to definitively discriminate and quantify the roles of internal diffusion and exchange to higher mobility interfacial layers in modeling the recovery kinetics within thin amorphous/amorphous phase-separated domains, with interfacial diffusion playing a critical role in recovery. These results have direct implications for the design of amorphous systems for stable storage and efficacious delivery of therapeutic molecules.

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D.

For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving.

How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g., object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.

Investigation of Electromechanical Properties on 3-D Printed Piezoelectric Composite Scaffold Structures.

Piezoelectric composites with 3-3 connectivity gathered attraction due to their potential application as an acoustic transducer in medical imaging, non-destructive testing, etc. In this contribution, piezoelectric composites were fabricated with a material extrusion-based additive manufacturing process (MEX), also well-known under the names fused deposition modeling (FDM), fused filament fabrication (FFF) or fused deposition ceramics (FDC). Thermoplastic filaments were used to achieve open and offset printed piezoelectric scaffold structures. Both scaffold structures were printed, debinded and sintered successfully using commercial PZT and BaTiO3 powder. For the first time, it could be demonstrated, that using the MEX processing method, closed pore ferroelectric structure can be achieved without pore-former additive. After ceramic processing, the PZT scaffold structures were impregnated with epoxy resin to convert them into composites with 3-3 connectivity. A series of composites with varying ceramic content were achieved by changing the infill parameter during the 3D printing process systematically, and their electromechanical properties were investigated using the electromechanical aix PES device. Also, the Figure of merit (FOM) of these composites was calculated to assess the potential of this material as a candidate for transducer applications. A maximum for the FOM at 25 vol.% of PZT could be observed in this study.

Fluorescence-Detected Mid-Infrared Photothermal Microscopy.

We demonstrate instrumentation and methods to enable fluorescence-detected photothermal infrared (F-PTIR) microscopy and then demonstrate the utility of F-PTIR to characterize the composition within phase-separated domains of model amorphous solid dispersions (ASDs) induced by water sorption. In F-PTIR, temperature-dependent changes in fluorescence quantum efficiency are shown to sensitively report on highly localized absorption of mid-infrared radiation. The spatial resolution with which infrared spectroscopy can be performed is dictated by fluorescence microscopy, rather than the infrared wavelength. Intrinsic ultraviolet autofluorescence of tryptophan and protein microparticles enabled label-free F-PTIR microscopy. Following proof of concept F-PTIR demonstration on model systems of polyethylene glycol (PEG) and silica gel, F-PTIR enabled the characterization of chemical composition within inhomogeneous ritonavir/polyvinylpyrrolidone-vinyl acetate (PVPVA) amorphous dispersions. Phase separation is implicated in the observation of critical behaviors in ASD dissolution kinetics, with the results of F-PTIR supporting the formation of phase-separated drug-rich domains upon water sorption in spin-cast films.

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking.

Multi-object tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association. To address this, we present a novel MOT evaluation metric, higher order tracking accuracy (HOTA), which explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately, which enables clear analysis of tracking performance. We evaluate the effectiveness of HOTA on the MOTChallenge benchmark, and show that it is able to capture important aspects of MOT performance not previously taken into account by established metrics. Furthermore, we show HOTA scores better align with human visual evaluation of tracking performance.

Learning Steering Kernels for Guided Depth Completion.

This paper addresses the guided depth completion task in which the goal is to predict a dense depth map given a guidance RGB image and sparse depth measurements. Recent advances on this problem nurture hopes that one day we can acquire accurate and dense depth at a very low cost. A major challenge of guided depth completion is to effectively make use of extremely sparse measurements, e.g., measurements covering less than 1% of the image pixels. In this paper, we propose a fully differentiable model that avoids convolving on sparse tensors by jointly learning depth interpolation and refinement. More specifically, we propose a differentiable kernel regression layer that interpolates the sparse depth measurements via learned kernels. We further refine the interpolated depth map using a residual depth refinement layer which leads to improved performance compared to learning absolute depth prediction using a vanilla network. We provide experimental evidence that our differentiable kernel regression layer not only enables end-to-end training from very sparse measurements using standard convolutional network architectures, but also leads to better depth interpolation results compared to existing heuristically motivated methods. We demonstrate that our method outperforms many state-of-the-art guided depth completion techniques on both NYUv2 and KITTI. We further show the generalization ability of our method with respect to the density and spatial statistics of the sparse depth measurements.

Anomalous Diffusion Characterization by Fourier Transform-FRAP with Patterned Illumination.

Fourier transform fluorescence recovery after photobleaching (FT-FRAP) with patterned illumination is theorized and demonstrated for quantitatively evaluating normal and anomalous diffusion. Diffusion characterization is routinely performed to assess mobility in cell biology, pharmacology, and food science. Conventional FRAP is noninvasive, has low sample volume requirements, and can rapidly measure diffusion over distances of a few micrometers. However, conventional point-bleach measurements are complicated by signal-to-noise limitations, the need for precise knowledge of the photobleach beam profile, potential for bias due to sample heterogeneity, and poor compatibility with multiphoton excitation because of local heating. In FT-FRAP with patterned illumination, the time-dependent fluorescence recovery signal is concentrated to puncta in the spatial Fourier domain, with substantial improvements in signal-to-noise, mathematical simplicity, representative sampling, and multiphoton compatibility. A custom nonlinear optical beam-scanning microscope enabled patterned illumination for photobleaching through two-photon excitation. Measurements in the spatial Fourier domain removed dependence on the photobleach profile, suppressing bias from imprecise knowledge of the point spread function. For normal diffusion, the fluorescence recovery produced a simple single-exponential decay in the spatial Fourier domain, in excellent agreement with theoretical predictions. Simultaneous measurement of diffusion at multiple length scales was enabled through analysis of multiple spatial harmonics of the photobleaching pattern. Anomalous diffusion was characterized by FT-FRAP through a nonlinear fit to multiple spatial harmonics of the fluorescence recovery. Constraining the fit to describe diffusion over multiple length scales resulted in higher confidence in the recovered fitting parameters. Additionally, phase analysis in FT-FRAP was shown to inform on flow/sample translation.

Antistatic Fibers for High-Visibility Workwear: Challenges of Melt-Spinning Industrial Fibers.

Safety workwear often requires antistatic protection to prevent the build-up of static electricity and sparks, which can be extremely dangerous in a working environment. In order to make synthetic antistatic fibers, electrically conducting materials such as carbon black are added to the fiber-forming polymer. This leads to unwanted dark colors in the respective melt-spun fibers. To attenuate the undesired dark color, we looked into various possibilities including the embedding of the conductive element inside a dull side-by-side bicomponent fiber. The bicomponent approach, with an antistatic compound as a minor element, also helped in preventing the severe loss of tenacity often caused by a high additive loading. We could melt-spin a bicomponent fiber with a specific resistance as low as 0.1 Ωm and apply it in a fabric that fulfills the requirements regarding the antistatic properties, luminance and flame retardancy of safety workwear.

Stochastic Differential Scanning Calorimetry by Nonlinear Optical Microscopy.

Stochastic phase transformations within individual crystalline particles were recorded by integration of second harmonic generation (SHG) imaging with differential scanning calorimetry (DSC). The SHG activity of a crystal is highly sensitive to the specific molecular packing arrangement within a noncentrosymmetric lattice, providing access to information otherwise unavailable by conventional imaging approaches. Consequently, lattice transformations associated with dehydration/desolvation events were readily observed by SHG imaging and directly correlated to the phase transformations detected by the DSC measurements. Following studies of a model system (urea), stochastic differential scanning calorimetry (SDSC) was performed on trehalose dihydrate, which has a more complex phase behavior. From these measurements, SDSC revealed a broad diversity of single-particle thermal trajectories and direct evidence of a "cold phase transformation" process not observable by the DSC measurements alone.

Map-Based Probabilistic Visual Self-Localization.

Accurate and efficient self-localization is a critical problem for autonomous systems. This paper describes an affordable solution to vehicle self-localization which uses odometry computed from two video cameras and road maps as the sole inputs. The core of the method is a probabilistic model for which an efficient approximate inference algorithm is derived. The inference algorithm is able to utilize distributed computation in order to meet the real-time requirements of autonomous systems in some instances. Because of the probabilistic nature of the model the method is capable of coping with various sources of uncertainty including noise in the visual odometry and inherent ambiguities in the map (e.g., in a Manhattan world). By exploiting freely available, community developed maps and visual odometry measurements, the proposed method is able to localize a vehicle to 4 m on average after 52 seconds of driving on maps which contain more than 2,150 km of drivable roads.

3D Traffic Scene Understanding From Movable Platforms.

In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, the scene topology, geometry, and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilities of humans, our model does not rely on GPS, lidar, or map knowledge. Instead, it takes advantage of a diverse set of visual cues in the form of vehicle tracklets, vanishing points, semantic scene labels, scene flow, and occupancy grids. For each of these cues, we propose likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method we are able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and cluttered urban environments.

Factors affecting wear of composite resin denture teeth--24-month results from a clinical study.

The objectives of this study were to measure the occlusal wear of composite resin denture teeth in patients wearing a complete denture and to evaluate factors affecting wear. Fifty participants provided with complete dentures in at least one jaw were included. Gypsum casts were made from preliminary vinyl polysiloxane impressions 4 weeks after insertion, then after 6 (t(1)), 12 (t(2)), and 24 months (t(3)). Three-hundred and three posterior denture teeth were evaluated after 24 months. Wear was measured indirectly, from the casts, by means of a three-dimensional laser scanner device. Sequential images of the occlusal surfaces were digitized and superimposed (occlusal matching). Statistical analysis was performed by the use of mixed regression models, with the patient being a random effect. Mean wear (median, interquartile range; micrometer) of the entire occlusal surface was 8 (19) at t(1), 18 (34) at t(2), and 40 (61) at t(3). Maximum vertical loss (median, interquartile range; micrometer) was 92 (112) at t(1), 146 (148) at t(2), and 226 (184) at t(3). The dental status of the opposing jaw and the nature of the opposing material significantly affected the wear of denture teeth at t (3). Gender, daily wearing time, jaw, and type of tooth had no significant effects on the extent of wear. Clinically relevant vertical loss of composite resin denture teeth occurs after 24 months. Considering the limitations of this study, wear of denture teeth was affected by dental status and opposing material. The results suggest that wear of composite resin denture teeth exceeds that of enamel.