Unbalanced visual cues do not affect search precision at the nest in desert ants (Cataglyphis nodus).

Desert ant foragers are well known for their visual navigation abilities, relying on visual cues in the environment to find their way along routes back to the nest. If the inconspicuous nest entrance is missed, ants engage in a highly structured systematic search until it is discovered. Searching ants continue to be guided by visual cues surrounding the nest, from which they derive a location estimate. The precision level of this estimate depends on the information content of the nest panorama. This study examines whether search precision is also affected by the directional distribution of visual information. The systematic searching behavior of ants is examined under laboratory settings. Two different visual scenarios are compared - a balanced one where visual information is evenly distributed, and an unbalanced one where all visual information is located on one side of an experimental arena. The identity and number of visual objects is similar over both conditions. The ants search with comparable precision in both conditions. Even in the visually unbalanced condition, searches are characterized by balanced precision on both sides of the arena. This finding lends support to the idea that ants memorize the visual scenery at the nest as panoramic views from different locations. A searching ant is thus able to estimate its location with equal precision in all directions, leading to symmetrical search paths.

Trail using ants follow idiosyncratic routes in complex landscapes.

A large volume of research on individually navigating ants has shown how path integration and visually guided navigation form a major part of the ant navigation toolkit for many species and are sufficient mechanisms for successful navigation. One of the behavioural markers of the interaction of these mechanisms is that experienced foragers develop idiosyncratic routes that require that individual ants have personal and unique visual memories that they use to guide habitual routes between the nest and feeding sites. The majority of ants, however, inhabit complex cluttered environments and social pheromone trails are often part of the collective recruitment, organisation and navigation of these foragers. We do not know how individual navigation interacts with collective behaviour along shared trails in complex natural environments. We thus asked here if wood ants that forage through densely cluttered woodlands where they travel along shared trails repeatedly follow the same routes or if they choose a spread of paths within the shared trail. We recorded three long homing trajectories of 20 individual wood ants in their natural woodland habitat. We found that wood ants follow idiosyncratic routes when navigating along shared trails through highly complex visual landscapes. This shows that ants rely on individual memories for habitual route guidance even in cluttered environments when chemical trail information is available. We argue that visual cues are likely to be the dominant sensory modality for the idiosyncratic routes. These experiments shed new light on how ants, or insects in general, navigate through complex multimodal environments.

Visual Navigation Algorithms for Aircraft Fusing Neural Networks in Denial Environments.

A lightweight aircraft visual navigation algorithm that fuses neural networks is proposed to address the limited computing power issue during the offline operation of aircraft edge computing platforms in satellite-denied environments with complex working scenarios. This algorithm utilizes object detection algorithms to label dynamic objects within complex scenes and performs dynamic feature point elimination to enhance the feature point extraction quality, thereby improving navigation accuracy. The algorithm was validated using an aircraft edge computing platform, and comparisons were made with existing methods through experiments conducted on the TUM public dataset and physical flight experiments. The experimental results show that the proposed algorithm not only improves the navigation accuracy but also has high robustness compared with the monocular ORB-SLAM2 method under the premise of satisfying the real-time operation of the system.

Exploring the Role of Video Playback Visual Cues in Object Retrieval Tasks.

Searching for objects is a common task in daily life and work. For augmented reality (AR) devices without spatial perception systems, the image of the object's last appearance serves as a common search assistance. Compared to using only images as visual cues, videos capturing the process of object placement can provide procedural guidance, potentially enhancing users' search efficiency. However, complete video playback capturing the entire object placement process as visual cues can be excessively lengthy, requiring users to invest significant viewing time. To explore whether segmented or accelerated video playback can still assist users in object retrieval tasks effectively, we conducted a user study. The results indicated that when video playback is covering the first appearance of the object's destination to the object's final appearance (referred to as the destination appearance, DA) and playing at normal speed, search time and cognitive load were significantly reduced. Subsequently, we designed a second user study to evaluate the performance of video playback compared to image cues in object retrieval tasks. The results showed that combining the DA playback starting point with images of the object's last appearance further reduced search time and cognitive load.

Visual navigation: properties, acquisition and use of views.

Panoramic views offer information on heading direction and on location to visually navigating animals. This review covers the properties of panoramic views and the information they provide to navigating animals, irrespective of image representation. Heading direction can be retrieved by alignment matching between memorized and currently experienced views, and a gradient descent in image differences can lead back to the location at which a view was memorized (positional image matching). Central place foraging insects, such as ants, bees and wasps, conduct distinctly choreographed learning walks and learning flights upon first leaving their nest that are likely to be designed to systematically collect scene memories tagged with information provided by path integration on the direction of and the distance to the nest. Equally, traveling along routes, ants have been shown to engage in scanning movements, in particular when routes are unfamiliar, again suggesting a systematic process of acquiring and comparing views. The review discusses what we know and do not know about how view memories are represented in the brain of insects, how they are acquired and how they are subsequently used for traveling along routes and for pinpointing places.

Impact of central complex lesions on innate and learnt visual navigation in ants.

Wood ants are excellent navigators, using a combination of innate and learnt navigational strategies to travel between their nest and feeding sites. Visual navigation in ants has been studied extensively, however, we have little direct evidence for the underlying neural mechanisms. Here, we perform lateralized mechanical lesions in the central complex (CX) of wood ants, a midline structure known to allow an insect to keep track of the direction of sensory cues relative to its own orientation and to control movement. We lesioned two groups of ants and observed their behaviour in an arena with a large visual landmark present. The first group of ants were naïve and when intact such ants show a clear innate attraction to the conspicuous landmark. The second group of ants were trained to aim to a food location to the side of the landmark. The general heading of naïve ants towards a visual cue was not altered by the lesions, but the heading of ants trained to a landmark adjacent food position was affected. Thus, CX lesions had a specific impact on learnt visual guidance. We also observed that lateralised lesions altered the fine details of turning with lesioned ants spending less time turning to the side ipsilateral of the lesion. The results confirm the role of the CX in turn control and highlight its important role in the implementation of learnt behaviours that rely on information from other brain regions.

State-dependent, visually guided behaviors in the nudibranch Berghia stephanieae.

Nudibranch mollusks have structurally simple eyes whose behavioral roles have not been established. We tested the effects of visual stimuli on the behavior of the nudibranch Berghia stephanieae under different food and hunger conditions. In an arena that was half-shaded, animals spent most of their time in the dark, where they also decreased their speed and made more changes in heading. These behavioral differences between the light and dark were less evident in uniformly illuminated or darkened arenas, suggesting that they were not caused by the level of illumination. Berghia stephanieae responded to distant visual targets; animals approached a black stripe that was at least 15 deg wide on a white background. They did not approach a stripe that was lighter than the background but approached a stripe that was isoluminant with the background, suggesting the detection of spatial information. Animals traveled in convoluted paths in a featureless arena but straightened their paths when a visual target was present even if they did not approach it, suggesting that visual cues were used for navigation. Individuals were less responsive to visual stimuli when food deprived or in the presence of food odor. Thus, B. stephanieae exhibits visually guided behaviors that are influenced by odors and hunger state.

Regularized Denoising Masked Visual Pretraining for Robust Embodied PointGoal Navigation.

Embodied PointGoal navigation is a fundamental task for embodied agents. Recent works have shown that the performance of the embodied navigation agent degrades significantly in the presence of visual corruption, including Spatter, Speckle Noise, and Defocus Blur, showing the weak robustness of the agent. To improve the robustness of embodied navigation agents to various visual corruptions, we propose a navigation framework called Regularized Denoising Masked AutoEncoders Navigation (RDMAE-Nav). In a nutshell, RDMAE-Nav mainly consists of two modules: a visual module and a policy module. In the visual module, a self-supervised pretraining method, dubbed Regularized Denoising Masked AutoEncoders (RDMAE), is designed to enable the Vision Transformers (ViT)-based visual encoder to learn robust representations. The bidirectional Kullback-Leibler divergence is introduced in RDMAE as the regularization term for a denoising masked modeling task. Specifically, RDMAE mitigates the gap between clean and noisy image representations by minimizing the bidirectional Kullback-Leibler divergence. Then, the visual encoder is pretrained by RDMAE. In contrast to existing works, RDMAE-Nav applies denoising masked visual pretraining for PointGoal navigation to improve robustness to various visual corruptions. Finally, the pretrained visual encoder with frozen weights is applied to extract robust visual representations for policy learning in the RDMAE-Nav. Extensive experiments show that RDMAE-Nav performs competitively compared with state of the arts (SOTAs) on various visual corruptions. In detail, RDMAE-Nav performs the absolute improvement: 28.2% in SR and 23.68% in SPL under Spatter; 2.28% in SR and 6.41% in SPL under Speckle Noise; and 9.46% in SR and 9.55% in SPL under Defocus Blur.

Multiple Self-Supervised Auxiliary Tasks for Target-Driven Visual Navigation Using Deep Reinforcement Learning.

Visual navigation based on deep reinforcement learning requires a large amount of interaction with the environment, and due to the reward sparsity, it requires a large amount of training time and computational resources. In this paper, we focus on sample efficiency and navigation performance and propose a framework for visual navigation based on multiple self-supervised auxiliary tasks. Specifically, we present an LSTM-based dynamics model and an attention-based image-reconstruction model as auxiliary tasks. These self-supervised auxiliary tasks enable agents to learn navigation strategies directly from the original high-dimensional images without relying on ResNet features by constructing latent representation learning. Experimental results show that without manually designed features and prior demonstrations, our method significantly improves the training efficiency and outperforms the baseline algorithms on the simulator and real-world image datasets.

A Benchmark Comparison of Four Off-the-Shelf Proprietary Visual-Inertial Odometry Systems.

Commercial visual-inertial odometry (VIO) systems have been gaining attention as cost-effective, off-the-shelf, six-degree-of-freedom (6-DoF) ego-motion-tracking sensors for estimating accurate and consistent camera pose data, in addition to their ability to operate without external localization from motion capture or global positioning systems. It is unclear from existing results, however, which commercial VIO platforms are the most stable, consistent, and accurate in terms of state estimation for indoor and outdoor robotic applications. We assessed four popular proprietary VIO systems (Apple ARKit, Google ARCore, Intel RealSense T265, and Stereolabs ZED 2) through a series of both indoor and outdoor experiments in which we showed their positioning stability, consistency, and accuracy. After evaluating four popular VIO sensors in challenging real-world indoor and outdoor scenarios, Apple ARKit showed the most stable and high accuracy/consistency, and the relative pose error was a drift error of about 0.02 m per second. We present our complete results as a benchmark comparison for the research community.

Object-Based Reliable Visual Navigation for Mobile Robot.

Visual navigation is of vital importance for autonomous mobile robots. Most existing practical perception-aware based visual navigation methods generally require prior-constructed precise metric maps, and learning-based methods rely on large training to improve their generality. To improve the reliability of visual navigation, in this paper, we propose a novel object-level topological visual navigation method. Firstly, a lightweight object-level topological semantic map is constructed to release the dependence on the precise metric map, where the semantic associations between objects are stored via graph memory and topological organization is performed. Then, we propose an object-based heuristic graph search method to select the global topological path with the optimal and shortest characteristics. Furthermore, to reduce the global cumulative error, a global path segmentation strategy is proposed to divide the global topological path on the basis of active visual perception and object guidance. Finally, to achieve adaptive smooth trajectory generation, a Bernstein polynomial-based smooth trajectory refinement method is proposed by transforming trajectory generation into a nonlinear planning problem, achieving smooth multi-segment continuous navigation. Experimental results demonstrate the feasibility and efficiency of our method on both simulation and real-world scenarios. The proposed method also obtains better navigation success rate (SR) and success weighted by inverse path length (SPL) than the state-of-the-art methods.

Visual imagination and cognitive mapping of a virtual building.

We investigated the contribution of visual imagination to the cognitive mapping of a building when initial exploration was simulated either visually by using a passive video walk-through, or mentally by using verbal guidance. Building layout had repeating elements with either rotational or mirror symmetry. Cognitive mapping of the virtual building, determined using questionnaires and map drawings, was present following verbal guidance but inferior to that following video guidance. Mapping was not affected by the building's structural symmetry. However, notably, it correlated with small-scale mental rotation scores for both video and verbal guidance conditions. There was no difference between males and females. A common factor that may have influenced cognitive mapping was the availability of visual information about the relationships of the building elements, either directly perceived (during the video walk-through) or imagined (during the verbal walk-through and/or during recall). Differences in visual imagination, particularly mental rotation, may thus account for some of the individual variance in cognitive mapping of complex built environments, which is relevant to how designers provide navigation-relevant information.

Minding the gap: learning and visual scanning behaviour in nocturnal bull ants.

Insects possess small brains but exhibit sophisticated behaviour, specifically their ability to learn to navigate within complex environments. To understand how they learn to navigate in a cluttered environment, we focused on learning and visual scanning behaviour in the Australian nocturnal bull ant, Myrmecia midas, which are exceptional visual navigators. We tested how individual ants learn to detour via a gap and how they cope with substantial spatial changes over trips. Homing M. midas ants encountered a barrier on their foraging route and had to find a 50 cm gap between symmetrical large black screens, at 1 m distance towards the nest direction from the centre of the releasing platform in both familiar (on-route) and semi-familiar (off-route) environments. Foragers were tested for up to 3 learning trips with the changed conditions in both environments. The results showed that on the familiar route, individual foragers learned the gap quickly compared with when they were tested in the semi-familiar environment. When the route was less familiar, and the panorama was changed, foragers were less successful at finding the gap and performed more scans on their way home. Scene familiarity thus played a significant role in visual scanning behaviour. In both on-route and off-route environments, panoramic changes significantly affected learning, initial orientation and scanning behaviour. Nevertheless, over a few trips, success at gap finding increased, visual scans were reduced, the paths became straighter, and individuals took less time to reach the goal.

Learning walks in an Australian desert ant, Melophorus bagoti.

The central Australian ant Melophorus bagoti is the most thermophilic ant in Australia and forages solitarily in the summer months during the hottest period of the day. For successful navigation, desert ants of many species are known to integrate a path and learn landmark cues around the nest. Ants perform a series of exploratory walks around the nest before their first foraging trip, during which they are presumed to learn about their landmark panorama. Here, we studied 15 naive M. bagoti ants transitioning from indoor work to foraging outside the nest. In 3-4 consecutive days, they performed 3-7 exploratory walks before heading off to forage. Naive ants increased the area of exploration around the nest and the duration of trips over successive learning walks. In their first foraging walk, the majority of the ants followed a direction explored on their last learning walk. During learning walks, the ants stopped and performed stereotypical orientation behaviours called pirouettes. They performed complete body rotations with stopping phases as well as small circular walks without stops known as voltes. After just one learning walk, these desert ants could head in the home direction from locations 2 m from the nest, although not from locations 4 m from the nest. These results suggest gradual learning of the visual landmark panorama around the foragers' nest. Our observations show that M. bagoti exhibit similar characteristics in their learning walks to other desert ants of the genera Ocymyrmex and Cataglyphis.

Uncertainty Estimation of Dense Optical Flow for Robust Visual Navigation.

This paper presents a novel dense optical-flow algorithm to solve the monocular simultaneous localisation and mapping (SLAM) problem for ground or aerial robots. Dense optical flow can effectively provide the ego-motion of the vehicle while enabling collision avoidance with the potential obstacles. Existing research has not fully utilised the uncertainty of the optical flow-at most, an isotropic Gaussian density model has been used. We estimate the full uncertainty of the optical flow and propose a new eight-point algorithm based on the statistical Mahalanobis distance. Combined with the pose-graph optimisation, the proposed method demonstrates enhanced robustness and accuracy for the public autonomous car dataset (KITTI) and aerial monocular dataset.

Sky and Ground Segmentation in the Navigation Visions of the Planetary Rovers.

Sky and ground are two essential semantic components in computer vision, robotics, and remote sensing. The sky and ground segmentation has become increasingly popular. This research proposes a sky and ground segmentation framework for the rover navigation visions by adopting weak supervision and transfer learning technologies. A new sky and ground segmentation neural network (network in U-shaped network (NI-U-Net)) and a conservative annotation method have been proposed. The pre-trained process achieves the best results on a popular open benchmark (the Skyfinder dataset) by evaluating seven metrics compared to the state-of-the-art. These seven metrics achieve 99.232%, 99.211%, 99.221%, 99.104%, 0.0077, 0.0427, and 98.223% on accuracy, precision, recall, dice score (F1), misclassification rate (MCR), root mean squared error (RMSE), and intersection over union (IoU), respectively. The conservative annotation method achieves superior performance with limited manual intervention. The NI-U-Net can operate with 40 frames per second (FPS) to maintain the real-time property. The proposed framework successfully fills the gap between the laboratory results (with rich idea data) and the practical application (in the wild). The achievement can provide essential semantic information (sky and ground) for the rover navigation vision.

Driven by Vision: Learning Navigation by Visual Localization and Trajectory Prediction.

When driving, people make decisions based on current traffic as well as their desired route. They have a mental map of known routes and are often able to navigate without needing directions. Current published self-driving models improve their performances when using additional GPS information. Here we aim to push forward self-driving research and perform route planning even in the complete absence of GPS at inference time. Our system learns to predict in real-time vehicle's current location and future trajectory, on a known map, given only the raw video stream and the final destination. Trajectories consist of instant steering commands that depend on present traffic, as well as longer-term navigation decisions towards a specific destination. Along with our novel proposed approach to localization and navigation from visual data, we also introduce a novel large dataset in an urban environment, which consists of video and GPS streams collected with a smartphone while driving. The GPS is automatically processed to obtain supervision labels and to create an analytical representation of the traversed map. In tests, our solution outperforms published state of the art methods on visual localization and steering and provides reliable navigation assistance between any two known locations. We also show that our system can adapt to short and long-term changes in weather conditions or the structure of the urban environment. We make the entire dataset and the code publicly available.

The role of attractive and repellent scene memories in ant homing (Myrmecia croslandi).

Solitary foraging ants rely on vision when travelling along routes and when pinpointing their nest. We tethered foragers of Myrmecia croslandi on a trackball and recorded their intended movements when the trackball was located on their normal foraging corridor (on-route), above their nest and at a location several metres away where they have never been before (off-route). We found that at on- and off-route locations, most ants walk in the nest or foraging direction and continue to do so for tens of metres in a straight line. In contrast, above the nest, ants walk in random directions and change walking direction frequently. In addition, the walking direction of ants above the nest oscillates on a fine scale, reflecting search movements that are absent from the paths of ants at the other locations. An agent-based simulation shows that the behaviour of ants at all three locations can be explained by the integration of attractive and repellent views directed towards or away from the nest, respectively. Ants are likely to acquire such views via systematic scanning movements during their learning walks. The model predicts that ants placed in a completely unfamiliar environment should behave as if at the nest, which our subsequent experiments confirmed. We conclude first, that the ants' behaviour at release sites is exclusively driven by what they currently see and not by information on expected outcomes of their behaviour; and second, that navigating ants might continuously integrate attractive and repellent visual memories. We discuss the benefits of such a procedure.

Effect of large visual changes on the navigation of the nocturnal bull ant, Myrmecia midas.

Nocturnal insects have remarkable visual capacities in dim light. They can navigate using both the surrounding panorama and celestial cues. Individual foraging ants are efficient navigators, able to accurately reach a variety of goal locations. During navigation, foragers compare the current panoramic view to previously learnt views. In this natural experiment, we observed the effects of large panorama changes, the addition of a fence and the removal of several trees near the nest site, on the navigation of the nocturnal bull ant Myrmecia midas. We examined how the ants' navigational efficiency and behaviour changed in response to changes in ~ 30% of the surrounding skyline, following them over multiple nights. Foragers were displaced locally off-route where we collected initial orientations and homing paths both before and after large panorama changes. We found that immediately after these changes, foragers were unable to initially orient correctly to the nest direction and foragers' return paths were less straight, suggesting increased navigational uncertainty. Continued testing showed rapid recovery in both initial orientation and path straightness.

For humans navigating without vision, navigation depends upon the layout of mechanically contacted ground surfaces.

Navigation can be haptically guided. In specific, tissue deformations arising from both limb motions during locomotion (i.e., gait patterns) and mechanical interactions between the limbs and the environment can convey information, detected by the haptic perceptual system, about how the body is moving relative to the environment. Here, we test hypotheses concerning the properties of mechanically contacted environments relevant to navigation of this kind. We studied blindfolded participants implicitly learning to perceive their location within environments that were physically encountered via walking on, stepping on, and probing ground surfaces with a cane. Environments were straight-line paths with elevated sections where the path either narrowed or remained the same width. We formed hypotheses concerning how these two environments would affect spatial updating and reorientation processes. In the constant pathwidth environment, homing task accuracy was higher and a manipulation of the elevated surface, to be either unchanged or (unbeknown to participants) shortened, biased the performance. This was consistent with our hypothesis of a metric recalibration scaled to elevated surface extent. In the narrowing pathwidth environment, elevated surface shortening did not bias performance. This supported our hypothesis of positional recalibration resulting from contact with the leading edge of the elevated surface. We discuss why certain environmental properties, such as path-narrowing, have significance for how one becomes implicitly oriented the surrounding environment.