Deep-Learning-Based Context-Aware Multi-Level Information Fusion Systems for Indoor Mobile Robots Safe Navigation.

Hazardous object detection (escalators, stairs, glass doors, etc.) and avoidance are critical functional safety modules for autonomous mobile cleaning robots. Conventional object detectors have less accuracy for detecting low-feature hazardous objects and have miss detection, and the false classification ratio is high when the object is under occlusion. Miss detection or false classification of hazardous objects poses an operational safety issue for mobile robots. This work presents a deep-learning-based context-aware multi-level information fusion framework for autonomous mobile cleaning robots to detect and avoid hazardous objects with a higher confidence level, even if the object is under occlusion. First, the image-level-contextual-encoding module was proposed and incorporated with the Faster RCNN ResNet 50 object detector model to improve the low-featured and occluded hazardous object detection in an indoor environment. Further, a safe-distance-estimation function was proposed to avoid hazardous objects. It computes the distance of the hazardous object from the robot's position and steers the robot into a safer zone using detection results and object depth data. The proposed framework was trained with a custom image dataset using fine-tuning techniques and tested in real-time with an in-house-developed mobile cleaning robot, BELUGA. The experimental results show that the proposed algorithm detected the low-featured and occluded hazardous object with a higher confidence level than the conventional object detector and scored an average detection accuracy of 88.71%.

AI-Enabled Mosquito Surveillance and Population Mapping Using Dragonfly Robot.

Mosquito-borne diseases can pose serious risks to human health. Therefore, mosquito surveillance and control programs are essential for the wellbeing of the community. Further, human-assisted mosquito surveillance and population mapping methods are time-consuming, labor-intensive, and require skilled manpower. This work presents an AI-enabled mosquito surveillance and population mapping framework using our in-house-developed robot, named 'Dragonfly', which uses the You Only Look Once (YOLO) V4 Deep Neural Network (DNN) algorithm and a two-dimensional (2D) environment map generated by the robot. The Dragonfly robot was designed with a differential drive mechanism and a mosquito trapping module to attract mosquitoes in the environment. The YOLO V4 was trained with three mosquito classes, namely Aedes aegypti, Aedes albopictus, and Culex, to detect and classify the mosquito breeds from the mosquito glue trap. The efficiency of the mosquito surveillance framework was determined in terms of mosquito classification accuracy and detection confidence level on offline and real-time field tests in a garden, drain perimeter area, and covered car parking area. The experimental results show that the trained YOLO V4 DNN model detects and classifies the mosquito classes with an 88% confidence level on offline mosquito test image datasets and scores an average of an 82% confidence level on the real-time field trial. Further, to generate the mosquito population map, the detection results are fused in the robot's 2D map, which will help to understand mosquito population dynamics and species distribution.

Object-of-Interest Perception in a Reconfigurable Rolling-Crawling Robot.

Cebrenus Rechenburgi, a member of the huntsman spider family have inspired researchers to adopt different locomotion modes in reconfigurable robotic development. Object-of-interest perception is crucial for such a robot to provide fundamental information on the traversed pathways and guide its locomotion mode transformation. Therefore, we present a object-of-interest perception in a reconfigurable rolling-crawling robot and identifying appropriate locomotion modes. We demonstrate it in Scorpio, our in-house developed robot with two locomotion modes: rolling and crawling. We train the locomotion mode recognition framework, named Pyramid Scene Parsing Network (PSPNet), with a self-collected dataset composed of two categories paths, unobstructed paths (e.g., floor) for rolling and obstructed paths (e.g., with person, railing, stairs, static objects and wall) for crawling, respectively. The efficiency of the proposed framework has been validated with evaluation metrics in offline and real-time field trial tests. The experiment results show that the trained model can achieve an mIOU score of 72.28 and 70.63 in offline and online testing, respectively for both environments. The proposed framework's performance is compared with semantic framework (HRNet and Deeplabv3) where the proposed framework outperforms in terms of mIOU and speed. Furthermore, the experimental results has revealed that the robot's maneuverability is stable, and the proposed framework can successfully determine the appropriate locomotion modes with enhanced accuracy during complex pathways.

Toward a Comprehensive Domestic Dirt Dataset Curation for Cleaning Auditing Applications.

Cleaning is an important task that is practiced in every domain and has prime importance. The significance of cleaning has led to several newfangled technologies in the domestic and professional cleaning domain. However, strategies for auditing the cleanliness delivered by the various cleaning methods remain manual and often ignored. This work presents a novel domestic dirt image dataset for cleaning auditing application including AI-based dirt analysis and robot-assisted cleaning inspection. One of the significant challenges in an AI-based robot-aided cleaning auditing is the absence of a comprehensive dataset for dirt analysis. We bridge this gap by identifying nine classes of commonly occurring domestic dirt and a labeled dataset consisting of 3000 microscope dirt images curated from a semi-indoor environment. The dirt dataset gathered using the adhesive dirt lifting method can enhance the current dirt sensing and dirt composition estimation for cleaning auditing. The dataset's quality is analyzed by AI-based dirt analysis and a robot-aided cleaning auditing task using six standard classification models. The models trained with the dirt dataset were capable of yielding a classification accuracy above 90% in the offline dirt analysis experiment and 82% in real-time test results.

A Novel Path Planning Strategy for a Cleaning Audit Robot Using Geometrical Features and Swarm Algorithms.

Robot-aided cleaning auditing is pioneering research that uses autonomous robots to assess a region's cleanliness level by analyzing the dirt samples collected from various locations. Since the dirt sample gathering process is more challenging, adapting a coverage planning strategy from a similar domain for cleaning is non-viable. Alternatively, a path planning approach to gathering dirt samples selectively at locations with a high likelihood of dirt accumulation is more feasible. This work presents a first-of-its-kind dirt sample gathering strategy for the cleaning auditing robots by combining the geometrical feature extraction and swarm algorithms. This combined approach generates an efficient optimal path covering all the identified dirt locations for efficient cleaning auditing. Besides being the foundational effort for cleaning audit, a path planning approach considering the geometric signatures that contribute to the dirt accumulation of a region has not been device so far. The proposed approach is validated systematically through experiment trials. The geometrical feature extraction-based dirt location identification method successfully identified dirt accumulated locations in our post-cleaning analysis as part of the experiment trials. The path generation strategies are validated in a real-world environment using an in-house developed cleaning auditing robot BELUGA. From the experiments conducted, the ant colony optimization algorithm generated the best cleaning auditing path with less travel distance, exploration time, and energy usage.

AI Enabled IoRT Framework for Rodent Activity Monitoring in a False Ceiling Environment.

Routine rodent inspection is essential to curbing rat-borne diseases and infrastructure damages within the built environment. Rodents find false ceilings to be a perfect spot to seek shelter and construct their habitats. However, a manual false ceiling inspection for rodents is laborious and risky. This work presents an AI-enabled IoRT framework for rodent activity monitoring inside a false ceiling using an in-house developed robot called "Falcon". The IoRT serves as a bridge between the users and the robots, through which seamless information sharing takes place. The shared images by the robots are inspected through a Faster RCNN ResNet 101 object detection algorithm, which is used to automatically detect the signs of rodent inside a false ceiling. The efficiency of the rodent activity detection algorithm was tested in a real-world false ceiling environment, and detection accuracy was evaluated with the standard performance metrics. The experimental results indicate that the algorithm detects rodent signs and 3D-printed rodents with a good confidence level.

sTetro-Deep Learning Powered Staircase Cleaning and Maintenance Reconfigurable Robot.

Staircase cleaning is a crucial and time-consuming task for maintenance of multistory apartments and commercial buildings. There are many commercially available autonomous cleaning robots in the market for building maintenance, but few of them are designed for staircase cleaning. A key challenge for automating staircase cleaning robots involves the design of Environmental Perception Systems (EPS), which assist the robot in determining and navigating staircases. This system also recognizes obstacles and debris for safe navigation and efficient cleaning while climbing the staircase. This work proposes an operational framework leveraging the vision based EPS for the modular re-configurable maintenance robot, called sTetro. The proposed system uses an SSD MobileNet real-time object detection model to recognize staircases, obstacles and debris. Furthermore, the model filters out false detection of staircases by fusion of depth information through the use of a MobileNet and SVM. The system uses a contour detection algorithm to localize the first step of the staircase and depth clustering scheme for obstacle and debris localization. The framework has been deployed on the sTetro robot using the Jetson Nano hardware from NVIDIA and tested with multistory staircases. The experimental results show that the entire framework takes an average of 310 ms to run and achieves an accuracy of 94.32% for staircase recognition tasks and 93.81% accuracy for obstacle and debris detection tasks during real operation of the robot.

False Ceiling Deterioration Detection and Mapping Using a Deep Learning Framework and the Teleoperated Reconfigurable 'Falcon' Robot.

Periodic inspection of false ceilings is mandatory to ensure building and human safety. Generally, false ceiling inspection includes identifying structural defects, degradation in Heating, Ventilation, and Air Conditioning (HVAC) systems, electrical wire damage, and pest infestation. Human-assisted false ceiling inspection is a laborious and risky task. This work presents a false ceiling deterioration detection and mapping framework using a deep-neural-network-based object detection algorithm and the teleoperated 'Falcon' robot. The object detection algorithm was trained with our custom false ceiling deterioration image dataset composed of four classes: structural defects (spalling, cracks, pitted surfaces, and water damage), degradation in HVAC systems (corrosion, molding, and pipe damage), electrical damage (frayed wires), and infestation (termites and rodents). The efficiency of the trained CNN algorithm and deterioration mapping was evaluated through various experiments and real-time field trials. The experimental results indicate that the deterioration detection and mapping results were accurate in a real false-ceiling environment and achieved an 89.53% detection accuracy.

An Autonomous Robot-Aided Auditing Scheme for Floor Cleaning.

Cleaning is an important factor in most aspects of our day-to-day life. This research work brings a solution to the fundamental question of "How clean is clean" by introducing a novel framework for auditing the cleanliness of built infrastructure using mobile robots. The proposed system presents a strategy for assessing the quality of cleaning in a given area and a novel exploration strategy that facilitates the auditing in a given location by a mobile robot. An audit sensor that works by the "touch and inspect" analogy that assigns an audit score corresponds to its area of inspection has been developed. A vision-based dirt-probability-driven exploration is proposed to empower a mobile robot with an audit sensor on-board to perform auditing tasks effectively. The quality of cleaning is quantified using a dirt density map representing location-wise audit scores, dirt distribution pattern obtained by kernel density estimation, and cleaning benchmark score representing the extent of cleanliness. The framework is realized in an in-house developed audit robot to perform the cleaning audit in indoor and semi-outdoor environments. The proposed method is validated by experiment trials to estimate the cleanliness in five different locations using the developed audit sensor and dirt-probability-driven exploration.

A Reinforcement Learning Based Dirt-Exploration for Cleaning-Auditing Robot.

Cleaning is one of the fundamental tasks with prime importance given in our day-to-day life. Moreover, the importance of cleaning drives the research efforts towards bringing leading edge technologies, including robotics, into the cleaning domain. However, an effective method to assess the quality of cleaning is an equally important research problem to be addressed. The primary footstep towards addressing the fundamental question of "How clean is clean" is addressed using an autonomous cleaning-auditing robot that audits the cleanliness of a given area. This research work focuses on a novel reinforcement learning-based experience-driven dirt exploration strategy for a cleaning-auditing robot. The proposed approach uses proximal policy approximation (PPO) based on-policy learning method to generate waypoints and sampling decisions to explore the probable dirt accumulation regions in a given area. The policy network is trained in multiple environments with simulated dirt patterns. Experiment trials have been conducted to validate the trained policy in both simulated and real-world environments using an in-house developed cleaning audit robot called BELUGA.

An Optimal Footprint Based Coverage Planning for Hydro Blasting Robots.

The hydro blasting of metallic surfaces is an essential maintenance task in various industrial sites. Its requirement of a considerable labour force and time, calls for automating the hydro blasting jobs through mobile robots. A hydro blasting robot should be able to cover the required area for a successful implementation. If a conventional robot footprint is chosen, the blasting may become inefficient, even though the concerned area is completely covered. In this work, the blasting arm's sweeping angle is chosen as the robot's footprint for hydro blasting task, and a multi-objective optimization-based framework is proposed to compute the optimal sweeping arc. The genetic algorithm (GA) methodology is exploited to compute the optimal footprint, which minimizes the blasting time and energy simultaneously. Multiple numerical simulations are performed to show the effectiveness of the proposed approach. Moreover, the strategy is successfully implemented on our hydro blasting robot named Hornbill, and the efficacy of the proposed approach is validated through experimental trials.

Drain Structural Defect Detection and Mapping Using AI-Enabled Reconfigurable Robot Raptor and IoRT Framework.

Human visual inspection of drains is laborious, time-consuming, and prone to accidents. This work presents an AI-enabled robot-assisted remote drain inspection and mapping framework using our in-house developed reconfigurable robot Raptor. The four-layer IoRT serves as a bridge between the users and the robots, through which seamless information sharing takes place. The Faster RCNN ResNet50, Faster RCNN ResNet101, and Faster RCNN Inception-ResNet-v2 deep learning frameworks were trained using a transfer learning scheme with six typical concrete defect classes and deployed in an IoRT framework remote defect detection task. The efficiency of the trained CNN algorithm and drain inspection robot Raptor was evaluated through various real-time drain inspection field trials using the SLAM technique. The experimental results indicate that robot's maneuverability was stable, and its mapping and localization were also accurate in different drain types. Finally, for effective drain maintenance, the SLAM-based defect map was generated by fusing defect detection results in the lidar-SLAM map.

Towards an Optimal Footprint Based Area Coverage Strategy for a False-Ceiling Inspection Robot.

False-ceiling inspection is a critical factor in pest-control management within a built infrastructure. Conventionally, the false-ceiling inspection is done manually, which is time-consuming and unsafe. A lightweight robot is considered a good solution for automated false-ceiling inspection. However, due to the constraints imposed by less load carrying capacity and brittleness of false ceilings, the inspection robots cannot rely upon heavy batteries, sensors, and computation payloads for enhancing task performance. Hence, the strategy for inspection has to ensure efficiency and best performance. This work presents an optimal functional footprint approach for the robot to maximize the efficiency of an inspection task. With a conventional footprint approach in path planning, complete coverage inspection may become inefficient. In this work, the camera installation parameters are considered as the footprint defining parameters for the false ceiling inspection. An evolutionary algorithm-based multi-objective optimization framework is utilized to derive the optimal robot footprint by minimizing the area missed and path-length taken for the inspection task. The effectiveness of the proposed approach is analyzed using numerical simulations. The results are validated on an in-house developed false-ceiling inspection robot-Raptor-by experiment trials on a false-ceiling test-bed.

Collision Avoidance and Stability Study of a Self-Reconfigurable Drainage Robot.

The inspection and maintenance of drains with varying heights necessitates a drain mapping robot with trained labour to maintain community hygiene and prevent the spread of diseases. For adapting to level changes and navigating in the narrow confined environments of drains, we developed a self-configurable hybrid robot, named Tarantula-II. The platform is a quadruped robot with hybrid locomotion and the ability to reconfigure to achieve variable height and width. It has four legs, and each leg is made of linear actuators and modular rolling wheel mechanisms with bi-directional movement. The platform has a fuzzy logic system for collision avoidance of the side wall in the drain environment. During level shifting, the platform achieves stability by using the pitch angle as the feedback from the inertial measuring unit (IMU) mounted on the platform. This feedback helps to adjust the accurate height of the platform. In this paper, we describe the detailed mechanical design and system architecture, kinematic models, control architecture, and stability of the platform. We deployed the platform both in a lab setting and in a real-time drain environment to demonstrate the wall collision avoidance, stability, and level shifting capabilities of the platform.

Adaptive Floor Cleaning Strategy by Human Density Surveillance Mapping with a Reconfigurable Multi-Purpose Service Robot.

Professional cleaning and safe social distance monitoring are often considered as demanding, time-consuming, repetitive, and labor-intensive tasks with the risk of getting exposed to the virus. Safe social distance monitoring and cleaning are emerging problems solved through robotics solutions. This research aims to develop a safe social distance surveillance system on an intra-reconfigurable robot with a multi-robot cleaning system for large population environments, like office buildings, hospitals, or shopping malls. We propose an adaptive multi-robot cleaning strategy based on zig-zag-based coverage path planning that works in synergy with the human interaction heat map generated by safe social distance monitoring systems. We further validate the proposed adaptive velocity model's efficiency for the multi-robot cleaning systems regarding time consumption and energy saved. The proposed method using sigmoid-based non-linear function has shown superior performance with 14.1 percent faster and energy consumption of 11.8 percent less than conventional cleaning methods.

Coverage Path Planning Using Reinforcement Learning-Based TSP for hTetran-A Polyabolo-Inspired Self-Reconfigurable Tiling Robot.

One of the critical challenges in deploying the cleaning robots is the completion of covering the entire area. Current tiling robots for area coverage have fixed forms and are limited to cleaning only certain areas. The reconfigurable system is the creative answer to such an optimal coverage problem. The tiling robot's goal enables the complete coverage of the entire area by reconfiguring to different shapes according to the area's needs. In the particular sequencing of navigation, it is essential to have a structure that allows the robot to extend the coverage range while saving energy usage during navigation. This implies that the robot is able to cover larger areas entirely with the least required actions. This paper presents a complete path planning (CPP) for hTetran, a polyabolo tiled robot, based on a TSP-based reinforcement learning optimization. This structure simultaneously produces robot shapes and sequential trajectories whilst maximizing the reward of the trained reinforcement learning (RL) model within the predefined polyabolo-based tileset. To this end, a reinforcement learning-based travel sales problem (TSP) with proximal policy optimization (PPO) algorithm was trained using the complementary learning computation of the TSP sequencing. The reconstructive results of the proposed RL-TSP-based CPP for hTetran were compared in terms of energy and time spent with the conventional tiled hypothetical models that incorporate TSP solved through an evolutionary based ant colony optimization (ACO) approach. The CPP demonstrates an ability to generate an ideal Pareto optima trajectory that enhances the robot's navigation inside the real environment with the least energy and time spent in the company of conventional techniques.

Reinforcement Learning-Based Complete Area Coverage Path Planning for a Modified hTrihex Robot.

One of the essential attributes of a cleaning robot is to achieve complete area coverage. Current commercial indoor cleaning robots have fixed morphology and are restricted to clean only specific areas in a house. The results of maximum area coverage are sub-optimal in this case. Tiling robots are innovative solutions for such a coverage problem. These new kinds of robots can be deployed in the cases of cleaning, painting, maintenance, and inspection, which require complete area coverage. Tiling robots' objective is to cover the entire area by reconfiguring to different shapes as per the area requirements. In this context, it is vital to have a framework that enables the robot to maximize the area coverage while minimizing energy consumption. That means it is necessary for the robot to cover the maximum area with the least number of shape reconfigurations possible. The current paper proposes a complete area coverage planning module for the modified hTrihex, a honeycomb-shaped tiling robot, based on the deep reinforcement learning technique. This framework simultaneously generates the tiling shapes and the trajectory with minimum overall cost. In this regard, a convolutional neural network (CNN) with long short term memory (LSTM) layer was trained using the actor-critic experience replay (ACER) reinforcement learning algorithm. The simulation results obtained from the current implementation were compared against the results that were generated through traditional tiling theory models that included zigzag, spiral, and greedy search schemes. The model presented in the current paper was also compared against other methods where this problem was considered as a traveling salesman problem (TSP) solved through genetic algorithm (GA) and ant colony optimization (ACO) approaches. Our proposed scheme generates a path with a minimized cost at a lesser time.

Deep Learning Based Pavement Inspection Using Self-Reconfigurable Robot.

The pavement inspection task, which mainly includes crack and garbage detection, is essential and carried out frequently. The human-based or dedicated system approach for inspection can be easily carried out by integrating with the pavement sweeping machines. This work proposes a deep learning-based pavement inspection framework for self-reconfigurable robot named Panthera. Semantic segmentation framework SegNet was adopted to segment the pavement region from other objects. Deep Convolutional Neural Network (DCNN) based object detection is used to detect and localize pavement defects and garbage. Furthermore, Mobile Mapping System (MMS) was adopted for the geotagging of the defects. The proposed system was implemented and tested with the Panthera robot having NVIDIA GPU cards. The experimental results showed that the proposed technique identifies the pavement defects and litters or garbage detection with high accuracy. The experimental results on the crack and garbage detection are presented. It is found that the proposed technique is suitable for deployment in real-time for garbage detection and, eventually, sweeping or cleaning tasks.

Modelling and Control of a Reconfigurable Robot for Achieving Reconfiguration and Locomotion with Different Shapes.

Area coverage is a crucial factor for a robot intended for applications such as floor cleaning, disinfection, and inspection. Robots with fixed shapes could not realize an adequate level of area coverage performance. Reconfigurable robots have been introduced to overcome the limitations of fixed-shape robots, such as accessing narrow spaces and cover obstacles. Although state-of-the-art reconfigurable robots used for coverage applications are capable of shape-changing for improving the area coverage, the reconfiguration is limited to a few predefined shapes. It has been proven that the ability of reconfiguration beyond a few shapes can significantly improve the area coverage performance of a reconfigurable robot. In this regard, this paper proposes a novel robot model and a low-level controller that can facilitate the reconfiguration beyond a small set of predefined shapes and locomotion per instructions while firmly maintaining the shape. A prototype of a robot that facilitates the aim mentioned above has been designed and developed. The proposed robot model and controller have been integrated into the prototype, and experiments have been conducted considering various reconfiguration and locomotion scenarios. Experimental results confirm the validity of the proposed model and controller during reconfiguration and locomotion of the robot. Moreover, the applicability of the proposed model and controller for achieving high-level autonomous capabilities has been proven.

Design of a Reconfigurable Wall Disinfection Robot.

During a viral outbreak, such as COVID-19, autonomously operated robots are in high demand. Robots effectively improve the environmental concerns of contaminated surfaces in public spaces, such as airports, public transport areas and hospitals, that are considered high-risk areas. Indoor spaces walls made up most of the indoor areas in these public spaces and can be easily contaminated. Wall cleaning and disinfection processes are therefore critical for managing and mitigating the spread of viruses. Consequently, wall cleaning robots are preferred to address the demands. A wall cleaning robot needs to maintain a close and consistent distance away from a given wall during cleaning and disinfection processes. In this paper, a reconfigurable wall cleaning robot with autonomous wall following ability is proposed. The robot platform, Wasp, possess inter-reconfigurability, which enables it to be physically reconfigured into a wall-cleaning robot. The wall following ability has been implemented using a Fuzzy Logic System (FLS). The design of the robot and the FLS are presented in the paper. The platform and the FLS are tested and validated in several test cases. The experimental outcomes validate the real-world applicability of the proposed wall following method for a wall cleaning robot.