Vision-based dirt distribution mapping using deep learning.

Cleaning is a fundamental routine task in human life that is now handed over to leading-edge technologies such as robotics and artificial intelligence. Various floor-cleaning robots have been developed with different cleaning functionalities, such as vacuuming and scrubbing. However, failures can occur when a robot tries to clean an incompatible dirt type. These situations will not only reduce the efficiency of the robot but also impose severe damage to the robots. Therefore, developing effective methods to classify the cleaning tasks performed in different regions and assign them to the respective cleaning agent has become a trending research domain. This article proposes a vision-based system that employs YOLOv5 and DeepSORT algorithms to detect and classify dirt to create a dirt distribution map that indicates the regions to be assigned for different cleaning requirements. This map would be useful for a collaborative cleaning framework for deploying each cleaning robot to its respective region to achieve an uninterrupted and energy-efficient operation. The proposed method can be executed with any mobile robot and on any surface and dirt, achieving high accuracy of 81.0%, for dirt indication in the dirt distribution map.

Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions.

This work presents the vision pipeline for our in-house developed autonomous reconfigurable pavement sweeping robot named Panthera. As the goal of Panthera is to be an autonomous self-reconfigurable robot, it has to understand the type of pavement it is moving in so that it can adapt smoothly to changing pavement width and perform cleaning operations more efficiently and safely. deep learning (DL) based vision pipeline is proposed for the Panthera robot to recognize pavement features, including pavement type identification, pavement surface condition prediction, and pavement width estimation. The DeepLabv3+ semantic segmentation algorithm was customized to identify the pavement type classification, an eight-layer CNN was proposed for pavement surface condition prediction. Furthermore, pavement width estimation was computed by fusing the segmented pavement region on the depth map. In the end, the fuzzy inference system was implemented by taking input as the pavement width and its conditions detected and output as the safe operational speed. The vision pipeline was trained using the DL provided with the custom pavement images dataset. The performance was evaluated using offline test and real-time field trial images captured through the reconfigurable robot Panthera stereo vision sensor. In the experimental analysis, the DL-based vision pipeline components scored 88.02% and 93.22% accuracy for pavement segmentation and pavement surface condition assessment, respectively, and took approximately 10 ms computation time to process the single image frame from the vision sensor using the onboard computer.

Toward a Comprehensive Domestic Dirt Dataset Curation for Cleaning Auditing Applications.

Cleaning is an important task that is practiced in every domain and has prime importance. The significance of cleaning has led to several newfangled technologies in the domestic and professional cleaning domain. However, strategies for auditing the cleanliness delivered by the various cleaning methods remain manual and often ignored. This work presents a novel domestic dirt image dataset for cleaning auditing application including AI-based dirt analysis and robot-assisted cleaning inspection. One of the significant challenges in an AI-based robot-aided cleaning auditing is the absence of a comprehensive dataset for dirt analysis. We bridge this gap by identifying nine classes of commonly occurring domestic dirt and a labeled dataset consisting of 3000 microscope dirt images curated from a semi-indoor environment. The dirt dataset gathered using the adhesive dirt lifting method can enhance the current dirt sensing and dirt composition estimation for cleaning auditing. The dataset's quality is analyzed by AI-based dirt analysis and a robot-aided cleaning auditing task using six standard classification models. The models trained with the dirt dataset were capable of yielding a classification accuracy above 90% in the offline dirt analysis experiment and 82% in real-time test results.

A Novel Path Planning Strategy for a Cleaning Audit Robot Using Geometrical Features and Swarm Algorithms.

Robot-aided cleaning auditing is pioneering research that uses autonomous robots to assess a region's cleanliness level by analyzing the dirt samples collected from various locations. Since the dirt sample gathering process is more challenging, adapting a coverage planning strategy from a similar domain for cleaning is non-viable. Alternatively, a path planning approach to gathering dirt samples selectively at locations with a high likelihood of dirt accumulation is more feasible. This work presents a first-of-its-kind dirt sample gathering strategy for the cleaning auditing robots by combining the geometrical feature extraction and swarm algorithms. This combined approach generates an efficient optimal path covering all the identified dirt locations for efficient cleaning auditing. Besides being the foundational effort for cleaning audit, a path planning approach considering the geometric signatures that contribute to the dirt accumulation of a region has not been device so far. The proposed approach is validated systematically through experiment trials. The geometrical feature extraction-based dirt location identification method successfully identified dirt accumulated locations in our post-cleaning analysis as part of the experiment trials. The path generation strategies are validated in a real-world environment using an in-house developed cleaning auditing robot BELUGA. From the experiments conducted, the ant colony optimization algorithm generated the best cleaning auditing path with less travel distance, exploration time, and energy usage.

A Reinforcement Learning Based Dirt-Exploration for Cleaning-Auditing Robot.

Cleaning is one of the fundamental tasks with prime importance given in our day-to-day life. Moreover, the importance of cleaning drives the research efforts towards bringing leading edge technologies, including robotics, into the cleaning domain. However, an effective method to assess the quality of cleaning is an equally important research problem to be addressed. The primary footstep towards addressing the fundamental question of "How clean is clean" is addressed using an autonomous cleaning-auditing robot that audits the cleanliness of a given area. This research work focuses on a novel reinforcement learning-based experience-driven dirt exploration strategy for a cleaning-auditing robot. The proposed approach uses proximal policy approximation (PPO) based on-policy learning method to generate waypoints and sampling decisions to explore the probable dirt accumulation regions in a given area. The policy network is trained in multiple environments with simulated dirt patterns. Experiment trials have been conducted to validate the trained policy in both simulated and real-world environments using an in-house developed cleaning audit robot called BELUGA.

Design of a Reconfigurable Wall Disinfection Robot.

During a viral outbreak, such as COVID-19, autonomously operated robots are in high demand. Robots effectively improve the environmental concerns of contaminated surfaces in public spaces, such as airports, public transport areas and hospitals, that are considered high-risk areas. Indoor spaces walls made up most of the indoor areas in these public spaces and can be easily contaminated. Wall cleaning and disinfection processes are therefore critical for managing and mitigating the spread of viruses. Consequently, wall cleaning robots are preferred to address the demands. A wall cleaning robot needs to maintain a close and consistent distance away from a given wall during cleaning and disinfection processes. In this paper, a reconfigurable wall cleaning robot with autonomous wall following ability is proposed. The robot platform, Wasp, possess inter-reconfigurability, which enables it to be physically reconfigured into a wall-cleaning robot. The wall following ability has been implemented using a Fuzzy Logic System (FLS). The design of the robot and the FLS are presented in the paper. The platform and the FLS are tested and validated in several test cases. The experimental outcomes validate the real-world applicability of the proposed wall following method for a wall cleaning robot.

sTetro-Deep Learning Powered Staircase Cleaning and Maintenance Reconfigurable Robot.

Staircase cleaning is a crucial and time-consuming task for maintenance of multistory apartments and commercial buildings. There are many commercially available autonomous cleaning robots in the market for building maintenance, but few of them are designed for staircase cleaning. A key challenge for automating staircase cleaning robots involves the design of Environmental Perception Systems (EPS), which assist the robot in determining and navigating staircases. This system also recognizes obstacles and debris for safe navigation and efficient cleaning while climbing the staircase. This work proposes an operational framework leveraging the vision based EPS for the modular re-configurable maintenance robot, called sTetro. The proposed system uses an SSD MobileNet real-time object detection model to recognize staircases, obstacles and debris. Furthermore, the model filters out false detection of staircases by fusion of depth information through the use of a MobileNet and SVM. The system uses a contour detection algorithm to localize the first step of the staircase and depth clustering scheme for obstacle and debris localization. The framework has been deployed on the sTetro robot using the Jetson Nano hardware from NVIDIA and tested with multistory staircases. The experimental results show that the entire framework takes an average of 310 ms to run and achieves an accuracy of 94.32% for staircase recognition tasks and 93.81% accuracy for obstacle and debris detection tasks during real operation of the robot.

Raptor: A Design of a Drain Inspection Robot.

Frequent inspections are essential for drains to maintain proper function to ensure public health and safety. Robots have been developed to aid the drain inspection process. However, existing robots designed for drain inspection require improvements in their design and autonomy. This paper proposes a novel design of a drain inspection robot named Raptor. The robot has been designed with a manually reconfigurable wheel axle mechanism, which allows the change of ground clearance height. Design aspects of the robot, such as mechanical design, control architecture and autonomy functions, are comprehensively described in the paper, and insights are included. Maintaining the robot's position in the middle of a drain when moving along the drain is essential for the inspection process. Thus, a fuzzy logic controller has been introduced to the robot to cater to this demand. Experiments have been conducted by deploying a prototype of the design to drain environments considering a set of diverse test scenarios. Experiment results show that the proposed controller effectively maintains the robot in the middle of a drain while moving along the drain. Therefore, the proposed robot design and the controller would be helpful in improving the productivity of robot-aided inspection of drains.

Towards an Optimal Footprint Based Area Coverage Strategy for a False-Ceiling Inspection Robot.

False-ceiling inspection is a critical factor in pest-control management within a built infrastructure. Conventionally, the false-ceiling inspection is done manually, which is time-consuming and unsafe. A lightweight robot is considered a good solution for automated false-ceiling inspection. However, due to the constraints imposed by less load carrying capacity and brittleness of false ceilings, the inspection robots cannot rely upon heavy batteries, sensors, and computation payloads for enhancing task performance. Hence, the strategy for inspection has to ensure efficiency and best performance. This work presents an optimal functional footprint approach for the robot to maximize the efficiency of an inspection task. With a conventional footprint approach in path planning, complete coverage inspection may become inefficient. In this work, the camera installation parameters are considered as the footprint defining parameters for the false ceiling inspection. An evolutionary algorithm-based multi-objective optimization framework is utilized to derive the optimal robot footprint by minimizing the area missed and path-length taken for the inspection task. The effectiveness of the proposed approach is analyzed using numerical simulations. The results are validated on an in-house developed false-ceiling inspection robot-Raptor-by experiment trials on a false-ceiling test-bed.

Falcon: A False Ceiling Inspection Robot.

Frequent inspections are essential for false ceilings to maintain the service infrastructures, such as mechanical, electrical, and plumbing, and the structure of false ceilings. Human-labor-based conventional inspection procedures for false ceilings suffer many shortcomings, including safety concerns. Thus, robot-aided solutions are demanded for false ceiling inspections similar to other building maintenance services. However, less work has been conducted on developing robot-aided solutions for false ceiling inspections. This paper proposes a novel design for a robot intended for false ceiling inspections named Falcon. The compact size and the tracked wheel design of the robot allow it to traverse obstacles such as runners and lighting fixtures. The robot's ability to autonomously follow the perimeter of a false ceiling can improve the productivity of the inspection process since the heading of the robot often changes due to the nature of the terrain, and continuous heading correction is an overhead for a teleoperator. Therefore, a Perimeter-Following Controller (PFC) based on fuzzy logic was integrated into the robot. Experimental results obtained by deploying a prototype of the robot design to a false ceiling testbed confirmed the effectiveness of the proposed PFC in perimeter following and the robot's features, such as the ability to traverse on runners and fixtures in a false ceiling.

Modelling and Control of a Reconfigurable Robot for Achieving Reconfiguration and Locomotion with Different Shapes.

Area coverage is a crucial factor for a robot intended for applications such as floor cleaning, disinfection, and inspection. Robots with fixed shapes could not realize an adequate level of area coverage performance. Reconfigurable robots have been introduced to overcome the limitations of fixed-shape robots, such as accessing narrow spaces and cover obstacles. Although state-of-the-art reconfigurable robots used for coverage applications are capable of shape-changing for improving the area coverage, the reconfiguration is limited to a few predefined shapes. It has been proven that the ability of reconfiguration beyond a few shapes can significantly improve the area coverage performance of a reconfigurable robot. In this regard, this paper proposes a novel robot model and a low-level controller that can facilitate the reconfiguration beyond a small set of predefined shapes and locomotion per instructions while firmly maintaining the shape. A prototype of a robot that facilitates the aim mentioned above has been designed and developed. The proposed robot model and controller have been integrated into the prototype, and experiments have been conducted considering various reconfiguration and locomotion scenarios. Experimental results confirm the validity of the proposed model and controller during reconfiguration and locomotion of the robot. Moreover, the applicability of the proposed model and controller for achieving high-level autonomous capabilities has been proven.

An Autonomous Robot-Aided Auditing Scheme for Floor Cleaning.

Cleaning is an important factor in most aspects of our day-to-day life. This research work brings a solution to the fundamental question of "How clean is clean" by introducing a novel framework for auditing the cleanliness of built infrastructure using mobile robots. The proposed system presents a strategy for assessing the quality of cleaning in a given area and a novel exploration strategy that facilitates the auditing in a given location by a mobile robot. An audit sensor that works by the "touch and inspect" analogy that assigns an audit score corresponds to its area of inspection has been developed. A vision-based dirt-probability-driven exploration is proposed to empower a mobile robot with an audit sensor on-board to perform auditing tasks effectively. The quality of cleaning is quantified using a dirt density map representing location-wise audit scores, dirt distribution pattern obtained by kernel density estimation, and cleaning benchmark score representing the extent of cleanliness. The framework is realized in an in-house developed audit robot to perform the cleaning audit in indoor and semi-outdoor environments. The proposed method is validated by experiment trials to estimate the cleanliness in five different locations using the developed audit sensor and dirt-probability-driven exploration.

Collision Avoidance and Stability Study of a Self-Reconfigurable Drainage Robot.

The inspection and maintenance of drains with varying heights necessitates a drain mapping robot with trained labour to maintain community hygiene and prevent the spread of diseases. For adapting to level changes and navigating in the narrow confined environments of drains, we developed a self-configurable hybrid robot, named Tarantula-II. The platform is a quadruped robot with hybrid locomotion and the ability to reconfigure to achieve variable height and width. It has four legs, and each leg is made of linear actuators and modular rolling wheel mechanisms with bi-directional movement. The platform has a fuzzy logic system for collision avoidance of the side wall in the drain environment. During level shifting, the platform achieves stability by using the pitch angle as the feedback from the inertial measuring unit (IMU) mounted on the platform. This feedback helps to adjust the accurate height of the platform. In this paper, we describe the detailed mechanical design and system architecture, kinematic models, control architecture, and stability of the platform. We deployed the platform both in a lab setting and in a real-time drain environment to demonstrate the wall collision avoidance, stability, and level shifting capabilities of the platform.

Locomotion with Pedestrian Aware from Perception Sensor by Pavement Sweeping Reconfigurable Robot.

Regular washing of public pavements is necessary to ensure that the public environment is sanitary for social activities. This is a challenge for autonomous cleaning robots, as they must adapt to the environment with varying pavement widths while avoiding pedestrians. A self-reconfigurable pavement sweeping robot, named Panthera, has the mechanisms to perform reconfiguration in width to enable smooth cleaning operations, and it changes its behavior based on environment dynamics of moving pedestrians and changing pavement widths. Reconfiguration in the robot's width is possible, due to the scissor mechanism at the core of the robot's body, which is driven by a lead screw motor. Panthera will perform locomotion and reconfiguration based on perception sensors feedback control proposed while using an Red Green Blue-D (RGB-D) camera. The proposed control scheme involves publishing robot kinematic parameters for reconfiguration during locomotion. Experiments were conducted in outdoor pavements to demonstrate the autonomous reconfiguration during locomotion to avoid pedestrians while complying with varying pavements widths in a real-world scenario.

Coverage Path Planning Using Reinforcement Learning-Based TSP for hTetran-A Polyabolo-Inspired Self-Reconfigurable Tiling Robot.

One of the critical challenges in deploying the cleaning robots is the completion of covering the entire area. Current tiling robots for area coverage have fixed forms and are limited to cleaning only certain areas. The reconfigurable system is the creative answer to such an optimal coverage problem. The tiling robot's goal enables the complete coverage of the entire area by reconfiguring to different shapes according to the area's needs. In the particular sequencing of navigation, it is essential to have a structure that allows the robot to extend the coverage range while saving energy usage during navigation. This implies that the robot is able to cover larger areas entirely with the least required actions. This paper presents a complete path planning (CPP) for hTetran, a polyabolo tiled robot, based on a TSP-based reinforcement learning optimization. This structure simultaneously produces robot shapes and sequential trajectories whilst maximizing the reward of the trained reinforcement learning (RL) model within the predefined polyabolo-based tileset. To this end, a reinforcement learning-based travel sales problem (TSP) with proximal policy optimization (PPO) algorithm was trained using the complementary learning computation of the TSP sequencing. The reconstructive results of the proposed RL-TSP-based CPP for hTetran were compared in terms of energy and time spent with the conventional tiled hypothetical models that incorporate TSP solved through an evolutionary based ant colony optimization (ACO) approach. The CPP demonstrates an ability to generate an ideal Pareto optima trajectory that enhances the robot's navigation inside the real environment with the least energy and time spent in the company of conventional techniques.

Deep Learning Based Pavement Inspection Using Self-Reconfigurable Robot.

The pavement inspection task, which mainly includes crack and garbage detection, is essential and carried out frequently. The human-based or dedicated system approach for inspection can be easily carried out by integrating with the pavement sweeping machines. This work proposes a deep learning-based pavement inspection framework for self-reconfigurable robot named Panthera. Semantic segmentation framework SegNet was adopted to segment the pavement region from other objects. Deep Convolutional Neural Network (DCNN) based object detection is used to detect and localize pavement defects and garbage. Furthermore, Mobile Mapping System (MMS) was adopted for the geotagging of the defects. The proposed system was implemented and tested with the Panthera robot having NVIDIA GPU cards. The experimental results showed that the proposed technique identifies the pavement defects and litters or garbage detection with high accuracy. The experimental results on the crack and garbage detection are presented. It is found that the proposed technique is suitable for deployment in real-time for garbage detection and, eventually, sweeping or cleaning tasks.

Adaptive Floor Cleaning Strategy by Human Density Surveillance Mapping with a Reconfigurable Multi-Purpose Service Robot.

Professional cleaning and safe social distance monitoring are often considered as demanding, time-consuming, repetitive, and labor-intensive tasks with the risk of getting exposed to the virus. Safe social distance monitoring and cleaning are emerging problems solved through robotics solutions. This research aims to develop a safe social distance surveillance system on an intra-reconfigurable robot with a multi-robot cleaning system for large population environments, like office buildings, hospitals, or shopping malls. We propose an adaptive multi-robot cleaning strategy based on zig-zag-based coverage path planning that works in synergy with the human interaction heat map generated by safe social distance monitoring systems. We further validate the proposed adaptive velocity model's efficiency for the multi-robot cleaning systems regarding time consumption and energy saved. The proposed method using sigmoid-based non-linear function has shown superior performance with 14.1 percent faster and energy consumption of 11.8 percent less than conventional cleaning methods.

An Optimal Footprint Based Coverage Planning for Hydro Blasting Robots.

The hydro blasting of metallic surfaces is an essential maintenance task in various industrial sites. Its requirement of a considerable labour force and time, calls for automating the hydro blasting jobs through mobile robots. A hydro blasting robot should be able to cover the required area for a successful implementation. If a conventional robot footprint is chosen, the blasting may become inefficient, even though the concerned area is completely covered. In this work, the blasting arm's sweeping angle is chosen as the robot's footprint for hydro blasting task, and a multi-objective optimization-based framework is proposed to compute the optimal sweeping arc. The genetic algorithm (GA) methodology is exploited to compute the optimal footprint, which minimizes the blasting time and energy simultaneously. Multiple numerical simulations are performed to show the effectiveness of the proposed approach. Moreover, the strategy is successfully implemented on our hydro blasting robot named Hornbill, and the efficacy of the proposed approach is validated through experimental trials.

Reinforcement Learning-Based Complete Area Coverage Path Planning for a Modified hTrihex Robot.

One of the essential attributes of a cleaning robot is to achieve complete area coverage. Current commercial indoor cleaning robots have fixed morphology and are restricted to clean only specific areas in a house. The results of maximum area coverage are sub-optimal in this case. Tiling robots are innovative solutions for such a coverage problem. These new kinds of robots can be deployed in the cases of cleaning, painting, maintenance, and inspection, which require complete area coverage. Tiling robots' objective is to cover the entire area by reconfiguring to different shapes as per the area requirements. In this context, it is vital to have a framework that enables the robot to maximize the area coverage while minimizing energy consumption. That means it is necessary for the robot to cover the maximum area with the least number of shape reconfigurations possible. The current paper proposes a complete area coverage planning module for the modified hTrihex, a honeycomb-shaped tiling robot, based on the deep reinforcement learning technique. This framework simultaneously generates the tiling shapes and the trajectory with minimum overall cost. In this regard, a convolutional neural network (CNN) with long short term memory (LSTM) layer was trained using the actor-critic experience replay (ACER) reinforcement learning algorithm. The simulation results obtained from the current implementation were compared against the results that were generated through traditional tiling theory models that included zigzag, spiral, and greedy search schemes. The model presented in the current paper was also compared against other methods where this problem was considered as a traveling salesman problem (TSP) solved through genetic algorithm (GA) and ant colony optimization (ACO) approaches. Our proposed scheme generates a path with a minimized cost at a lesser time.

Wall-Following Behavior for a Disinfection Robot Using Type 1 and Type 2 Fuzzy Logic Systems.

Infectious diseases are caused by pathogenic microorganisms, whose transmission can lead to global pandemics like COVID-19. Contact with contaminated surfaces or objects is one of the major channels of spreading infectious diseases among the community. Therefore, the typical contaminable surfaces, such as walls and handrails, should often be cleaned using disinfectants. Nevertheless, safety and efficiency are the major concerns of the utilization of human labor in this process. Thereby, attention has drifted toward developing robotic solutions for the disinfection of contaminable surfaces. A robot intended for disinfecting walls should be capable of following the wall concerned, while maintaining a given distance, to be effective. The ability to operate in an unknown environment while coping with uncertainties is crucial for a wall disinfection robot intended for deployment in public spaces. Therefore, this paper contributes to the state-of-the-art by proposing a novel method of establishing the wall-following behavior for a wall disinfection robot using fuzzy logic. A non-singleton Type 1 Fuzzy Logic System (T1-FLS) and a non-singleton Interval Type 2 Fuzzy Logic System (IT2-FLS) are developed in this regard. The wall-following behavior of the two fuzzy systems was evaluated through simulations by considering heterogeneous wall arrangements. The simulation results validate the real-world applicability of the proposed FLSs for establishing the wall-following behavior for a wall disinfection robot. Furthermore, the statistical outcomes show that the IT2-FLS has significantly superior performance than the T1-FLS in this application.