Deep-Learning-Based Context-Aware Multi-Level Information Fusion Systems for Indoor Mobile Robots Safe Navigation.

Hazardous object detection (escalators, stairs, glass doors, etc.) and avoidance are critical functional safety modules for autonomous mobile cleaning robots. Conventional object detectors have less accuracy for detecting low-feature hazardous objects and have miss detection, and the false classification ratio is high when the object is under occlusion. Miss detection or false classification of hazardous objects poses an operational safety issue for mobile robots. This work presents a deep-learning-based context-aware multi-level information fusion framework for autonomous mobile cleaning robots to detect and avoid hazardous objects with a higher confidence level, even if the object is under occlusion. First, the image-level-contextual-encoding module was proposed and incorporated with the Faster RCNN ResNet 50 object detector model to improve the low-featured and occluded hazardous object detection in an indoor environment. Further, a safe-distance-estimation function was proposed to avoid hazardous objects. It computes the distance of the hazardous object from the robot's position and steers the robot into a safer zone using detection results and object depth data. The proposed framework was trained with a custom image dataset using fine-tuning techniques and tested in real-time with an in-house-developed mobile cleaning robot, BELUGA. The experimental results show that the proposed algorithm detected the low-featured and occluded hazardous object with a higher confidence level than the conventional object detector and scored an average detection accuracy of 88.71%.

Optimal selective floor cleaning using deep learning algorithms and reconfigurable robot hTetro.

Floor cleaning robots are widely used in public places like food courts, hospitals, and malls to perform frequent cleaning tasks. However, frequent cleaning tasks adversely impact the robot's performance and utilize more cleaning accessories (such as brush, scrubber, and mopping pad). This work proposes a novel selective area cleaning/spot cleaning framework for indoor floor cleaning robots using RGB-D vision sensor-based Closed Circuit Television (CCTV) network, deep learning algorithms, and an optimal complete waypoints path planning method. In this scheme, the robot will clean only dirty areas instead of the whole region. The selective area cleaning/spot cleaning region is identified based on the combination of two strategies: tracing the human traffic patterns and detecting stains and trash on the floor. Here, a deep Simple Online and Real-time Tracking (SORT) human tracking algorithm was used to trace the high human traffic region and Single Shot Detector (SSD) MobileNet object detection framework for detecting the dirty region. Further, optimal shortest waypoint coverage path planning using evolutionary-based optimization was incorporated to traverse the robot efficiently to the designated selective area cleaning/spot cleaning regions. The experimental results show that the SSD MobileNet algorithm scored 90% accuracy for stain and trash detection on the floor. Further, compared to conventional methods, the evolutionary-based optimization path planning scheme reduces 15% percent of navigation time and 10% percent of energy consumption.

Object-of-Interest Perception in a Reconfigurable Rolling-Crawling Robot.

Cebrenus Rechenburgi, a member of the huntsman spider family have inspired researchers to adopt different locomotion modes in reconfigurable robotic development. Object-of-interest perception is crucial for such a robot to provide fundamental information on the traversed pathways and guide its locomotion mode transformation. Therefore, we present a object-of-interest perception in a reconfigurable rolling-crawling robot and identifying appropriate locomotion modes. We demonstrate it in Scorpio, our in-house developed robot with two locomotion modes: rolling and crawling. We train the locomotion mode recognition framework, named Pyramid Scene Parsing Network (PSPNet), with a self-collected dataset composed of two categories paths, unobstructed paths (e.g., floor) for rolling and obstructed paths (e.g., with person, railing, stairs, static objects and wall) for crawling, respectively. The efficiency of the proposed framework has been validated with evaluation metrics in offline and real-time field trial tests. The experiment results show that the trained model can achieve an mIOU score of 72.28 and 70.63 in offline and online testing, respectively for both environments. The proposed framework's performance is compared with semantic framework (HRNet and Deeplabv3) where the proposed framework outperforms in terms of mIOU and speed. Furthermore, the experimental results has revealed that the robot's maneuverability is stable, and the proposed framework can successfully determine the appropriate locomotion modes with enhanced accuracy during complex pathways.

Patient-Specific Finite Element Modeling of the Whole Lumbar Spine Using Clinical Routine Multi-Detector Computed Tomography (MDCT) Data-A Pilot Study.

(1) Background: To study the feasibility of developing finite element (FE) models of the whole lumbar spine using clinical routine multi-detector computed tomography (MDCT) scans to predict failure load (FL) and range of motion (ROM) parameters. (2) Methods: MDCT scans of 12 subjects (6 healthy controls (HC), mean age ± standard deviation (SD): 62.16 ± 10.24 years, and 6 osteoporotic patients (OP), mean age ± SD: 65.83 ± 11.19 years) were included in the current study. Comprehensive FE models of the lumbar spine (5 vertebrae + 4 intervertebral discs (IVDs) + ligaments) were generated (L1−L5) and simulated. The coefficients of correlation (ρ) were calculated to investigate the relationship between FE-based FL and ROM parameters and bone mineral density (BMD) values of L1−L3 derived from MDCT (BMDQCT-L1-3). Finally, Mann−Whitney U tests were performed to analyze differences in FL and ROM parameters between HC and OP cohorts. (3) Results: Mean FE-based FL value of the HC cohort was significantly higher than that of the OP cohort (1471.50 ± 275.69 N (HC) vs. 763.33 ± 166.70 N (OP), p < 0.01). A strong correlation of 0.8 (p < 0.01) was observed between FE-based FL and BMDQCT-L1-L3 values. However, no significant differences were observed between ROM parameters of HC and OP cohorts (p = 0.69 for flexion; p = 0.69 for extension; p = 0.47 for lateral bending; p = 0.13 for twisting). In addition, no statistically significant correlations were observed between ROM parameters and BMDQCT- L1-3. (4) Conclusions: Clinical routine MDCT data can be used for patient-specific FE modeling of the whole lumbar spine. ROM parameters do not seem to be significantly altered between HC and OP. In contrast, FE-derived FL may help identify patients with increased osteoporotic fracture risk in the future.

AI-Enabled Mosquito Surveillance and Population Mapping Using Dragonfly Robot.

Mosquito-borne diseases can pose serious risks to human health. Therefore, mosquito surveillance and control programs are essential for the wellbeing of the community. Further, human-assisted mosquito surveillance and population mapping methods are time-consuming, labor-intensive, and require skilled manpower. This work presents an AI-enabled mosquito surveillance and population mapping framework using our in-house-developed robot, named 'Dragonfly', which uses the You Only Look Once (YOLO) V4 Deep Neural Network (DNN) algorithm and a two-dimensional (2D) environment map generated by the robot. The Dragonfly robot was designed with a differential drive mechanism and a mosquito trapping module to attract mosquitoes in the environment. The YOLO V4 was trained with three mosquito classes, namely Aedes aegypti, Aedes albopictus, and Culex, to detect and classify the mosquito breeds from the mosquito glue trap. The efficiency of the mosquito surveillance framework was determined in terms of mosquito classification accuracy and detection confidence level on offline and real-time field tests in a garden, drain perimeter area, and covered car parking area. The experimental results show that the trained YOLO V4 DNN model detects and classifies the mosquito classes with an 88% confidence level on offline mosquito test image datasets and scores an average of an 82% confidence level on the real-time field trial. Further, to generate the mosquito population map, the detection results are fused in the robot's 2D map, which will help to understand mosquito population dynamics and species distribution.

Remote drain inspection framework using the convolutional neural network and re-configurable robot Raptor.

Drain blockage is a crucial problem in the urban environment. It heavily affects the ecosystem and human health. Hence, routine drain inspection is essential for urban environment. Manual drain inspection is a tedious task and prone to accidents and water-borne diseases. This work presents a drain inspection framework using convolutional neural network (CNN) based object detection algorithm and in house developed reconfigurable teleoperated robot called 'Raptor'. The CNN based object detection model was trained using a transfer learning scheme with our custom drain-blocking objects data-set. The efficiency of the trained CNN algorithm and drain inspection robot Raptor was evaluated through various real-time drain inspection field trial. The experimental results indicate that our trained object detection algorithm has detect and classified the drain blocking objects with 91.42% accuracy for both offline and online test images and is able to process 18 frames per second (FPS). Further, the maneuverability of the robot was evaluated from various open and closed drain environment. The field trial results ensure that the robot maneuverability was stable, and its mapping and localization is also accurate in a complex drain environment.

Drain Structural Defect Detection and Mapping Using AI-Enabled Reconfigurable Robot Raptor and IoRT Framework.

Human visual inspection of drains is laborious, time-consuming, and prone to accidents. This work presents an AI-enabled robot-assisted remote drain inspection and mapping framework using our in-house developed reconfigurable robot Raptor. The four-layer IoRT serves as a bridge between the users and the robots, through which seamless information sharing takes place. The Faster RCNN ResNet50, Faster RCNN ResNet101, and Faster RCNN Inception-ResNet-v2 deep learning frameworks were trained using a transfer learning scheme with six typical concrete defect classes and deployed in an IoRT framework remote defect detection task. The efficiency of the trained CNN algorithm and drain inspection robot Raptor was evaluated through various real-time drain inspection field trials using the SLAM technique. The experimental results indicate that robot's maneuverability was stable, and its mapping and localization were also accurate in different drain types. Finally, for effective drain maintenance, the SLAM-based defect map was generated by fusing defect detection results in the lidar-SLAM map.

AI Enabled IoRT Framework for Rodent Activity Monitoring in a False Ceiling Environment.

Routine rodent inspection is essential to curbing rat-borne diseases and infrastructure damages within the built environment. Rodents find false ceilings to be a perfect spot to seek shelter and construct their habitats. However, a manual false ceiling inspection for rodents is laborious and risky. This work presents an AI-enabled IoRT framework for rodent activity monitoring inside a false ceiling using an in-house developed robot called "Falcon". The IoRT serves as a bridge between the users and the robots, through which seamless information sharing takes place. The shared images by the robots are inspected through a Faster RCNN ResNet 101 object detection algorithm, which is used to automatically detect the signs of rodent inside a false ceiling. The efficiency of the rodent activity detection algorithm was tested in a real-world false ceiling environment, and detection accuracy was evaluated with the standard performance metrics. The experimental results indicate that the algorithm detects rodent signs and 3D-printed rodents with a good confidence level.

False Ceiling Deterioration Detection and Mapping Using a Deep Learning Framework and the Teleoperated Reconfigurable 'Falcon' Robot.

Periodic inspection of false ceilings is mandatory to ensure building and human safety. Generally, false ceiling inspection includes identifying structural defects, degradation in Heating, Ventilation, and Air Conditioning (HVAC) systems, electrical wire damage, and pest infestation. Human-assisted false ceiling inspection is a laborious and risky task. This work presents a false ceiling deterioration detection and mapping framework using a deep-neural-network-based object detection algorithm and the teleoperated 'Falcon' robot. The object detection algorithm was trained with our custom false ceiling deterioration image dataset composed of four classes: structural defects (spalling, cracks, pitted surfaces, and water damage), degradation in HVAC systems (corrosion, molding, and pipe damage), electrical damage (frayed wires), and infestation (termites and rodents). The efficiency of the trained CNN algorithm and deterioration mapping was evaluated through various experiments and real-time field trials. The experimental results indicate that the deterioration detection and mapping results were accurate in a real false-ceiling environment and achieved an 89.53% detection accuracy.

Remote Insects Trap Monitoring System Using Deep Learning Framework and IoT.

Insect detection and control at an early stage are essential to the built environment (human-made physical spaces such as homes, hotels, camps, hospitals, parks, pavement, food industries, etc.) and agriculture fields. Currently, such insect control measures are manual, tedious, unsafe, and time-consuming labor dependent tasks. With the recent advancements in Artificial Intelligence (AI) and the Internet of things (IoT), several maintenance tasks can be automated, which significantly improves productivity and safety. This work proposes a real-time remote insect trap monitoring system and insect detection method using IoT and Deep Learning (DL) frameworks. The remote trap monitoring system framework is constructed using IoT and the Faster RCNN (Region-based Convolutional Neural Networks) Residual neural Networks 50 (ResNet50) unified object detection framework. The Faster RCNN ResNet 50 object detection framework was trained with built environment insects and farm field insect images and deployed in IoT. The proposed system was tested in real-time using four-layer IoT with built environment insects image captured through sticky trap sheets. Further, farm field insects were tested through a separate insect image database. The experimental results proved that the proposed system could automatically identify the built environment insects and farm field insects with an average of 94% accuracy.

Table Cleaning Task by Human Support Robot Using Deep Learning Technique.

This work presents a table cleaning and inspection method using a Human Support Robot (HSR) which can operate in a typical food court setting. The HSR is able to perform a cleanliness inspection and also clean the food litter on the table by implementing a deep learning technique and planner framework. A lightweight Deep Convolutional Neural Network (DCNN) has been proposed to recognize the food litter on top of the table. In addition, the planner framework was proposed to HSR for accomplishing the table cleaning task which generates the cleaning path according to the detection of food litter and then the cleaning action is carried out. The effectiveness of the food litter detection module is verified with the cleanliness inspection task using Toyota HSR, and its detection results are verified with standard quality metrics. The experimental results show that the food litter detection module achieves an average of 96 % detection accuracy, which is more suitable for deploying the HSR robots for performing the cleanliness inspection and also helps to select the different cleaning modes. Further, the planner part has been tested through the table cleaning tasks. The experimental results show that the planner generated the cleaning path in real time and its generated path is optimal which reduces the cleaning time by grouping based cleaning action for removing the food litters from the table.

Evolutionary Algorithm-Based Complete Coverage Path Planning for Tetriamond Tiling Robots.

Tiling robots with fixed morphology face major challenges in terms of covering the cleaning area and generating the optimal trajectory during navigation. Developing a self-reconfigurable autonomous robot is a probable solution to these issues, as it adapts various forms and accesses narrow spaces during navigation. The total navigation energy includes the energy expenditure during locomotion and the shape-shifting of the platform. Thus, during motion planning, the optimal navigation sequence of a self-reconfigurable robot must include the components of the navigation energy and the area coverage. This paper addresses the framework to generate an optimal navigation path for reconfigurable cleaning robots made of tetriamonds. During formulation, the cleaning environment is filled with various tiling patterns of the tetriamond-based robot, and each tiling pattern is addressed by a waypoint. The objective is to minimize the amount of shape-shifting needed to fill the workspace. The energy cost function is formulated based on the travel distance between waypoints, which considers the platform locomotion inside the workspace. The objective function is optimized based on evolutionary algorithms such as the genetic algorithm (GA) and ant colony optimization (ACO) of the traveling salesman problem (TSP) and estimates the shortest path that connects all waypoints. The proposed path planning technique can be extended to other polyamond-based reconfigurable robots.

A System-of-Systems Bio-Inspired Design Process: Conceptual Design and Physical Prototype of a Reconfigurable Robot Capable of Multi-Modal Locomotion.

Modern engineering problems require solutions with multiple functionalities in order to meet their practical needs to handle a variety of applications in different scenarios. Conventional design paradigms for single design purpose may not be able to satisfy this requirement efficiently. This paper proposes a novel system-of-systems bio-inspired design method framed in a solution-driven bio-inspired design paradigm. The whole design process consists of eight steps, that is, (1) biological solutions identification, (2) biological solutions definition/champion biological solutions, (3) principle extraction from each champion biological solution, (4) merging of extracted principles, (5) solution reframing, (6) problem search, (7) problem definition, and (8) principles application & implementation. The steps are elaborated and a case study of reconfigurable robots is presented following these eight steps. The design originates from the multimodal locomotion capabilities of two species (i.e., spiders and primates) and is analyzed based on the Pugh analysis. The resulting robotic platform could be potentially used for urban patrolling purposes.

Methods for evaluating effects of unloader knee braces on joint health: a review.

The paper aims to provide a state-of-the-art review of methods for evaluating the effectiveness and effect of unloader knee braces on the knee joint and discuss their limitations and future directions. Unloader braces are prescribed as a non-pharmacological conservative treatment option for patients with medial knee osteoarthritis to provide relief in terms of pain reduction, returning to regular physical activities, and enhancing the quality of life. Methods used to evaluate and monitor the effectiveness of these devices on patients' health are categorized into three broad categories (perception-, biochemical-, and morphology-based), depending upon the process and tools used. The main focus of these methods is on the short-term clinical outcome (pain or unloading efficiency). There is a significant technical, research, and clinical literature gap in understanding the short- and long-term consequences of these braces on the tissues in the knee joint, including the cartilage and ligaments. Future research directions may complement existing methods with advanced quantitative imaging (morphological, biochemical, and molecular) and numerical simulation are discussed as they offer potential in assessing long-term and post-bracing effects on the knee joint.

Inspiration From Games and Entertainment Artifacts: A Rising Paradigm for Designing Mechanisms and Algorithms in Robotics.

Games and toys have been serving as entertainment tools to humans for a long period of time. While except for entertainment, they can also trigger inspiration and enhance productivity in many other domains such as healthcare and general workplaces. The concept of the game is referred to a series of structured procedures (e.g., card games) and virtual programs. The entertainment artifacts could be a toy or even a handicraft, such as origami and kirigami, for entertainment purposes in a broader sense. Recently, the design of robots and relevant applications in robotics has been emerging in taking inspiration from Games and Entertainment Artifacts (GEA). However, there is a lack of systematic and general process for implementing a GEA-inspired design for developing robot-related applications. In this article, we put forward a design paradigm based on the inspiration of game and entertainment artifacts which is a systematic design approach. The design paradigm could follow two different processes which are driven by problems and solutions, respectively, using analogies of games and entertainment artifacts to build robotic solutions for solving real problems. The problem-driven process starts with an existing real-world problem, which follows the sequences of robotics problem search, robotics problem identification, GEA solution search, GEA solution identification, GEA principle extraction, and the principle implementation. Reversely, the solution-driven process follows the sequence of GEA solution search, GEA solution identification, GEA principle extraction, robotics problem search, robotics problem identification, and principle implementation. We demonstrate the application of the design paradigm using the case study of a new type of reconfigurable floor cleaning robot and its path planning algorithm.

Modified A-Star Algorithm for Efficient Coverage Path Planning in Tetris Inspired Self-Reconfigurable Robot with Integrated Laser Sensor.

Advancing an efficient coverage path planning in robots set up for application such as cleaning, painting and mining are becoming more crucial. Such drive in the coverage path planning field proposes numerous techniques over the past few decades. However, the proposed approaches were only applied and tested with a fixed morphological robot in which the coverage performance was significantly degraded in a complex environment. To this end, an A-star based zigzag global planner for a novel self-reconfigurable Tetris inspired cleaning robot (hTetro) presented in this paper. Unlike the traditional A-star algorithm, the presented approach can generate waypoints in order to cover the narrow spaces while assuming appropriate morphology of the hTtero robot with the objective of maximizing the coverage area. We validated the efficiency of the proposed planning approach in the Robot Operation System (ROS) Based simulated environment and tested with the hTetro robot in real-time under the controlled scenarios. Our experiments demonstrate the efficiency of the proposed coverage path planning approach resulting in superior area coverage performance in all considered experimental scenarios.