RESUMO
The identification of viruses from negative staining transmission electron microscopy (TEM) images has mainly depended on experienced experts. Recent advances in artificial intelligence have enabled virus recognition using deep learning techniques. However, most of the existing methods only perform virus classification or semantic segmentation, and few studies have addressed the challenge of virus instance segmentation in TEM images. In this paper, we focus on the instance segmentation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) and other respiratory viruses and provide experts with more effective information about viruses. We propose an effective virus instance segmentation network based on the You Only Look At CoefficienTs backbone, which integrates the Swin Transformer, dense connections and the coordinate-spatial attention mechanism, to identify SARS-CoV-2, H1N1 influenza virus, respiratory syncytial virus, Herpes simplex virus-1, Human adenovirus type 5 and Vaccinia virus. We also provide a public TEM virus dataset and conduct extensive comparative experiments. Our method achieves a mean average precision score of 83.8 and F1 score of 0.920, outperforming other state-of-the-art instance segmentation algorithms. The proposed automated method provides virologists with an effective approach for recognizing and identifying SARS-CoV-2 and assisting in the diagnosis of viruses. Our dataset and code are accessible at https://github.com/xiaochiHNU/Virus-Instance-Segmentation-Transformer-Network.
Assuntos
Vírus da Influenza A Subtipo H1N1 , Vírus da Influenza A , Humanos , Inteligência Artificial , Algoritmos , SARS-CoV-2RESUMO
Within the current process of large-scale dairy-cattle breeding, to address the problems of low recognition-accuracy and significant recognition-error associated with existing visual methods, we propose a method for recognizing the feeding behavior of dairy cows, one based on an improved RefineMask instance-segmentation model, and using high-quality detection and segmentation results to realize the recognition of the feeding behavior of dairy cows. Firstly, the input features are better extracted by incorporating the convolutional block attention module into the residual module of the feature extraction network. Secondly, an efficient channel attention module is incorporated into the neck design to achieve efficient integration of feature extraction while avoiding the surge of parameter volume computation. Subsequently, the GIoU loss function is used to increase the area of the prediction frame to optimize the convergence speed of the loss function, thus improving the regression accuracy. Finally, the logic of using mask information to recognize foraging behavior was designed, and the accurate recognition of foraging behavior was achieved according to the segmentation results of the model. We constructed, trained, and tested a cow dataset consisting of 1000 images from 50 different individual cows at peak feeding times. The method's effectiveness, robustness, and accuracy were verified by comparing it with example segmentation algorithms such as MSRCNN, Point_Rend, Cascade_Mask, and ConvNet_V2. The experimental results show that the accuracy of the improved RefineMask algorithm in recognizing the bounding box and accurately determining the segmentation mask is 98.3%, which is higher than that of the benchmark model by 0.7 percentage points; for this, the model parameter count size was 49.96 M, which meets the practical needs of local deployment. In addition, the technologies under study performed well in a variety of scenarios and adapted to various light environments; this research can provide technical support for the analysis of the relationship between cow feeding behavior and feed intake during peak feeding periods.
Assuntos
Algoritmos , Comportamento Alimentar , Bovinos , Animais , Comportamento Alimentar/fisiologia , Feminino , Redes Neurais de Computação , Indústria de Laticínios/métodosRESUMO
A novel approach for video instance segmentation is presented using semisupervised learning. Our Cluster2Former model leverages scribble-based annotations for training, significantly reducing the need for comprehensive pixel-level masks. We augment a video instance segmenter, for example, the Mask2Former architecture, with similarity-based constraint loss to handle partial annotations efficiently. We demonstrate that despite using lightweight annotations (using only 0.5% of the annotated pixels), Cluster2Former achieves competitive performance on standard benchmarks. The approach offers a cost-effective and computationally efficient solution for video instance segmentation, especially in scenarios with limited annotation resources.
RESUMO
Mechanical weed management is a drudging task that requires manpower and has risks when conducted within rows of orchards. However, intrarow weeding must still be conducted by manual labor due to the restricted movements of riding mowers within the rows of orchards due to their confined structures with nets and poles. However, autonomous robotic weeders still face challenges identifying uncut weeds due to the obstruction of Global Navigation Satellite System (GNSS) signals caused by poles and tree canopies. A properly designed intelligent vision system would have the potential to achieve the desired outcome by utilizing an autonomous weeder to perform operations in uncut sections. Therefore, the objective of this study is to develop a vision module using a custom-trained dataset on YOLO instance segmentation algorithms to support autonomous robotic weeders in recognizing uncut weeds and obstacles (i.e., fruit tree trunks, fixed poles) within rows. The training dataset was acquired from a pear orchard located at the Tsukuba Plant Innovation Research Center (T-PIRC) at the University of Tsukuba, Japan. In total, 5000 images were preprocessed and labeled for training and testing using YOLO models. Four versions of edge-device-dedicated YOLO instance segmentation were utilized in this research-YOLOv5n-seg, YOLOv5s-seg, YOLOv8n-seg, and YOLOv8s-seg-for real-time application with an autonomous weeder. A comparison study was conducted to evaluate all YOLO models in terms of detection accuracy, model complexity, and inference speed. The smaller YOLOv5-based and YOLOv8-based models were found to be more efficient than the larger models, and YOLOv8n-seg was selected as the vision module for the autonomous weeder. In the evaluation process, YOLOv8n-seg had better segmentation accuracy than YOLOv5n-seg, while the latter had the fastest inference time. The performance of YOLOv8n-seg was also acceptable when it was deployed on a resource-constrained device that is appropriate for robotic weeders. The results indicated that the proposed deep learning-based detection accuracy and inference speed can be used for object recognition via edge devices for robotic operation during intrarow weeding operations in orchards.
Assuntos
Algoritmos , Cultura , Frutas , Inteligência , Japão , Plantas DaninhasRESUMO
Crack detection plays a critical role in ensuring road safety and maintenance. Traditional, manual, and semi-automatic detection methods have proven inefficient. Nowadays, the emergence of deep learning techniques has opened up new possibilities for automatic crack detection. However, there are few methods with both localization and segmentation abilities, and most perform poorly. The consistent nature of pavement over a small mileage range gives us the opportunity to make improvements. A novel data-augmentation strategy called CrackMover, specifically tailored for crack detection methods, is proposed. Experiments demonstrate the effectiveness of CrackMover for various methods. Moreover, this paper presents a new instance segmentation method for crack detection. It adopts a redesigned backbone network and incorporates a cascade structure for the region-based convolutional network (R-CNN) part. The experimental evaluation showcases significant performance improvements achieved by these approaches in crack detection. The proposed method achieves an average precision of 33.3%, surpassing Mask R-CNN with a Residual Network 50 backbone by 8.6%, proving its effectiveness in detecting crack distress.
RESUMO
Real-world understanding serves as a medium that bridges the information world and the physical world, enabling the realization of virtual-real mapping and interaction. However, scene understanding based solely on 2D images faces problems such as a lack of geometric information and limited robustness against occlusion. The depth sensor brings new opportunities, but there are still challenges in fusing depth with geometric and semantic priors. To address these concerns, our method considers the repeatability of video stream data and the sparsity of newly generated data. We introduce a sparsely correlated network architecture (SCN) designed explicitly for online RGBD instance segmentation. Additionally, we leverage the power of object-level RGB-D SLAM systems, thereby transcending the limitations of conventional approaches that solely emphasize geometry or semantics. We establish correlation over time and leverage this correlation to develop rules and generate sparse data. We thoroughly evaluate the system's performance on the NYU Depth V2 and ScanNet V2 datasets, demonstrating that incorporating frame-to-frame correlation leads to significantly improved accuracy and consistency in instance segmentation compared to existing state-of-the-art alternatives. Moreover, using sparse data reduces data complexity while ensuring the real-time requirement of 18 fps. Furthermore, by utilizing prior knowledge of object layout understanding, we showcase a promising application of augmented reality, showcasing its potential and practicality.
RESUMO
Lineament is a unique geological structure. The study of Lunar lineament structure has great significance on understanding its history and evolution of Lunar surface. However, the existing geographic feature extraction methods are not suitable for the extraction of Lunar lineament structure. In this paper, a new lineament extraction method is proposed based on improved-UNet++ and YOLOv5. Firstly, new lineament dataset is created containing lineaments structure based on CCD data from LROC. At same time the residual blocks are replaced with the VGG blocks in the down sample part of the UNet++ with adding the attention block between each layer. Secondly, the improved-UNet++ and YOLO networks are trained to execute the object detection and semantic segmentation of lineament structure respectively. Finally, a polygon-match strategy is proposed to combine the results of object detection and semantic segmentation. The experiment result indicate that this new method has relatively better and more stable performance compared with current mainstream networks and the original UNet++ network in the instance segmentation of lineament structure. Additionally, the polygon-match strategy is able to perform preciser edge detail in the instance segmentation of lineament structure result.
RESUMO
To address the challenges of reduced localization accuracy and incomplete map construction demonstrated using classical semantic simultaneous localization and mapping (SLAM) algorithms in dynamic environments, this study introduces a dynamic scene SLAM technique that builds upon direct sparse odometry (DSO) and incorporates instance segmentation and video completion algorithms. While prioritizing the algorithm's real-time performance, we leverage the rapid matching capabilities of Direct Sparse Odometry (DSO) to link identical dynamic objects in consecutive frames. This association is achieved through merging semantic and geometric data, thereby enhancing the matching accuracy during image tracking through the inclusion of semantic probability. Furthermore, we incorporate a loop closure module based on video inpainting algorithms into our mapping thread. This allows our algorithm to rely on the completed static background for loop closure detection, further enhancing the localization accuracy of our algorithm. The efficacy of this approach is validated using the TUM and KITTI public datasets and the unmanned platform experiment. Experimental results show that, in various dynamic scenes, our method achieves an improvement exceeding 85% in terms of localization accuracy compared with the DSO system.
RESUMO
Quickly and accurately assessing the damage level of buildings is a challenging task for post-disaster emergency response. Most of the existing research mainly adopts semantic segmentation and object detection methods, which have yielded good results. However, for high-resolution Unmanned Aerial Vehicle (UAV) imagery, these methods may result in the problem of various damage categories within a building and fail to accurately extract building edges, thus hindering post-disaster rescue and fine-grained assessment. To address this issue, we proposed an improved instance segmentation model that enhances classification accuracy by incorporating a Mixed Local Channel Attention (MLCA) mechanism in the backbone and improving small object segmentation accuracy by refining the Neck part. The method was tested on the Yangbi earthquake UVA images. The experimental results indicated that the modified model outperformed the original model by 1.07% and 1.11% in the two mean Average Precision (mAP) evaluation metrics, mAPbbox50 and mAPseg50, respectively. Importantly, the classification accuracy of the intact category was improved by 2.73% and 2.73%, respectively, while the collapse category saw an improvement of 2.58% and 2.14%. In addition, the proposed method was also compared with state-of-the-art instance segmentation models, e.g., Mask-R-CNN and YOLO V9-Seg. The results demonstrated that the proposed model exhibits advantages in both accuracy and efficiency. Specifically, the efficiency of the proposed model is three times faster than other models with similar accuracy. The proposed method can provide a valuable solution for fine-grained building damage evaluation.
RESUMO
The calculation of land surface temperatures (LSTs) via low-altitude thermal infrared remote (TIR) sensing images at a block scale is gaining attention. However, the accurate calculation of LSTs requires a precise determination of the range of various underlying surfaces in the TIR images, and existing approaches face challenges in effectively segmenting the underlying surfaces in the TIR images. To address this challenge, this study proposes a deep learning (DL) methodology to complete the instance segmentation and quantification of underlying surfaces through the low-altitude TIR image dataset. Mask region-based convolutional neural networks were utilized for pixel-level classification and segmentation with an image dataset of 1350 annotated TIR images of an urban rail transit hub with a complex distribution of underlying surfaces. Subsequently, the hyper-parameters and architecture were optimized for the precise classification of the underlying surfaces. The algorithms were validated using 150 new TIR images, and four evaluation indictors demonstrated that the optimized algorithm outperformed the other algorithms. High-quality segmented masks of the underlying surfaces were generated, and the area of each instance was obtained by counting the true-positive pixels with values of 1. This research promotes the accurate calculation of LSTs based on the low-altitude TIR sensing images.
RESUMO
Considering the complex structure of Chinese characters, particularly the connections and intersections between strokes, there are challenges in low accuracy of Chinese character stroke extraction and recognition, as well as unclear segmentation. This study builds upon the YOLOv8n-seg model to propose the YOLOv8n-seg-CAA-BiFPN Chinese character stroke fine segmentation model. The proposed Coordinate-Aware Attention mechanism (CAA) divides the backbone network input feature map into four parts, applying different weights for horizontal, vertical, and channel attention to compute and fuse key information, thus capturing the contextual regularity of closely arranged stroke positions. The network's neck integrates an enhanced weighted bi-directional feature pyramid network (BiFPN), enhancing the fusion effect for features of strokes of various sizes. The Shape-IoU loss function is adopted in place of the traditional CIoU loss function, focusing on the shape and scale of stroke bounding boxes to optimize the bounding box regression process. Finally, the Grad-CAM++ technique is used to generate heatmaps of segmentation predictions, facilitating the visualization of effective features and a deeper understanding of the model's focus areas. Trained and tested on the public Chinese character stroke datasets CCSE-Kai and CCSE-HW, the model achieves an average accuracy of 84.71%, an average recall rate of 83.65%, and a mean average precision of 80.11%. Compared to the original YOLOv8n-seg and existing mainstream segmentation models like SegFormer, BiSeNetV2, and Mask R-CNN, the average accuracy improved by 3.50%, 4.35%, 10.56%, and 22.05%, respectively; the average recall rates improved by 4.42%, 9.32%, 15.64%, and 24.92%, respectively; and the mean average precision improved by 3.11%, 4.15%, 8.02%, and 19.33%, respectively. The results demonstrate that the YOLOv8n-seg-CAA-BiFPN network can accurately achieve Chinese character stroke segmentation.
RESUMO
This paper addresses the critical environmental issue of effectively managing construction and demolition waste (CDW), which has seen a global surge due to rapid urbanization. With the advent of deep learning-based computer vision, this study focuses on improving intelligent identification of valuable recyclables from cluttered and heterogeneous CDW streams in material recovery facilities (MRFs) by optimally leveraging both visual and spatial features (depth). A high-quality CDW RGB-D dataset was curated to capture MRF stream complexities often overlooked in prior studies, and comprises over 3500 images for each modality and more than 160,000 dense object instances of diverse CDW materials with high resource value. In contrast to former studies which directly concatenate RGB and depth features, this study introduces a new depth fusion strategy that utilizes computationally efficient convolutional operations at the end of the conventional waste segmentation architecture to effectively fuse colour and depth information. This avoids cross-modal interference and maximizes the use of distinct information present in the two different modalities. Despite the high clutter and diversity of waste objects, the proposed RGB-DL architecture achieves a 13% increase in segmentation accuracy and a 36% reduction in inference time when compared to the direct concatenation of features. The findings of this study emphasize the benefit of effectively incorporating geometrical features to complement visual cues. This approach helps to deal with the cluttered and varied nature of CDW streams, enhancing automated waste recognition accuracy to improve resource recovery in MRFs. This, in turn, promotes intelligent solid waste management for efficiently managing environmental concerns.
Assuntos
Indústria da Construção , Gerenciamento de Resíduos , Indústria da Construção/métodos , Materiais de Construção , Reciclagem/métodos , Gerenciamento de Resíduos/métodos , Resíduos Sólidos/análise , Resíduos Industriais/análiseRESUMO
OBJECTIVE: To propose a novel neural network to achieve tooth instance segmentation and recognition based on cone-beam computed tomography (CBCT) voxel data. METHODS: The proposed methods included three different convolutional neural network models. The architecture was based on the Resnet module and built according to the structure of "Encoder-Decoder" and U-Net. The CBCT image was de-sampled and a fixed-size region of interest (ROI) containing all the teeth was determined. ROI would first through a two-branch "encoder and decoder" structure of the network, the network could predict each voxel in the input data of the spatial embedding. The post-processing algorithm would cluster the prediction results of the relevant spatial location information according to the two-branch network to realize the tooth instance segmentation. The tooth position identification was realized by another U-Net model based on the multi-classification segmentation task. According to the predicted results of the network, the post-processing algorithm would classify the tooth position according to the voting results of each tooth instance segmentation. At the original spatial resolution, a U-Net network model for the fine-tooth segmentation was trained using the region corresponding to each tooth as the input. According to the results of instance segmentation and tooth position identification, the model would process the correspon-ding positions on the high-resolution CBCT images to obtain the high-resolution tooth segmentation results. In this study, CBCT data of 59 cases with simple crown prostheses and implants were collected for manual labeling as the database, and statistical indicators were evaluated for the prediction results of the algorithm. To assess the performance of tooth segmentation and classification, instance Dice similarity coefficient (IDSC) and the average Dice similarity coefficient (ADSC) were calculated. RESULTS: The experimental results showed that the IDSC was 89.35%, and the ADSC was 84. 74%. After eliminating the data with prostheses artifacts, the database of 43 samples was generated, and the performance of the training network was better, with 90.34% for IDSC and 87.88% for ADSC. The framework achieved excellent performance on tooth segmentation and identification. Voxels near intercuspation surfaces and fuzzy boundaries could be separated into correct instances by this framework. CONCLUSIONS: The results show that this method can not only successfully achieve 3D tooth instance segmentation but also identify all teeth notation numbers accurately, which has clinical practicability.
Assuntos
Algoritmos , Tomografia Computadorizada de Feixe Cônico , Redes Neurais de Computação , Dente , Tomografia Computadorizada de Feixe Cônico/métodos , Humanos , Dente/diagnóstico por imagem , Processamento de Imagem Assistida por Computador/métodosRESUMO
There are some problems in positron emission tomography/ computed tomography (PET/CT) lung images, such as little information of feature pixels in lesion regions, complex and diverse shapes, and blurred boundaries between lesions and surrounding tissues, which lead to inadequate extraction of tumor lesion features by the model. To solve the above problems, this paper proposes a dense interactive feature fusion Mask RCNN (DIF-Mask RCNN) model. Firstly, a feature extraction network with cross-scale backbone and auxiliary structures was designed to extract the features of lesions at different scales. Then, a dense interactive feature enhancement network was designed to enhance the lesion detail information in the deep feature map by interactively fusing the shallowest lesion features with neighboring features and current features in the form of dense connections. Finally, a dense interactive feature fusion feature pyramid network (FPN) network was constructed, and the shallow information was added to the deep features one by one in the bottom-up path with dense connections to further enhance the model's perception of weak features in the lesion region. The ablation and comparison experiments were conducted on the clinical PET/CT lung image dataset. The results showed that the APdet, APseg, APdet_s and APseg_s indexes of the proposed model were 67.16%, 68.12%, 34.97% and 37.68%, respectively. Compared with Mask RCNN (ResNet50), APdet and APseg indexes increased by 7.11% and 5.14%, respectively. DIF-Mask RCNN model can effectively detect and segment tumor lesions. It provides important reference value and evaluation basis for computer-aided diagnosis of lung cancer.
Assuntos
Neoplasias Pulmonares , Tomografia por Emissão de Pósitrons combinada à Tomografia Computadorizada , Humanos , Neoplasias Pulmonares/diagnóstico por imagem , Tomografia por Emissão de Pósitrons combinada à Tomografia Computadorizada/métodos , Pulmão/diagnóstico por imagem , Algoritmos , Processamento de Imagem Assistida por Computador/métodos , Redes Neurais de ComputaçãoRESUMO
New image-derived biomarkers for patients affected by autosomal dominant polycystic kidney disease are needed to improve current clinical management. The measurement of total kidney volume (TKV) provides critical information for clinicians to drive care decisions. However, patients with similar TKV may present with very different phenotypes, often requiring subjective decisions based on other factors (e.g., appearance of healthy kidney parenchyma, a few cysts contributing significantly to overall TKV, etc.). In this study, we describe a new technique to individually segment cysts and quantify biometric parameters including cyst volume, cyst number, parenchyma volume, and cyst parenchyma surface area. Using data from the Consortium for Radiologic Imaging Studies of Polycystic Kidney Disease (CRISP) study the utility of these new parameters was explored, both quantitatively as well as visually. Total cyst number and cyst parenchyma surface area showed superior prediction of the slope of estimated glomerular filtration rate decline, kidney failure and chronic kidney disease stages 3A, 3B, and 4, compared to TKV. In addition, presentations such as a few large cysts contributing significantly to overall kidney volume were shown to be much better stratified in terms of outcome predictions. Thus, these new image biomarkers, which can be obtained automatically, will have great utility in future studies and clinical care for patients affected by autosomal dominant polycystic kidney disease.
Assuntos
Rim Policístico Autossômico Dominante , Humanos , Rim Policístico Autossômico Dominante/complicações , Rim Policístico Autossômico Dominante/diagnóstico por imagem , Progressão da Doença , Imageamento por Ressonância Magnética/métodos , Prognóstico , Rim/diagnóstico por imagem , Biomarcadores , Taxa de Filtração GlomerularRESUMO
As a fundamental computer vision task, instance segmentation is widely used in the field of autonomous driving because it can perform both instance-level distinction and pixel-level segmentation. We propose CompleteInst based on QueryInst as a solution to the problems of missed detection with a network structure designed from the feature level and the instance level. At the feature level, we propose Global Pyramid Networks (GPN) to collect global information of missed instances. Then, we introduce the semantic branch to complete the semantic features of the missed instances. At the instance level, we implement the query-based optimal transport assignment (OTA-Query) sample allocation strategy which enhances the quality of positive samples of missed instances. Both the semantic branch and OTA-Query are parallel, meaning that there is no interference between stages, and they are compatible with the parallel supervision mechanism of QueryInst. We also compare their performance to that of non-parallel structures, highlighting the superiority of the proposed parallel structure. Experiments were conducted on the Cityscapes and COCO dataset, and the recall of CompleteInst reached 56.7% and 54.2%, a 3.5% and 3.2% improvement over the baseline, outperforming other methods.
RESUMO
The core body temperature serves as a pivotal physiological metric indicative of sow health, with rectal thermometry prevailing as a prevalent method for estimating core body temperature within sow farms. Nonetheless, employing contact thermometers for rectal temperature measurement proves to be time-intensive, labor-demanding, and hygienically suboptimal. Addressing the issues of minimal automation and temperature measurement accuracy in sow temperature monitoring, this study introduces an automatic temperature monitoring method for sows, utilizing a segmentation network amalgamating YOLOv5s and DeepLabv3+, complemented by an adaptive genetic algorithm-random forest (AGA-RF) regression algorithm. In developing the sow vulva segmenter, YOLOv5s was synergized with DeepLabv3+, and the CBAM attention mechanism and MobileNetv2 network were incorporated to ensure precise localization and expedited segmentation of the vulva region. Within the temperature prediction module, an optimized regression algorithm derived from the random forest algorithm facilitated the construction of a temperature inversion model, predicated upon environmental parameters and vulva temperature, for the rectal temperature prediction in sows. Testing revealed that vulvar segmentation IoU was 91.50%, while the predicted MSE, MAE, and R2 for rectal temperature were 0.114 °C, 0.191 °C, and 0.845, respectively. The automatic sow temperature monitoring method proposed herein demonstrates substantial reliability and practicality, facilitating an autonomous sow temperature monitoring.
Assuntos
Temperatura Corporal , Termômetros , Suínos , Animais , Feminino , Temperatura , Reprodutibilidade dos Testes , Aprendizado de MáquinaRESUMO
In this study, we propose an algorithm to improve the accuracy of tiny object segmentation for precise pothole detection on asphalt pavements. The approach comprises a three-step process: MOED, VAPOR, and Exception Processing, designed to extract pothole edges, validate the results, and manage detected abnormalities. The proposed algorithm addresses the limitations of previous methods and offers several advantages, including wider coverage. We experimentally evaluated the performance of the proposed algorithm by filming roads in various regions of South Korea using a UAV at high altitudes of 30-70 m. The results show that our algorithm outperforms previous methods in terms of instance segmentation performance for small objects such as potholes. Our study offers a practical and efficient solution for pothole detection and contributes to road safety maintenance and monitoring.
Assuntos
Algoritmos , Gases , Hidrocarbonetos , Filmes CinematográficosRESUMO
Building extraction from high-resolution remote sensing images has various applications, such as urban planning and population estimation. However, buildings have intraclass heterogeneity and interclass homogeneity in high-resolution remote sensing images with complex backgrounds, which makes the accurate extraction of building instances challenging and regular building boundaries difficult to maintain. In this paper, an attention-gated and direction-field-optimized building instance extraction network (AGDF-Net) is proposed. Two refinements are presented, including an Attention-Gated Feature Pyramid Network (AG-FPN) and a Direction Field Optimization Module (DFOM), which are used to improve information flow and optimize the mask, respectively. The AG-FPN promotes complementary semantic and detail information by measuring information importance to control the addition of low-level and high-level features. The DFOM predicts the pixel-level direction field of each instance and iteratively corrects the direction field based on the initial segmentation. Experimental results show that the proposed method outperforms the six state-of-the-art instance segmentation methods and three semantic segmentation methods. Specifically, AGDF-Net improves the objective-level metric AP and the pixel-level metric IoU by 1.1%~9.4% and 3.55%~5.06%.
RESUMO
The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block's pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.