Rethinking Cross-Domain Pedestrian Detection: A Background-Focused Distribution Alignment Framework for Instance-Free One-Stage Detectors.

Cross-domain pedestrian detection aims to generalize pedestrian detectors from one label-rich domain to another label-scarce domain, which is crucial for various real-world applications. Most recent works focus on domain alignment to train domain-adaptive detectors either at the instance level or image level. From a practical point of view, one-stage detectors are faster. Therefore, we concentrate on designing a cross-domain algorithm for rapid one-stage detectors that lacks instance-level proposals and can only perform image-level feature alignment. However, pure image-level feature alignment causes the foreground-background misalignment issue to arise, i.e., the foreground features in the source domain image are falsely aligned with background features in the target domain image. To address this issue, we systematically analyze the importance of foreground and background in image-level cross-domain alignment, and learn that background plays a more critical role in image-level cross-domain alignment. Therefore, we focus on cross-domain background feature alignment while minimizing the influence of foreground features on the cross-domain alignment stage. This paper proposes a novel framework, namely, background-focused distribution alignment (BFDA), to train domain adaptive one-stage pedestrian detectors. Specifically, BFDA first decouples the background features from the whole image feature maps and then aligns them via a novel long-short-range discriminator. Extensive experiments demonstrate that compared to mainstream domain adaptation technologies, BFDA significantly enhances cross-domain pedestrian detection performance for either one-stage or two-stage detectors. Moreover, by employing the efficient one-stage detector (YOLOv5), BFDA can reach 217.4 FPS ( 640×480 pixels) on NVIDIA Tesla V100 (7~12 times the FPS of the existing frameworks), which is highly significant for practical applications. The code from this study will be made publicly available.

OTP-NMS: Toward Optimal Threshold Prediction of NMS for Crowded Pedestrian Detection.

Pedestrian detection is still a challenging task for computer vision, especially in crowded scenes where the overlaps between pedestrians tend to be large. The non-maximum suppression (NMS) plays an important role in removing the redundant false positive detection proposals while retaining the true positive detection proposals. However, the highly overlapped results may be suppressed if the threshold of NMS is lower. Meanwhile, a higher threshold of NMS will introduce a larger number of false positive results. To solve this problem, we propose an optimal threshold prediction (OTP) based NMS method that predicts a suitable threshold of NMS for each human instance. First, a visibility estimation module is designed to obtain the visibility ratio. Then, we propose a threshold prediction subnet to determine the optimal threshold of NMS automatically according to the visibility ratio and classification score. Finally, we re-formulate the objective function of the subnet and utilize the reward-guided gradient estimation algorithm to update the subnet. Comprehensive experiments on CrowdHuman and CityPersons show the superior performance of the proposed method in pedestrian detection, especially in crowded scenes.

A Closer Look at the Joint Training of Object Detection and Re-identification in Multi-Object Tracking.

Unifying object detection and re-identification (ReID) into a single network enables faster multi-object tracking (MOT), while this multi-task setting poses challenges for training. In this work, we dissect the joint training of detection and ReID from two dimensions: label assignment and loss function. We find previous works generally overlook them and directly borrow the practices from object detection, inevitably causing inferior performance. Specifically, we identify a qualified label assignment for MOT should: 1) have the assignment cost aware of ReID cost, not just detection cost; 2) provide sufficient positive samples for robust feature learning while avoiding ambiguous positives (i.e., the positives shared by different ground-truth objects). To achieve the above goals, we first propose Identity-aware Label Assignment, which jointly considers the assignment cost of detection and ReID to select positive samples for each instance without ambiguities. Moreover, we advance a novel Discriminative Focal Loss that integrates ReID predictions with Focal Loss to focus the training on the discriminative samples. Finally, we upgrade the strong baseline FairMOT with our techniques and achieve up to 7.0 MOTA / 54.1% IDs improvements on MOT16/17/20 benchmarks under favorable inference speed, which verifies our tailored label assignment and loss function for MOT are superior to those inherited from object detection.

HTD: Heterogeneous Task Decoupling for Two-Stage Object Detection.

Decoupling the sibling head has recently shown great potential in relieving the inherent task-misalignment problem in two-stage object detectors. However, existing works design similar structures for the classification and regression, ignoring task-specific characteristics and feature demands. Besides, the shared knowledge that may benefit the two branches is neglected, leading to potential excessive decoupling and semantic inconsistency. To address these two issues, we propose Heterogeneous task decoupling (HTD) framework for object detection, which utilizes a Progressive Graph (PGraph) module and a Border-aware Adaptation (BA) module for task-decoupling. Specifically, we first devise a Semantic Feature Aggregation (SFA) module to aggregate global semantics with image-level supervision, serving as the shared knowledge for the task-decoupled framework. Then, the PGraph module performs progressive graph reasoning, including local spatial aggregation and global semantic interaction, to enhance semantic representations of region proposals for classification. The proposed BA module integrates multi-level features adaptively, focusing on the low-level border activation to obtain representations with spatial and border perception for regression. Finally, we utilize the aggregated knowledge from SFA to keep the instance-level semantic consistency (ISC) of decoupled frameworks. Extensive experiments demonstrate that HTD outperforms existing detection works by a large margin, and achieves single-model 50.4%AP and 33.2% APs on COCO test-dev set using ResNet-101-DCN backbone, which is the best entry among state-of-the-arts under the same configuration. Our code is available at https://github.com/CityU-AIM-Group/HTD.

AutoPedestrian: An Automatic Data Augmentation and Loss Function Search Scheme for Pedestrian Detection.

Pedestrian detection is a challenging and hot research topic in the field of computer vision, especially for the crowded scenes where occlusion happens frequently. In this paper, we propose a novel AutoPedestrian scheme that automatically augments the pedestrian data and searches for suitable loss functions, aiming for better performance of pedestrian detection especially in crowded scenes. To our best knowledge, it is the first work to automatically search the optimal policy of data augmentation and loss function jointly for the pedestrian detection. To achieve the goal of searching the optimal augmentation scheme and loss function jointly, we first formulate the data augmentation policy and loss function as probability distributions based on different hyper-parameters. Then, we apply a double-loop scheme with importance-sampling to solve the optimization problem of data augmentation and loss function types efficiently. Comprehensive experiments on two popular benchmarks of CrowdHuman and CityPersons show the effectiveness of our proposed method. In particular, we achieve 40.58% in MR on CrowdHuman datasets and 11.3% in MR on CityPersons reasonable subset, yielding new state-of-the-art results on these two datasets.

Dual Color Space Guided Sketch Colorization.

Automatic sketch colorization is a challenging task in both computer graphics and computer vision since all the color, texture, shading generation have to be created based on the abstract sketch. Besides, it is a subjective task in painting process, which needs illustrators to comprehend drawing priori (DP), such as hue variation, saturation contrast and gray contrast and utilize them in the HSV color space which is closer to human visual cognition system. As such, incorporating supplementary supervision in the HSV color space may be beneficial to sketch colorization. However, previous methods improve the colorization quality only in the RGB color space without considering the HSV color space, often causing results with dull color, inappropriate saturation contrast, and artifacts. To address this issue, we propose a novel sketch colorization method, dual color space guided generative adversarial network (DCSGAN), that considers the complementary information contained in both the RGB and HSV color space. Specifically, we incorporate the HSV color space to construct dual color spaces for supervising our method with a color space transformation (CST) network that learns transformation from the RGB to HSV color space. Then, we propose a DP loss that enables the DCSGAN to generate vivid color images with pixel level supervision. Additionally, a novel dual color space adversarial (DCSA) loss is designed to guide the generator at global level to reduce the artifacts to meet audiences' aesthetic expectations. Extensive experiments and ablation studies demonstrate the superiority of the proposed method over previous state-of-the-art (SOTA) methods.

Bleeding Frame and Region Detection in the Wireless Capsule Endoscopy Video.

Wireless capsule endoscopy (WCE) enables noninvasive and painless direct visual inspection of a patient's whole digestive tract, but at the price of long time reviewing large amount of images by clinicians. Thus, an automatic computer-aided technique to reduce the burden of physicians is highly demanded. In this paper, we propose a novel color feature extraction method to discriminate the bleeding frames from the normal ones, with further localization of the bleeding regions. Our proposal is based on a twofold system. First, we make full use of the color information of WCE images and utilize K-means clustering method on the pixel represented images to obtain the cluster centers, with which we characterize WCE images as words-based color histograms. Then, we judge the status of a WCE frame by applying the support vector machine (SVM) and K-nearest neighbor methods. Comprehensive experimental results reveal that the best classification performance is obtained with YCbCr color space, cluster number 80 and the SVM. The achieved classification performance reaches 95.75% in accuracy, 0.9771 for AUC, validating that the proposed scheme provides an exciting performance for bleeding classification. Second, we propose a two-stage saliency map extraction method to highlight bleeding regions, where the first-stage saliency map is created by means of different color channels mixer and the second-stage saliency map is obtained from the visual contrast. Followed by an appropriate fusion strategy and threshold, we localize the bleeding areas. Quantitative as well as qualitative results show that our methods could differentiate the bleeding areas from neighborhoods correctly.

Saliency based ulcer detection for wireless capsule endoscopy diagnosis.

Ulcer is one of the most common symptoms of many serious diseases in the human digestive tract. Especially for the ulcers in the small bowel where other procedures cannot adequately visualize, wireless capsule endoscopy (WCE) is increasingly being used in the diagnosis and clinical management. Because WCE generates large amount of images from the whole process of inspection, computer-aided detection of ulcer is considered an indispensable relief to clinicians. In this paper, a two-staged fully automated computer-aided detection system is proposed to detect ulcer from WCE images. In the first stage, we propose an effective saliency detection method based on multi-level superpixel representation to outline the ulcer candidates. To find the perceptually and semantically meaningful salient regions, we first segment the image into multi-level superpixel segmentations. Each level corresponds to different initial region sizes of the superpixels. Then we evaluate the corresponding saliency according to the color and texture features in superpixel region of each level. In the end, we fuse the saliency maps from all levels together to obtain the final saliency map. In the second stage, we apply the obtained saliency map to better encode the image features for the ulcer image recognition tasks. Because the ulcer mainly corresponds to the saliency region, we propose a saliency max-pooling method integrated with the Locality-constrained Linear Coding (LLC) method to characterize the images. Experiment results achieve promising 92.65% accuracy and 94.12% sensitivity, validating the effectiveness of the proposed method. Moreover, the comparison results show that our detection system outperforms the state-of-the-art methods on the ulcer classification task.

Computer aided wireless capsule endoscopy video segmentation.

PURPOSE: Wireless capsule endoscopy (WCE) opens a new door for the digestive tract examination and diagnosis. However, the examination of its video data is tedious. This study aims to assist a physician to interpret a WCE video by segmenting it into different anatomic parts in the digestive tract. METHODS: A two level WCE video segmentation scheme is proposed to locate the boundary between the stomach, small intestine, and large intestine. In the rough level, the authors utilize color feature to draw a dissimilarity curve for a WCE video and obtain an approximate boundary. Meanwhile, training data for the fine level segmentation can be collected automatically between the two approximate boundaries of organs to overcome the difficulty of training data collection in traditional approaches. In the fine level, color histogram in the HSI color space is used to segment the stomach and small intestine. Then, color uniform local binary pattern (CULBP) algorithm is applied for discrimination of the small intestine and large intestine, which includes two patterns, namely, color norm and color angle pattern. The CULBP feature is robust to variation of illumination and discriminative for classification. In order to increase the performance of support vector machine, the authors integrate it with the Adaboost approach. Finally, the authors refine the classification results to segment a WCE video into different parts, that is, the stomach, small intestine, and large intestine. RESULTS: The average precision and recall are 91.2% and 90.6% for the stomach/small intestine classification, 89.2% and 88.7% for the small/large intestine discrimination. Paired t-test also demonstrates a significant better performance of the proposed scheme compared to some traditional methods. The average segmentation error is 8 frames for the stomach/small intestine discrimination, and 14 frames for the small/large intestine segmentation. CONCLUSIONS: The results have demonstrated that the new video segmentation method can accurately locate the boundary between different organ regions in a WCE video. Such a segmentation result may enhance the efficiency of WCE examination.

Robust object tracking with reacquisition ability using online learned detector.

Long term tracking is a challenging task for many applications. In this paper, we propose a novel tracking approach that can adapt various appearance changes such as illumination, motion, and occlusions, and owns the ability of robust reacquisition after drifting. We utilize a condensation-based method with an online support vector machine as a reliable observation model to realize adaptive tracking. To redetect the target when drifting, a cascade detector based on random ferns is proposed. It can detect the target robustly in real time. After redetection, we also come up with a new refinement strategy to improve the tracker's performance by removing the support vectors corresponding to possible wrong updates by a matching template. Extensive comparison experiments on typical and challenging benchmark dataset illustrate a robust and encouraging performance of the proposed approach.

A deep learning based framework for accurate segmentation of cervical cytoplasm and nuclei.

In this paper, a superpixel and convolution neural network (CNN) based segmentation method is proposed for cervical cancer cell segmentation. Since the background and cytoplasm contrast is not relatively obvious, cytoplasm segmentation is first performed. Deep learning based on CNN is explored for region of interest detection. A coarse-to-fine nucleus segmentation for cervical cancer cell segmentation and further refinement is also developed. Experimental results show that an accuracy of 94.50% is achieved for nucleus region detection and a precision of 0.9143±0.0202 and a recall of 0.8726±0.0008 are achieved for nucleus cell segmentation. Furthermore, our comparative analysis also shows that the proposed method outperforms the related methods.

Tumor recognition in wireless capsule endoscopy images using textural features and SVM-based feature selection.

Tumor in digestive tract is a common disease and wireless capsule endoscopy (WCE) is a relatively new technology to examine diseases for digestive tract especially for small intestine. This paper addresses the problem of automatic recognition of tumor for WCE images. Candidate color texture feature that integrates uniform local binary pattern and wavelet is proposed to characterize WCE images. The proposed features are invariant to illumination change and describe multiresolution characteristics of WCE images. Two feature selection approaches based on support vector machine, sequential forward floating selection and recursive feature elimination, are further employed to refine the proposed features for improving the detection accuracy. Extensive experiments validate that the proposed computer-aided diagnosis system achieves a promising tumor recognition accuracy of 92.4% in WCE images on our collected data.

Comparison of several texture features for tumor detection in CE images.

Capsule endoscopy (CE) has been widely used as a new technology to diagnose gastrointestinal tract diseases, especially for small intestine. However, the large number of images in each test is a great burden for physicians. As such, computer aided detection (CAD) scheme is needed to relieve the workload of clinicians. In this paper, automatic differentiation of tumor CE image and normal CE image is investigated through comparative textural feature analysis. Four different color textures are studied in this work, i.e., texture spectrum histogram, color wavelet covariance, rotation invariant uniform local binary pattern and curvelet based local binary pattern. With support vector machine being the classifier, the discrimination ability of these four different color textures for tumor detection in CE images is extensively compared in RGB, Lab and HSI color space through ten-fold cross-validation experiments on our CE image data. It is found that HSI color space is the most suitable color space for all these texture based CAD systems. Moreover, the best performance achieved is 83.50% in terms of average accuracy, which is obtained by the scheme based on rotation invariant uniform local binary pattern.

Contourlet-based features for computerized tumor detection in capsule endoscopy images.

This article presents a computer-aided detection system for capsule endoscopy (CE) images using contourlet-based color textural features to recognize tumors in the digestive tract. As tumor exhibits rich information in color texture, a novel color texture feature based on contourlet transform is proposed to describe characteristics of tumor in CE images. The proposed features are a hybrid of contourlet transform and uniform local binary pattern, yielding detailed and robust color texture features in multi-directions for CE images. Sequential floating forward search approach is further applied to refine the proposed features. With support vector machine for classification, comprehensive experiments on our present data reveal an encouraging accuracy of 93.6% for tumor detection in CE images using the proposed features.

Computer-aided small bowel tumor detection for capsule endoscopy.

OBJECTIVE: Capsule endoscopy is useful in the diagnosis of small bowel diseases. However, the large number of images produced in each test is a tedious task for physicians. To relieve burden of physicians, a new computer-aided detection scheme is developed in this study, which aims to detect small bowel tumors for capsule endoscopy. METHODS AND MATERIALS: A novel textural feature based on multi-scale local binary pattern is proposed to discriminate tumor images from normal images. Since tumor in small bowel exhibit great diversities in appearance, multiple classifiers are employed to improve detection accuracy. 1200 capsule endoscopy images chosen from 10 patients' data constitute test data in our experiment. RESULTS: Multiple classifiers based on k-nearest neighbor, multilayer perceptron neural network and support vector machine, which are built from six different ensemble rules, are experimented in three different color spaces. The results demonstrate an encouraging detection accuracy of 90.50%, together with a sensitivity of 92.33% and a specificity of 88.67%. CONCLUSION: The proposed scheme using color texture features and classifier ensemble is promising for small bowel tumor detection in capsule endoscopy images.

A novel method for informative frame selection in wireless capsule endoscopy video.

Wireless capsule endoscopy (WCE) has been validated to be an important tool in the evaluation of gastrointestinal (GI) tract. Compared with traditional endoscope technologies, its non-invasiveness property meets with great favor of patients. However, from physician's point of view, WCE video suffers from low resolution, limited illumination, irregular movement, more importantly, imbalanced rate of abnormality which brings a lot of challenges for diagnosis. These challenges motivate us to devise an approach to guide the physicians to focus on the informative frames which could be convenient for review of the content of the GI tract. This paper presents a novel approach for automatic selection of the WCE video frames with lumen and coherent motility. We adopt lumen detection based on mean shift to provide robust and reliable selection of lumen. Together with the evaluation of coherent motility, we can provide a full and fast approach to select the informative frames for diagnosis. The experiments on real date are presented to show the performance of our proposed method.

3D reconstruction of wireless capsule endoscopy images.

Wireless capsule endoscopy (WCE) has been gradually applied for inspecting the gastrointestinal (GI) tract. However, WCE can only provide monocular view. Moreover, only a small part of GI wall is visible frame by frame due to the limited illumination and irregular motion of the capsule endoscope. The perception of entire GI structure could be hard even for the experienced endoscopists. A realistic friendly three dimension view is needed to help the physicians to get a better perception of the GI tract. In this paper, we present a method to reconstruct the three dimension surface of the intestinal wall by applying the SIFT feature detector and descriptor to a sequence of WCE images. Epipolar geometry is employed to further constrain the matching feature points in order to obtain a more accurate 3D view. The experiments on real data are presented to show the performance of our proposed method.

A comparative study of shape features for polyp detection in wireless capsule endoscopy images.

Wireless capsule endoscopy (WCE) has been gradually employed in hospitals because it can directly view the entire small bowel of a human body for the first time. However, a troublesome problem related to this new technology is that too many images produced by WCE will take a lot of efforts for doctors to inspect. In this paper, we propose a comparative study of shape features aiming for intestinal polyp detection for WCE images. As polyps exhibit strong shape characteristics, also a powerful clue used by physicians, we investigate two kinds of shape features, MEPG-7 region-based shape descriptor and Zernike moments, in our study. With multi-layer perceptron neural network as the classifier, experiments on our present image data show that it is promising to employ both Zernike moments and MEPG-7 region-based shape descriptor as the shape features to recognize the intestinal polyp regions, and a better performance is obtained by the Zernike moments based shape features.

Computer-aided detection of bleeding regions for capsule endoscopy images.

Capsule endoscopy (CE) has been widely used to diagnose diseases in human digestive tract. However, a tough problem of this new technology is that too many images to be inspected by eyes cause a huge burden to physicians, so it is significant to investigate computerized diagnosis methods. In this paper, a new computer-aided system aimed for bleeding region detection in CE images is proposed. This new system exploits color texture feature, an important clue used by physicians, to analyze status of gastrointestinal tract. We put forward a new idea of chrominance moment as the color part of color texture feature, which makes full use of Tchebichef polynomials and illumination invariant of hue/saturation/intensity color space. Combined with uniform local binary pattern, a current texture representation model, it can be applied to discriminate normal regions and bleeding regions in CE images. Classification of bleeding regions using multilayer perceptron neural network is then deployed to verify performance of the proposed color texture features. Experimental results on our bleeding image data show that the proposed scheme is promising in detecting bleeding regions.

Computer-based detection of bleeding and ulcer in wireless capsule endoscopy images by chromaticity moments.

The wireless capsule endoscopy (WCE) invented by Given Imaging has been gradually used in hospitals due to its great breakthrough that it can view the entire small bowel for gastrointestinal diseases. However, a tough problem associated with this new technology is that too many images to be examined by eyes cause a huge burden to physicians, so it is significant if we can help physicians do diagnosis using computerized methods. In this paper, a new method aimed for bleeding and ulcer detection in WCE images is proposed. This new approach mainly focuses on color feature, also a very powerful clue used by physicians for diagnosis, to judge the status of gastrointestinal tract. We propose a new idea of chromaticity moment as the features to discriminate normal regions and abnormal regions, which make full use of the Tchebichef polynomials and the illumination invariant of HSI color space, and we verify performances of the proposed features by employing neural network classifier. Experimental results on our present image data of bleeding and ulcer show that it is feasible to exploit the proposed chromaticity moments to detect bleeding and ulcer for WCE images.