Efficient Image Retrieval Using Hierarchical K-Means Clustering.

The objective of content-based image retrieval (CBIR) is to locate samples from a database that are akin to a query, relying on the content embedded within the images. A contemporary strategy involves calculating the similarity between compact vectors by encoding both the query and the database images as global descriptors. In this work, we propose an image retrieval method by using hierarchical K-means clustering to efficiently organize the image descriptors within the database, which aims to optimize the subsequent retrieval process. Then, we compute the similarity between the descriptor set within the leaf nodes and the query descriptor to rank them accordingly. Three tree search algorithms are presented to enable a trade-off between search accuracy and speed that allows for substantial gains at the expense of a slightly reduced retrieval accuracy. Our proposed method demonstrates enhancement in image retrieval speed when applied to the CLIP-based model, UNICOM, designed for category-level retrieval, as well as the CNN-based R-GeM model, tailored for particular object retrieval by validating its effectiveness across various domains and backbones. We achieve an 18-times speed improvement while preserving over 99% accuracy when applied to the In-Shop dataset, the largest dataset in the experiments.

Reading of small bowel capsule endoscopy after frame reduction using an artificial intelligence algorithm.

OBJECTIVES: Poorly visualized images that appear during small bowel capsule endoscopy (SBCE) can confuse the interpretation of small bowel lesions and increase the physician's workload. Using a validated artificial intelligence (AI) algorithm that can evaluate the mucosal visualization, we aimed to assess whether SBCE reading after the removal of poorly visualized images could affect the diagnosis of SBCE. METHODS: A study was conducted to analyze 90 SBCE cases in which a small bowel examination was completed. Two experienced endoscopists alternately performed two types of readings. They used the AI algorithm to remove poorly visualized images for the frame reduction reading (AI user group) and conducted whole frame reading without AI (AI non-user group) for the same patient. A poorly visualized image was defined as an image with < 50% mucosal visualization. The study outcomes were diagnostic concordance and reading time between the two groups. The SBCE diagnosis was classified as Crohn's disease, bleeding, polyp, angiodysplasia, and nonspecific finding. RESULTS: The final SBCE diagnoses between the two groups showed statistically significant diagnostic concordance (k = 0.954, p < 0.001). The mean number of lesion images was 3008.5 ± 9964.9 in the AI non-user group and 1401.7 ± 4811.3 in the AI user group. There were no cases in which lesions were completely removed. Compared with the AI non-user group (120.9 min), the reading time was reduced by 35.6% in the AI user group (77.9 min). CONCLUSIONS: SBCE reading after reducing poorly visualized frames using the AI algorithm did not have a negative effect on the final diagnosis. SBCE reading method integrated with frame reduction and mucosal visualization evaluation will help improve AI-assisted SBCE interpretation.

Video Analysis of Small Bowel Capsule Endoscopy Using a Transformer Network.

Although wireless capsule endoscopy (WCE) detects small bowel diseases effectively, it has some limitations. For example, the reading process can be time consuming due to the numerous images generated per case and the lesion detection accuracy may rely on the operators' skills and experiences. Hence, many researchers have recently developed deep-learning-based methods to address these limitations. However, they tend to select only a portion of the images from a given WCE video and analyze each image individually. In this study, we note that more information can be extracted from the unused frames and the temporal relations of sequential frames. Specifically, to increase the accuracy of lesion detection without depending on experts' frame selection skills, we suggest using whole video frames as the input to the deep learning system. Thus, we propose a new Transformer-architecture-based neural encoder that takes the entire video as the input, exploiting the power of the Transformer architecture to extract long-term global correlation within and between the input frames. Subsequently, we can capture the temporal context of the input frames and the attentional features within a frame. Tests on benchmark datasets of four WCE videos showed 95.1% sensitivity and 83.4% specificity. These results may significantly advance automated lesion detection techniques for WCE images.

Lossless Reconstruction of Convolutional Neural Network for Channel-Based Network Pruning.

Network pruning reduces the number of parameters and computational costs of convolutional neural networks while maintaining high performance. Although existing pruning methods have achieved excellent results, they do not consider reconstruction after pruning in order to apply the network to actual devices. This study proposes a reconstruction process for channel-based network pruning. For lossless reconstruction, we focus on three components of the network: the residual block, skip connection, and convolution layer. Union operation and index alignment are applied to the residual block and skip connection, respectively. Furthermore, we reconstruct a compressed convolution layer by considering batch normalization. We apply our method to existing channel-based pruning methods for downstream tasks such as image classification, object detection, and semantic segmentation. Experimental results show that compressing a large model has a 1.93% higher accuracy in image classification, 2.2 higher mean Intersection over Union (mIoU) in semantic segmentation, and 0.054 higher mean Average Precision (mAP) in object detection than well-designed small models. Moreover, we demonstrate that our method can reduce the actual latency by 8.15× and 5.29× on Raspberry Pi and Jetson Nano, respectively.

Reduced detection rate of artificial intelligence in images obtained from untrained endoscope models and improvement using domain adaptation algorithm.

A training dataset that is limited to a specific endoscope model can overfit artificial intelligence (AI) to its unique image characteristics. The performance of the AI may degrade in images of different endoscope model. The domain adaptation algorithm, i.e., the cycle-consistent adversarial network (cycleGAN), can transform the image characteristics into AI-friendly styles. We attempted to confirm the performance degradation of AIs in images of various endoscope models and aimed to improve them using cycleGAN transformation. Two AI models were developed from data of esophagogastroduodenoscopies collected retrospectively over 5 years: one for identifying the endoscope models, Olympus CV-260SL, CV-290 (Olympus, Tokyo, Japan), and PENTAX EPK-i (PENTAX Medical, Tokyo, Japan), and the other for recognizing the esophagogastric junction (EGJ). The AIs were trained using 45,683 standardized images from 1,498 cases and validated on 624 separate cases. Between the two endoscope manufacturers, there was a difference in image characteristics that could be distinguished without error by AI. The accuracy of the AI in recognizing gastroesophageal junction was >0.979 in the same endoscope-examined validation dataset as the training dataset. However, they deteriorated in datasets from different endoscopes. Cycle-consistent adversarial network can successfully convert image characteristics to ameliorate the AI performance. The improvements were statistically significant and greater in datasets from different endoscope manufacturers [original → AI-trained style, increased area under the receiver operating characteristic (ROC) curve, P-value: CV-260SL → CV-290, 0.0056, P = 0.0106; CV-260SL → EPK-i, 0.0182, P = 0.0158; CV-290 → CV-260SL, 0.0134, P < 0.0001; CV-290 → EPK-i, 0.0299, P = 0.0001; EPK-i → CV-260SL, 0.0215, P = 0.0024; and EPK-i → CV-290, 0.0616, P < 0.0001]. In conclusion, cycleGAN can transform the diverse image characteristics of endoscope models into an AI-trained style to improve the detection performance of AI.

Small bowel cleanliness in capsule endoscopy: a case-control study using validated artificial intelligence algorithm.

Small bowel capsule endoscopy (SBCE) may need to be performed immediately after colonoscopy without additional bowel preparation if active small bowel diseases are suspected. However, it is unclear whether the small bowel cleanliness is adequately maintained even after SBCE is performed immediately after colonoscopy. We compared the small bowel cleanliness scores of the study group (SBCE immediately after colonoscopy) and control group (SBCE alone) using a validated artificial intelligence (AI) algorithm (cut-off score > 3.25 for adequate). Cases of SBCE in which polyethylene glycol was used were included retrospectively. Among 85 enrolled cases, 50 cases (58.8%) were the study group. The mean time from the last dose of purgative administration to SBCE was 6.86 ± 0.94 h in the study group and 3.00 ± 0.18 h in the control group. Seventy-five cases (88.2%) were adequate small bowel cleanliness, which was not different between the two groups. The mean small bowel cleanliness score for the study group was 3.970 ± 0.603, and for the control group was 3.937 ± 0.428. In the study group, better colon preparation resulted in a higher small bowel cleanliness score (p = 0.015). Small bowel cleanliness was also adequately maintained in SBCE immediately after colonoscopy. There was no difference between the time and volume of purgative administration and small bowel cleanliness.

Gastric examination using a novel three-dimensional magnetically assisted capsule endoscope and a hand-held magnetic controller: A porcine model study.

Magnetically assisted capsule endoscopy (MACE) is a noninvasive procedure and can overcome passive capsule movement that limits gastric examination. MACE has been studied in many trials as an alternative to upper endoscopy. However, to increase diagnostic accuracy of various gastric lesions, MACE should be able to provide stereoscopic, clear images and to measure the size of a lesion. So, we conducted the animal experiment using a novel three-dimensional (3D) MACE and a new hand-held magnetic controller for gastric examination. The purpose of this study is to assess the performance and safety of 3D MACE and hand-held magnetic controller through the animal experiment. Subsequently, via the dedicated viewer, we evaluate whether 3D reconstruction images and clear images can be obtained and accurate lesion size can be measured. During real-time gastric examination, the maneuverability and visualization of 3D MACE were adequate. A polypoid mass lesion was incidentally observed at the lesser curvature side of the prepyloric antrum. The mass lesion was estimated to be 10.9 x 11.5 mm in the dedicated viewer, nearly the same size and shape as confirmed by upper endoscopy and postmortem examination. Also, 3D and clear images of the lesion were successfully reconstructed. This animal experiment demonstrates the accuracy and safety of 3D MACE. Further clinical studies are warranted to confirm the feasibility of 3D MACE for human gastric examination.

Efficacy of a comprehensive binary classification model using a deep convolutional neural network for wireless capsule endoscopy.

The manual reading of capsule endoscopy (CE) videos in small bowel disease diagnosis is time-intensive. Algorithms introduced to automate this process are premature for real clinical applications, and multi-diagnosis using these methods has not been sufficiently validated. Therefore, we developed a practical binary classification model, which selectively identifies clinically meaningful images including inflamed mucosa, atypical vascularity or bleeding, and tested it with unseen cases. Four hundred thousand CE images were randomly selected from 84 cases in which 240,000 images were used to train the algorithm to categorize images binarily. The remaining images were utilized for validation and internal testing. The algorithm was externally tested with 256,591 unseen images. The diagnostic accuracy of the trained model applied to the validation set was 98.067%. In contrast, the accuracy of the model when applied to a dataset provided by an independent hospital that did not participate during training was 85.470%. The area under the curve (AUC) was 0.922. Our model showed excellent internal test results, and the misreadings were slightly increased when the model was tested in unseen external cases while the classified 'insignificant' images contain ambiguous substances. Once this limitation is solved, the proposed CNN-based binary classification will be a promising candidate for developing clinically-ready computer-aided reading methods.

A Current and Newly Proposed Artificial Intelligence Algorithm for Reading Small Bowel Capsule Endoscopy.

Small bowel capsule endoscopy (SBCE) is one of the most useful methods for diagnosing small bowel mucosal lesions. However, it takes a long time to interpret the capsule images. To solve this problem, artificial intelligence (AI) algorithms for SBCE readings are being actively studied. In this article, we analyzed several studies that applied AI algorithms to SBCE readings, such as automatic lesion detection, automatic classification of bowel cleanliness, and automatic compartmentalization of small bowels. In addition to automatic lesion detection using AI algorithms, a new direction of AI algorithms related to shorter reading times and improved lesion detection accuracy should be considered. Therefore, it is necessary to develop an integrated AI algorithm composed of algorithms with various functions in order to be used in clinical practice.

Development of a deep learning-based software for calculating cleansing score in small bowel capsule endoscopy.

A standardized small bowel (SB) cleansing scale is currently not available. The aim of this study was to develop an automated calculation software for SB cleansing score using deep learning. Consecutively performed capsule endoscopy cases were enrolled from three hospitals. A 5-step scoring system based on mucosal visibility was trained for deep learning in the training set. Performance of the trained software was evaluated in the validation set. Average cleansing score (1.0 to 5.0) by deep learning was compared to clinical grading (A to C) reviewed by clinicians. Cleansing scores decreased as clinical grading worsened (scores of 4.1, 3.5, and 2.9 for grades A, B, and C, respectively, P < 0.001). Adequate preparation was achieved for 91.7% of validation cases. The average cleansing score was significantly different between adequate and inadequate group (4.0 vs. 2.9, P < 0.001). ROC curve analysis revealed that a cut-off value of cleansing score at 3.25 had an AUC of 0.977. Diagnostic yields for small, hard-to-find lesions were associated with high cleansing scores (4.3 vs. 3.8, P < 0.001). We developed a novel scoring software which calculates objective, automated cleansing scores for SB preparation. The cut-off value we suggested provides a standard criterion for adequate bowel preparation as a quality indicator.

Artificial intelligence that determines the clinical significance of capsule endoscopy images can increase the efficiency of reading.

Artificial intelligence (AI), which has demonstrated outstanding achievements in image recognition, can be useful for the tedious capsule endoscopy (CE) reading. We aimed to develop a practical AI-based method that can identify various types of lesions and tried to evaluate the effectiveness of the method under clinical settings. A total of 203,244 CE images were collected from multiple centers selected considering the regional distribution. The AI based on the Inception-Resnet-V2 model was trained with images that were classified into two categories according to their clinical significance. The performance of AI was evaluated with a comparative test involving two groups of reviewers with different experiences. The AI summarized 67,008 (31.89%) images with a probability of more than 0.8 for containing lesions in 210,100 frames of 20 selected CE videos. Using the AI-assisted reading model, reviewers in both the groups exhibited increased lesion detection rates compared to those achieved using the conventional reading model (experts; 34.3%-73.0%; p = 0.029, trainees; 24.7%-53.1%; p = 0.029). The improved result for trainees was comparable to that for the experts (p = 0.057). Further, the AI-assisted reading model significantly shortened the reading time for trainees (1621.0-746.8 min; p = 0.029). Thus, we have developed an AI-assisted reading model that can detect various lesions and can successfully summarize CE images according to clinical significance. The assistance rendered by AI can increase the lesion detection rates of reviewers. Especially, trainees could improve their efficiency of reading as a result of reduced reading time using the AI-assisted model.

Iterative K-Closest Point Algorithms for Colored Point Cloud Registration.

We present two algorithms for aligning two colored point clouds. The two algorithms are designed to minimize a probabilistic cost based on the color-supported soft matching of points in a point cloud to their K-closest points in the other point cloud. The first algorithm, like prior iterative closest point algorithms, refines the pose parameters to minimize the cost. Assuming that the point clouds are obtained from RGB-depth images, our second algorithm regards the measured depth values as variables and minimizes the cost to obtain refined depth values. Experiments with our synthetic dataset show that our pose refinement algorithm gives better results compared to the existing algorithms. Our depth refinement algorithm is shown to achieve more accurate alignments from the outputs of the pose refinement step. Our algorithms are applied to a real-world dataset, providing accurate and visually improved results.

Noise-Aware and Light-Weight VLSI Design of Bilateral Filter for Robust and Fast Image Denoising in Mobile Systems.

The range kernel of bilateral filter degrades image quality unintentionally in real environments because the pixel intensity varies randomly due to the noise that is generated in image sensors. Furthermore, the range kernel increases the complexity due to the comparisons with neighboring pixels and the multiplications with the corresponding weights. In this paper, we propose a noise-aware range kernel, which estimates noise using an intensity difference-based image noise model and dynamically adjusts weights according to the estimated noise, in order to alleviate the quality degradation of bilateral filters by noise. In addition, to significantly reduce the complexity, an approximation scheme is introduced, which converts the proposed noise-aware range kernel into a binary kernel while using the statistical hypothesis test method. Finally, blue a fully parallelized and pipelined very-large-scale integration (VLSI) architecture of a noise-aware bilateral filter (NABF) that is based on the proposed binary range kernel is presented, which was successfully implemented in field-programmable gate array (FPGA). The experimental results show that the proposed NABF is more robust to noise than the conventional bilateral filter under various noise conditions. Furthermore, the proposed VLSI design of the NABF achieves 10.5 and 95.7 times higher throughput and uses 63.6-97.5% less internal memory than state-of-the-art bilateral filter designs.

3D reconstruction of small bowel lesions using stereo camera-based capsule endoscopy.

Three-dimensional (3D) reconstruction of capsule endoscopic images has been attempted for a long time to obtain more information on small bowel structures. Due to the limited hardware resources of capsule size and battery capacity, software approaches have been studied but have mainly exhibited inherent limitations. Recently, stereo camera-based capsule endoscopy, which can perform hardware-enabled 3D reconstruction, has been developed. We aimed to evaluate the feasibility of newly developed 3D capsule endoscopy in clinical practice. This study was a prospective, single-arm, feasibility study conducted at two university-affiliated hospitals in South Korea. Small bowel evaluation was performed using a newly developed 3D capsule endoscope for patients with obscure gastrointestinal bleeding, suspected or established Crohn's disease, small bowel tumors, and abdominal pain of unknown origin. We assessed the technical limitations, performance, and safety of the new capsule endoscope. Thirty-one patients (20 men and 11 women; mean age: 44.5 years) were enrolled. There was no technical defect preventing adequate visualization of the small bowel. The overall completion rate was 77.4%, the detection rate was 64.5%, and there was no capsule retention. All capsule endoscopic procedures were completed uneventfully. In conclusion, newly developed 3D capsule endoscopy was safe and feasible, showing similar performance as conventional capsule endoscopy. Newly added features of 3D reconstruction and size measurement are expected to be useful in the characterization of subepithelial tumours.

Multi-Cue-Based Circle Detection and Its Application to Robust Extrinsic Calibration of RGB-D Cameras.

RGB-Depth (RGB-D) cameras are widely used in computer vision and robotics applications such as 3D modeling and human⁻computer interaction. To capture 3D information of an object from different viewpoints simultaneously, we need to use multiple RGB-D cameras. To minimize costs, the cameras are often sparsely distributed without shared scene features. Due to the advantage of being visible from different viewpoints, spherical objects have been used for extrinsic calibration of widely-separated cameras. Assuming that the projected shape of the spherical object is circular, this paper presents a multi-cue-based method for detecting circular regions in a single color image. Experimental comparisons with existing methods show that our proposed method accurately detects spherical objects with cluttered backgrounds under different illumination conditions. The circle detection method is then applied to extrinsic calibration of multiple RGB-D cameras, for which we propose to use robust cost functions to reduce errors due to misdetected sphere centers. Through experiments, we show that the proposed method provides accurate calibration results in the presence of outliers and performs better than a least-squares-based method.

Recent Development of Computer Vision Technology to Improve Capsule Endoscopy.

Capsule endoscopy (CE) is a preferred diagnostic method for analyzing small bowel diseases. However, capsule endoscopes capture a sparse number of images because of their mechanical limitations. Post-procedural management using computational methods can enhance image quality. Additional information, including depth, can be obtained by using recently developed computer vision techniques. It is possible to measure the size of lesions and track the trajectory of capsule endoscopes using the computer vision technology, without requiring additional equipment. Moreover, the computational analysis of CE images can help detect lesions more accurately within a shorter time. Newly introduced deep leaning-based methods have shown more remarkable results over traditional computerized approaches. A large-scale standard dataset should be prepared to develop an optimal algorithms for improving the diagnostic yield of CE. The close collaboration between information technology and medical professionals is needed.

Application of Artificial Intelligence in Capsule Endoscopy: Where Are We Now?

Unlike wired endoscopy, capsule endoscopy requires additional time for a clinical specialist to review the operation and examine the lesions. To reduce the tedious review time and increase the accuracy of medical examinations, various approaches have been reported based on artificial intelligence for computer-aided diagnosis. Recently, deep learning-based approaches have been applied to many possible areas, showing greatly improved performance, especially for image-based recognition and classification. By reviewing recent deep learning-based approaches for clinical applications, we present the current status and future direction of artificial intelligence for capsule endoscopy.

Difference-Based Image Noise Modeling Using Skellam Distribution.

By the laws of quantum physics, pixel intensity does not have a true value, but should be a random variable. Contrary to the conventional assumptions, the distribution of intensity may not be an additive Gaussian. We propose to directly model the intensity difference and show its validity by an experimental comparison to the conventional additive model. As a model of the intensity difference, we present a Skellam distribution derived from the Poisson photon noise model. This modeling induces a linear relationship between intensity and Skellam parameters, while conventional variance computation methods do not yield any significant relationship between these parameters under natural illumination. The intensity-Skellam line is invariant to scene, illumination, and even most of camera parameters. We also propose practical methods to obtain the line using a color pattern and an arbitrary image under natural illumination. Because the Skellam parameters that can be obtained from this linearity determine a noise distribution for each intensity value, we can statistically determine whether any intensity difference is caused by an underlying signal difference or by noise. We demonstrate the effectiveness of this new noise model by applying it to practical applications of background subtraction and edge detection.