Fine-Grained Image Analysis With Deep Learning: A Survey.

Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas - fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community.

Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges.

In he past decade, object detection has achieved significant progress in natural images but not in aerial images, due to the massive variations in the scale and orientation of objects caused by the bird's-eye view of aerial images. More importantly, the lack of large-scale benchmarks has become a major obstacle to the development of object detection in aerial images (ODAI). In this paper, we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI. The proposed DOTA dataset contains 1,793,658 object instances of 18 categories of oriented-bounding-box annotations collected from 11,268 aerial images. Based on this large-scale and well-annotated dataset, we build baselines covering 10 state-of-the-art algorithms with over 70 configurations, where the speed and accuracy performances of each model have been evaluated. Furthermore, we provide a code library for ODAI and build a website for evaluating different algorithms. Previous challenges run on DOTA have attracted more than 1300 teams worldwide. We believe that the expanded large-scale DOTA dataset, the extensive baselines, the code library and the challenges can facilitate the designs of robust algorithms and reproducible research on the problem of object detection in aerial images.

The Herbarium 2021 Half-Earth Challenge Dataset and Machine Learning Competition.

Herbarium sheets present a unique view of the world's botanical history, evolution, and biodiversity. This makes them an all-important data source for botanical research. With the increased digitization of herbaria worldwide and advances in the domain of fine-grained visual classification which can facilitate automatic identification of herbarium specimen images, there are many opportunities for supporting and expanding research in this field. However, existing datasets are either too small, or not diverse enough, in terms of represented taxa, geographic distribution, and imaging protocols. Furthermore, aggregating datasets is difficult as taxa are recognized under a multitude of names and must be aligned to a common reference. We introduce the Herbarium 2021 Half-Earth dataset: the largest and most diverse dataset of herbarium specimen images, to date, for automatic taxon recognition. We also present the results of the Herbarium 2021 Half-Earth challenge, a competition that was part of the Eighth Workshop on Fine-Grained Visual Categorization (FGVC8) and hosted by Kaggle to encourage the development of models to automatically identify taxa from herbarium sheet images.

The Plant Pathology Challenge 2020 data set to classify foliar disease of apples.

PREMISE: Apple orchards in the United States are under constant threat from a large number of pathogens and insects. Appropriate and timely deployment of disease management depends on early disease detection. Incorrect and delayed diagnosis can result in either excessive or inadequate use of chemicals, with increased production costs and increased environmental and health impacts. METHODS AND RESULTS: We have manually captured 3651 high-quality, real-life symptom images of multiple apple foliar diseases, with variable illumination, angles, surfaces, and noise. A subset of images, expert-annotated to create a pilot data set for apple scab, cedar apple rust, and healthy leaves, was made available to the Kaggle community for the Plant Pathology Challenge as part of the Fine-Grained Visual Categorization (FGVC) workshop at the 2020 Computer Vision and Pattern Recognition conference (CVPR 2020). Participants were asked to use the image data set to train a machine learning model to classify disease categories and develop an algorithm for disease severity quantification. The top three area under the ROC curve (AUC) values submitted to the private leaderboard were 0.98445, 0.98182, and 0.98089. We also trained an off-the-shelf convolutional neural network on this data for disease classification and achieved 97% accuracy on a held-out test set. DISCUSSION: This data set will contribute toward development and deployment of machine learning-based automated plant disease classification algorithms to ultimately realize fast and accurate disease detection. We will continue to add images to the pilot data set for a larger, more comprehensive expert-annotated data set for future Kaggle competitions and to explore more advanced methods for disease classification and quantification.

An algorithm competition for automatic species identification from herbarium specimens.

PREMISE: Plant biodiversity is threatened, yet many species remain undescribed. It is estimated that >50% of undescribed species have already been collected and are awaiting discovery in herbaria. Robust automatic species identification algorithms using machine learning could accelerate species discovery. METHODS: To encourage the development of an automatic species identification algorithm, we submitted our Herbarium 2019 data set to the Fine-Grained Visual Categorization sub-competition (FGVC6) hosted on the Kaggle platform. We chose to focus on the flowering plant family Melastomataceae because we have a large collection of imaged herbarium specimens (46,469 specimens representing 683 species) and taxonomic expertise in the family. As is common for herbarium collections, some species in this data set are represented by few specimens and others by many. RESULTS: In less than three months, the FGVC6 Herbarium 2019 Challenge drew 22 teams who entered 254 models for Melastomataceae species identification. The four best algorithms identified species with >88% accuracy. DISCUSSION: The FGVC competitions provide a unique opportunity for computer vision and machine learning experts to address difficult species-recognition problems. The Herbarium 2019 Challenge brought together a novel combination of collections resources, taxonomic expertise, and collaboration between botanists and computer scientists.

Automatic coregistration of volumetric images based on implanted fiducial markers.

The accurate delivery of external beam radiation therapy is often facilitated through the implantation of radio-opaque fiducial markers (gold seeds). Before the delivery of each treatment fraction, seed positions can be determined via low dose volumetric imaging. By registering these seed locations with the corresponding locations in the previously acquired treatment planning computed tomographic (CT) scan, it is possible to adjust the patient position so that seed displacement is accommodated. The authors present an unsupervised automatic algorithm that identifies seeds in both planning and pretreatment images and subsequently determines a rigid geometric transformation between the two sets. The algorithm is applied to the imaging series of ten prostate cancer patients. Each test series is comprised of a single multislice planning CT and multiple megavoltage conebeam (MVCB) images. Each MVCB dataset is obtained immediately prior to a subsequent treatment session. Seed locations were determined to within 1 mm with an accuracy of 97 +/- 6.1% for datasets obtained by application of a mean imaging dose of 3.5 cGy per study. False positives occurred in three separate instances, but only when datasets were obtained at imaging doses too low to enable fiducial resolution by a human operator, or when the prostate gland had undergone large displacement or significant deformation. The registration procedure requires under nine seconds of computation time on a typical contemporary computer workstation.

Yum-Me: A Personalized Nutrient-Based Meal Recommender System.

Nutrient-based meal recommendations have the potential to help individuals prevent or manage conditions such as diabetes and obesity. However, learning people's food preferences and making recommendations that simultaneously appeal to their palate and satisfy nutritional expectations are challenging. Existing approaches either only learn high-level preferences or require a prolonged learning period. We propose Yum-me, a personalized nutrient-based meal recommender system designed to meet individuals' nutritional expectations, dietary restrictions, and fine-grained food preferences. Yum-me enables a simple and accurate food preference profiling procedure via a visual quiz-based user interface and projects the learned profile into the domain of nutritionally appropriate food options to find ones that will appeal to the user. We present the design and implementation of Yum-me and further describe and evaluate two innovative contributions. The first contriution is an open source state-of-the-art food image analysis model, named FoodDist. We demonstrate FoodDist's superior performance through careful benchmarking and discuss its applicability across a wide array of dietary applications. The second contribution is a novel online learning framework that learns food preference from itemwise and pairwise image comparisons. We evaluate the framework in a field study of 227 anonymous users and demonstrate that it outperforms other baselines by a significant margin. We further conducted an end-to-end validation of the feasibility and effectiveness of Yum-me through a 60-person user study, in which Yum-me improves the recommendation acceptance rate by 42.63%.

Framework for parsing, visualizing and scoring tissue microarray images.

Increasingly automated techniques for arraying, immunostaining, and imaging tissue sections led us to design software for convenient management, display, and scoring. Demand for molecular marker data derived in situ from tissue has driven histology informatics automation to the point where one can envision the computer, rather than the microscope, as the primary viewing platform for histopathological scoring and diagnoses. Tissue microarrays (TMAs), with hundreds or even thousands of patients' tissue sections on each slide, were the first step in this wave of automation. Via TMAs, increasingly rapid identification of the molecular patterns of cancer that define distinct clinical outcome groups among patients has become possible. TMAs have moved the bottleneck of acquiring molecular pattern information away from sampling and processing the tissues to the tasks of scoring and results analyses. The need to read large numbers of new slides, primarily for research purposes, is driving continuing advances in commercially available automated microscopy instruments that already do or soon will automatically image hundreds of slides per day. We reviewed strategies for acquiring, collating, and storing histological images with the goal of streamlining subsequent data analyses. As a result of this work, we report an implementation of software for automated preprocessing, organization, storage, and display of high resolution composite TMA images.

Efficient shape matching using shape contexts.

We demonstrate that shape contexts can be used to quickly prune a search for similar shapes. We present two algorithms for rapid shape retrieval: representative shape contexts, performing comparisons based on a small number of shape contexts, and shapemes, using vector quantization in the space of shape contexts to obtain prototypical shape pieces.

Three-dimensional sperm surface reconstruction: a novel approach to assessing sperm morphology.

OBJECTIVE: To create a rapid, inexpensive, efficient, and reproducible real-time three-dimensional (3-D) analysis of viable spermatozoa. Previous studies have demonstrated that abnormal semen profiles are associated with a modest increase in the frequency of sperm chromosomal abnormalities, and that sperm with aberrations in the shape and contours of the head may be carriers of chromatinic defects. Although high-power magnification and enhanced video-generated magnification have been suggested, these techniques are inherently limited by the clarity of the image, the time required for the analysis, and the risk of variable head-positioning during imaging. DESIGN: In vitro experiment. SETTING: University-affiliated infertility research laboratory. PATIENT(S): Anonymous sperm donors. INTERVENTION(S): Individual motile sperm were identified, analyzed at ×600 magnification, and a 10-second digital video was obtained. MAIN OUTCOME MEASURE(S): Image-tracking software captured serial photographs of sperm from recorded videos. Images were automatically extracted from each video frame using enhanced correlation coefficient maximization; the general shape of the sperm was extracted via space-carving. The reconstructed image was rotated to permit viewing from any direction, and the final image was rendered through interpolation. RESULT(S): This technique yielded images that enable noninvasive, 3-D, real-time, in vitro assessment of sperm surface morphology. CONCLUSION(S): This proof-of-principle demonstrates that by keeping spermatozoa in a fluid environment, a 3-D sperm-surface reconstruction can be created. This technique can be automated, requires minimal computing power, and utilizes equipment already available in most embryology laboratories.

Normalized cuts in 3-D for spinal MRI segmentation.

Segmentation of medical images has become an indispensable process to perform quantitative analysis of images of human organs and their functions. Normalized Cuts (NCut) is a spectral graph theoretic method that readily admits combinations of different features for image segmentation. The computational demand imposed by NCut has been successfully alleviated with the Nyström approximation method for applications different than medical imaging. In this paper we discuss the application of NCut with the Nyström approximation method to segment vertebral bodies from sagittal T1-weighted magnetic resonance images of the spine. The magnetic resonance images were preprocessed by the anisotropic diffusion algorithm, and three-dimensional local histograms of brightness was chosen as the segmentation feature. Results of the segmentation as well as limitations and challenges in this area are presented.

Spectral grouping using the Nyström method.

Spectral graph theoretic methods have recently shown great promise for the problem of image segmentation. However, due to the computational demands of these approaches, applications to large problems such as spatiotemporal data and high resolution imagery have been slow to appear. The contribution of this paper is a method that substantially reduces the computational requirements of grouping algorithms based on spectral partitioning making it feasible to apply them to very large grouping problems. Our approach is based on a technique for the numerical solution of eigenfunction problems known as the Nyström method. This method allows one to extrapolate the complete grouping solution using only a small number of samples. In doing so, we leverage the fact that there are far fewer coherent groups in a scene than pixels.

Fast Feature Pyramids for Object Detection.

Multi-resolution image features may be approximated via extrapolation from nearby scales, rather than being computed explicitly. This fundamental insight allows us to design object detection algorithms that are as accurate, and considerably faster, than the state-of-the-art. The computational bottleneck of many modern detectors is the computation of features at every scale of a finely-sampled image pyramid. Our key insight is that one may compute finely sampled feature pyramids at a fraction of the cost, without sacrificing performance: for a broad family of features we find that features computed at octave-spaced scale intervals are sufficient to approximate features on a finely-sampled pyramid. Extrapolation is inexpensive as compared to direct feature computation. As a result, our approximation yields considerable speedups with negligible loss in detection accuracy. We modify three diverse visual recognition systems to use fast feature pyramids and show results on both pedestrian detection (measured on the Caltech, INRIA, TUD-Brussels and ETH data sets) and general object detection (measured on the PASCAL VOC). The approach is general and is widely applicable to vision algorithms requiring fine-grained multi-scale analysis. Our approximation is valid for images with broad spectra (most natural images) and fails for images with narrow band-pass spectra (e.g., periodic textures).

Regression and classification methods for nasolabial folds: a possible paradigm for computer-aided diagnosis of skin diseases.

Classification of facial features, for example, nasolabial folds, still relies mainly on clinical assessment, resulting in significant costs because of high intra- and interrater variability. Further, diagnosing skin diseases, for example, malignant melanoma, also can present challenges. In an attempt to reduce cost of medical care in future, we determined the utility of methods in image processing and statistical analysis to automatically quantify, for example, the structure of nasolabial folds. To the best of our knowledge, this is the first report of the application of computer technology to grading of nasolabial folds. When classifying severity of wrinkles on a scale of 1-5, the computer achieved an accuracy of 87% compared to the dermatologist, taken as the gold standard. Further, the computer program's capacity to sort the order of wrinkles from least to most wrinkled was 98% as accurate as the clinician(s). We conclude that by using computer technology, nasolabial folds can be categorized almost as accurately as by using grading by dermatologists, suggesting that computer technology may be a useful tool to grade nasolabial folds because a computer is always consistent. We hypothesize that, after additional studies, this technology also may be a useful tool to aid in diagnosing skin diseases.

Robust Object Tracking with Online Multiple Instance Learning.

In this paper, we address the problem of tracking an object in a video given its location in the first frame and no other information. Recently, a class of tracking techniques called "tracking by detection" has been shown to give promising results at real-time speeds. These methods train a discriminative classifier in an online manner to separate the object from the background. This classifier bootstraps itself by using the current tracker state to extract positive and negative examples from the current frame. Slight inaccuracies in the tracker can therefore lead to incorrectly labeled training examples, which degrade the classifier and can cause drift. In this paper, we show that using Multiple Instance Learning (MIL) instead of traditional supervised learning avoids these problems and can therefore lead to a more robust tracker with fewer parameter tweaks. We propose a novel online MIL algorithm for object tracking that achieves superior results with real-time performance. We present thorough experimental results (both qualitative and quantitative) on a number of challenging video clips.