Enhancing current guidance for psoriatic arthritis and its comorbidities: recommendations from an expert consensus panel.

OBJECTIVES: Existing guidelines for psoriatic arthritis (PsA) cover many aspects of management. Some gaps remain relating to routine practice application. An expert group aimed to enhance current guidance and develop recommendations for clinical practice that are complementary to existing guidelines. METHODS: A steering committee comprising experienced, research-active clinicians in rheumatology, dermatology and primary care agreed on themes and relevant questions. A targeted literature review of PubMed and Embase following a PICO framework was conducted. At a second meeting, recommendations were drafted and subsequently an extended faculty comprising rheumatologists, dermatologists, primary care clinicians, specialist nurses, allied health professionals, non-clinical academic participants and members of the Brit-PACT patient group, was recruited. Consensus was achieved via an online voting platform when 75% of respondents agreed in the range of 7-9 on a 9-point scale. RESULTS: The guidance comprised 34 statements covering four PsA themes. Diagnosis focused on strategies to identify PsA early and refer appropriately, assessment of diagnostic indicators, use of screening tools and use of imaging. Disease assessment centred on holistic consideration of disease activity, physical functioning and impact from a patient perspective, and on how to implement shared decision-making. For comorbidities, recommendations included specific guidance for high-impact conditions such as depression and obesity. Management statements (which excluded extant guidance on pharmacological therapies) covered multidisciplinary team working, implementation of lifestyle modifications and treat-to-target strategies. Minimising corticosteroid use was recommended where feasible. CONCLUSION: The consensus group have made evidence-based best practice recommendations for the management of PsA to enhance the existing guidelines.

Understanding Children's Attention to Dental Caries through Eye-Tracking.

Visual attention is a significant gateway to a child's mind, and looking is one of the first behaviors young children develop. Untreated caries and the resulting poor dental aesthetics can have adverse emotional and social impacts on children's oral health-related quality of life due to its detrimental effects on self-esteem and self-concept. Therefore, we explored preschool children's eye movement patterns and visual attention to images with and without dental caries via eye movement analysis using hidden Markov models (EMHMM). We calibrated a convenience sample of 157 preschool children to the eye-tracker (Tobii Nano Pro) to ensure standardization. Consequently, each participant viewed the same standardized pictures with and without dental caries while an eye-tracking device tracked their eye movements. Subsequently, based on the sequence of viewed regions of interest (ROIs), a transition matrix was developed where the participants' previously viewed ROI informed their subsequently considered ROI. Hence, an individual's HMM was estimated from their eye movement data using a variational Bayesian approach to determine the optimal number of ROIs automatically. Consequently, this data-driven approach generated the visual task participants' most representative eye movement patterns. Preschool children exhibited two different eye movement patterns, distributed (78%) and selective (21%), which was statistically significant. Children switched between images with more similar probabilities in the distributed pattern while children remained looking at the same ROI than switching to the other ROI in the selective pattern. Nevertheless, all children exhibited an equal starting fixation on the right or left image and noticed teeth. The study findings reveal that most preschool children did not have an attentional bias to images with and without dental caries. Furthermore, only a few children selectively fixated on images with dental caries. Therefore, selective eye-movement patterns may strongly predict preschool children's sustained visual attention to dental caries. Nevertheless, future studies are essential to fully understand the developmental origins of differences in visual attention to common oral health presentations in children. Finally, EMHMM is appropriate for assessing inter-individual differences in children's visual attention.

Understanding children's attention to traumatic dental injuries using eye-tracking.

BACKGROUND/AIM: Traumatic dental injuries (TDIs) in the primary dentition may result in tooth discolouration and fractures. The aim of this child-centred study was to explore the differences between preschool children's eye movement patterns and visual attention to typical outcomes following TDIs to primary teeth. MATERIALS AND METHODS: An eye-tracker recorded 155 healthy preschool children's eye movements when they viewed clinical images of healthy teeth, tooth fractures and discolourations. The visual search pattern was analysed using the eye movement analysis with the Hidden Markov Models (EMHMM) approach and preference for the various regions of interest (ROIs). RESULTS: Two different eye movement patterns (distributed and selective) were identified (p < .05). Children with the distributed pattern shifted their fixations between the presented images, while those with the selective pattern remained focused on the same image they first saw. CONCLUSIONS: Preschool children noticed teeth. However, most of them did not have an attentional bias, implying that they did not interpret these TDI outcomes negatively. Only a few children avoided looking at images with TDIs indicating a potential negative impact. The EMHMM approach is appropriate for assessing inter-individual differences in children's visual attention to TDI outcomes.

Eye movement analysis with hidden Markov models (EMHMM) with co-clustering.

The eye movement analysis with hidden Markov models (EMHMM) method provides quantitative measures of individual differences in eye-movement pattern. However, it is limited to tasks where stimuli have the same feature layout (e.g., faces). Here we proposed to combine EMHMM with the data mining technique co-clustering to discover participant groups with consistent eye-movement patterns across stimuli for tasks involving stimuli with different feature layouts. Through applying this method to eye movements in scene perception, we discovered explorative (switching between the foreground and background information or different regions of interest) and focused (mainly looking at the foreground with less switching) eye-movement patterns among Asian participants. Higher similarity to the explorative pattern predicted better foreground object recognition performance, whereas higher similarity to the focused pattern was associated with better feature integration in the flanker task. These results have important implications for using eye tracking as a window into individual differences in cognitive abilities and styles. Thus, EMHMM with co-clustering provides quantitative assessments on eye-movement patterns across stimuli and tasks. It can be applied to many other real-life visual tasks, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.

Understanding visual attention to face emotions in social anxiety using hidden Markov models.

Theoretical models propose that attentional biases might account for the maintenance of social anxiety symptoms. However, previous eye-tracking studies have yielded mixed results. One explanation is that existing studies quantify eye-movements using arbitrary, experimenter-defined criteria such as time segments and regions of interests that do not capture the dynamic nature of overt visual attention. The current study adopted the Eye Movement analysis with Hidden Markov Models (EMHMM) approach for eye-movement analysis, a machine-learning, data-driven approach that can cluster people's eye-movements into different strategy groups. Sixty participants high and low in self-reported social anxiety symptoms viewed angry and neutral faces in a free-viewing task while their eye-movements were recorded. EMHMM analyses revealed novel associations between eye-movement patterns and social anxiety symptoms that were not evident with standard analytical approaches. Participants who adopted the same face-viewing strategy when viewing both angry and neutral faces showed higher social anxiety symptoms than those who transitioned between strategies when viewing angry versus neutral faces. EMHMM can offer novel insights into psychopathology-related attention processes.

Eye movement analysis with switching hidden Markov models.

Here we propose the eye movement analysis with switching hidden Markov model (EMSHMM) approach to analyzing eye movement data in cognitive tasks involving cognitive state changes. We used a switching hidden Markov model (SHMM) to capture a participant's cognitive state transitions during the task, with eye movement patterns during each cognitive state being summarized using a regular HMM. We applied EMSHMM to a face preference decision-making task with two pre-assumed cognitive states-exploration and preference-biased periods-and we discovered two common eye movement patterns through clustering the cognitive state transitions. One pattern showed both a later transition from the exploration to the preference-biased cognitive state and a stronger tendency to look at the preferred stimulus at the end, and was associated with higher decision inference accuracy at the end; the other pattern entered the preference-biased cognitive state earlier, leading to earlier above-chance inference accuracy in a trial but lower inference accuracy at the end. This finding was not revealed by any other method. As compared with our previous HMM method, which assumes no cognitive state change (i.e., EMHMM), EMSHMM captured eye movement behavior in the task better, resulting in higher decision inference accuracy. Thus, EMSHMM reveals and provides quantitative measures of individual differences in cognitive behavior/style, making a significant impact on the use of eyetracking to study cognitive behavior across disciplines.

Scanpath modeling and classification with hidden Markov models.

How people look at visual information reveals fundamental information about them; their interests and their states of mind. Previous studies showed that scanpath, i.e., the sequence of eye movements made by an observer exploring a visual stimulus, can be used to infer observer-related (e.g., task at hand) and stimuli-related (e.g., image semantic category) information. However, eye movements are complex signals and many of these studies rely on limited gaze descriptors and bespoke datasets. Here, we provide a turnkey method for scanpath modeling and classification. This method relies on variational hidden Markov models (HMMs) and discriminant analysis (DA). HMMs encapsulate the dynamic and individualistic dimensions of gaze behavior, allowing DA to capture systematic patterns diagnostic of a given class of observers and/or stimuli. We test our approach on two very different datasets. Firstly, we use fixations recorded while viewing 800 static natural scene images, and infer an observer-related characteristic: the task at hand. We achieve an average of 55.9% correct classification rate (chance = 33%). We show that correct classification rates positively correlate with the number of salient regions present in the stimuli. Secondly, we use eye positions recorded while viewing 15 conversational videos, and infer a stimulus-related characteristic: the presence or absence of original soundtrack. We achieve an average 81.2% correct classification rate (chance = 50%). HMMs allow to integrate bottom-up, top-down, and oculomotor influences into a single model of gaze behavior. This synergistic approach between behavior and machine learning will open new avenues for simple quantification of gazing behavior. We release SMAC with HMM, a Matlab toolbox freely available to the community under an open-source license agreement.

Silencing or inhibition of endoplasmic reticulum aminopeptidase 1 (ERAP1) suppresses free heavy chain expression and Th17 responses in ankylosing spondylitis.

OBJECTIVE: Human leucocyte antigen (HLA)-B27 and endoplasmic reticulum aminopeptidase 1 (ERAP1) are strongly associated with ankylosing spondylitis (AS). ERAP1 is a key aminopeptidase in HLA class I presentation and can potentially alter surface expression of HLA-B27 free heavy chains (FHCs). We studied the effects of ERAP1 silencing/inhibition/variations on HLA-B27 FHC expression and Th17 responses in AS. METHODS: Flow cytometry was used to measure surface expression of HLA class I in peripheral blood mononuclear cells (PBMCs) from patients with AS carrying different ERAP1 genotypes (rs2287987, rs30187 and rs27044) and in ERAP1-silenced/inhibited/mutated HLA-B27-expressing antigen presenting cells (APCs). ERAP1-silenced/inhibited APCs were cocultured with KIR3DL2CD3ε-reporter cells or AS CD4+ T cells. Th17 responses of AS CD4+ T cells were measured by interleukin (IL)-17A ELISA and Th17 intracellular cytokine staining. FHC cell surface expression and Th17 responses were also measured in AS PBMCs following ERAP1 inhibition. RESULTS: The AS-protective ERAP1 variants, K528R and Q730E, were associated with reduced surface FHC expression by monocytes from patients with AS and HLA-B27-expressing APCs. ERAP1 silencing or inhibition in APCs downregulated HLA-B27 FHC surface expression, reduced IL-2 production by KIR3DL2CD3ε-reporter cells and suppressed the Th17 expansion and IL-17A secretion by AS CD4+ T cells. ERAP1 inhibition of AS PBMCs reduced HLA class I FHC surface expression by monocytes and B cells, and suppressed Th17 expansion. CONCLUSIONS: ERAP1 activity determines surface expression of HLA-B27 FHCs and potentially promotes Th17 responses in AS through binding of HLA-B27 FHCs to KIR3DL2. Our data suggest that ERAP1 inhibition has potential for AS treatment.

Understanding eye movements in face recognition using hidden Markov models.

We use a hidden Markov model (HMM) based approach to analyze eye movement data in face recognition. HMMs are statistical models that are specialized in handling time-series data. We conducted a face recognition task with Asian participants, and model each participant's eye movement pattern with an HMM, which summarized the participant's scan paths in face recognition with both regions of interest and the transition probabilities among them. By clustering these HMMs, we showed that participants' eye movements could be categorized into holistic or analytic patterns, demonstrating significant individual differences even within the same culture. Participants with the analytic pattern had longer response times, but did not differ significantly in recognition accuracy from those with the holistic pattern. We also found that correct and wrong recognitions were associated with distinctive eye movement patterns; the difference between the two patterns lies in the transitions rather than locations of the fixations alone.

Generalized Characteristic Function Loss for Crowd Analysis in the Frequency Domain.

Typical approaches that learn crowd density maps are limited to extracting the supervisory information from the loosely organized spatial information in the crowd dot/density maps. This paper tackles this challenge by performing the supervision in the frequency domain. More specifically, we devise a new loss function for crowd analysis called generalized characteristic function loss (GCFL). This loss carries out two steps: 1) transforming the spatial information in density or dot maps to the frequency domain; 2) calculating a loss value between their frequency contents. For step 1, we establish a series of theoretical fundaments by extending the definition of the characteristic function for probability distributions to density maps, as well as proving some vital properties of the extended characteristic function. After taking the characteristic function of the density map, its information in the frequency domain is well-organized and hierarchically distributed, while in the spatial domain it is loose-organized and dispersed everywhere. In step 2, we design a loss function that can fit the information organization in the frequency domain, allowing the exploitation of the well-organized frequency information for the supervision of crowd analysis tasks. The loss function can be adapted to various crowd analysis tasks through the specification of its window functions. In this paper, we demonstrate its power in three tasks: Crowd Counting, Crowd Localization and Noisy Crowd Counting. We show the advantages of our GCFL compared to other SOTA losses and its competitiveness to other SOTA methods by theoretical analysis and empirical results on benchmark datasets. Our codes are available at https://github.com/wbshu/Crowd_Counting_in_the_Frequency_Domain.

Gradient-Based Instance-Specific Visual Explanations for Object Specification and Object Discrimination.

We propose the gradient-weighted Object Detector Activation Maps (ODAM), a visual explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, ODAM produces heat maps that show the influence of regions on the detector's decision for each predicted attribute. Compared to previous works on classification activation maps (CAM), ODAM generates instance-specific explanations rather than class-specific ones. We show that ODAM is applicable to one-stage, two-stage, and transformer-based detectors with different types of detector backbones and heads, and produces higher-quality visual explanations than the state-of-the-art in terms of both effectiveness and efficiency. We discuss two explanation tasks for object detection: 1) object specification: what is the important region for the prediction? 2) object discrimination: which object is detected? Aiming at these two aspects, we present a detailed analysis of the visual explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM. Furthermore, we investigate user trust on the explanation maps, how well the visual explanations of object detectors agrees with human explanations, as measured through human eye gaze, and whether this agreement is related with user trust. Finally, we also propose two applications, ODAM-KD and ODAM-NMS, based on these two abilities of ODAM. ODAM-KD utilizes the object specification of ODAM to generate top-down attention for key predictions and instruct the knowledge distillation of object detection. ODAM-NMS considers the location of the model's explanation for each prediction to distinguish the duplicate detected objects. A training scheme, ODAM-Train, is proposed to improve the quality on object discrimination, and help with ODAM-NMS.

Another Perspective of Over-Smoothing: Alleviating Semantic Over-Smoothing in Deep GNNs.

Graph neural networks (GNNs) are widely used for analyzing graph-structural data and solving graph-related tasks due to their powerful expressiveness. However, existing off-the-shelf GNN-based models usually consist of no more than three layers. Deeper GNNs usually suffer from severe performance degradation due to several issues including the infamous "over-smoothing" issue, which restricts the further development of GNNs. In this article, we investigate the over-smoothing issue in deep GNNs. We discover that over-smoothing not only results in indistinguishable embeddings of graph nodes, but also alters and even corrupts their semantic structures, dubbed semantic over-smoothing. Existing techniques, e.g., graph normalization, aim at handling the former concern, but neglect the importance of preserving the semantic structures in the spatial domain, which hinders the further improvement of model performance. To alleviate the concern, we propose a cluster-keeping sparse aggregation strategy to preserve the semantic structure of embeddings in deep GNNs (especially for spatial GNNs). Particularly, our strategy heuristically redistributes the extent of aggregations for all the nodes from layers, instead of aggregating them equally, so that it enables aggregate concise yet meaningful information for deep layers. Without any bells and whistles, it can be easily implemented as a plug-and-play structure of GNNs via weighted residual connections. Last, we analyze the over-smoothing issue on the GNNs with weighted residual structures and conduct experiments to demonstrate the performance comparable to the state-of-the-arts.

Human attention guided explainable artificial intelligence for computer vision models.

Explainable artificial intelligence (XAI) has been increasingly investigated to enhance the transparency of black-box artificial intelligence models, promoting better user understanding and trust. Developing an XAI that is faithful to models and plausible to users is both a necessity and a challenge. This work examines whether embedding human attention knowledge into saliency-based XAI methods for computer vision models could enhance their plausibility and faithfulness. Two novel XAI methods for object detection models, namely FullGrad-CAM and FullGrad-CAM++, were first developed to generate object-specific explanations by extending the current gradient-based XAI methods for image classification models. Using human attention as the objective plausibility measure, these methods achieve higher explanation plausibility. Interestingly, all current XAI methods when applied to object detection models generally produce saliency maps that are less faithful to the model than human attention maps from the same object detection task. Accordingly, human attention-guided XAI (HAG-XAI) was proposed to learn from human attention how to best combine explanatory information from the models to enhance explanation plausibility by using trainable activation functions and smoothing kernels to maximize the similarity between XAI saliency map and human attention map. The proposed XAI methods were evaluated on widely used BDD-100K, MS-COCO, and ImageNet datasets and compared with typical gradient-based and perturbation-based XAI methods. Results suggest that HAG-XAI enhanced explanation plausibility and user trust at the expense of faithfulness for image classification models, and it enhanced plausibility, faithfulness, and user trust simultaneously and outperformed existing state-of-the-art XAI methods for object detection models.

Granulocyte-macrophage colony-stimulating factor neutralisation in patients with axial spondyloarthritis in the UK (NAMASTE): a randomised, double-blind, placebo-controlled, phase 2 trial.

BACKGROUND: Granulocyte-macrophage colony-stimulating factor (GM-CSF) is a proinflammatory cytokine overproduced in several inflammatory and autoimmune diseases, including axial spondyloarthritis. Namilumab is a human IgG1 monoclonal anti-GM-CSF antibody that potently neutralises human GM-CSF. We aimed to assess the efficacy of namilumab in participants with moderate-to-severe active axial spondyloarthritis. METHODS: This proof-of-concept, randomised, double-blind, placebo-controlled, phase 2, Bayesian (NAMASTE) trial was done at nine hospitals in the UK. Participants aged 18-75 years with axial spondyloarthritis, meeting the Assessment in SpondyloArthritis international Society (ASAS) criteria and the ASAS-defined MRI criteria, with active disease as defined by a Bath Ankylosing Spondylitis Disease Activity Index (BASDAI), were eligible. Those who had inadequately responded or had intolerance to previous treatment with an anti-TNF agent were included. Participants were randomly assigned (6:1) to receive subcutaneous namilumab 150 mg or placebo at weeks 0, 2, 6, and 10. Participants, site staff (except pharmacy staff), and central study staff were masked to treatment assignment. The primary endpoint was the proportion of participants who had an ASAS ≥20% improvement (ASAS20) clinical response at week 12 in the full analysis set (all randomly assigned participants). This trial is registered with ClinicalTrials.gov (NCT03622658). FINDINGS: From Sept 6, 2018, to July 25, 2019, 60 patients with moderate-to-severe active axial spondyloarthritis were assessed for eligibility and 42 were randomly assigned to receive namilumab (n=36) or placebo (n=six). The mean age of participants was 39·5 years (SD 13·3), 17 were women, 25 were men, 39 were White, and seven had previously received anti-TNF therapy. The primary endpoint was not met. At week 12, the proportion of patients who had an ASAS20 clinical response was lower in the namilumab group (14 of 36) than in the placebo group (three of six; estimated between-group difference 6·8%). The Bayesian posterior probability η was 0·72 (>0·927 suggests high clinical significance). The rates of any treatment-emergent adverse events in the namilumab group were similar to those in the placebo group (31 vs five). INTERPRETATION: Namilumab did not show efficacy compared with placebo in patients with active axial spondyloarthritis, but the treatment was generally well tolerated. FUNDING: Izana Bioscience, NIHR Oxford Biomedical Research Centre (BRC), NIHR Birmingham BRC, and Clinical Research Facility.

Th17 cells expressing KIR3DL2+ and responsive to HLA-B27 homodimers are increased in ankylosing spondylitis.

CD4 Th cells producing the proinflammatory cytokine IL-17 (Th17) have been implicated in a number of inflammatory arthritides including the spondyloarthritides. Th17 development is promoted by IL-23. Ankylosing spondylitis, the most common spondyloarthritis (SpA), is genetically associated with both HLA-B27 (B27) and IL-23R polymorphisms; however, the link remains unexplained. We have previously shown that B27 can form H chain dimers (termed B27(2)), which, unlike classical HLA-B27, bind the killer-cell Ig-like receptor KIR3DL2. In this article, we show that B27(2)-expressing APCs stimulate the survival, proliferation, and IL-17 production of KIR3DL2(+) CD4 T cells. KIR3DL2(+) CD4 T cells are expanded and enriched for IL-17 production in the blood and synovial fluid of patients with SpA. Despite KIR3DL2(+) cells comprising a mean of just 15% of CD4 T in the peripheral blood of SpA patients, this subset accounted for 70% of the observed increase in Th17 numbers in SpA patients compared with control subjects. TCR-stimulated peripheral blood KIR3DL2(+) CD4 T cell lines from SpA patients secreted 4-fold more IL-17 than KIR3DL2(+) lines from controls or KIR3DL2(-) CD4 T cells. Strikingly, KIR3DL2(+) CD4 T cells account for the majority of peripheral blood CD4 T cell IL-23R expression and produce more IL-17 in the presence of IL-23. Our findings link HLA-B27 with IL-17 production and suggest new therapeutic strategies in ankylosing spondylitis/SpA.

Single-Frame-Based Deep View Synchronization for Unsynchronized Multicamera Surveillance.

Multicamera surveillance has been an active research topic for understanding and modeling scenes. Compared to a single camera, multicameras provide larger field-of-view and more object cues, and the related applications are multiview counting, multiview tracking, 3-D pose estimation or 3-D reconstruction, and so on. It is usually assumed that the cameras are all temporally synchronized when designing models for these multicamera-based tasks. However, this assumption is not always valid, especially for multicamera systems with network transmission delay and low frame rates due to limited network bandwidth, resulting in desynchronization of the captured frames across cameras. To handle the issue of unsynchronized multicameras, in this article, we propose a synchronization model that works in conjunction with existing deep neural network (DNN)-based multiview models, thus avoiding the redesign of the whole model. We consider two variants of the model, based on where in the pipeline the synchronization occurs, scene-level synchronization and camera-level synchronization. The view synchronization step and the task-specific view fusion and prediction step are unified in the same framework and trained in an end-to-end fashion. Our view synchronization models are applied to different DNNs-based multicamera vision tasks under the unsynchronized setting, including multiview counting and 3-D pose estimation, and achieve good performance compared to baselines.

Visual attention to own- versus other-race faces: Perspectives from learning mechanisms and task demands.

Multiple factors have been proposed to contribute to the other-race effect in face recognition, including perceptual expertise and social-cognitive accounts. Here, we propose to understand the effect and its contributing factors from the perspectives of learning mechanisms that involve joint learning of visual attention strategies and internal representations for faces, which can be modulated by quality of contact with other-race individuals including emotional and motivational factors. Computational simulations of this process will enhance our understanding of interactions among factors and help resolve inconsistent results in the literature. In particular, since learning is driven by task demands, visual attention effects observed in different face-processing tasks, such as passive viewing or recognition, are likely to be task specific (although may be associated) and should be examined and compared separately. When examining visual attention strategies, the use of more data-driven and comprehensive eye movement measures, taking both spatial-temporal pattern and consistency of eye movements into account, can lead to novel discoveries in other-race face processing. The proposed framework and analysis methods may be applied to other tasks of real-life significance such as face emotion recognition, further enhancing our understanding of the relationship between learning and visual cognition.

On Distinctive Image Captioning via Comparing and Reweighting.

Recent image captioning models are achieving impressive results based on popular metrics, i.e., BLEU, CIDEr, and SPICE. However, focusing on the most popular metrics that only consider the overlap between the generated captions and human annotation could result in using common words and phrases, which lacks distinctiveness, i.e., many similar images have the same caption. In this paper, we aim to improve the distinctiveness of image captions via comparing and reweighting with a set of similar images. First, we propose a distinctiveness metric-between-set CIDEr (CIDErBtw) to evaluate the distinctiveness of a caption with respect to those of similar images. Our metric reveals that the human annotations of each image in the MSCOCO dataset are not equivalent based on distinctiveness; however, previous works normally treat the human annotations equally during training, which could be a reason for generating less distinctive captions. In contrast, we reweight each ground-truth caption according to its distinctiveness during training. We further integrate a long-tailed weight strategy to highlight the rare words that contain more information, and captions from the similar image set are sampled as negative examples to encourage the generated sentence to be unique. Finally, extensive experiments are conducted, showing that our proposed approach significantly improves both distinctiveness (as measured by CIDErBtw and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide variety of image captioning baselines. These results are further confirmed through a user study.

Modeling Noisy Annotations for Point-Wise Supervision.

Point-wise supervision is widely adopted in computer vision tasks such as crowd counting and human pose estimation. In practice, the noise in point annotations may affect the performance and robustness of algorithm significantly. In this paper, we investigate the effect of annotation noise in point-wise supervision and propose a series of robust loss functions for different tasks. In particular, the point annotation noise includes spatial-shift noise, missing-point noise, and duplicate-point noise. The spatial-shift noise is the most common one, and exists in crowd counting, pose estimation, visual tracking, etc, while the missing-point and duplicate-point noises usually appear in dense annotations, such as crowd counting. In this paper, we first consider the shift noise by modeling the real locations as random variables and the annotated points as noisy observations. The probability density function of the intermediate representation (a smooth heat map generated from dot annotations) is derived and the negative log likelihood is used as the loss function to naturally model the shift uncertainty in the intermediate representation. The missing and duplicate noise are further modeled by an empirical way with the assumption that the noise appears at high density region with a high probability. We apply the method to crowd counting, human pose estimation and visual tracking, propose robust loss functions for those tasks, and achieve superior performance and robustness on widely used datasets.