Gradient-Based Instance-Specific Visual Explanations for Object Specification and Object Discrimination.

We propose the gradient-weighted Object Detector Activation Maps (ODAM), a visual explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, ODAM produces heat maps that show the influence of regions on the detector's decision for each predicted attribute. Compared to previous works on classification activation maps (CAM), ODAM generates instance-specific explanations rather than class-specific ones. We show that ODAM is applicable to one-stage, two-stage, and transformer-based detectors with different types of detector backbones and heads, and produces higher-quality visual explanations than the state-of-the-art in terms of both effectiveness and efficiency. We discuss two explanation tasks for object detection: 1) object specification: what is the important region for the prediction? 2) object discrimination: which object is detected? Aiming at these two aspects, we present a detailed analysis of the visual explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM. Furthermore, we investigate user trust on the explanation maps, how well the visual explanations of object detectors agrees with human explanations, as measured through human eye gaze, and whether this agreement is related with user trust. Finally, we also propose two applications, ODAM-KD and ODAM-NMS, based on these two abilities of ODAM. ODAM-KD utilizes the object specification of ODAM to generate top-down attention for key predictions and instruct the knowledge distillation of object detection. ODAM-NMS considers the location of the model's explanation for each prediction to distinguish the duplicate detected objects. A training scheme, ODAM-Train, is proposed to improve the quality on object discrimination, and help with ODAM-NMS. The code of ODAM is available: https://github.com/Cyang-Zhao/ODAM.

Visual attention to own- versus other-race faces: Perspectives from learning mechanisms and task demands.

Multiple factors have been proposed to contribute to the other-race effect in face recognition, including perceptual expertise and social-cognitive accounts. Here, we propose to understand the effect and its contributing factors from the perspectives of learning mechanisms that involve joint learning of visual attention strategies and internal representations for faces, which can be modulated by quality of contact with other-race individuals including emotional and motivational factors. Computational simulations of this process will enhance our understanding of interactions among factors and help resolve inconsistent results in the literature. In particular, since learning is driven by task demands, visual attention effects observed in different face-processing tasks, such as passive viewing or recognition, are likely to be task specific (although may be associated) and should be examined and compared separately. When examining visual attention strategies, the use of more data-driven and comprehensive eye movement measures, taking both spatial-temporal pattern and consistency of eye movements into account, can lead to novel discoveries in other-race face processing. The proposed framework and analysis methods may be applied to other tasks of real-life significance such as face emotion recognition, further enhancing our understanding of the relationship between learning and visual cognition.

Studying the effect of self-selected background music on reading task with eye movements.

Using background music (BGM) during learning is a common behavior, yet whether BGM can facilitate or hinder learning remains inconclusive and the underlying mechanism is largely an open question. This study aims to elucidate the effect of self-selected BGM on reading task for learners with different characteristics. Particularly, learners' reading task performance, metacognition, and eye movements were examined, in relation to their personal traits including language proficiency, working memory capacity, music experience and personality. Data were collected from a between-subject experiment with 100 non-native English speakers who were randomly assigned into two groups. Those in the experimental group read English passages with music of their own choice played in the background, while those in the control group performed the same task in silence. Results showed no salient differences on passage comprehension accuracy or metacognition between the two groups. Comparisons on fine-grained eye movement measures reveal that BGM imposed heavier cognitive load on post-lexical processes but not on lexical processes. It was also revealed that students with higher English proficiency level or more frequent BGM usage in daily self-learning/reading experienced less cognitive load when reading with their BGM, whereas students with higher working memory capacity (WMC) invested more mental effort than those with lower WMC in the BGM condition. These findings further scientific understanding of how BGM interacts with cognitive tasks in the foreground, and provide practical guidance for learners and learning environment designers on making the most of BGM for instruction and learning.

On becoming socially anxious: Toddlers' attention bias to fearful faces.

Early attention bias to threat-related negative emotions may lead children to overestimate dangers in social situations. This study examined its emergence and how it might develop in tandem with a known predictor namely temperamental shyness for toddlers' fear of strangers in 168 Chinese toddlers. Measurable individual differences in such attention bias to fearful faces were found and remained stable from age 12 to 18 months. When shown photos of paired happy versus fearful or happy versus angry faces, toddlers initially gazed more and had longer initial fixation and total fixation at fearful faces compared with happy faces consistently. However, they initially gazed more at happy faces compared with angry faces consistently and had a longer total fixation at angry faces only at 18 months. Stranger anxiety at 12 months predicted attention bias to fearful faces at 18 months. Temperamentally shyer 12-month-olds went on to show stronger attention bias to fearful faces at 18 months, and their fear of strangers also increased more from 12 to 18 months. Together with prior research suggesting attention bias to angry or fearful faces foretelling social anxiety, the present findings point to likely positive feedback loops among attention bias to fearful faces, temperamental shyness, and stranger anxiety in early childhood. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Clustering Hidden Markov Models With Variational Bayesian Hierarchical EM.

The hidden Markov model (HMM) is a broadly applied generative model for representing time-series data, and clustering HMMs attract increased interest from machine learning researchers. However, the number of clusters ( K ) and the number of hidden states ( S ) for cluster centers are still difficult to determine. In this article, we propose a novel HMM-based clustering algorithm, the variational Bayesian hierarchical EM algorithm, which clusters HMMs through their densities and priors and simultaneously learns posteriors for the novel HMM cluster centers that compactly represent the structure of each cluster. The numbers K and S are automatically determined in two ways. First, we place a prior on the pair (K,S) and approximate their posterior probabilities, from which the values with the maximum posterior are selected. Second, some clusters and states are pruned out implicitly when no data samples are assigned to them, thereby leading to automatic selection of the model complexity. Experiments on synthetic and real data demonstrate that our algorithm performs better than using model selection techniques with maximum likelihood estimation.

Differential audiovisual information processing in emotion recognition: An eye-tracking study.

Recent research has suggested that dynamic emotion recognition involves strong audiovisual association; that is, facial or vocal information alone automatically induces perceptual processes in the other modality. We hypothesized that different emotions may differ in the automaticity of audiovisual association, resulting in differential audiovisual information processing. Participants judged the emotion of a talking-head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over the unimodal conditions in recognition accuracy. In addition, in the recognition of all the emotions except for disgust, participants' eye-movement patterns did not change significantly across the three conditions, suggesting mandatory audiovisual information processing. In contrast, in disgust recognition, participants' eye movements in the audiovisual condition were less eyes-focused than the video-only condition and more eyes-focused than the audio-only condition, suggesting that audio information in the audiovisual condition interfered with eye-movement planning for important features (eyes) for disgust. In addition, those whose eye-movement pattern was affected less by concurrent disgusted voice information benefited more in recognition accuracy. Disgust recognition is learned later in life and thus may involve a reduced amount of audiovisual associative learning. Consequently, audiovisual association in disgust recognition is less automatic and demands more attentional resources than other emotions. Thus, audiovisual information processing in emotion recognition depends on the automaticity of audiovisual association of the emotion resulting from associative learning. This finding has important implications for real-life emotion recognition and multimodal learning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Understanding the role of eye movement consistency in face recognition and autism through integrating deep neural networks and hidden Markov models.

Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children's face recognition involves visual routine development through social exposure, indexed by eye movement consistency.

How does face mask in COVID-19 pandemic disrupt face learning and recognition in adults with autism spectrum disorder?

Use of face masks is one of the measures adopted by the general community to stop the transmission of disease during this ongoing COVID-19 pandemic. This wide use of face masks has indeed been shown to disrupt day-to-day face recognition. People with autism spectrum disorder (ASD) often have predisposed impairment in face recognition and are expected to be more vulnerable to this disruption in face recognition. Here, we recruited typically developing adult participants and those with ASD, and we measured their non-verbal intelligence, autism spectrum quotient, empathy quotient, and recognition performances of faces with and without a face mask covering the lower halves of the face. When faces were initially learned unobstructed, we showed that participants had a general reduced face recognition performance for masked faces. In contrast, when masked faces were first learned, typically developing adults benefit with an overall advantage in recognizing both masked and unmasked faces; while adults with ASD recognized unmasked faces with a significantly more reduced level of performance than masked faces-this face recognition discrepancy is predicted by a higher level of autistic traits. This paper also discusses how autistic traits influence processing of faces with and without face masks.

Eye movement analysis of children's attention for midline diastema.

No previous studies have investigated eye-movement patterns to show children's information processing while viewing clinical images. Therefore, this study aimed to explore children and their educators' perception of a midline diastema by applying eye-movement analysis using the hidden Markov models (EMHMM). A total of 155 children between 2.5 and 5.5 years of age and their educators (n = 34) viewed pictures with and without a midline diastema while Tobii Pro Nano eye-tracker followed their eye movements. Fixation data were analysed using data-driven, and fixed regions of interest (ROIs) approaches with EMHMM. Two different eye-movement patterns were identified: explorative pattern (76%), where the children's ROIs were predominantly around the nose and mouth, and focused pattern (26%), where children's ROIs were precise, locating on the teeth with and without a diastema, and fixations transited among the ROIs with similar frequencies. Females had a significantly higher eye-movement preference for without diastema image than males. Comparisons between the different age groups showed a statistically significant difference for overall entropies. The 3.6-4.5y age groups exhibited higher entropies, indicating lower eye-movement consistency. In addition, children and their educators exhibited two specific eye-movement patterns. Children in the explorative pattern saw the midline diastema more often while their educators focussed on the image without diastema. Thus, EMHMMs are valuable in analysing eye-movement patterns in children and adults.

Understanding children's attention to traumatic dental injuries using eye-tracking.

BACKGROUND/AIM: Traumatic dental injuries (TDIs) in the primary dentition may result in tooth discolouration and fractures. The aim of this child-centred study was to explore the differences between preschool children's eye movement patterns and visual attention to typical outcomes following TDIs to primary teeth. MATERIALS AND METHODS: An eye-tracker recorded 155 healthy preschool children's eye movements when they viewed clinical images of healthy teeth, tooth fractures and discolourations. The visual search pattern was analysed using the eye movement analysis with the Hidden Markov Models (EMHMM) approach and preference for the various regions of interest (ROIs). RESULTS: Two different eye movement patterns (distributed and selective) were identified (p < .05). Children with the distributed pattern shifted their fixations between the presented images, while those with the selective pattern remained focused on the same image they first saw. CONCLUSIONS: Preschool children noticed teeth. However, most of them did not have an attentional bias, implying that they did not interpret these TDI outcomes negatively. Only a few children avoided looking at images with TDIs indicating a potential negative impact. The EMHMM approach is appropriate for assessing inter-individual differences in children's visual attention to TDI outcomes.

Understanding Children's Attention to Dental Caries through Eye-Tracking.

Visual attention is a significant gateway to a child's mind, and looking is one of the first behaviors young children develop. Untreated caries and the resulting poor dental aesthetics can have adverse emotional and social impacts on children's oral health-related quality of life due to its detrimental effects on self-esteem and self-concept. Therefore, we explored preschool children's eye movement patterns and visual attention to images with and without dental caries via eye movement analysis using hidden Markov models (EMHMM). We calibrated a convenience sample of 157 preschool children to the eye-tracker (Tobii Nano Pro) to ensure standardization. Consequently, each participant viewed the same standardized pictures with and without dental caries while an eye-tracking device tracked their eye movements. Subsequently, based on the sequence of viewed regions of interest (ROIs), a transition matrix was developed where the participants' previously viewed ROI informed their subsequently considered ROI. Hence, an individual's HMM was estimated from their eye movement data using a variational Bayesian approach to determine the optimal number of ROIs automatically. Consequently, this data-driven approach generated the visual task participants' most representative eye movement patterns. Preschool children exhibited two different eye movement patterns, distributed (78%) and selective (21%), which was statistically significant. Children switched between images with more similar probabilities in the distributed pattern while children remained looking at the same ROI than switching to the other ROI in the selective pattern. Nevertheless, all children exhibited an equal starting fixation on the right or left image and noticed teeth. The study findings reveal that most preschool children did not have an attentional bias to images with and without dental caries. Furthermore, only a few children selectively fixated on images with dental caries. Therefore, selective eye-movement patterns may strongly predict preschool children's sustained visual attention to dental caries. Nevertheless, future studies are essential to fully understand the developmental origins of differences in visual attention to common oral health presentations in children. Finally, EMHMM is appropriate for assessing inter-individual differences in children's visual attention.

The effects of attentional and interpretation biases on later pain outcomes among younger and older adults: A prospective study.

BACKGROUND: Studies examining the effect of biased cognitions on later pain outcomes have primarily focused on attentional biases, leaving the role of interpretation biases largely unexplored. Also, few studies have examined pain-related cognitive biases in elderly persons. The current study aims to fill these research gaps. METHODS: Younger and older adults with and without chronic pain (N = 126) completed an interpretation bias task and a free-viewing task of injury and neutral scenes at baseline. Participants' pain intensity and disability were assessed at baseline and at a 6-month follow-up. A machine-learning data-driven approach to analysing eye movement data was adopted. RESULTS: Eye movement analyses revealed two common attentional pattern subgroups for scene-viewing: an "explorative" group and a "focused" group. At baseline, participants with chronic pain endorsed more injury-/illness-related interpretations compared to pain-free controls, but they did not differ in eye movements on scene images. Older adults interpreted illness-related scenarios more negatively compared to younger adults, but there was also no difference in eye movements between age groups. Moreover, negative interpretation biases were associated with baseline but not follow-up pain disability, whereas a focused gaze tendency for injury scenes was associated with follow-up but not baseline pain disability. Additionally, there was an indirect effect of interpretation biases on pain disability 6 months later through attentional bias for pain-related images. CONCLUSIONS: The present study provided evidence for pain status and age group differences in injury-/illness-related interpretation biases. Results also revealed distinct roles of interpretation and attentional biases in pain chronicity. SIGNIFICANCE: Adults with chronic pain endorsed more injury-/illness-related interpretations than pain-free controls. Older adults endorsed more illness interpretations than younger adults. A more negative interpretation bias indirectly predicted pain disability 6 months later through hypervigilance towards pain.

PRIMAL-GMM: PaRametrIc MAnifold Learning of Gaussian Mixture Models.

We propose a ParametRIc MAnifold Learning (PRIMAL) algorithm for Gaussian mixtures models (GMM), assuming that GMMs lie on or near to a manifold of probability distributions that is generated from a low-dimensional hierarchical latent space through parametric mappings. Inspired by principal component analysis (PCA), the generative processes for priors, means and covariance matrices are modeled by their respective latent space and parametric mapping. Then, the dependencies between latent spaces are captured by a hierarchical latent space by a linear or kernelized mapping. The function parameters and hierarchical latent space are learned by minimizing the reconstruction error between ground-truth GMMs and manifold-generated GMMs, measured by Kullback-Leibler Divergence (KLD). Variational approximation is employed to handle the intractable KLD between GMMs and a variational EM algorithm is derived to optimize the objective function. Experiments on synthetic data, flow cytometry analysis, eye-fixation analysis and topic models show that PRIMAL learns a continuous and interpretable manifold of GMM distributions and achieves a minimum reconstruction error.

Understanding the collinear masking effect in visual search through eye tracking.

Recent research has reported that, while both orientation contrast and collinearity increase target salience in visual search, a combination of the two counterintuitively masks a local target. Through eye-tracking and eye-movement analysis with hidden Markov models (EMHMM), here we showed that this collinear masking effect was associated with reduced eye-fixation consistency (as measured in entropy) at the central fixation cross prior to the search display presentation. As a decreased precision of saccade landing position is shown to be related to attention shift away from the saccadic target, our result suggested that the collinear masking effect may be related to attention shift to a non-saccadic-goal location in expectation of the search display before saccading to the central fixation cross. This attention shift may consequently interfere with attention capture by the collinear distractor containing the target, resulting in the masking effect. In contrast, although older adults had longer response times, more dispersed eye-movement pattern, and lower eye-movement consistency than young adults during visual search, the two age groups did not differ in the masking effect, suggesting limited contribution from ageing-related cognitive decline. Thus, participants' pre-saccadic attention shift prior to search may be an important factor influencing their search behavior.

Eye movement analysis with hidden Markov models (EMHMM) with co-clustering.

The eye movement analysis with hidden Markov models (EMHMM) method provides quantitative measures of individual differences in eye-movement pattern. However, it is limited to tasks where stimuli have the same feature layout (e.g., faces). Here we proposed to combine EMHMM with the data mining technique co-clustering to discover participant groups with consistent eye-movement patterns across stimuli for tasks involving stimuli with different feature layouts. Through applying this method to eye movements in scene perception, we discovered explorative (switching between the foreground and background information or different regions of interest) and focused (mainly looking at the foreground with less switching) eye-movement patterns among Asian participants. Higher similarity to the explorative pattern predicted better foreground object recognition performance, whereas higher similarity to the focused pattern was associated with better feature integration in the flanker task. These results have important implications for using eye tracking as a window into individual differences in cognitive abilities and styles. Thus, EMHMM with co-clustering provides quantitative assessments on eye-movement patterns across stimuli and tasks. It can be applied to many other real-life visual tasks, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.

Do portrait artists have enhanced face processing abilities? Evidence from hidden Markov modeling of eye movements.

Recent research has suggested the importance of part-based information in face recognition in addition to global, whole-face information. Nevertheless, face drawing experience was reported to enhance selective attention to the eyes but did not improve face recognition performance, leading to speculations about limited plasticity in adult face recognition. Here we examined the mechanism underlying the limited advantage of face drawing experience in face recognition through the Eye Movement analysis with Hidden Markov Models (EMHMM) approach. We found that portrait artists showed more eyes-focused eye movement patterns and outperformed novices in face matching, and participants' drawing rating was correlated with both eye movement pattern and performance. In contrast, portrait artists did not outperform novices and did not differ from novices in eye movement pattern in either the face recognition or part-whole tasks, although the eyes-focused pattern was associated with better recognition performance and longer response times in the whole condition relative to the part condition. Interestingly, in contrast to the face recognition and part-whole tasks, participants' performance in face matching was predicted by their drawing rating but not eye movement pattern. These results suggested that artists' advantage in face processing is specific to tasks similar to their drawing experience such as face matching, and may be related to their better ability in extracting identity-invariant information between two faces rather than more eyes-focused eye movement patterns.

Modulation of mood on eye movement and face recognition performance.

In face recognition, looking at the eyes has been associated with engagement of local attention, as well as better recognition performance. As recent research has suggested negative mood facilitates local attention while positive mood facilitates global attention, negative mood changes may lead to more eyes-focused eye movement patterns and consequently enhance recognition performance. Here we test this hypothesis using mood induction. Through eye movement analysis with hidden Markov models, we discovered eyes-focused and nose-focused eye movement strategies in the participants, and the eyes-focused strategy was associated with better recognition performance. During the recognition phase, participants with a negative mood change had increased eye movement pattern similarity to the eyes-focused strategy, and participants' mood change was correlated with eye movement pattern similarity change. Nevertheless, mood change did not significantly change participants' eye movement strategy classification despite changes in eye movement pattern similarity, and the eye movement pattern similarity change did not modulate recognition performance. These results suggest that mood changes through mood induction lead to slight changes in eye movement pattern that may not be sufficient to modulate recognition performance. Thus, individuals may have preferred eye movement strategies in face recognition impervious to transitory mood changes. This finding is consistent with a recent speculation on limited plasticity in adult face recognition and suggests that eye movements in face recognition may provide reliable information about an individual's cognitive abilities. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The interrelation between interpretation biases, threat expectancies and pain-related attentional processing.

BACKGROUND: Few studies examining the effect of pain-related threat on eye movements have incorporated a measure of interpretation bias. However, theories suggest that interpretation biases also play an important role in the anticipation of harm in situations where pain could be imminent. The current study investigates the association between interpretation biases and pain-related threat expectancies and their associations with eye movements to pain-related imagery. METHODS: Healthy adults' (N = 91) fear of pain, emotional functioning and interpretation biases were assessed prior to a threat manipulation where they were given either threatening or reassuring information about an upcoming cold pressor task. Participants were then asked to freely view scene images that were either pain-related or neutral. RESULTS: We used a data-driven machine learning method to analyse eye movements. We identified an explorative (i.e. greater dispersal of eye fixations) and a focused eye movement pattern subgroup (i.e. mainly focusing on foreground information) for scene viewing in the sample. Participants with more negative interpretation biases expected that the cold pressor task would be more harmful, and those with higher levels of anticipated harm used a more explorative strategy when viewing injury scene images. Subsequent analysis confirmed an indirect effect of interpretation biases on eye movements through expected bodily harm. No difference in eye movements was found between participants given threatening and reassuring information. CONCLUSIONS: Interpretation biases may play a prominent role in threat-related attentional processing. By adopting a novel eye movement analysis approach, our results revealed interesting associations among interpretations, threat expectancies and eye movements. SIGNIFICANCE: Negative interpretation biases may be associated with greater threat expectancies for an upcoming experimental pain task. Anticipation of bodily harm may induce a stimulus non-specific hypervigilant style of scanning of pain-related scenes.

Understanding visual attention to face emotions in social anxiety using hidden Markov models.

Theoretical models propose that attentional biases might account for the maintenance of social anxiety symptoms. However, previous eye-tracking studies have yielded mixed results. One explanation is that existing studies quantify eye-movements using arbitrary, experimenter-defined criteria such as time segments and regions of interests that do not capture the dynamic nature of overt visual attention. The current study adopted the Eye Movement analysis with Hidden Markov Models (EMHMM) approach for eye-movement analysis, a machine-learning, data-driven approach that can cluster people's eye-movements into different strategy groups. Sixty participants high and low in self-reported social anxiety symptoms viewed angry and neutral faces in a free-viewing task while their eye-movements were recorded. EMHMM analyses revealed novel associations between eye-movement patterns and social anxiety symptoms that were not evident with standard analytical approaches. Participants who adopted the same face-viewing strategy when viewing both angry and neutral faces showed higher social anxiety symptoms than those who transitioned between strategies when viewing angry versus neutral faces. EMHMM can offer novel insights into psychopathology-related attention processes.