Influence of training and expertise on deep neural network attention and human attention during a medical image classification task.

In many different domains, experts can make complex decisions after glancing very briefly at an image. However, the perceptual mechanisms underlying expert performance are still largely unknown. Recently, several machine learning algorithms have been shown to outperform human experts in specific tasks. But these algorithms often behave as black boxes and their information processing pipeline remains unknown. This lack of transparency and interpretability is highly problematic in applications involving human lives, such as health care. One way to "open the black box" is to compute an artificial attention map from the model, which highlights the pixels of the input image that contributed the most to the model decision. In this work, we directly compare human visual attention to machine visual attention when performing the same visual task. We have designed a medical diagnosis task involving the detection of lesions in small bowel endoscopic images. We collected eye movements from novices and gastroenterologist experts while they classified medical images according to their relevance for Crohn's disease diagnosis. We trained three state-of-the-art deep learning models on our carefully labeled dataset. Both humans and machine performed the same task. We extracted artificial attention with six different post hoc methods. We show that the model attention maps are significantly closer to human expert attention maps than to novices', especially for pathological images. As the model gets trained and its performance gets closer to the human experts, the similarity between model and human attention increases. Through the understanding of the similarities between the visual decision-making process of human experts and deep neural networks, we hope to inform both the training of new doctors and the architecture of new algorithms.

Landmark-dependent Navigation Strategy Declines across the Human Life-Span: Evidence from Over 37,000 Participants.

Humans show a remarkable capacity to navigate various environments using different navigation strategies, and we know that strategy changes across the life span. However, this observation has been based on studies of small sample sizes. To this end, we used a mobile app-based video game (Sea Hero Quest) to test virtual navigation strategies and memory performance within a distinct radial arm maze level in over 37,000 participants. Players were presented with six pathways (three open and three closed) and were required to navigate to the three open pathways to collect a target. Next, all six pathways were made available and the player was required to visit the pathways that were previously unavailable. Both reference memory and working memory errors were calculated. Crucially, at the end of the level, the player was asked a multiple-choice question about how they found the targets (i.e., a counting-dependent strategy vs. a landmark-dependent strategy). As predicted from previous laboratory studies, we found the use of landmarks declined linearly with age. Those using landmark-based strategies also performed better on reference memory than those using a counting-based strategy. These results extend previous observations in the laboratory showing a decreased use of landmark-dependent strategies with age.

Influence of prior knowledge on eye movements to scenes as revealed by hidden Markov models.

Human visual experience usually provides ample opportunity to accumulate knowledge about events unfolding in the environment. In typical scene perception experiments, however, participants view images that are unrelated to each other and, therefore, they cannot accumulate knowledge relevant to the upcoming visual input. Consequently, the influence of such knowledge on how this input is processed remains underexplored. Here, we investigated this influence in the context of gaze control. We used sequences of static film frames arranged in a way that allowed us to compare eye movements to identical frames between two groups: a group that accumulated prior knowledge relevant to the situations depicted in these frames and a group that did not. We used a machine learning approach based on hidden Markov models fitted to individual scanpaths to demonstrate that the gaze patterns from the two groups differed systematically and, thereby, showed that recently accumulated prior knowledge contributes to gaze control. Next, we leveraged the interpretability of hidden Markov models to characterize these differences. Additionally, we report two unexpected and interesting caveats of our approach. Overall, our results highlight the importance of recently acquired prior knowledge for oculomotor control and the potential of hidden Markov models as a tool for investigating it.

Examining individual learning patterns using generalised linear mixed models.

Everyone learns differently, but individual performance is often ignored in favour of a group-level analysis. Using data from four different experiments, we show that generalised linear mixed models (GLMMs) and extensions can be used to examine individual learning patterns. Producing ellipsoids and cluster analyses based on predicted random effects, individual learning patterns can be identified, clustered and used for comparisons across various experimental conditions or groups. This analysis can handle a range of datasets including discrete, continuous, censored and non-censored, as well as different experimental conditions, sample sizes and trial numbers. Using this approach, we show that learning a face-named paired associative task produced individuals that can learn quickly, with the performance of some remaining high, but with a drop-off in others, whereas other individuals show poor performance throughout the learning period. We see this more clearly in a virtual navigation spatial learning task (NavWell). Two prominent clusters of learning emerged, one showing individuals who produced a rapid learning and another showing a slow and gradual learning pattern. Using data from another spatial learning task (Sea Hero Quest), we show that individuals' performance generally reflects their age category, but not always. Overall, using this analytical approach may help practitioners in education and medicine to identify those individuals who might need extra help and attention. In addition, identifying learning patterns may enable further investigation of the underlying neural, biological, environmental and other factors associated with these individuals.

Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer's disease.

Spatial navigation is emerging as a critical factor in identifying preclinical Alzheimer's disease (AD). However, the impact of interindividual navigation ability and demographic risk factors (e.g., APOE, age, and sex) on spatial navigation make it difficult to identify persons "at high risk" of AD in the preclinical stages. In the current study, we use spatial navigation big data (n = 27,108) from the Sea Hero Quest (SHQ) game to overcome these challenges by investigating whether big data can be used to benchmark a highly phenotyped healthy aging laboratory cohort into high- vs. low-risk persons based on their genetic (APOE) and demographic (sex, age, and educational attainment) risk factors. Our results replicate previous findings in APOE ε4 carriers, indicative of grid cell coding errors in the entorhinal cortex, the initial brain region affected by AD pathophysiology. We also show that although baseline navigation ability differs between men and women, sex does not interact with the APOE genotype to influence the manifestation of AD-related spatial disturbance. Most importantly, we demonstrate that such high-risk preclinical cases can be reliably distinguished from low-risk participants using big-data spatial navigation benchmarks. By contrast, participants were undistinguishable on neuropsychological episodic memory tests. Taken together, we present evidence to suggest that, in the future, SHQ normative benchmark data can be used to more accurately classify spatial impairments in at-high-risk of AD healthy participants at a more individual level, therefore providing the steppingstone for individualized diagnostics and outcome measures of cognitive symptoms in preclinical AD.

Semantic content outweighs low-level saliency in determining children's and adults' fixation of movies.

To make sense of the visual world, we need to move our eyes to focus regions of interest on the high-resolution fovea. Eye movements, therefore, give us a way to infer mechanisms of visual processing and attention allocation. Here, we examined age-related differences in visual processing by recording eye movements from 37 children (aged 6-14years) and 10 adults while viewing three 5-min dynamic video clips taken from child-friendly movies. The data were analyzed in two complementary ways: (a) gaze based and (b) content based. First, similarity of scanpaths within and across age groups was examined using three different measures of variance (dispersion, clusters, and distance from center). Second, content-based models of fixation were compared to determine which of these provided the best account of our dynamic data. We found that the variance in eye movements decreased as a function of age, suggesting common attentional orienting. Comparison of the different models revealed that a model that relies on faces generally performed better than the other models tested, even for the youngest age group (<10years). However, the best predictor of a given participant's eye movements was the average of all other participants' eye movements both within the same age group and in different age groups. These findings have implications for understanding how children attend to visual information and highlight similarities in viewing strategies across development.

Personality Traits Do Not Predict How We Look at Faces.

While personality has typically been considered to influence gaze behaviour, literature relating to the topic is mixed. Previously, we found no evidence of self-reported personality traits on preferred gaze duration between a participant and a person looking at them via a video. In this study, 77 of the original participants answered an in-depth follow-up survey containing a more comprehensive assessment of personality traits (Big Five Inventory) than was initially used, to check whether earlier findings were caused by the personality measure being too coarse. In addition to preferred mutual gaze duration, we also examined two other factors linked to personality traits: number of blinks and total fixation duration in the eye region of observed faces. No significant correlations were found between any of these measures and participant personality traits. We suggest that effects previously reported in the literature may stem from contextual differences or modulation of arousal.

Scanpath modeling and classification with hidden Markov models.

How people look at visual information reveals fundamental information about them; their interests and their states of mind. Previous studies showed that scanpath, i.e., the sequence of eye movements made by an observer exploring a visual stimulus, can be used to infer observer-related (e.g., task at hand) and stimuli-related (e.g., image semantic category) information. However, eye movements are complex signals and many of these studies rely on limited gaze descriptors and bespoke datasets. Here, we provide a turnkey method for scanpath modeling and classification. This method relies on variational hidden Markov models (HMMs) and discriminant analysis (DA). HMMs encapsulate the dynamic and individualistic dimensions of gaze behavior, allowing DA to capture systematic patterns diagnostic of a given class of observers and/or stimuli. We test our approach on two very different datasets. Firstly, we use fixations recorded while viewing 800 static natural scene images, and infer an observer-related characteristic: the task at hand. We achieve an average of 55.9% correct classification rate (chance = 33%). We show that correct classification rates positively correlate with the number of salient regions present in the stimuli. Secondly, we use eye positions recorded while viewing 15 conversational videos, and infer a stimulus-related characteristic: the presence or absence of original soundtrack. We achieve an average 81.2% correct classification rate (chance = 50%). HMMs allow to integrate bottom-up, top-down, and oculomotor influences into a single model of gaze behavior. This synergistic approach between behavior and machine learning will open new avenues for simple quantification of gazing behavior. We release SMAC with HMM, a Matlab toolbox freely available to the community under an open-source license agreement.

Face exploration dynamics differentiate men and women.

The human face is central to our everyday social interactions. Recent studies have shown that while gazing at faces, each one of us has a particular eye-scanning pattern, highly stable across time. Although variables such as culture or personality have been shown to modulate gaze behavior, we still don't know what shapes these idiosyncrasies. Moreover, most previous observations rely on static analyses of small-sized eye-position data sets averaged across time. Here, we probe the temporal dynamics of gaze to explore what information can be extracted about the observers and what is being observed. Controlling for any stimuli effect, we demonstrate that among many individual characteristics, the gender of both the participant (gazer) and the person being observed (actor) are the factors that most influence gaze patterns during face exploration. We record and exploit the largest set of eye-tracking data (405 participants, 58 nationalities) from participants watching videos of another person. Using novel data-mining techniques, we show that female gazers follow a much more exploratory scanning strategy than males. Moreover, female gazers watching female actresses look more at the eye on the left side. These results have strong implications in every field using gaze-based models from computer vision to clinical psychology.

How saliency, faces, and sound influence gaze in dynamic social scenes.

Conversation scenes are a typical example in which classical models of visual attention dramatically fail to predict eye positions. Indeed, these models rarely consider faces as particular gaze attractors and never take into account the important auditory information that always accompanies dynamic social scenes. We recorded the eye movements of participants viewing dynamic conversations taking place in various contexts. Conversations were seen either with their original soundtracks or with unrelated soundtracks (unrelated speech and abrupt or continuous natural sounds). First, we analyze how auditory conditions influence the eye movement parameters of participants. Then, we model the probability distribution of eye positions across each video frame with a statistical method (Expectation-Maximization), allowing the relative contribution of different visual features such as static low-level visual saliency (based on luminance contrast), dynamic low level visual saliency (based on motion amplitude), faces, and center bias to be quantified. Through experimental and modeling results, we show that regardless of the auditory condition, participants look more at faces, and especially at talking faces. Hearing the original soundtrack makes participants follow the speech turn-taking more closely. However, we do not find any difference between the different types of unrelated soundtracks. These eyetracking results are confirmed by our model that shows that faces, and particularly talking faces, are the features that best explain the gazes recorded, especially in the original soundtrack condition. Low-level saliency is not a relevant feature to explain eye positions made on social scenes, even dynamic ones. Finally, we propose groundwork for an audiovisual saliency model.

Shorter self-reported sleep duration is associated with worse virtual spatial navigation performance in men.

Sleep has been shown to impact navigation ability. However, it remains unclear how different sleep-related variables may be independently associated with spatial navigation performance, and as to whether gender may play a role in these associations. We used a mobile video game app, Sea Hero Quest (SHQ), to measure wayfinding ability in US-based participants. Wayfinding performance on SHQ has been shown to correlate with real-world wayfinding. Participants were asked to report their sleep duration, quality, daytime sleepiness and nap frequency and duration on a typical night (n = 766, 335 men, 431 women, mean age = 26.5 years, range = 18-59 years). A multiple linear regression was used to identify which self-reported sleep variables were independently associated with wayfinding performance. Shorter self-reported sleep durations were significantly associated with worse wayfinding performance in men only. Other self-reported sleep variables showed non-significant trends of association with wayfinding performance. When removing non-typical sleepers (< 6 or > 9 h of sleep on a typical night), the significant association between sleep duration and spatial navigation performance in men was no longer present. These findings from U.S.-based participants suggest that a longer self-reported sleep duration may be an important contributor to successful navigation ability in men.

Idiosyncratic fixation patterns generalize across dynamic and static facial expression recognition.

Facial expression recognition (FER) is crucial for understanding the emotional state of others during human social interactions. It has been assumed that humans share universal visual sampling strategies to achieve this task. However, recent studies in face identification have revealed striking idiosyncratic fixation patterns, questioning the universality of face processing. More importantly, very little is known about whether such idiosyncrasies extend to the biological relevant recognition of static and dynamic facial expressions of emotion (FEEs). To clarify this issue, we tracked observers' eye movements categorizing static and ecologically valid dynamic faces displaying the six basic FEEs, all normalized for time presentation (1 s), contrast and global luminance across exposure time. We then used robust data-driven analyses combining statistical fixation maps with hidden Markov Models to explore eye-movements across FEEs and stimulus modalities. Our data revealed three spatially and temporally distinct equally occurring face scanning strategies during FER. Crucially, such visual sampling strategies were mostly comparably effective in FER and highly consistent across FEEs and modalities. Our findings show that spatiotemporal idiosyncratic gaze strategies also occur for the biologically relevant recognition of FEEs, further questioning the universality of FER and, more generally, face processing.

Combining patient-lesion and big data approaches to reveal hippocampal contributions to spatial memory and navigation.

Classic findings of impaired allocentric spatial learning and memory following hippocampal lesions indicate that the hippocampus supports cognitive maps of one's environment. Many studies assess navigation in vista space virtual reality environments and compare hippocampal-lesioned individuals' performance to that of small control samples, potentially stifling detection of preserved and impaired performance. Using the mobile app Sea Hero Quest, we examined navigation in diverse complex environments in two individuals with hippocampal lesions relative to demographically matched controls (N = 17,734). We found surprisingly accurate navigation in several environments, particularly those containing a constrained set of sub-goals, paths, and/or turns. Areas of impaired performance may reflect a role for the hippocampus in anterograde memory and more flexible and/or precise spatial representations, even when the need for allocentric processing is minimal. The results emphasize the value of combining single cases with big data and illustrate navigation performance profiles in individuals with hippocampal compromise.

The relationship between object-based spatial ability and virtual navigation performance.

Spatial navigation is a multi-faceted behaviour drawing on many different aspects of cognition. Visuospatial abilities, such as mental rotation and visuospatial working memory, in particular, may be key factors. A range of tests have been developed to assess visuospatial processing and memory, but how such tests relate to navigation ability remains unclear. This understanding is important to advance tests of navigation for disease monitoring in various disorders (e.g., Alzheimer's disease) where spatial impairment is an early symptom. Here, we report the use of an established mobile gaming app, Sea Hero Quest (SHQ), as a measure of navigation ability in a sample of young, predominantly female university students (N = 78; 20; female = 74.3%; mean age = 20.33 years). We used three separate tests of navigation embedded in SHQ: wayfinding, path integration and spatial memory in a radial arm maze. In the same participants, we also collected measures of mental rotation (Mental Rotation Test), visuospatial processing (Design Organization Test) and visuospatial working memory (Digital Corsi). We found few strong correlations across our measures. Being good at wayfinding in a virtual navigation test does not mean an individual will also be good at path integration, have a superior memory in a radial arm maze, or rate themself as having a strong sense of direction. However, we observed that participants who were good in the wayfinding task of SHQ tended to perform well on the three visuospatial tasks examined here, and to also use a landmark strategy in the radial maze task. These findings help clarify the associations between different abilities involved in spatial navigation.

Explaining World-Wide Variation in Navigation Ability from Millions of People: Citizen Science Project Sea Hero Quest.

Navigation ability varies widely across humans. Prior studies have reported that being younger and a male has an advantage for navigation ability. However, these studies have generally involved small numbers of participants from a handful of western countries. Here, we review findings from our project Sea Hero Quest, which used a video game for mobile and tablet devices to test 3.9 million people on their navigation ability, sampling across every nation-state and from 18 to 99 years of age. Results revealed that the task has good ecological validity and across all countries sufficiently sampled (N = 63), age is linked to a near-linear decline in navigation ability from the early 20s. All countries showed a male advantage, but this varied considerably and could be partly predicted by gender inequality. We found that those who reported growing up in a city were on average worse at navigating than those who grew up outside cities and that navigation performance helped identify those at greater genetic risk of Alzheimer's disease. We discuss the advantages and challenges of using a mobile app to study cognition and the future avenues for understanding individual differences in navigation ability arising from this research.

Prior knowledge about events depicted in scenes decreases oculomotor exploration.

The visual input that the eyes receive usually contains temporally continuous information about unfolding events. Therefore, humans can accumulate knowledge about their current environment. Typical studies on scene perception, however, involve presenting multiple unrelated images and thereby render this accumulation unnecessary. Our study, instead, facilitated it and explored its effects. Specifically, we investigated how recently-accumulated prior knowledge affects gaze behavior. Participants viewed sequences of static film frames that contained several 'context frames' followed by a 'critical frame'. The context frames showed either events from which the situation depicted in the critical frame naturally followed, or events unrelated to this situation. Therefore, participants viewed identical critical frames while possessing prior knowledge that was either relevant or irrelevant to the frames' content. In the former case, participants' gaze behavior was slightly more exploratory, as revealed by seven gaze characteristics we analyzed. This result demonstrates that recently-gained prior knowledge reduces exploratory eye movements.

Wayfinding and path integration deficits detected using a virtual reality mobile app in patients with traumatic brain injury.

The ability to navigate is supported by a wide network of brain areas which are particularly vulnerable to disruption brain injury, including traumatic brain injury (TBI). Wayfinding and the ability to orient back to the direction you have recently come (path integration) may likely be impacted in daily life but have so far not been tested with patients with TBI. Here, we assessed spatial navigation in thirty-eight participants, fifteen of whom had a history of TBI, and twenty-three control participants. Self-estimated spatial navigation ability was assessed using the Santa Barbara Sense of Direction (SBSOD) scale. No significant difference between TBI patients and a control group was identified. Rather, results indicated that both participant groups demonstrated 'good' self-inferred spatial navigational ability on the SBSOD scale. Objective navigation ability was tested via the virtual mobile app test Sea Hero Quest (SHQ), which has been shown to predict real-world navigation difficulties and assesses (a) wayfinding across several environments and (b) path integration. Compared to a sub-sample of 13 control participants, a matched subsample of 10 TBI patients demonstrated generally poorer performance on all wayfinding environments tested. Further analysis revealed that TBI participants consistently spent a shorter duration viewing a map prior to navigating to goals. Patients showed mixed performance on the path integration task, with poor performance evident when proximal cues were absent. Our results provide preliminary evidence that TBI impacts both wayfinding and, to some extent, path integration. The findings suggest long-lasting clinical difficulties experienced in TBI patients affect both wayfinding and to some degree path integration ability.

Communicating and reading emotion with masked faces in the Covid era: A short review of the literature.

Face masks have proven to be key to slowing down the SARS-Cov2 virus spread in the COVID-19 pandemic context. However, wearing face masks is not devoid of "side-effects", at both the physical and psychosocial levels. In particular, masks hinder emotion reading from facial expressions as they hide a significant part of the face. This disturbs both holistic and featural processing of facial expressions and, therefore, impairs emotion recognition, and influences many aspects of human social behavior. Communication in general is disrupted by face masks, as they modify the wearer's voice and prevent the audience from using lip reading or other non-verbal cues for speech comprehension. Individuals suffering from psychiatric conditions with impairment of communication, are at higher risk of distress because masks increase their difficulties to read emotions from faces. The identification and acknowledgement of these "side-effects" on communication are necessary because they warrant further work on adaptive solutions that will help foster the use of face masks by the greatest number.

Does culture shape our understanding of others' thoughts and emotions? An investigation across 12 countries.

Measures of social cognition have now become central in neuropsychology, being essential for early and differential diagnoses, follow-up, and rehabilitation in a wide range of conditions. With the scientific world becoming increasingly interconnected, international neuropsychological and medical collaborations are burgeoning to tackle the global challenges that are mental health conditions. These initiatives commonly merge data across a diversity of populations and countries, while ignoring their specificity. OBJECTIVE: In this context, we aimed to estimate the influence of participants' nationality on social cognition evaluation. This issue is of particular importance as most cognitive tasks are developed in highly specific contexts, not representative of that encountered by the world's population. METHOD: Through a large international study across 18 sites, neuropsychologists assessed core aspects of social cognition in 587 participants from 12 countries using traditional and widely used tasks. RESULTS: Age, gender, and education were found to impact measures of mentalizing and emotion recognition. After controlling for these factors, differences between countries accounted for more than 20% of the variance on both measures. Importantly, it was possible to isolate participants' nationality from potential translation issues, which classically constitute a major limitation. CONCLUSIONS: Overall, these findings highlight the need for important methodological shifts to better represent social cognition in both fundamental research and clinical practice, especially within emerging international networks and consortia. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Multi-expert annotation of Crohn's disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network.

Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn's disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network. Methods Images of capsule were annotated by a reader first and then reviewed by three experts in inflammatory bowel disease. Concordance analysis between experts was evaluated by Fleiss' kappa and all the discordant images were, again, read by all the endoscopists to obtain a consensus annotation. A recurrent attention neural network developed for the study was tested before and after the consensus annotation. Available neural networks (ResNet and VGGNet) were also tested under the same conditions. Results The final dataset included 3498 images with 2124 non-pathological (60.7 %), 1360 pathological (38.9 %), and 14 (0.4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0.79 ( P  < 0.0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93.7 %, sensitivity of 93 %, and specificity of 95 %. Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset.