Visual search and real-image similarity: An empirical assessment through the lens of deep learning.

The ability to predict how efficiently a person finds an object in the environment is a crucial goal of attention research. Central to this issue are the similarity principles initially proposed by Duncan and Humphreys, which outline how the similarity between target and distractor objects (TD) and between distractor objects themselves (DD) affect search efficiency. However, the search principles lack direct quantitative support from an ecological perspective, being a summary approximation of a wide range of lab-based results poorly generalisable to real-world scenarios. This study exploits deep convolutional neural networks to predict human search efficiency from computational estimates of similarity between objects populating, potentially, any visual scene. Our results provide ecological evidence supporting the similarity principles: search performance continuously varies across tasks and conditions and improves with decreasing TD similarity and increasing DD similarity. Furthermore, our results reveal a crucial dissociation: TD and DD similarities mainly operate at two distinct layers of the network: DD similarity at the intermediate layers of coarse object features and TD similarity at the final layers of complex features used for classification. This suggests that these different similarities exert their major effects at two distinct perceptual levels and demonstrates our methodology's potential to offer insights into the depth of visual processing on which the search relies. By combining computational techniques with visual search principles, this approach aligns with modern trends in other research areas and fulfils longstanding demands for more ecologically valid research in the field of visual search.

Visual experience modulates the sensitivity to the distributional history of words in natural language.

Different experiential traces (i.e., linguistic, motor, and perceptual) are likely contributing to the organization of human semantic knowledge. Here, we aimed to address this issue by investigating whether visual experience may affect the sensitivity to distributional priors from natural language. We conducted an independent reanalysis of data from Bottini et al., in which early blind and sighted participants performed an auditory lexical decision task. Since previous research has shown that semantic neighborhood density-the mean distance between a target word and its closest semantic neighbors-can influence performance in lexical decision tasks, we investigated whether vision may alter the reliance on this semantic index. We demonstrate that early blind participants are more sensitive to semantic neighborhood density than sighted participants, as indicated by the significantly faster response times for words with higher levels of semantic neighborhood density shown by the blind group. These findings suggest that an early lack of visual experience may lead to enhanced sensitivity to the distributional history of words in natural language, deepening in turn our understanding of the strict interplay between linguistic and perceptual experience in the organization of conceptual knowledge.

On humans' (explicit) intuitions about the meaning of novel words.

Pseudowords offer a unique opportunity to investigate how humans deal with new (verbal) information. Within this framework, previous studies have shown that, at the implicit level, humans exploit systematic associations in the form-meaning interface to process new information by relying on (sub-lexical) contents already mapped in semantic memory. However, whether speakers exploit such processes in explicit decisions about the meanings elicited by unfamiliar terms remains an open, important question. Here, we tested this by leveraging computational models that are able to induce semantic representations for out-of-vocabulary stimuli. Across two experiments, we demonstrate that participants' guesses about pseudoword meanings in a 2AFC task consistently align with the model's predictions. This indicates that humans' ability to extract meaningful knowledge from complex statistical patterns can affect explicit decisions.

Decomposing geographical judgments into spatial, temporal and linguistic components.

When mentally exploring maps representing large-scale environments (e.g., countries or continents), humans are assumed to mainly rely on spatial information derived from direct perceptual experience (e.g., prior visual experience with the geographical map itself). In the present study, we rather tested whether also temporal and linguistic information could account for the way humans explore and ultimately represent this type of maps. We quantified temporal distance as the minimum time needed to travel by train across Italian cities, while linguistic distance was retrieved from natural language through cognitively plausible AI models based on non-spatial associative learning mechanisms (i.e., distributional semantic models). In a first experiment, we show that temporal and linguistic distances capture with high-confidence real geographical distances. Next, in a second behavioral experiment, we show that linguistic information can account for human performance over and above real spatial information (which plays the major role in explaining participants' performance) in a task in which participants have to judge the distance between cities (while temporal information was found to be not relevant). These findings indicate that, when exploring maps representing large-scale environments, humans do take advantage of both perceptual and linguistic information, suggesting in turn that the formation of cognitive maps possibly relies on a strict interplay between spatial and non-spatial learning principles.

Taboo language across the globe: A multi-lab study.

The use of taboo words represents one of the most common and arguably universal linguistic behaviors, fulfilling a wide range of psychological and social functions. However, in the scientific literature, taboo language is poorly characterized, and how it is realized in different languages and populations remains largely unexplored. Here we provide a database of taboo words, collected from different linguistic communities (Study 1, N = 1046), along with their speaker-centered semantic characterization (Study 2, N = 455 for each of six rating dimensions), covering 13 languages and 17 countries from all five permanently inhabited continents. Our results show that, in all languages, taboo words are mainly characterized by extremely low valence and high arousal, and very low written frequency. However, a significant amount of cross-country variability in words' tabooness and offensiveness proves the importance of community-specific sociocultural knowledge in the study of taboo language.

From vector spaces to DRM lists: False Memory Generator, a software for automated generation of lists of stimuli inducing false memories.

The formation of false memories is one of the most widely studied topics in cognitive psychology. The Deese-Roediger-McDermott (DRM) paradigm is a powerful tool for investigating false memories and revealing the cognitive mechanisms subserving their formation. In this task, participants first memorize a list of words (encoding phase) and next have to indicate whether words presented in a new list were part of the initially memorized one (recognition phase). By employing DRM lists optimized to investigate semantic effects, previous studies highlighted a crucial role of semantic processes in false memory generation, showing that new words semantically related to the studied ones tend to be more erroneously recognized (compared to new words less semantically related). Despite the strengths of the DRM task, this paradigm faces a major limitation in list construction due to its reliance on human-based association norms, posing both practical and theoretical concerns. To address these issues, we developed the False Memory Generator (FMG), an automated and data-driven tool for generating DRM lists, which exploits similarity relationships between items populating a vector space. Here, we present FMG and demonstrate the validity of the lists generated in successfully replicating well-known semantic effects on false memory production. FMG potentially has broad applications by allowing for testing false memory production in domains that go well beyond the current possibilities, as it can be in principle applied to any vector space encoding properties related to word referents (e.g., lexical, orthographic, phonological, sensory, affective, etc.) or other type of stimuli (e.g., images, sounds, etc.).

Time is -ending: Sublexical information activates the horizontal mental time line in word processing.

The mental time line (MTL) is a spatial continuum on which earlier events are generally associated with the left space and later events with the right space. Accordingly, past- and future-related words receive faster responses with, respectively, the left and the right hand. Yet, it is currently unclear whether the MTL is activated by the whole word or whether it can be triggered by more subtle sublexical cues, such as verb-endings, and whether the activation of this spatial continuum is an automatic phenomenon. The aim of this study is to test whether verb-endings do bring conceptual information that is in turn capable to activate the MTL and whether this activation holds also when the temporal information is not explicitly processed. We designed three experiments. In Experiment 1, consisting of a temporal categorization task, and in Experiment 2, consisting of a lexical decision task, we tested Italian tensed verbs (trov-avo "I found," trov-erò "I will find") and pseudo-verbs (trop-avo, trop-erò). Results of Experiment 1 showed that both tensed verbs and pseudo-verbs were spatially coded on the MTL. Results from Experiment 2 showed that the MTL is activated by the verb-endings also when temporal information was task-irrelevant (i.e., lexical decision task). Experiment 3 further clarified that the spatial-temporal congruency effect does not emerge during the evaluation of an inhomogeneous set of stimuli (i.e., when adding to the stimuli time-unrelated fillers). Overall, the present findings indicate that sublexical strings carry specific semantic information that comes into play in the generation of spatial-temporal associations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Predicting Hand Movements With Distributional Semantics: Evidence From Mouse-Tracking.

Although mouse-tracking has been taken as a real-time window on different aspects of human decision-making processes, whether purely semantic information affects response conflict at the level of motor output as measured through mouse movements is still unknown. Here, across two experiments, we investigated the effects of semantic knowledge by predicting participants' performance in a standard keyboard task and in a mouse-tracking task through distributional semantics, a usage-based modeling approach to meaning. In Experiment 1, participants were shown word pairs and were required to perform a two-alternative forced choice task selecting either the more abstract or the more concrete word, using standard keyboard presses. In Experiment 2, participants performed the same task, yet this time response selection was achieved by moving the computer mouse. Results showed that the involvement of semantic components in the task at hand is observable using both standard reaction times (Experiment 1) as well as using indexes extracted from mouse trajectories (Experiment 2). In particular, mouse trajectories reflected the response conflict and its temporal evolution, with a larger deviation for increasing word semantic relatedness. These findings support the validity of mouse-tracking as a method to detect deep and implicit decision-making features. Additionally, by demonstrating that a usage-based model of meaning can account for the different degrees of cognitive conflict associated with task achievement, these findings testify the impact of the human semantic memory on decision-making processes.

Visual Intuitions in the Absence of Visual Experience: The Role of Direct Experience in Concreteness and Imageability Judgements.

The strongest formulations of grounded cognition assume that perceptual intuitions about concepts involve the re-activation of sensorimotor experience we have made with their referents in the world. Within this framework, concreteness and imageability ratings are indeed of crucial importance by operationalising the amount of perceptual interaction we have made with objects. Here we tested such an assumption by asking whether visual intuitions about concepts are provided accurately even when direct visual experience is absent. To this aim, we considered concreteness and imageability intuitions in blind people and tested whether these judgments are predicted by Image-based Frequency (IF, i.e. a data-driven estimate approximating the availability of the word referent in the visual environment). Results indicated that IF predicts perceptual intuitions with a larger extent in sighted compared to blind individuals, thus suggesting a role of direct experience in shaping our judgements. However, the effect of IF was significant not only in sighted but also in blind individuals. This indicates that having direct visual experience with objects does not play a critical role in making them concrete and imageable in a person's intuitions: people do not need visual experience to develop intuition about the availability of things in the external visual environment and use this intuition to inform concreteness/imageability judgments. Our findings fit closely the idea that perceptual judgments are the outcome of introspection/abstraction tasks invoking high-level conceptual knowledge that is not necessarily acquired via direct perceptual experience.

Surfing the OCEAN: The machine learning psycholexical approach 2.0 to detect personality traits in texts.

OBJECTIVE: We aimed to develop a machine learning model to infer OCEAN traits from text. BACKGROUND: The psycholexical approach allows retrieving information about personality traits from human language. However, it has rarely been applied because of methodological and practical issues that current computational advancements could overcome. METHOD: Classical taxonomies and a large Yelp corpus were leveraged to learn an embedding for each personality trait. These embeddings were used to train a feedforward neural network for predicting trait values. Their generalization performances have been evaluated through two external validation studies involving experts (N = 11) and laypeople (N = 100) in a discrimination task about the best markers of each trait and polarity. RESULTS: Intrinsic validation of the model yielded excellent results, with R2 values greater than 0.78. The validation studies showed a high proportion of matches between participants' choices and model predictions, confirming its efficacy in identifying new terms related to the OCEAN traits. The best performance was observed for agreeableness and extraversion, especially for their positive polarities. The model was less efficient in identifying the negative polarity of openness and conscientiousness. CONCLUSIONS: This innovative methodology can be considered a "psycholexical approach 2.0," contributing to research in personality and its practical applications in many fields.

The Children and Young People's Books Lexicon (CYP-LEX): A large-scale lexical database of books read by children and young people in the United Kingdom.

This article introduces the Children and Young People's Books-Lexicon (CYP-LEX), a large-scale lexical database derived from books popular with children and young people in the United Kingdom. CYP-LEX includes 1,200 books evenly distributed across three age bands (7-9, 10-12, 13+) and comprises over 70 million tokens and over 105,000 types. For each word in each age band, we provide its raw and Zipf-transformed frequencies, all parts-of-speech in which it occurs with raw frequency and lemma for each occurrence, and measures of count-based contextual diversity. Together and individually, the three CYP-LEX age bands contain substantially more words than any other publicly available database of books for primary and secondary school children. Most of these words are very low in frequency, and a substantial proportion of the words in each age band do not occur on British television. Although the three age bands share some very frequent words, they differ substantially regarding words that occur less frequently, and this pattern also holds at the level of individual books. Initial analyses of CYP-LEX illustrate why independent reading constitutes a challenge for children and young people, and they also underscore the importance of reading widely for the development of reading expertise. Overall, CYP-LEX provides unprecedented information into the nature of vocabulary in books that British children aged 7+ read, and is a highly valuable resource for those studying reading and language development.

Beyond quantity of experience: Exploring the role of semantic consistency in Chinese character knowledge.

Most printed Chinese words are compounds built from the combination of meaningful characters. Yet, there is a poor understanding of how individual characters contribute to the recognition of compounds. Using a megastudy of Chinese word recognition (Tse et al., 2017), we examined how the lexical decision of existing and novel Chinese compounds was influenced by two properties of individual characters: family size (the number of distinct words that embed a character) and family semantic consistency (the average semantic relatedness between a character and all words containing it). Results revealed that both variables influence word and nonword processing: Words are recognized more quickly and accurately when they contain characters that occur frequently across different words and that make consistent meaningful contributions to those words, while nonwords containing those types of characters are rejected more slowly. These findings suggest that the learning of individual characters is based not only on the quantity of experience with them but also on the reliability of the semantic information they communicate. In addition, readers are able to generalize character knowledge acquired from previous word experiences to their daily encounters with familiar and unfamiliar words. We close by discussing how word experience shapes character knowledge when different ways of calculating family properties are considered. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Embodied Processing at Six Linguistic Granularity Levels: A Consensus Paper.

Language processing is influenced by sensorimotor experiences. Here, we review behavioral evidence for embodied and grounded influences in language processing across six linguistic levels of granularity. We examine (a) sub-word features, discussing grounded influences on iconicity (systematic associations between word form and meaning); (b) words, discussing boundary conditions and generalizations for the simulation of color, sensory modality, and spatial position; (c) sentences, discussing boundary conditions and applications of action direction simulation; (d) texts, discussing how the teaching of simulation can improve comprehension in beginning readers; (e) conversations, discussing how multi-modal cues improve turn taking and alignment; and (f) text corpora, discussing how distributional semantic models can reveal how grounded and embodied knowledge is encoded in texts. These approaches are converging on a convincing account of the psychology of language, but at the same time, there are important criticisms of the embodied approach and of specific experimental paradigms. The surest way forward requires the adoption of a wide array of scientific methods. By providing complimentary evidence, a combination of multiple methods on various levels of granularity can help us gain a more complete understanding of the role of embodiment and grounding in language processing.

Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data.

We release a database of cloze probability values, predictability ratings, and computational estimates for a sample of 205 English sentences (1726 words), aligned with previously released word-by-word reading time data (both self-paced reading and eye-movement records; Frank et al., Behavior Research Methods, 45(4), 1182-1190. 2013) and EEG responses (Frank et al., Brain and Language, 140, 1-11. 2015). Our analyses show that predictability ratings are the best predictors of the EEG signal (N400, P600, LAN) self-paced reading times, and eye movement patterns, when spillover effects are taken into account. The computational estimates are particularly effective at explaining variance in the eye-tracking data without spillover. Cloze probability estimates have decent overall psychometric accuracy and are the best predictors of early fixation patterns (first fixation duration). Our results indicate that the choice of the best measurement of word predictability in context critically depends on the processing index being considered.

Embodied language of emotions: Predicting human intuitions with linguistic distributions in blind and sighted individuals.

Recent constructionist theories have suggested that language and sensory experience play a crucial role not only in how individuals categorise emotions but also in how they experience and shape them, helping to acquire abstract concepts that are used to make sense of bodily perceptions associated with specific emotions. Here, we aimed to investigate the role of sensory experience in conceptualising bodily felt emotions by asking 126 Italian blind participants to freely recall in which part of the body they commonly feel specific emotions (N = 15). Participants varied concerning visual experience in terms of blindness onset (i.e., congenital vs late) and degree of visual experience (i.e., total vs partial sensory loss). Using an Italian semantic model to estimate to what extent discrete emotions are associated with body parts in language experience, we found that all participants' reports correlated with the model predictions. Interestingly, blind - and especially congenitally blind - participants' responses were more strongly correlated with the model, suggesting that language might be one of the possible compensative mechanisms for the lack of visual feedback in constructing bodily felt emotions. Our findings present theoretical implications for the study of emotions, as well as potential real-world applications for blind individuals, by revealing, on the one hand, that vision plays an essential role in the construction of felt emotions and the way we talk about our related bodily (emotional) experiences. On the other hand, evidence that blind individuals rely more strongly on linguistic cues suggests that vision is a strong cue to acquire emotional information from the surrounding world, influencing how we experience emotions. While our findings do not suggest that blind individuals experience emotions in an atypical and dysfunctional way, they nonetheless support the view that promoting the use of non-visual emotional signs and body language since early on might help the blind child to develop a good emotional awareness as well as good emotion regulation abilities.

Form to meaning mapping and the impact of explicit morpheme combination in novel word processing.

In the present study, we leveraged computational methods to explore the extent to which, relative to direct access to semantics from orthographic cues, the additional appreciation of morphological cues is advantageous while inducing the meaning of affixed pseudo-words. We re-analyzed data from a study on a lexical decision task for affixed pseudo-words. We considered a parsimonious model only including semantic variables (namely, semantic neighborhood density, entropy, magnitude, stem proximity) derived through a word-form-to-meaning approach (ngram-based). We then explored the extent to which the addition of equivalent semantic variables derived by combining semantic information from morphemes (combination-based) improved the fit of the statistical model explaining human data. Results suggest that semantic information can be extracted from arbitrary clusters of letters, yet a computational model of semantic access also including a combination-based strategy based on explicit morphological information better captures the cognitive mechanisms underlying human performance. This is particularly evident when participants recognize affixed pseudo-words as meaningful stimuli.

Lateralized reading in the healthy brain: A behavioral and computational study on the nature of the visual field effect.

Despite its widespread use to measure functional lateralization of language in healthy subjects, the neurocognitive bases of the visual field effect in lateralized reading are still debated. Crucially, the lack of knowledge on the nature of the visual field effect is accompanied by a lack of knowledge on the relative impact of psycholinguistic factors on its measurement, thus potentially casting doubts on its validity as a functional laterality measure. In this study, an eye-tracking-controlled tachistoscopic lateralized lexical decision task (Experiment 1) was administered to 60 right-handed and 60 left-handed volunteers and word length, orthographic neighborhood, word frequency, and imageability were manipulated. The magnitude of visual field effect was bigger in right-handed than in left-handed participants. Across the whole sample, a visual field-by-frequency interaction was observed, whereby a comparatively smaller effect of word frequency was detected in the left visual field/right hemisphere (LVF/RH) than in the right visual field/left hemisphere (RVF/LH). In a subsequent computational study (Experiment 2), efficient (LH) and inefficient (RH) activation of lexical orthographic nodes was modelled by means of the Naïve Discriminative Learning approach. Computational data simulated the effect of visual field and its interaction with frequency observed in the Experiment 1. Data suggest that the visual field effect can be biased by word frequency. Less distinctive connections between orthographic cues and lexical/semantic output units in the RH than in the LH can account for the emergence of the visual field effect and its interaction with word frequency.

Prediction at the intersection of sentence context and word form: Evidence from eye-movements and self-paced reading.

A key issue in language processing is how we recognize and understand words in sentences. Research on sentence reading indicates that the time we need to read a word depends on how (un)expected it is. Research on single word recognition shows that each word also has its own recognition dynamics based on the relation between its orthographic form and its meaning. It is not clear, however, how these sentence-level and word-level dynamics interact. In the present study, we examine the joint impact of these sources of information during sentence reading. We analyze existing eye-tracking and self-paced reading data (Frank et al., 2013, Behavior Research Methods, 45[4], 1182-1190) to investigate the interplay of sentence-level prediction (operationalized as Surprisal) and word Orthography-Semantics Consistency in activating word meaning in sentence processing. Results indicate that both Surprisal and Orthography-Semantics Consistency exert an influence on several reading measures. The shape of the observed interaction differs, but the results give compelling indication for a general trade-off between expectations based on sentence context and cues to meaning from word orthography.

Hands-on false memories: a combined study with distributional semantics and mouse-tracking.

Although mouse-tracking has been seen as a real-time window into different aspects of human decision-making processes, currently little is known about how the decision process unfolds in veridical and false memory retrieval. Here, we directly investigated decision-making processes by predicting participants' performance in a mouse-tracking version of a typical Deese-Roediger-McDermott (DRM) task through distributional semantic models, a usage-based approach to meaning. Participants were required to study lists of associated words and then to perform a recognition task with the mouse. Results showed that mouse trajectories were extensively affected by the semantic similarity between the words presented in the recognition phase and the ones previously studied. In particular, the higher the semantic similarity, the larger the conflict driving the choice and the higher the irregularity in the trajectory when correctly rejecting new words (i.e., the false memory items). Conversely, on the temporal evolution of the decision, our results showed that semantic similarity affects more complex temporal measures indexing the online decision processes subserving task performance. Together, these findings demonstrate that semantic similarity can affect human behavior at the level of motor control, testifying its influence on online decision-making processes. More generally, our findings complement previous seminal theories on false memory and provide insights into the impact of the semantic memory structure on different decision-making components.

ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation.

Quantitative, data-driven models for mental representations have long enjoyed popularity and success in psychology (e.g., distributional semantic models in the language domain), but have largely been missing for the visual domain. To overcome this, we present ViSpa (Vision Spaces), high-dimensional vector spaces that include vision-based representation for naturalistic images as well as concept prototypes. These vectors are derived directly from visual stimuli through a deep convolutional neural network trained to classify images and allow us to compute vision-based similarity scores between any pair of images and/or concept prototypes. We successfully evaluate these similarities against human behavioral data in a series of large-scale studies, including off-line judgments-visual similarity judgments for the referents of word pairs (Study 1) and for image pairs (Study 2), and typicality judgments for images given a label (Study 3)-as well as online processing times and error rates in a discrimination (Study 4) and priming task (Study 5) with naturalistic image material. ViSpa similarities predict behavioral data across all tasks, which renders ViSpa a theoretically appealing model for vision-based representations and a valuable research tool for data analysis and the construction of experimental material: ViSpa allows for precise control over experimental material consisting of images and/or words denoting imageable concepts and introduces a specifically vision-based similarity for word pairs. To make ViSpa available to a wide audience, this article (a) includes (video) tutorials on how to use ViSpa in R and (b) presents a user-friendly web interface at http://vispa.fritzguenther.de. (PsycInfo Database Record (c) 2023 APA, all rights reserved).