Mutual influence between language and perception in multi-agent communication games.

Language interfaces with many other cognitive domains. This paper explores how interactions at these interfaces can be studied with deep learning methods, focusing on the relation between language emergence and visual perception. To model the emergence of language, a sender and a receiver agent are trained on a reference game. The agents are implemented as deep neural networks, with dedicated vision and language modules. Motivated by the mutual influence between language and perception in cognition, we apply systematic manipulations to the agents' (i) visual representations, to analyze the effects on emergent communication, and (ii) communication protocols, to analyze the effects on visual representations. Our analyses show that perceptual biases shape semantic categorization and communicative content. Conversely, if the communication protocol partitions object space along certain attributes, agents learn to represent visual information about these attributes more accurately, and the representations of communication partners align. Finally, an evolutionary analysis suggests that visual representations may be shaped in part to facilitate the communication of environmentally relevant distinctions. Aside from accounting for co-adaptation effects between language and perception, our results point out ways to modulate and improve visual representation learning and emergent communication in artificial agents.

Mutual Exclusivity in Pragmatic Agents.

One of the great challenges in word learning is that words are typically uttered in a context with many potential referents. Children's tendency to associate novel words with novel referents, which is taken to reflect a mutual exclusivity (ME) bias, forms a useful disambiguation mechanism. We study semantic learning in pragmatic agents-combining the Rational Speech Act model with gradient-based learning-and explore the conditions under which such agents show an ME bias. This approach provides a framework for investigating a pragmatic account of the ME bias in humans but also for building artificial agents that display an ME bias. A series of analyses demonstrates striking parallels between our model and human word learning regarding several aspects relevant to the ME bias phenomenon: online inference, long-term learning, and developmental effects. By testing different implementations, we find that two components, pragmatic online inference and incremental collection of evidence for one-to-one correspondences between words and referents, play an important role in modeling the developmental trajectory of the ME bias. Finally, we outline an extension of our model to a deep neural network architecture that can process more naturalistic visual and linguistic input. Until now, in contrast to children, deep neural networks have needed indirect access to (supposed to be novel) test inputs during training to display an ME bias. Our model is the first one to do so without using this manipulation.

Infants Understand Others' Needs.

Infants begin to help other individuals in the second year of life. However, it is still unclear whether early helping behavior is based on an understanding of other individuals' needs and is thus motivated prosocially. In the present eye-tracking study, 9- to 18-month-old infants (N= 71) saw a character in need of help, unable to reach its goal because of an obstacle, and a second character that was able to achieve a goal on its own. When a third individual (a helper) initiated an action, the infants expected the helper to help the character in need (as indicated during the anticipatory-looking and violation-of-expectation phases). Their prosocial understanding did not differ between age groups and was not related to their helping behavior (measured in two behavioral tasks). Thus, infants understand other individuals' needs even before they start to help others themselves. This indicates that early helping may indeed be motivated prosocially and raises the question of which other competences underlie the ontogeny of helping behavior.