Scaffolding the human partner by contrastive guidance in an explanatory human-robot dialogue.

Explanation has been identified as an important capability for AI-based systems, but research on systematic strategies for achieving understanding in interaction with such systems is still sparse. Negation is a linguistic strategy that is often used in explanations. It creates a contrast space between the affirmed and the negated item that enriches explaining processes with additional contextual information. While negation in human speech has been shown to lead to higher processing costs and worse task performance in terms of recall or action execution when used in isolation, it can decrease processing costs when used in context. So far, it has not been considered as a guiding strategy for explanations in human-robot interaction. We conducted an empirical study to investigate the use of negation as a guiding strategy in explanatory human-robot dialogue, in which a virtual robot explains tasks and possible actions to a human explainee to solve them in terms of gestures on a touchscreen. Our results show that negation vs. affirmation 1) increases processing costs measured as reaction time and 2) increases several aspects of task performance. While there was no significant effect of negation on the number of initially correctly executed gestures, we found a significantly lower number of attempts-measured as breaks in the finger movement data before the correct gesture was carried out-when being instructed through a negation. We further found that the gestures significantly resembled the presented prototype gesture more following an instruction with a negation as opposed to an affirmation. Also, the participants rated the benefit of contrastive vs. affirmative explanations significantly higher. Repeating the instructions decreased the effects of negation, yielding similar processing costs and task performance measures for negation and affirmation after several iterations. We discuss our results with respect to possible effects of negation on linguistic processing of explanations and limitations of our study.

The time course of salience: not entirely caused by salience.

Visual salience is a key component of attentional selection, the process that guards the scarce resources needed for conscious recognition and perception. In previous works, we proposed a measure of visual salience based on a formal theory of visual selection. However, the strength of visual salience depends on the time course as well as local physical contrasts. Evidence from multiple experimental designs in the literature suggests that the strength of salience rises initially and declines after approximately 150 ms. The present article amends the theory-based salience measure beyond local physical contrasts to the time course of salience. It does so through a first experiment which reveals that-contrary to expectations-salience is not reduced during the first 150 ms after onset. Instead, the overall visual processing capacity is severely reduced, which corresponds to a reduced processing speed of all stimuli in the visual field. A second experiment confirms this conclusion by replicating the result. We argue that the slower stimulus processing may have been overlooked previously because the attentional selection mechanism had not yet been modeled in studies on the time course of salience.

The Role of Saliency in Learning First Words.

In word learning, one key accomplishment is the reference, that is, the linking of a word to its referent. According to classical theories, the term reference captures a mental event: A person uses a word to mentally recall a concept of an entity (an object or event) in order to bring it into the mental focus of an interaction. The developmental literature proposes different approaches regarding how children accomplish this link. Although researchers agree that multiple processes (within and across phonological, lexical, and semantic areas) are responsible for word learning, recent research has highlighted the role of saliency and perception as crucial factors in the early phases of word learning. Generally speaking, whereas some approaches to solving the reference problem attribute a greater role to the referent's properties being salient, others emphasize the social context that is needed to select the appropriate referent. In this review, we aim to systematize terminology and propose that the reason why assessments of the impact of saliency on word learning are controversial is that definitions of the term saliency reveal different weightings of the importance that either perceptual or social stimuli have for the learning process. We propose that defining early word learning in terms of paying attention to salient stimuli is too narrow. Instead, we emphasize that a new link between a word and its referent will succeed if a stimulus is relevant for the child.

Poking Left To Be Right? A Model-Based Analysis of Temporal Order Judged by Mice.

The theory of visual attention (TVTVA) provides a formal framework for the assessment of visual attention and related processes. Its center is a mathematical model of visual encoding processes and discretely defined components of attention. Building on this model, TVTVA offers quantitative and process-related explanations for a variety of phenomena in the domain of visual attention. Because the theory relies on very general assumptions which might hold true for other domains of sensory processing, we tested its possible explanatory value for tactile processing in mice. Reanalyzing published data of temporal-order judgments by mice, we show how a TVTVA-based analysis identifies the processes which drive observable behavior and that it comes to conclusions quite different from those of conventional analyses of temporal-order judgments. According to this analysis, despite the same overall capacity dedicated to the task, some mice assume attentional biases toward one side, possibly to optimize their overall performance. We suggest that TVTVA's concepts provide a powerful point of vantage to find explanations for observable behavior where conventional analysis easily leads to dead ends.

Stuck on a Plateau? A Model-Based Approach to Fundamental Issues in Visual Temporal-Order Judgments.

Humans are incapable of judging the temporal order of visual events at brief temporal separations with perfect accuracy. Their performance-which is of much interest in visual cognition and attention research-can be measured with the temporal-order judgment (TOJ) task, which typically produces S-shaped psychometric functions. Occasionally, researchers reported plateaus within these functions, and some theories predict such deviation from the basic S shape. However, the centers of the psychometric functions result from the weakest performance at the most difficult presentations and therefore fluctuate strongly, leaving the existence and exact shapes of plateaus unclear. This study set out to investigate whether plateaus disappear if the data accuracy is enhanced, or if we are "stuck on a plateau", or rather with it. For this purpose, highly accurate data were assessed by model-based analysis. The existence of plateaus is confidently confirmed and two plausible mechanisms derived from very different models are presented. Neither model, however, performs well in the presence of a strong attention manipulation, and model comparison remains unclear on the question of which of the models describes the data best. Nevertheless, the present study includes the highest accuracy in visual TOJ data and the most explicit models of plateaus in TOJ studied so far.

Measuring and modeling salience with the theory of visual attention.

For almost three decades, the theory of visual attention (TVA) has been successful in mathematically describing and explaining a wide variety of phenomena in visual selection and recognition with high quantitative precision. Interestingly, the influence of feature contrast on attention has been included in TVA only recently, although it has been extensively studied outside the TVA framework. The present approach further develops this extension of TVA's scope by measuring and modeling salience. An empirical measure of salience is achieved by linking different (orientation and luminance) contrasts to a TVA parameter. In the modeling part, the function relating feature contrasts to salience is described mathematically and tested against alternatives by Bayesian model comparison. This model comparison reveals that the power function is an appropriate model of salience growth in the dimensions of orientation and luminance contrast. Furthermore, if contrasts from the two dimensions are combined, salience adds up additively.

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments.

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen's Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level. To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

Peripheral Visual Cues: Their Fate in Processing and Effects on Attention and Temporal-Order Perception.

Peripheral visual cues lead to large shifts in psychometric distributions of temporal-order judgments. In one view, such shifts are attributed to attention speeding up processing of the cued stimulus, so-called prior entry. However, sometimes these shifts are so large that it is unlikely that they are caused by attention alone. Here we tested the prevalent alternative explanation that the cue is sometimes confused with the target on a perceptual level, bolstering the shift of the psychometric function. We applied a novel model of cued temporal-order judgments, derived from Bundesen's Theory of Visual Attention. We found that cue-target confusions indeed contribute to shifting psychometric functions. However, cue-induced changes in the processing rates of the target stimuli play an important role, too. At smaller cueing intervals, the cue increased the processing speed of the target. At larger intervals, inhibition of return was predominant. Earlier studies of cued TOJs were insensitive to these effects because in psychometric distributions they are concealed by the conjoint effects of cue-target confusions and processing rate changes.

Fast and Conspicuous? Quantifying Salience With the Theory of Visual Attention.

Particular differences between an object and its surrounding cause salience, guide attention, and improve performance in various tasks. While much research has been dedicated to identifying which feature dimensions contribute to salience, much less regard has been paid to the quantitative strength of the salience caused by feature differences. Only a few studies systematically related salience effects to a common salience measure, and they are partly outdated in the light of new findings on the time course of salience effects. We propose Bundesen's Theory of Visual Attention (TVA) as a theoretical basis for measuring salience and introduce an empirical and modeling approach to link this theory to data retrieved from temporal-order judgments. With this procedure, TVA becomes applicable to a broad range of salience-related stimulus material. Three experiments with orientation pop-out displays demonstrate the feasibility of the method. A 4th experiment substantiates its applicability to the luminance dimension.

Does attention speed up processing? Decreases and increases of processing rates in visual prior entry.

Selective visual attention improves performance in many tasks. Among others, it leads to "prior entry"--earlier perception of an attended compared to an unattended stimulus. Whether this phenomenon is purely based on an increase of the processing rate of the attended stimulus or if a decrease in the processing rate of the unattended stimulus also contributes to the effect is, up to now, unanswered. Here we describe a novel approach to this question based on Bundesen's Theory of Visual Attention, which we use to overcome the limitations of earlier prior-entry assessment with temporal order judgments (TOJs) that only allow relative statements regarding the processing speed of attended and unattended stimuli. Prevalent models of prior entry in TOJs either indirectly predict a pure acceleration or cannot model the difference between acceleration and deceleration. In a paradigm that combines a letter-identification task with TOJs, we show that indeed acceleration of the attended and deceleration of the unattended stimuli conjointly cause prior entry.

Attention and the speed of information processing: posterior entry for unattended stimuli instead of prior entry for attended stimuli.

Why are nearly simultaneous stimuli frequently perceived in reversed order? The origin of errors in temporal judgments is a question older than experimental psychology itself. One of the earliest suspects is attention. According to the concept of prior entry, attention accelerates attended stimuli; thus they have "prior entry" to perceptive processing stages, including consciousness. Although latency advantages for attended stimuli have been revealed in psychophysical studies many times, these measures (e.g. temporal order judgments, simultaneity judgments) cannot test the prior-entry hypothesis completely. Since they assess latency differences between an attended and an unattended stimulus, they cannot distinguish between faster processing of attended stimuli and slower processing of unattended stimuli. Therefore, we present a novel paradigm providing separate estimates for processing advantages respectively disadvantages of attended and unattended stimuli. We found that deceleration of unattended stimuli contributes more strongly to the prior-entry illusion than acceleration of attended stimuli. Thus, in the temporal domain, attention fulfills its selective function primarily by deceleration of unattended stimuli. That means it is actually posterior entry, not prior entry which accounts for the largest part of the effect.

At the mercy of prior entry: Prior entry induced by invisible primes is not susceptible to current intentions.

If one of two events is attended to, it will be perceived earlier than a simultaneously occurring unattended event. Since 150 years, this effect has been ascribed to the facilitating influence of attention, also known as prior entry. Yet, the attentional origin of prior-entry effects(1) has been repeatedly doubted. One criticism is that prior-entry effects might be due to biased decision processes that would mimic a temporal advantage for attended stimuli. Although most obvious biases have already been excluded experimentally (e.g. judgment criteria, response compatibility) and prior-entry effects have shown to persist (Shore, Spence, & Klein, 2001), many other biases are conceivable, which makes it difficult to put the debate to an end. Thus, we approach this problem the other way around by asking whether prior-entry effects can be biased voluntarily. Observers were informed about prior entry and instructed to reduce it as far as possible. For this aim they received continuous feedback about the correctness of their temporal judgments. If elicited by invisible primes the effect could not be reduced at all and only by 12 ms if elicited by visible cues. This challenges decision biases as primary source of prior-entry effects - at least if attention is oriented exogenously.

Prior entry and temporal attention: cueing affects order errors in RSVP.

The law of prior entry states that attended objects come to consciousness more quickly than unattended ones. This has been well established in spatial cueing paradigms, where two task-relevant stimuli are presented near-simultaneously at two different locations. Here, we suggest that prior entry also plays a pivotal role in temporal attention paradigms, where stimuli appear at the same location but at distinct moments in time, in rapid serial presentation (RSVP). Specifically, we hypothesize that prior entry can explain temporal order reversals in reporting two targets from RSVP. In support of this, three experiments show that cueing attention toward either of the targets has a strong influence on order errors. We conclude that prior entry provides a viable explanation of the way in which relevant information is prioritized in RSVP.

Spatial mislocalization as a consequence of sequential coding of stimuli.

In three experiments, we tested whether sequentially coding two visual stimuli can create a spatial misperception of a visual moving stimulus. In Experiment 1, we showed that a spatial misperception, the flash-lag effect, is accompanied by a similar temporal misperception of first perceiving the flash and only then a change of the moving stimulus, when in fact the two events were exactly simultaneous. In Experiment 2, we demonstrated that when the spatial misperception of a flash-lag effect is absent, the temporal misperception is also absent. In Experiment 3, we extended these findings and showed that if the stimulus conditions require coding first a flash and subsequently a nearby moving stimulus, a spatial flash-lag effect is found, with the position of the moving stimulus being misperceived as shifted in the direction of its motion, whereas this spatial misperception is reversed so that the moving stimulus is misperceived as shifted in a direction opposite to its motion when the conditions require coding first the moving stimulus and then the flash. Together, the results demonstrate that sequential coding of two stimuli can lead to a spatial misperception whose direction can be predicted from the order of coding the moving object versus the flash. We propose an attentional sequential-coding explanation for the flash-lag effect and discuss its explanatory power with respect to related illusions (e.g., the Fröhlich effect) and other explanations.

Prior entry explains order reversals in the attentional blink.

When two targets are presented in rapid succession, the first target (T1) is usually identified, but the second target (T2) is often missed. A remarkable exception to this "attentional blink" occurs when T2 immediately follows the first T1, at lag 1. It is then often spared but reported in the wrong order--that is, before T1. These order reversals have led to the hypothesis that "lag 1 sparing" occurs because the two targets merge into a single episodic representation. Here, we report evidence consistent with an alternative theory: T2 receives more attention than T1, leading to prior entry into working memory. Two experiments showed that the more T2 performance exceeded that for T1, the more order reversals were made. Furthermore, precuing T1 led to a shift in performance benefits from T2 to T1 and to an equivalent reduction in order reversals. We conclude that it is not necessary to assume episodic integration to explain lag 1 sparing or the accompanying order reversals.

Simultaneity and temporal order perception: Different sides of the same coin? Evidence from a visual prior-entry study.

Attended stimuli are perceived as occurring earlier than unattended stimuli. This phenomenon of prior entry is usually identified by a shift in the point of subjective simultaneity (PSS) in temporal order judgements (TOJs). According to its traditional psychophysical interpretation, the PSS coincides with the perception of simultaneity. This assumption is, however, questionable. Technically, the PSS represents the temporal interval between two stimuli at which the two alternative TOJs are equally likely. Thus it also seems possible that observers perceive not simultaneity, but uncertainty of temporal order. This possibility is supported by prior-entry studies, which find that perception of simultaneity is not very likely at the PSS. The present study tested the percept at the PSS in prior entry, using peripheral cues to orient attention. We found that manipulating attention caused varying temporal perceptions around the PSS. On some occasions observers perceived the two stimuli as simultaneous, but on others they were simply uncertain about the order in which they had been presented. This finding contradicts the implicit assumption of most models of temporal order perception, that perception of simultaneity inevitably results if temporal order cannot be discriminated.

Top-down contingent feature-specific orienting with and without awareness of the visual input.

In the present article, the role of endogenous feature-specific orienting for conscious and unconscious vision is reviewed. We start with an overview of orienting. We proceed with a review of masking research, and the definition of the criteria of experimental protocols that demonstrate endogenous and exogenous orienting, respectively. Against this background of criteria, we assess studies of unconscious orienting and come to the conclusion that so far studies of unconscious orienting demonstrated endogenous feature-specific orienting. The review closes with a discussion of the role of unconscious orienting in action control.

Enhancing implicit change detection through action.

Implicit change detection demonstrates how the visual system can benefit from stored information that is not immediately available to conscious awareness. We investigated the role of motor action in this context. In the first two experiments, using a one-shot implicit change-detection paradigm, participants responded to unperceived changes either with an action (jabbing the screen at the guessed location of a change) or with words (verbal report), and sat either 60 cm or 300 cm (with a laser pointer) away from the display. Our observers guessed the locations of changes at a reachable distance better with an action than with a verbal judgment. At 300 cm, beyond reach, the motor advantage disappeared. In experiment 3, this advantage was also unavailable when participants sat at a reachable distance but responded with hand-held laser pointers near their bodies. We conclude that a motor system specialized for real-time visually guided behavior has access to additional visual information. Importantly, this system is not activated by merely executing an action (experiment 2) or presenting stimuli in one's near space (experiment 3). It is activated only when both conditions are fulfilled, which implies that it is the actual contact that matters to the visual system.

Temporal processes in prime-mask interaction: Assessing perceptual consequences of masked information.

Visual backward masking is frequently used to study the temporal dynamics of visual perception. These dynamics may include the temporal features of conscious percepts, as suggested, for instance, by the asynchronous-updating model (Neumann, 1982) and perceptual-retouch theory ((Bachmann, 1994). These models predict that the perceptual latency of a visual backward mask is shorter than that of a like reference stimulus that was not preceded by a masked stimulus. The prediction has been confirmed by studies using temporal-order judgments: For certain asynchronies between mask and reference stimulus, temporal-order reversals are quite frequent (e.g. Scharlau, & Neumann, 2003a). However, it may be argued that these reversals were due to a response bias in favour of the mask rather than true temporal-perceptual effects. I introduce two measures for assessing latency effects that (1) are not prone to such a response bias, (2) allow to quantify the latency gain, and (3) extend the perceptual evidence from order reversals to duration/interval perception, that is, demonstrate that the perceived interval between a mask and a reference stimulus may be shortened as well as prolonged by the presence of a masked stimulus. Consequences for theories of visual masking such as asynchronous-updating, perceptual-retouch, and reentrant models are discussed.