Keep your finger on the pulse: Better rate perception and gap detection with vibrotactile compared to visual stimuli.

Important characteristics of the environment can be represented in the temporal pattern of sensory stimulation. In two experiments, we compared accuracy of temporal processing by different modalities. Experiment 1 examined binary categorization of rate for visual (V) or vibrotactile (T) stimulus pulses presented at either 4 or 6 Hz. Inter-pulse intervals were either constant or variable, perturbed by random Gaussian variates. Subjects categorized the rate of T pulse sequences more accurately than V sequences. In V conditions only, subjects disproportionately tended to mis-categorize 4-Hz pulse rates, for all but the most variable sequences. In Experiment 2, we compared gap detection thresholds across modalities, using the same V and T pulses from Experiment 1, as well as with bimodal (VT) pulses. Visual gap detection thresholds were larger (3[Formula: see text]) than tactile thresholds. Additionally, performance with VT stimuli seemed to be nearly completely dominated by their T components. Together, these results suggest (i) that vibrotactile temporal acuity surpasses visual temporal acuity, and (ii) that vibrotactile stimulation has considerable, untapped potential to convey temporal information like that needed for eyes-free alerting signals.

Audiovisual interaction with rate-varying signals.

A task-irrelevant, amplitude-modulating sound influences perception of a size-modulating visual stimulus. To probe the limits of this audiovisual interaction we vary the second temporal derivative of object size and of sound amplitude. In the study's first phase subjects see a visual stimulus size-modulating with f ″ ( x ) > 0, 0, or <0, and judge each one's rate as increasing, constant, or decreasing. Visual stimuli are accompanied by a steady, non-modulated auditory stimulus. The novel combination of multiple stimuli and multi-alternative responses allows subjects' similarity space to be estimated from the stimulus-response confusion matrix. In the study's second phase, rate-varying visual stimuli are presented in concert with auditory stimuli whose second derivative also varied. Subjects identified each visual stimuli as one of the three types, while trying to ignore the accompanying sound. Unlike some previous results with f ″ ( x ) fixed at 0, performance benefits relatively little when visual and auditory stimuli share the same directional change in modulation. However, performance does drop when visual and auditory stimului differ in their directions of rate change. Our task's computational demands may make it particularly vulnerable to the effects of a dynamic task-irrelevant stimulus.

Costs and benefits of audiovisual interactions.

A strong temporal correlation promotes integration of concurrent sensory signals, either within a single sensory modality, or from different modalities. Although the benefits of such integration are well known, far less attention has been given to possible costs incurred when concurrent sensory signals are uncorrelated. In two experiments, subjects categorized the rate at which a visual object modulated in size, while they also tried to ignore a concurrent task-irrelevant broadband sound. Overall, the experiments showed that (i) losses in accuracy from mismatched auditory and visual rates were larger than gains from matched rates and (ii) mismatched auditory and visual rates slowed responses more than they were sped up when rates matched. Experiment One showed that audiovisual interaction varied with the difference between the visual modulation rate and the modulation rate of a concurrent auditory stimulus. Experiment Two showed that audiovisual interaction depended upon the strength of the task-irrelevant auditory modulation. Although our stimuli involved abstract, low-dimensional stimuli, not speech, the effects we observed parallel key findings on interference in multi-speaker settings.

Feeling rushed? Perceived time pressure impacts executive function and stress.

Executive function (EF) is critical to everyday life, but it can be undermined by adverse psychological states like stress and negative affect. For example, inadequate time to perform a task is a common stressor that can disrupt EF. Although the impact of actual time pressure on EF has been established, little is known about how self-generated, perceived time pressure (PTP) affects EF in the absence of objective time limits. We chose Eriksen's Flanker task as an index of cognitive inhibition, a key component of EF, and we varied the interval between successive trials, the inter-trial interval (ITI), to proxy PTP. This manipulation strongly impacted task performance: shrinking the ITI to increase PTP diminished cognitive inhibition and increased both stress and negative affect. Subsequently lengthening the ITI to decrease PTP reversed nearly all of these effects, except stress, which persisted. Multilevel linear regression modeling revealed that ITI and stress predicted inhibition, and exploratory mediation modeling suggested that stress mediates the relationship between ITI and inhibition. These findings validate perceived time pressure as an empirical stressor and demonstrate EF's sensitivity to changes in PTP.

Spatial frequency impacts perceptual and attentional ERP components across cultures.

Culture impacts visual perception in several ways.To identify stages of perceptual processing that differ between cultures, we usedelectroencephalography measures of perceptual and attentional responses to simple visual stimuli.Gabor patches of higher or lower spatialfrequencywere presented at high contrast to 25 American and 31 East Asian participants while they were watching for the onset of aninfrequent, oddball stimulus. Region of interest and mass univariate analyses assessed how cultural background and stimuli spatial frequency affected the visual evoked response potentials. Across both groups, the Gabor of lower spatial frequency produced stronger evoked response potentials in the anterior N1 and P3 than did the higher frequency Gabor. The mass univariate analyses also revealed effects of spatial frequency, including a frontal negativity around 150 ms and a widespread posterior positivity around 300 ms. The effects of spatial frequency generally differed little across cultures; although there was some evidence for cultural differences in the P3 response to different frequencies at the Pz electrode, this effect did not emerge in the mass univariate analyses. We discuss these results in relation to those from previous studies, and explore the potential advantages of mass univariate analyses for cultural neuroscience.

Decision-Making and Multisensory Combination Under Time Stress.

Using a video game platform, we examined how vision-based decision making was affected by a concurrent, potentially conflicting auditory stimulus. Electroencephalographic responses showed that by 150 milliseconds of stimulus onset, the brain had detected the conflict between visual and auditory stimuli. Systematically reducing the intertrial interval (ITI), which subjects described as stressful, undermined decision making. Subjects' arterial pulse variance decreased along with ITI, signaling increased parasympathetic influence on the heart. When successive trials required a shift in processing mode, short ITIs significantly boosted one trial's influence on the next, suggesting that stress reduces cognitive flexibility. Finally, our study demonstrates the heart's and the brain's important influence on decision making.

Perceptual timing precision with vibrotactile, auditory, and multisensory stimuli.

The growing use of vibrotactile signaling devices makes it important to understand the perceptual limits on vibrotactile information processing. To promote that understanding, we carried out a pair of experiments on vibrotactile, auditory, and bimodal (synchronous vibrotactile and auditory) temporal acuity. On each trial, subjects experienced a set of isochronous, standard intervals (400 ms each), followed by one interval of variable duration (400 ± 1-80 ms). Intervals were demarcated by short vibrotactile, auditory, or bimodal pulses. Subjects categorized the timing of the last interval by describing the final pulse as either "early" or "late" relative to its predecessors. In Experiment 1, each trial contained three isochronous standard intervals, followed by an interval of variable length. In Experiment 2, the number of isochronous standard intervals per trial varied, from one to four. Psychometric modeling revealed that vibrotactile stimulation produced poorer temporal discrimination than either auditory or bimodal stimulation. Moreover, auditory signals dominated bimodal sensitivity, and inter-individual differences in temporal discriminability were reduced with bimodal stimulation. Additionally, varying the number of isochronous intervals in a trial failed to improve temporal sensitivity in either modality, suggesting that memory played a key role in judgments of interval duration.

Cultural differences in performance on Eriksen's flanker task.

Eriksen's zoom model of attention implies a trade-off between the breadth and resolution of representations of information. Following this perspective, we used Eriksen's flanker task to investigate culture's influence on attentional allocation and attentional resolution. In Experiment 1, the spatial distance of the flankers was varied to test whether people from Eastern cultures (here, Turks) experienced more interference than people from Western cultures (here, Americans) when flankers were further from the target. In Experiment 2, the contrast of the flankers was varied. The pattern of results shows that congruency of the flankers (Experiment 1) as well as the degree of contrast of the flankers compared with the target (Experiment 2) interact with participants' cultural background to differentially influence accuracy or reaction times. In addition, we used evidence accumulation modeling to jointly consider measures of speed and accuracy. Results indicate that to make decisions in the Eriksen flanker task, Turks both accumulate evidence faster and require more evidence than Americans do. These cultural differences in visual attention and decision-making have implications for a wide variety of cognitive processes.

Feeling the Beat (and Seeing It, Too): Vibrotactile, Visual, and Bimodal Rate Discrimination.

Beats are among the basic units of perceptual experience. Produced by regular, intermittent stimulation, beats are most commonly associated with audition, but the experience of a beat can result from stimulation in other modalities as well. We studied the robustness of visual, vibrotactile, and bimodal signals as sources of beat perception. Subjects attempted to discriminate between pulse trains delivered at 3 Hz or at 6 Hz. To investigate signal robustness, we intentionally degraded signals on two-thirds of the trials using temporal-domain noise. On these trials, inter-pulse intervals (IPIs) were stochastic, perturbed independently from the nominal IPI by random samples from zero-mean Gaussian distributions with different variances. These perturbations produced directional changes in the IPIs, which either increased or decreased the likelihood of confusing the two pulse rates. In addition to affording an assay of signal robustness, this paradigm made it possible to gauge how subjects' judgments were influenced by successive IPIs. Logistic regression revealed a strong primacy effect: subjects' decisions were disproportionately influenced by a trial's initial IPIs. Response times and parameter estimates from drift-diffusion modeling showed that information accumulates more rapidly with bimodal stimulation than with either unimodal stimulus alone. Analysis of error rates within each condition suggested consistently optimal decision making, even with increased IPI variability. Finally, beat information delivered by vibrotactile signals proved just as robust as information conveyed by visual signals, confirming vibrotactile stimulation's potential as a communication channel.

What makes a prototype a prototype? Averaging visual features in a sequence.

After viewing a series of sequentially presented visual stimuli, subjects can readily generate mean representations of various visual features. Although these average representations seem to be formed effortlessly and obligatorily, it is unclear how such averages are actually computed. According to conventional prototype models, the computation entails an equally weighted average taken over all the stimuli. To test this hypothesis, we had subjects estimate the running averages of some feature in a series of sequentially presented stimuli. Part way through the series, we perturbed the distribution from which stimuli were drawn, which allowed us to test alternative models of the computations behind subjects' estimates. In both explanatory and predictive tests, a model in which the most recent items had disproportionate high weight outperformed a model in which all items carried equal weight. Such recency-weighted behavior was shown consistently in multiple experiments in which subjects estimated running averages of length of vertical lines. However, the degree to which recent items were prioritized varied with the type of stimulus, such that when estimating the running averages of a series of numerals, subjects showed less recency prioritization. We conclude that previous evaluations of prototype models have made unrealistic assumptions about the nature of a prototype, and that a reassessment of prototype models of visual memory and perceptual categorization may be in order.

Older adults show impaired modulation of attentional alpha oscillations: Evidence from dichotic listening.

Auditory attention is critical for selectively listening to speech from a single talker in a multitalker environment (e.g., Cherry, 1953). Listening in such situations is notoriously more difficult and more poorly encoded to long-term memory in older than in young adults (Tun, O'Kane, & Wingfield, 2002). Recent work by Payne, Rogers, Wingfield, and Sekuler (2017) in young adults demonstrated a neural correlate of auditory attention in the directed dichotic listening task (DDLT), where listeners attend to one ear while ignoring the other. Measured using electroencephalography, differences in alpha band power (8-14 Hz) between left and right hemisphere parietal regions mark the direction to which auditory attention is focused. Little prior research has been conducted on alpha power modulations in older adults, particularly with regard to auditory attention directed toward speech stimuli. In the current study, an older adult sample was administered the DDLT and delayed recognition procedures used by Payne et al. (2017). Compared to young adults, older adults showed reduced selective attention in the DDLT, evidenced by a higher rate of intrusions from the unattended ear. Moreover, older adults did not exhibit attention-related alpha modulation evidenced by young adults, nor did their event-related potentials (ERPs) to recognition probes differentiate between attended or unattended probes. Older adults' delayed recognition did not reveal a pattern of suppression of unattended items evidenced by young adults. These results serve as evidence for an age-related decline in selective auditory attention, potentially mediated by age-related decline in the ability to modulate alpha oscillations. (PsycINFO Database Record

Multisensory Integration in Short-term Memory: Musicians do Rock.

Demonstrated interactions between seeing and hearing led us to assess the link between music training and short-term memory for auditory, visual and audiovisual sequences of rapidly presented, quasi-random components. Visual sequences' components varied in luminance; auditory sequences' components varied in frequency. Concurrent components in audiovisual sequences were either congruent (the frequency of an auditory item increased monotonically with the luminance of the visual item it accompanied), or incongruent (an item's frequency was uncorrelated with luminance of the item it accompanied). Subjects judged whether the last four items in a sequence replicated its first four items. With audiovisual sequences, subjects were instructed to ignore the sequence's auditory components, basing their judgments solely on the visual input. Subjects with prior instrumental training significantly outperformed their untrained counterparts, with both auditory and visual sequences, and with sequences of correlated auditory and visual items. Reverse correlation showed that the presence of a correlated, concurrent auditory stream altered subjects' reliance on particular visual items in a sequence. Moreover, congruence between auditory and visual items produced performance above what would be predicted from simple summation of information from the two modalities, a result that might reflect a contribution from special-purpose, multimodal neural mechanisms.

Catching Audiovisual Interactions With a First-Person Fisherman Video Game.

The human brain is excellent at integrating information from different sources across multiple sensory modalities. To examine one particularly important form of multisensory interaction, we manipulated the temporal correlation between visual and auditory stimuli in a first-person fisherman video game. Subjects saw rapidly swimming fish whose size oscillated, either at 6 or 8 Hz. Subjects categorized each fish according to its rate of size oscillation, while trying to ignore a concurrent broadband sound seemingly emitted by the fish. In three experiments, categorization was faster and more accurate when the rate at which a fish oscillated in size matched the rate at which the accompanying, task-irrelevant sound was amplitude modulated. Control conditions showed that the difference between responses to matched and mismatched audiovisual signals reflected a performance gain in the matched condition, rather than a cost from the mismatched condition. The performance advantage with matched audiovisual signals was remarkably robust over changes in task demands between experiments. Performance with matched or unmatched audiovisual signals improved over successive trials at about the same rate, emblematic of perceptual learning in which visual oscillation rate becomes more discriminable with experience. Finally, analysis at the level of individual subjects' performance pointed to differences in the rates at which subjects can extract information from audiovisual stimuli.

Bi-directional audiovisual influences on temporal modulation discrimination.

Cross-modal interactions of auditory and visual temporal modulation were examined in a game-like experimental framework. Participants observed an audiovisual stimulus (an animated, sound-emitting fish) whose sound intensity and/or visual size oscillated sinusoidally at either 6 or 7 Hz. Participants made speeded judgments about the modulation rate in either the auditory or visual modality while doing their best to ignore information from the other modality. Modulation rate in the task-irrelevant modality matched the modulation rate in the task-relevant modality (congruent conditions), was at the other rate (incongruent conditions), or had no modulation (unmodulated conditions). Both performance accuracy and parameter estimates from drift-diffusion decision modeling indicated that (1) the presence of temporal modulation in both modalities, regardless of whether modulations were matched or mismatched in rate, resulted in audiovisual interactions; (2) congruence in audiovisual temporal modulation resulted in more reliable information processing; and (3) the effects of congruence appeared to be stronger when judging visual modulation rates (i.e., audition influencing vision), than when judging auditory modulation rates (i.e., vision influencing audition). The results demonstrate that audiovisual interactions from temporal modulations are bi-directional in nature, but with potential asymmetries in the size of the effect in each direction.

Characterizing the roles of alpha and theta oscillations in multisensory attention.

Cortical alpha oscillations (8-13Hz) appear to play a role in suppressing distractions when just one sensory modality is being attended, but do they also contribute when attention is distributed over multiple sensory modalities? For an answer, we examined cortical oscillations in human subjects who were dividing attention between auditory and visual sequences. In Experiment 1, subjects performed an oddball task with auditory, visual, or simultaneous audiovisual sequences in separate blocks, while the electroencephalogram was recorded using high-density scalp electrodes. Alpha oscillations were present continuously over posterior regions while subjects were attending to auditory sequences. This supports the idea that the brain suppresses processing of visual input in order to advantage auditory processing. During a divided-attention audiovisual condition, an oddball (a rare, unusual stimulus) occurred in either the auditory or the visual domain, requiring that attention be divided between the two modalities. Fronto-central theta band (4-7Hz) activity was strongest in this audiovisual condition, when subjects monitored auditory and visual sequences simultaneously. Theta oscillations have been associated with both attention and with short-term memory. Experiment 2 sought to distinguish these possible roles of fronto-central theta activity during multisensory divided attention. Using a modified version of the oddball task from Experiment 1, Experiment 2 showed that differences in theta power among conditions were independent of short-term memory load. Ruling out theta's association with short-term memory, we conclude that fronto-central theta activity is likely a marker of multisensory divided attention.

Online Modulation of Selective Attention is not Impaired in Healthy Aging.

Background/Study Context: Reduced processing speed pervades a great many aspects of human aging and cognition. However, little is known about one aspect of cognitive aging in which speed is of the essence, namely, the speed with which older adults can deploy attention in response to a cue. METHODS: The authors compared rapid temporal modulation of cued visual attention in younger (Mage = 22.3 years) and older (Mage = 68.9 years) adults. On each trial of a short-term memory task, a cue identified which of two briefly presented stimuli was task relevant and which one should be ignored. After a short delay, subjects demonstrated recall by reproducing from memory the task-relevant stimulus. This produced estimates of (i) accuracy with which the task-relevant stimulus was recalled, (ii) the influence of stimuli encountered on previous trials (a prototype effect), and (iii) the influence of the trial's task-irrelevant stimulus. RESULTS: For both groups, errors in recall were considerably smaller when selective attention was cued before rather than after presentation of the stimuli. Both groups showed serial position effects to the same degree, and both seemed equally adept at exploiting the stimuli encountered on previous trials as a means of supplementing recall accuracy on the current trial. CONCLUSION: Younger and older subjects may not differ reliably in capacity for cue-directed temporal modulation of selective attention, or in ability to draw on previously seen stimuli as memory support.

Memory and learning for visual signals in time and space.

Vision is often characterized as a spatial sense, but what does that characterization imply about the relative ease of processing visual information distributed over time rather than over space? Three experiments addressed this question, using stimuli comprising random luminances. For some stimuli, individual items were presented sequentially, at 8 Hz; for other stimuli, individual items were presented simultaneously, as horizontal spatial arrays. For temporal sequences, subjects judged whether each of the last four luminances matched the corresponding luminance in the first four; for spatial arrays, they judged whether each of the right-hand four luminances matched the corresponding left-hand luminance. Overall, performance was far better with spatial presentations, even when the entire spatial array was presented for just tens of milliseconds. Experiment 2 demonstrated that there was no gain in performance from combining spatial and temporal information within a single stimulus. In a final experiment, particular spatial arrays or temporal sequences were made to recur intermittently, interspersed among, non-recurring stimuli. Performance improved steadily as particular stimulus exemplars recurred, with spatial and temporal stimuli being learned at equivalent rates. Logistic regression identified several shortcut strategies that subjects may have exploited while performing our task.

A right-ear bias of auditory selective attention is evident in alpha oscillations.

Auditory selective attention makes it possible to pick out one speech stream that is embedded in a multispeaker environment. We adapted a cued dichotic listening task to examine suppression of a speech stream lateralized to the nonattended ear, and to evaluate the effects of attention on the right ear's well-known advantage in the perception of linguistic stimuli. After being cued to attend to input from either their left or right ear, participants heard two different four-word streams presented simultaneously to the separate ears. Following each dichotic presentation, participants judged whether a spoken probe word had been in the attended ear's stream. We used EEG signals to track participants' spatial lateralization of auditory attention, which is marked by interhemispheric differences in EEG alpha (8-14 Hz) power. A right-ear advantage (REA) was evident in faster response times and greater sensitivity in distinguishing attended from unattended words. Consistent with the REA, we found strongest parietal and right frontotemporal alpha modulation during the attend-right condition. These findings provide evidence for a link between selective attention and the REA during directed dichotic listening.

Memory and learning with rapid audiovisual sequences.

We examined short-term memory for sequences of visual stimuli embedded in varying multisensory contexts. In two experiments, subjects judged the structure of the visual sequences while disregarding concurrent, but task-irrelevant auditory sequences. Stimuli were eight-item sequences in which varying luminances and frequencies were presented concurrently and rapidly (at 8 Hz). Subjects judged whether the final four items in a visual sequence identically replicated the first four items. Luminances and frequencies in each sequence were either perceptually correlated (Congruent) or were unrelated to one another (Incongruent). Experiment 1 showed that, despite encouragement to ignore the auditory stream, subjects' categorization of visual sequences was strongly influenced by the accompanying auditory sequences. Moreover, this influence tracked the similarity between a stimulus's separate audio and visual sequences, demonstrating that task-irrelevant auditory sequences underwent a considerable degree of processing. Using a variant of Hebb's repetition design, Experiment 2 compared musically trained subjects and subjects who had little or no musical training on the same task as used in Experiment 1. Test sequences included some that intermittently and randomly recurred, which produced better performance than sequences that were generated anew for each trial. The auditory component of a recurring audiovisual sequence influenced musically trained subjects more than it did other subjects. This result demonstrates that stimulus-selective, task-irrelevant learning of sequences can occur even when such learning is an incidental by-product of the task being performed.

Obligatory and adaptive averaging in visual short-term memory.

Visual memory can draw upon averaged perceptual representations, a dependence that could be both adaptive and obligatory. In support of this idea, we review a wide range of evidence, including findings from our own lab. This evidence shows that time- and space-averaged memory representations influence detection and recognition responses, and do so without instruction to compute or report an average. Some of the work reviewed exploits fine-grained measures of retrieval from visual short-term memory to closely track the influence of stored averages on recall and recognition of briefly presented visual textures. Results show that reliance on perceptual averages is greatest when memory resources are taxed or when subjects are uncertain about the fidelity of their memory representation. We relate these findings to models of how summary statistics impact visual short-term memory, and discuss a neural signature for contexts in which perceptual averaging exerts maximal influence.