The Promises and Pitfalls of Virtual Reality.

Increasingly, Virtual Reality technologies are finding a place in psychology and behavioral neuroscience labs. Immersing participants in virtual worlds enables researchers to investigate empirical questions in realistic or imaginary environments while measuring a wide range of behavioral responses, without sacrificing experimental control. In this chapter, we aim to provide a balanced appraisal of VR research methods. We describe how VR can help advance psychological science by opening pathways for addressing many pernicious challenges currently facing science (e.g., direct replication, prioritizing ecological validity). We also outline a range of unique and perhaps unanticipated obstacles and provide practical recommendations to overcome them.

Launching Your VR Neuroscience Laboratory.

The proliferation and refinement of affordable virtual reality (VR) technologies and wearable sensors have opened new frontiers in cognitive and behavioral neuroscience. This chapter offers a broad overview of VR for anyone interested in leveraging it as a research tool. In the first section, it examines the fundamental functionalities of VR and outlines important considerations that inform the development of immersive content that stimulates the senses. In the second section, the focus of the discussion shifts to the implementation of VR in the context of the neuroscience lab. Practical advice is offered on adapting commercial, off-the-shelf devices to a researcher's specific purposes. Further, methods are explored for recording, synchronizing, and fusing heterogeneous forms of data obtained through the VR system or add-on sensors, as well as for labeling events and capturing game play. The reader should come away with an understanding of fundamental considerations that need to be addressed in order to launch a successful VR neuroscience research program.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension.

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

Verbal working memory and co-speech gesture processing.

Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.

Iconic Gestures Facilitate Discourse Comprehension in Individuals With Superior Immediate Memory for Body Configurations.

To understand a speaker's gestures, people may draw on kinesthetic working memory (KWM)-a system for temporarily remembering body movements. The present study explored whether sensitivity to gesture meaning was related to differences in KWM capacity. KWM was evaluated through sequences of novel movements that participants viewed and reproduced with their own bodies. Gesture sensitivity was assessed through a priming paradigm. Participants judged whether multimodal utterances containing congruent, incongruent, or no gestures were related to subsequent picture probes depicting the referents of those utterances. Individuals with low KWM were primarily inhibited by incongruent speech-gesture primes, whereas those with high KWM showed facilitation-that is, they were able to identify picture probes more quickly when preceded by congruent speech and gestures than by speech alone. Group differences were most apparent for discourse with weakly congruent speech and gestures. Overall, speech-gesture congruency effects were positively correlated with KWM abilities, which may help listeners match spatial properties of gestures to concepts evoked by speech.

Neural correlates of mathematical problem solving.

This study explores electroencephalography (EEG) brain dynamics associated with mathematical problem solving. EEG and solution latencies (SLs) were recorded as 11 neurologically healthy volunteers worked on intellectually challenging math puzzles that involved combining four single-digit numbers through basic arithmetic operators (addition, subtraction, division, multiplication) to create an arithmetic expression equaling 24. Estimates of EEG spectral power were computed in three frequency bands - θ (4-7 Hz), α (8-13 Hz) and ß (14-30 Hz) - over a widely distributed montage of scalp electrode sites. The magnitude of power estimates was found to change in a linear fashion with SLs - that is, relative to a base of power spectrum, theta power increased with longer SLs, while alpha and beta power tended to decrease. Further, the topographic distribution of spectral fluctuations was characterized by more pronounced asymmetries along the left-right and anterior-posterior axes for solutions that involved a longer search phase. These findings reveal for the first time the topography and dynamics of EEG spectral activities important for sustained solution search during arithmetical problem solving.

Co-speech iconic gestures and visuo-spatial working memory.

Three experiments tested the role of verbal versus visuo-spatial working memory in the comprehension of co-speech iconic gestures. In Experiment 1, participants viewed congruent discourse primes in which the speaker's gestures matched the information conveyed by his speech, and incongruent ones in which the semantic content of the speaker's gestures diverged from that in his speech. Discourse primes were followed by picture probes that participants judged as being either related or unrelated to the preceding clip. Performance on this picture probe classification task was faster and more accurate after congruent than incongruent discourse primes. The effect of discourse congruency on response times was linearly related to measures of visuo-spatial, but not verbal, working memory capacity, as participants with greater visuo-spatial WM capacity benefited more from congruent gestures. In Experiments 2 and 3, participants performed the same picture probe classification task under conditions of high and low loads on concurrent visuo-spatial (Experiment 2) and verbal (Experiment 3) memory tasks. Effects of discourse congruency and verbal WM load were additive, while effects of discourse congruency and visuo-spatial WM load were interactive. Results suggest that congruent co-speech gestures facilitate multi-modal language comprehension, and indicate an important role for visuo-spatial WM in these speech-gesture integration processes.

A psychometric measure of working memory capacity for configured body movement.

Working memory (WM) models have traditionally assumed at least two domain-specific storage systems for verbal and visuo-spatial information. We review data that suggest the existence of an additional slave system devoted to the temporary storage of body movements, and present a novel instrument for its assessment: the movement span task. The movement span task assesses individuals' ability to remember and reproduce meaningless configurations of the body. During the encoding phase of a trial, participants watch short videos of meaningless movements presented in sets varying in size from one to five items. Immediately after encoding, they are prompted to reenact as many items as possible. The movement span task was administered to 90 participants along with standard tests of verbal WM, visuo-spatial WM, and a gesture classification test in which participants judged whether a speaker's gestures were congruent or incongruent with his accompanying speech. Performance on the gesture classification task was not related to standard measures of verbal or visuo-spatial working memory capacity, but was predicted by scores on the movement span task. Results suggest the movement span task can serve as an assessment of individual differences in WM capacity for body-centric information.

Resting state and task-related brain dynamics supporting insight.

Problems can be solved in a variety of ways. One might systematically evaluate a known space of possible solutions until the right one is found. Alternatively, it may prove necessary to enlarge or restructure the expected problem space - so called "thinking outside the box." This approach can yield an experience of unexpected insight or feeling of Aha!. Current challenges to understanding this phenomenon from a neurocognitive perspective include the vast diversity of problem domains and time scales for solutions. Whereas the subjective suddenness of an "Aha!" moment may lead to the impression that insight must be precipitated by a set of discrete, short-lived neural events, this report outlines research revealing that even before a problem is presented, scalp-recorded measures of resting or baseline brain states are linked with future performance and likelihood of experiencing insight during the search for a solution. Additionally, this study also shows that compared to more systematic problem solving approaches, insight is accompanied by differences in cortical and likely cognitive engagement that are detectable throughout much of the problem solving phase, rather than being confined to a distinct interval immediately preceding the dawn of a solution. These findings are important for the development of therapies targeting problem solving and reasoning skills, such as those used in cognitive training interventions to mitigate the effects of cognitive decline.

Brain dynamics of mathematical problem solving.

The purpose of this study is to examine brain activities of participants solving mental math problems. The research investigated how problem difficulty affected the subjects' responses and electroencephalogram (EEG) in different brain regions. In general, it was found that solution latencies (SL) to the math problems increased with difficulty. The EEG results showed that across subjects, the right-central beta, left-parietal theta, left-occipital theta and alpha, right-parietal alpha and beta, medial-frontal beta and medial central theta power decreased as task difficulty increased. This study further explored the effects of problem-solving performance on the EEG. Slow solvers exhibited greater frontal theta activities in the right hemisphere, whereas an inverse pattern of hemispheric asymmetry was found in fast solvers. Furthermore, analyses of spatio-temporal brain dynamics during problem solving show progressively stronger alpha- and beta-power suppression and theta-power augmentation as subjects were reaching a solution. These findings provide a better understanding of cortical activities mediating math-based problem solving and knowledge acquisition that can ultimately benefit math learning and education.

Are depictive gestures like pictures? commonalities and differences in semantic processing.

Conversation is multi-modal, involving both talk and gesture. Does understanding depictive gestures engage processes similar to those recruited in the comprehension of drawings or photographs? Event-related brain potentials (ERPs) were recorded from neurotypical adults as they viewed spontaneously produced depictive gestures preceded by congruent and incongruent contexts. Gestures were presented either dynamically in short, soundless video-clips, or statically as freeze frames extracted from gesture videos. In a separate ERP experiment, the same participants viewed related or unrelated pairs of photographs depicting common real-world objects. Both object photos and gesture stimuli elicited less negative ERPs from 400 to 600ms post-stimulus when preceded by matching versus mismatching contexts (dN450). Object photos and static gesture stills also elicited less negative ERPs between 300 and 400ms post-stimulus (dN300). Findings demonstrate commonalities between the conceptual integration processes underlying the interpretation of iconic gestures and other types of image-based representations of the visual world.

Gestures modulate speech processing early in utterances.

Electroencephalogram was recorded as healthy adults viewed short videos of spontaneous discourse in which a speaker used depictive gestures to complement information expressed through speech. Event-related potentials were computed time-locked to content words in the speech stream and to subsequent related and unrelated picture probes. Gestures modulated event-related potentials to content words co-timed with the first gesture in a discourse segment, relative to the same words presented with static freeze frames of the speaker. Effects were observed 200-550 ms after speech onset, a time interval associated with semantic processing. Gestures also increased sensitivity to picture probe relatedness. Effects of gestures on picture probe and spoken word analysis were inversely correlated, suggesting that gestures differentially impact verbal and image-based processes.

How iconic gestures enhance communication: an ERP study.

EEG was recorded as adults watched short segments of spontaneous discourse in which the speaker's gestures and utterances contained complementary information. Videos were followed by one of four types of picture probes: cross-modal related probes were congruent with both speech and gestures; speech-only related probes were congruent with information in the speech, but not the gesture; and two sorts of unrelated probes were created by pairing each related probe with a different discourse prime. Event-related potentials (ERPs) elicited by picture probes were measured within the time windows of the N300 (250-350 ms post-stimulus) and N400 (350-550 ms post-stimulus). Cross-modal related probes elicited smaller N300 and N400 than speech-only related ones, indicating that pictures were easier to interpret when they corresponded with gestures. N300 and N400 effects were not due to differences in the visual complexity of each probe type, since the same cross-modal and speech-only picture probes elicited N300 and N400 with similar amplitudes when they appeared as unrelated items. These findings extend previous research on gesture comprehension by revealing how iconic co-speech gestures modulate conceptualization, enabling listeners to better represent visuo-spatial aspects of the speaker's meaning.

Meaningful gestures: electrophysiological indices of iconic gesture comprehension.

To assess semantic processing of iconic gestures, EEG (29 scalp sites) was recorded as adults watched cartoon segments paired with soundless videos of congruous and incongruous gestures followed by probe words. Event-related potentials time-locked to the onset of gestures and probe words were measured in two experiments. In Experiment 1, participants judged the congruency between gestures and cartoons. Gestures elicited an N400-like component (gesture N450) that was larger for incongruent than congruent items, as well as a late positivity that was larger for congruent items. In Experiment 2, participants assessed the relatedness between probe words and preceding cartoon-gesture pairs. N450 effects to gestures were observed without overlapping positivity. These findings suggest that iconic gestures are subject to semantic processes analogous to those evoked by other meaningful representations, such as pictures and words.

Right hemisphere activation of joke-related information: an event-related brain potential study.

Two studies tested the hypothesis that the right hemisphere engages in relatively coarse semantic coding that aids high-level language tasks such as joke comprehension. Scalprecorded event-related brain potentials (ERPs) were collected as healthy adults read probe words (CRAZY) preceded either by jokes or nonfunny controls ("Everyone had so much fun jumping into the swimming pool, we decided to put in a little water/platform"). Probes were related to the meaning of the jokes, but not the controls. In Experiment 1a, with central presentation, probes following jokes (related) elicited less negative ERPs 300-700 msec postonset (N400) than did probes following nonfunny controls (unrelated). This finding suggests related probes were primed by the jokes. In addition, unrelated probes elicited a larger anterior positivity 700-900 msec than did related, as irrelevant stimuli impacted control processes invoked by task demands. In Experiment 1b, probes (CRAZY) were preceded only by sentence-final words from jokes (water) or controls (platform). No ERP effects were observed in Experiment 1b, suggesting the N400 priming effect and the anterior positivity observed in Experiment 1a ref lect semantic activations at the discourse level. To assess hemispheric differences in semantic activations, in Experiment 2, ERPs were recorded as participants read probe words presented in their left and right visual fields (LVF and RVF, respectively). Probes elicited a smaller N400 component when preceded by jokes than controls. This N400 priming effect was larger with presentation to the LVF, suggesting joke-relevant information was more active in the right hemisphere. The anterior positivity was observed with RVF but not LVF presentation, suggesting an important role for the left hemisphere in controlled retrieval in language comprehension.