Phase-encoded fMRI tracks down brainstorms of natural language processing with subsecond precision.

Natural language processing unfolds information overtime as spatially separated, multimodal, and interconnected neural processes. Existing noninvasive subtraction-based neuroimaging techniques cannot simultaneously achieve the spatial and temporal resolutions required to visualize ongoing information flows across the whole brain. Here we have developed rapid phase-encoded designs to fully exploit the temporal information latent in functional magnetic resonance imaging data, as well as overcoming scanner noise and head-motion challenges during overt language tasks. We captured real-time information flows as coherent hemodynamic waves traveling over the cortical surface during listening, reading aloud, reciting, and oral cross-language interpreting tasks. We were able to observe the timing, location, direction, and surge of traveling waves in all language tasks, which were visualized as "brainstorms" on brain "weather" maps. The paths of hemodynamic traveling waves provide direct evidence for dual-stream models of the visual and auditory systems as well as logistics models for crossmodal and cross-language processing. Specifically, we have tracked down the step-by-step processing of written or spoken sentences first being received and processed by the visual or auditory streams, carried across language and domain-general cognitive regions, and finally delivered as overt speeches monitored through the auditory cortex, which gives a complete picture of information flows across the brain during natural language functioning. PRACTITIONER POINTS: Phase-encoded fMRI enables simultaneous imaging of high spatial and temporal resolution, capturing continuous spatiotemporal dynamics of the entire brain during real-time overt natural language tasks. Spatiotemporal traveling wave patterns provide direct evidence for constructing comprehensive and explicit models of human information processing. This study unlocks the potential of applying rapid phase-encoded fMRI to indirectly track the underlying neural information flows of sequential sensory, motor, and high-order cognitive processes.

Unraveling the spatiotemporal brain dynamics during a simulated reach-to-eat task.

The reach-to-eat task involves a sequence of action components including looking, reaching, grasping, and feeding. While cortical representations of individual action components have been mapped in human functional magnetic resonance imaging (fMRI) studies, little is known about the continuous spatiotemporal dynamics among these representations during the reach-to-eat task. In a periodic event-related fMRI experiment, subjects were scanned while they reached toward a food image, grasped the virtual food, and brought it to their mouth within each 16-s cycle. Fourier-based analysis of fMRI time series revealed periodic signals and noise distributed across the brain. Independent component analysis was used to remove periodic or aperiodic motion artifacts. Time-frequency analysis was used to analyze the temporal characteristics of periodic signals in each voxel. Circular statistics was then used to estimate mean phase angles of periodic signals and select voxels based on the distribution of phase angles. By sorting mean phase angles across regions, we were able to show the real-time spatiotemporal brain dynamics as continuous traveling waves over the cortical surface. The activation sequence consisted of approximately the following stages: (1) stimulus related activations in occipital and temporal cortices; (2) movement planning related activations in dorsal premotor and superior parietal cortices; (3) reaching related activations in primary sensorimotor cortex and supplementary motor area; (4) grasping related activations in postcentral gyrus and sulcus; (5) feeding related activations in orofacial areas. These results suggest that phase-encoded design and analysis can be used to unravel sequential activations among brain regions during a simulated reach-to-eat task.

Spatiotemporal integration of looming visual and tactile stimuli near the face.

Real-world objects approaching or passing by an observer often generate visual, auditory, and tactile signals with different onsets and durations. Prompt detection and avoidance of an impending threat depend on precise binding of looming signals across modalities. Here we constructed a multisensory apparatus to study the spatiotemporal integration of looming visual and tactile stimuli near the face. In a psychophysical experiment, subjects assessed the subjective synchrony between a looming ball and an air puff delivered to the same side of the face with a varying temporal offset. Multisensory stimuli with similar onset times were perceived as completely out of sync and assessed with the lowest subjective synchrony index (SSI). Across subjects, the SSI peaked at an offset between 800 and 1,000 ms, where the multisensory stimuli were perceived as optimally in sync. In an fMRI experiment, tactile, visual, tactile-visual out-of-sync (TVoS), and tactile-visual in-sync (TViS) stimuli were delivered to either side of the face in randomized events. Group-average statistical responses to different stimuli were compared within each surface-based region of interest (sROI) outlined on the cortical surface. Most sROIs showed a preference for contralateral stimuli and higher responses to multisensory than unisensory stimuli. In several bilateral sROIs, particularly the human MT+ complex and V6A, responses to spatially aligned multisensory stimuli (TVoS) were further enhanced when the stimuli were in-sync (TViS), as expressed by TVoS < TViS. This study demonstrates the perceptual and neural mechanisms of multisensory integration near the face, which has potential applications in the development of multisensory entertainment systems and media.

Mapping the complex topological organization of the human parietal face area.

The macaque monkey ventral intraparietal area (VIP) contains neurons with aligned visual-tactile receptive fields anchored to the face and upper body. Our previous fMRI studies using standard head coils found a human parietal face area (VIP+ complex; putative macaque VIP homologue) containing superimposed topological maps of the face and near-face visual space. Here, we construct high signal-to-noise surface coils and used phase-encoded air puffs and looming stimuli to map topological organization of the parietal face area at higher resolution. This area is consistently identified as a region extending between the superior postcentral sulcus and the upper bank of the anterior intraparietal sulcus (IPS), avoiding the fundus of IPS. Using smaller voxel sizes, our surface coils picked up strong fMRI signals in response to tactile and visual stimuli. By analyzing tactile and visual maps in our current and previous studies, we constructed a set of topological models illustrating commonalities and differences in map organization across subjects. The most consistent topological feature of the VIP+ complex is a central-anterior upper face (and upper visual field) representation adjoined by lower face (and lower visual field) representations ventrally (laterally) and/or dorsally (medially), potentially forming two subdivisions VIPv (ventral) and VIPd (dorsal). The lower visual field representations typically extend laterally into the anterior IPS to adjoin human area AIP, and medially to overlap with the parietal body areas at the superior parietal ridge. Significant individual variations are then illustrated to provide an accurate and comprehensive view of the topological organization of the parietal face area.

Validation of periodic fMRI signals in response to wearable tactile stimulation.

To map cortical representations of the body, we recently developed a wearable technology for automatic tactile stimulation in human functional magnetic resonance imaging (fMRI) experiments. In a two-condition block design experiment, air puffs were delivered to the face and hands periodically. Surface-based regions of interest (S-ROIs) were initially identified by thresholding a linear statistical measure of signal-to-noise ratio of periodic response. Across subjects, S-ROIs were found in the frontal, primary sensorimotor, posterior parietal, insular, temporal, cingulate, and occipital cortices. To validate and differentiate these S-ROIs, we develop a measure of temporal stability of response based on the assumption that a periodic stimulation evokes stable (low-variance) periodic fMRI signals throughout the entire scan. Toward this end, we apply time-frequency analysis to fMRI time series and use circular statistics to characterize the distribution of phase angles for data selection. We then assess the temporal variability of a periodic signal by measuring the path length of its trajectory in the complex plane. Both within and outside the primary sensorimotor cortex, S-ROIs with high temporal variability and deviant phase angles are rejected. A surface-based probabilistic group-average map is constructed for spatial screening of S-ROIs with low to moderate temporal variability in non-sensorimotor regions. Areas commonly activated across subjects are also summarized in the group-average map. In summary, this study demonstrates that analyzing temporal characteristics of the entire fMRI time series is essential for second-level selection and interpretation of S-ROIs initially defined by an overall linear statistical measure.

Neural substrates underlying the passive observation and active control of translational egomotion.

Moving or static obstacles often get in the way while walking in daily life. Avoiding obstacles involves both perceptual processing of motion information and controlling appropriate defensive movements. Several higher-level motion areas, including the ventral intraparietal area (VIP), medial superior temporal area, parieto-insular vestibular cortex (PIVC), areas V6 and V6A, and cingulate sulcus visual area, have been identified in humans by passive viewing of optic flow patterns that simulate egomotion and object motion. However, the roles of these areas in the active control of egomotion in the real world remain unclear. Here, we used functional magnetic resonance imaging (fMRI) to map the neural substrates underlying the passive observation and active control of translational egomotion in humans. A wide-field virtual reality environment simulated a daily scenario where doors randomly swing outward while walking in a hallway. The stimuli of door-dodging events were essentially the same in two event-related fMRI experiments, which compared passive and active dodges in response to swinging doors. Passive dodges were controlled by a computer program, while active dodges were controlled by the subject. Passive dodges activated several higher-level areas distributed across three dorsal motion streams in the temporal, parietal, and cingulate cortex. Active dodges most strongly activated the temporal-vestibular stream, with peak activation located in the right PIVC. Other higher-level motion areas including VIP showed weaker to no activation in active dodges. These results suggest that PIVC plays an active role in sensing and guiding translational egomotion that moves an observer aside from impending obstacles.

Mapping multisensory parietal face and body areas in humans.

Detection and avoidance of impending obstacles is crucial to preventing head and body injuries in daily life. To safely avoid obstacles, locations of objects approaching the body surface are usually detected via the visual system and then used by the motor system to guide defensive movements. Mediating between visual input and motor output, the posterior parietal cortex plays an important role in integrating multisensory information in peripersonal space. We used functional MRI to map parietal areas that see and feel multisensory stimuli near or on the face and body. Tactile experiments using full-body air-puff stimulation suits revealed somatotopic areas of the face and multiple body parts forming a higher-level homunculus in the superior posterior parietal cortex. Visual experiments using wide-field looming stimuli revealed retinotopic maps that overlap with the parietal face and body areas in the postcentral sulcus at the most anterior border of the dorsal visual pathway. Starting at the parietal face area and moving medially and posteriorly into the lower-body areas, the median of visual polar-angle representations in these somatotopic areas gradually shifts from near the horizontal meridian into the lower visual field. These results suggest the parietal face and body areas fuse multisensory information in peripersonal space to guard an individual from head to toe.

Phase-encoded fMRI tracks down brainstorms of natural language processing with sub-second precision.

The human language system interacts with cognitive and sensorimotor regions during natural language processing. However, where, when, and how these processes occur remain unclear. Existing noninvasive subtraction-based neuroimaging techniques cannot simultaneously achieve the spatial and temporal resolutions required to visualize ongoing information flows across the whole brain. Here we have developed phase-encoded designs to fully exploit the temporal information latent in functional magnetic resonance imaging (fMRI) data, as well as overcoming scanner noise and head-motion challenges during overt language tasks. We captured neural information flows as coherent waves traveling over the cortical surface during listening, reciting, and oral cross-language interpreting. The timing, location, direction, and surge of traveling waves, visualized as 'brainstorms' on brain 'weather' maps, reveal the functional and effective connectivity of the brain in action. These maps uncover the functional neuroanatomy of language perception and production and motivate the construction of finer-grained models of human information processing.

Co-modulatory spectral changes in independent brain processes are correlated with task performance.

This study investigates the independent modulators that mediate the power spectra of electrophysiological processes, measured by electroencephalogram (EEG), in a sustained-attention experiment. EEG and behavioral data were collected during 1-2 hour virtual-reality based driving experiments in which subjects were instructed to maintain their cruising position and compensate for randomly induced drift using the steering wheel. Independent component analysis (ICA) applied to 30-channel EEG data separated the recorded EEG signals into a sum of maximally temporally independent components (ICs) for each of 30 subjects. Logarithmic spectra of resultant IC activities were then decomposed by principal component analysis, followed by ICA, to find spectrally fixed and temporally independent modulators (IM). Across subjects, the spectral ICA consistently found four performance-related independent modulators: delta, delta-theta, alpha, and beta modulators that multiplicatively affected the spectra of spatially distinct IC processes when the participants experienced waves of alternating alertness and drowsiness during long-hour simulated driving. The activation of the delta-theta modulator increased monotonically as subjects' task performances decreased. Furthermore, the time courses of the theta-beta modulator were highly correlated with concurrent changes in driving errors across subjects (r=0.77±0.13).

Topological Maps and Brain Computations From Low to High.

We first briefly summarize data from microelectrode studies on visual maps in non-human primates and other mammals, and characterize differences among the features of the approximately topological maps in the three main sensory modalities. We then explore the almost 50% of human neocortex that contains straightforward topological visual, auditory, and somatomotor maps by presenting a new parcellation as well as a movie atlas of cortical area maps on the FreeSurfer average surface, fsaverage. Third, we review data on moveable map phenomena as well as a recent study showing that cortical activity during sensorimotor actions may involve spatially locally coherent traveling wave and bump activity. Finally, by analogy with remapping phenomena and sensorimotor activity, we speculate briefly on the testable possibility that coherent localized spatial activity patterns might be able to 'escape' from topologically mapped cortex during 'serial assembly of content' operations such as scene and language comprehension, to form composite 'molecular' patterns that can move across some cortical areas and possibly return to topologically mapped cortex to generate motor output there.

Multiple parietal reach regions in humans: cortical representations for visual and proprioceptive feedback during on-line reaching.

Reaching toward a visual target involves at least two sources of information. One is the visual feedback from the hand as it approaches the target. Another is proprioception from the moving limb, which informs the brain of the location of the hand relative to the target even when the hand is not visible. Where these two sources of information are represented in the human brain is unknown. In the present study, we investigated the cortical representations for reaching with or without visual feedback from the moving hand, using functional magnetic resonance imaging. To identify reach-dominant areas, we compared reaching with saccades. Our results show that a reach-dominant region in the anterior precuneus (aPCu), extending into medial intraparietal sulcus, is equally active in visual and nonvisual reaching. A second region, at the superior end of the parieto-occipital sulcus (sPOS), is more active for visual than for nonvisual reaching. These results suggest that aPCu is a sensorimotor area whose sensory input is primarily proprioceptive, while sPOS is a visuomotor area that receives visual feedback during reaching. In addition to the precuneus, medial, anterior intraparietal, and superior parietal cortex were also activated during both visual and nonvisual reaching, with more anterior areas responding to hand movements only and more posterior areas responding to both hand and eye movements. Our results suggest that cortical networks for reaching are differentially activated depending on the sensory conditions during reaching. This indicates the involvement of multiple parietal reach regions in humans, rather than a single homogenous parietal reach region.

A human parietal face area contains aligned head-centered visual and tactile maps.

Visually guided eating, biting and kissing, and avoiding objects moving toward the face and toward which the face moves require prompt, coordinated processing of spatial visual and somatosensory information in order to protect the face and the brain. Single-cell recordings in parietal cortex have identified multisensory neurons with spatially restricted, aligned visual and somatosensory receptive fields, but so far, there has been no evidence for a topographic map in this area. Here we mapped the organization of a multisensory parietal face area in humans by acquiring functional magnetic resonance images while varying the polar angle of facial air puffs and close-up visual stimuli. We found aligned maps of tactile and near-face visual stimuli at the highest level of human association cortex-namely, in the superior part of the postcentral sulcus. We show that this area may code the location of visual stimuli with respect to the face, not with respect to the retina.

Visual stimulus presentation using fiber optics in the MRI scanner.

Imaging the neural basis of visuomotor actions using fMRI is a topic of increasing interest in the field of cognitive neuroscience. One challenge is to present realistic three-dimensional (3-D) stimuli in the subject's peripersonal space inside the MRI scanner. The stimulus generating apparatus must be compatible with strong magnetic fields and must not interfere with image acquisition. Virtual 3-D stimuli can be generated with a stereo image pair projected onto screens or via binocular goggles. Here, we describe designs and implementations for automatically presenting physical 3-D stimuli (point-light targets) in peripersonal and near-face space using fiber optics in the MRI scanner. The feasibility of fiber-optic based displays was demonstrated in two experiments. The first presented a point-light array along a slanted surface near the body, and the second presented multiple point-light targets around the face. Stimuli were presented using phase-encoded paradigms in both experiments. The results suggest that fiber-optic based displays can be a complementary approach for visual stimulus presentation in the MRI scanner.

Multisensory and sensorimotor maps.

The parietal lobe plays a major role in sensorimotor integration and action. Recent neuroimaging studies have revealed more than 40 retinotopic areas distributed across five visual streams in the human brain, two of which enter the parietal lobe. A series of retinotopic areas occupy the length of the intraparietal sulcus and continue into the postcentral sulcus. On the medial wall, retinotopy extends across the parieto-occipital sulcus into the precuneus and reaches the cingulate sulcus. Full-body tactile stimulation revealed a multisensory homunculus lying along the postcentral sulcus just posterior to primary somatosensory cortical areas and overlapping with the anteriormost retinotopic maps. These topologically organized higher-level maps lay the foundation for actions in peripersonal space (e.g., reaching and grasping) as well as navigation through space. A preliminary yet comprehensive multilayer functional atlas was constructed to specify the relative locations of cortical unisensory, multisensory, and action representations. We expect that those areal and functional definitions will be refined by future studies using more sophisticated stimuli and tasks tailored to regions with different specificity. The long-term goal is to construct an online surface-based atlas containing layered maps of multiple modalities that can be used as a reference to understand the functions and disorders of the parietal lobe.

Wide-field retinotopy defines human cortical visual area v6.

The retinotopic organization of a newly identified visual area near the midline in the dorsalmost part of the human parieto-occipital sulcus was mapped using high-field functional magnetic resonance imaging, cortical surface-based analysis, and wide-field retinotopic stimulation. This area was found in all 34 subjects that were mapped. It represents the contralateral visual hemifield in both hemispheres of all subjects, with upper fields located anterior and medial to areas V2/V3, and lower fields medial and slightly anterior to areas V3/V3A. It contains a representation of the center of gaze distinct from V3A, a large representation of the visual periphery, and a mirror-image representation of the visual field. Based on similarity in position, visuotopic organization, and relationship with the neighboring extrastriate visual areas, we suggest it might be the human homolog of macaque area V6, and perhaps of area M (medial) or DM (dorsomedial) of New World primates.

Multisensory maps in parietal cortex.

Parietal cortex has long been known to be a site of sensorimotor integration. Recent findings in humans have shown that it is divided up into a number of small areas somewhat specialized for eye movements, reaching, and hand movements, but also face-related movements (avoidance, eating), lower body movements, and movements coordinating multiple body parts. The majority of these areas contain rough sensory (receptotopic) maps, including a substantial multisensory representation of the lower body and lower visual field immediately medial to face VIP. There is strong evidence for retinotopic remapping in LIP and face-centered remapping in VIP, and weaker evidence for hand-centered remapping. The larger size of the functionally distinct inferior parietal default mode network in humans compared to monkeys results in a superior and medial displacement of middle parietal areas (e.g., the saccade-related LIP's). Multisensory superior parietal areas located anterior to the angular gyrus such as AIP and VIP are less medially displaced relative to macaque monkeys, so that human LIP paradoxically ends up medial to human VIP.

Bottom-up Retinotopic Organization Supports Top-down Mental Imagery.

Finding a path between locations is a routine task in daily life. Mental navigation is often used to plan a route to a destination that is not visible from the current location. We first used functional magnetic resonance imaging (fMRI) and surface-based averaging methods to find high-level brain regions involved in imagined navigation between locations in a building very familiar to each participant. This revealed a mental navigation network that includes the precuneus, retrosplenial cortex (RSC), parahippocampal place area (PPA), occipital place area (OPA), supplementary motor area (SMA), premotor cortex, and areas along the medial and anterior intraparietal sulcus. We then visualized retinotopic maps in the entire cortex using wide-field, natural scene stimuli in a separate set of fMRI experiments. This revealed five distinct visual streams or 'fingers' that extend anteriorly into middle temporal, superior parietal, medial parietal, retrosplenial and ventral occipitotemporal cortex. By using spherical morphing to overlap these two data sets, we showed that the mental navigation network primarily occupies areas that also contain retinotopic maps. Specifically, scene-selective regions RSC, PPA and OPA have a common emphasis on the far periphery of the upper visual field. These results suggest that bottom-up retinotopic organization may help to efficiently encode scene and location information in an eye-centered reference frame for top-down, internally generated mental navigation. This study pushes the border of visual cortex further anterior than was initially expected.

Tonic and phasic electroencephalographic dynamics during continuous compensatory tracking.

Tonic and phasic dynamics of electroencephalographic (EEG) activities during a continuous compensatory tracking task (CTT) were analyzed using time-frequency analysis of EEG sources identified by independent component analysis (ICA). In 1-hour sessions, 70-channel EEG data were recorded while participants attempted to use frequent compensatory trackball movements to maintain a drifting disc close to a bulls-eye at screen center. Disc trajectories were converted into two moving-average performance measures, root mean square distance of the disc from screen center in 4-s ('local') and in 20-s ('global') moving time windows. Maximally independent EEG processes and their equivalent dipole source locations were obtained using the EEGLAB toolbox (http://sccn.ucsd.edu/eeglab). Across subjects and sessions, independent EEG processes in occipital, somatomotor, and supplementary motor cortices exhibited tonic power increases during periods of high tracking error, plus additional phasic power increases in several frequency bands before and after trackball movements following disc 'perigees' (moments at which the disc began to drift away from the bulls-eye). These phasic activity increases, which were larger during high-error periods, reveal an intimate relation between EEG dynamics and top-down recognition of responding to threatening events. Thus during a continuous tracking task without impulsive stimulus onsets, sub-second scale EEG dynamics related to visuomotor task could be dissociated from slower spectral modulations linked to changes in performance and arousal. We tentatively interpret the observed EEG signal increases as indexing tonic and phasic modulations of the levels of task attention and engagement required to maintain visuomotor performance during sustained performance.

Dodecapus: An MR-compatible system for somatosensory stimulation.

Somatotopic mapping of human body surface using fMRI is challenging. First, it is difficult to deliver tactile stimuli in the scanner. Second, multiple stimulators are often required to cover enough area of the complex-shaped body surface, such as the face. In this study, a computer-controlled pneumatic system was constructed to automatically deliver air puffs to 12 locations on the body surface through an MR-compatible manifold (Dodecapus) mounted on a head coil inside the scanner bore. The timing of each air-puff channel is completely programmable and this allows systematic and precise stimulation on multiple locations on the body surface during functional scans. Three two-condition block-design "Localizer" paradigms were employed to localize the cortical representations of the face, lips, and fingers, respectively. Three "Phase-encoded" paradigms were employed to map the detailed somatotopic organizations of the face, lips, and fingers following each "Localizer" paradigm. Multiple somatotopic representations of the face, lips, and fingers were localized and mapped in primary motor cortex (MI), ventral premotor cortex (PMv), polysensory zone (PZ), primary (SI) and secondary (SII) somatosensory cortex, parietal ventral area (PV) and 7b, as well as anterior and ventral intraparietal areas (AIP and VIP). The Dodecapus system is portable, easy to setup, generates no radio frequency interference, and can also be used for EEG and MEG experiments. This system could be useful for non-invasive somatotopic mapping in both basic and clinical studies.

Analyzing event-related brain dynamics in continuous compensatory tracking tasks.

The dynamics of electroencephalographic (EEG) activity in continuous compensatory tracking tasks were analyzed by independent component analysis (ICA) and time-frequency techniques. In one-hour sessions, 72-channel EEG was recorded while a healthy volunteer attempted to use a trackball to keep a drifting disc in a bulls-eye in the center of screen. Disc trajectory was converted into a moving measure of disc error. Local minima (perigees) indicated moments when the disc started to drift away from the center. Subject performance was indexed by root mean square disc error in a 20s epoch centered on each perigee, high error generally indicating drowsiness. Maximally independent EEG processes and their equivalent dipole source locations were obtained using the EEGLAB toolbox (http://sccn.ucsd.edu/eeglab). Component activations were epoched in 5s time intervals time locked to perigees. Following disk perigees during (drowsy) periods of high disk error, significant spectral changes were observed. One of the 70 independent components was located in or near primary visual cortex. During periods of poor (drowsy) performance, it had increased mean tonic alpha/theta activity, with a further phasic alpha/theta increase following perigees [1,2]. At the same time, low alpha activity of a second component located in or near cingulate gyrus increased, and 10-30 Hz EEG activity of a third component in the left somatomotor cortex increased briefly. The alpha activity of the somatomotor component persisted through the following distance maximum. These spatiotemporal phenomena were consistently observed across three sessions within subjects. Thus, event-related EEG brain dynamics can be detected and modeled in a continuous behavioral task without impulsive event onsets.