Exploring Cognition with Brain-Machine Interfaces.

Traditional brain-machine interfaces decode cortical motor commands to control external devices. These commands are the product of higher-level cognitive processes, occurring across a network of brain areas, that integrate sensory information, plan upcoming motor actions, and monitor ongoing movements. We review cognitive signals recently discovered in the human posterior parietal cortex during neuroprosthetic clinical trials. These signals are consistent with small regions of cortex having a diverse role in cognitive aspects of movement control and body monitoring, including sensorimotor integration, planning, trajectory representation, somatosensation, action semantics, learning, and decision making. These variables are encoded within the same population of cells using structured representations that bind related sensory and motor variables, an architecture termed partially mixed selectivity. Diverse cognitive signals provide complementary information to traditional motor commands to enable more natural and intuitive control of external devices.

From thought to action: The brain-machine interface in posterior parietal cortex.

A dramatic example of translational monkey research is the development of neural prosthetics for assisting paralyzed patients. A neuroprosthesis consists of implanted electrodes that can record the intended movement of a paralyzed part of the body, a computer algorithm that decodes the intended movement, and an assistive device such as a robot limb or computer that is controlled by these intended movement signals. This type of neuroprosthetic system is also referred to as a brain-machine interface (BMI) since it interfaces the brain with an external machine. In this review, we will concentrate on BMIs in which microelectrode recording arrays are implanted in the posterior parietal cortex (PPC), a high-level cortical area in both humans and monkeys that represents intentions to move. This review will first discuss the basic science research performed in healthy monkeys that established PPC as a good source of intention signals. Next, it will describe the first PPC implants in human patients with tetraplegia from spinal cord injury. From these patients the goals of movements could be quickly decoded, and the rich number of action variables found in PPC indicates that it is an appropriate BMI site for a very wide range of neuroprosthetic applications. We will discuss research on learning to use BMIs in monkeys and humans and the advances that are still needed, requiring both monkey and human research to enable BMIs to be readily available in the clinic.

Hand Shape Representations in the Human Posterior Parietal Cortex.

Humans shape their hands to grasp, manipulate objects, and to communicate. From nonhuman primate studies, we know that visual and motor properties for grasps can be derived from cells in the posterior parietal cortex (PPC). Are non-grasp-related hand shapes in humans represented similarly? Here we show for the first time how single neurons in the PPC of humans are selective for particular imagined hand shapes independent of graspable objects. We find that motor imagery to shape the hand can be successfully decoded from the PPC by implementing a version of the popular Rock-Paper-Scissors game and its extension Rock-Paper-Scissors-Lizard-Spock. By simultaneous presentation of visual and auditory cues, we can discriminate motor imagery from visual information and show differences in auditory and visual information processing in the PPC. These results also demonstrate that neural signals from human PPC can be used to drive a dexterous cortical neuroprosthesis. SIGNIFICANCE STATEMENT: This study shows for the first time hand-shape decoding from human PPC. Unlike nonhuman primate studies in which the visual stimuli are the objects to be grasped, the visually cued hand shapes that we use are independent of the stimuli. Furthermore, we can show that distinct neuronal populations are activated for the visual cue and the imagined hand shape. Additionally we found that auditory and visual stimuli that cue the same hand shape are processed differently in PPC. Early on in a trial, only the visual stimuli and not the auditory stimuli can be decoded. During the later stages of a trial, the motor imagery for a particular hand shape can be decoded for both modalities.

Neural representations of economic decision variables in human posterior parietal cortex.

Decision making has been intensively studied in the posterior parietal cortex in non-human primates on a single neuron level. In humans decision making has mainly been studied with psychophysical tools or with fMRI. Here, we investigated how single neurons from human posterior parietal cortex represent numeric values informing future decisions during a complex two-player game. The tetraplegic study participant was implanted with a Utah electrode array in the anterior intraparietal area (AIP). We played a simplified variant of Black Jack with the participant while neuronal data was recorded. During the game two players are presented with numbers which are added up. Each time a number is presented the player has to decide to proceed or to stop. Once the first player stops or the score reaches a limit the turn passes on to the second player who tries to beat the score of the first player. Whoever is closer to the limit (without overshooting) wins the game. We found that many AIP neurons selectively responded to the face value of the presented number. Other neurons tracked the cumulative score or were selectively active for the upcoming decision of the study participant. Interestingly, some cells also kept track of the opponent's score. Our findings show that parietal regions engaged in hand action control also represent numbers and their complex transformations. This is also the first demonstration of complex economic decisions being possible to track in single neuron activity in human AIP. Our findings show how tight are the links between parietal neural circuits underlying hand control, numerical cognition and complex decision-making.

Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex.

Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis.Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain-machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands.Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands.Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.

Preserved cortical somatotopic and motor representations in tetraplegic humans.

A rich literature has documented changes in cortical representations of the body in somatosensory and motor cortex. Recent clinical studies of brain-machine interfaces designed to assist paralyzed patients have afforded the opportunity to record from and stimulate human somatosensory, motor, and action-related areas of the posterior parietal cortex. These studies show considerable preserved structure in the cortical somato-motor system. Motor cortex can immediately control assistive devices, stimulation of somatosensory cortex produces sensations in an orderly somatotopic map, and the posterior parietal cortex shows a high-dimensional representation of cognitive action variables. These results are strikingly similar to what would be expected in a healthy subject, demonstrating considerable stability of adult cortex even after severe injury and despite potential plasticity-induced new activations within the same region of cortex. Clinically, these results emphasize the importance of targeting cortical areas for BMI control signals that are consistent with their normal functional role.

Implicit mechanisms of intention.

High-level cortical regions encode motor decisions before or even absent awareness, suggesting that neural processes predetermine behavior before conscious choice. Such early neural encoding challenges popular conceptions of human agency. It also raises fundamental questions for brain-machine interfaces (BMIs) that traditionally assume that neural activity reflects the user's conscious intentions. Here, we study the timing of human posterior parietal cortex single-neuron activity recorded from implanted microelectrode arrays relative to the explicit urge to initiate movement. Participants were free to choose when to move, whether to move, and what to move, and they retrospectively reported the time they felt the urge to move. We replicate prior studies by showing that posterior parietal cortex (PPC) neural activity sharply rises hundreds of milliseconds before the reported urge. However, we find that this "preconscious" activity is part of a dynamic neural population response that initiates much earlier, when the participant first chooses to perform the task. Together with details of neural timing, our results suggest that PPC encodes an internal model of the motor planning network that transforms high-level task objectives into appropriate motor behavior. These new data challenge traditional interpretations of early neural activity and offer a more holistic perspective on the interplay between choice, behavior, and their neural underpinnings. Our results have important implications for translating BMIs into more complex real-world environments. We find that early neural dynamics are sufficient to drive BMI movements before the participant intends to initiate movement. Appropriate algorithms ensure that BMI movements align with the subject's awareness of choice.

Stability of motor representations after paralysis.

Neural plasticity allows us to learn skills and incorporate new experiences. What happens when our lived experiences fundamentally change, such as after a severe injury? To address this question, we analyzed intracortical population activity in the posterior parietal cortex (PPC) of a tetraplegic adult as she controlled a virtual hand through a brain-computer interface (BCI). By attempting to move her fingers, she could accurately drive the corresponding virtual fingers. Neural activity during finger movements exhibited robust representational structure similar to fMRI recordings of able-bodied individuals' motor cortex, which is known to reflect able-bodied usage patterns. The finger representational structure was consistent throughout multiple sessions, even though the structure contributed to BCI decoding errors. Within individual BCI movements, the representational structure was dynamic, first resembling muscle activation patterns and then resembling the anticipated sensory consequences. Our results reveal that motor representations in PPC reflect able-bodied motor usage patterns even after paralysis, and BCIs can re-engage these stable representations to restore lost motor functions.

Mapping behavioral repertoire onto the cortex.

A traditional view of the motor cortex in the primate brain is that it contains a map of the body arranged across the cortical surface. This traditional topographic scheme, however, does not capture the actual pattern of overlaps, fractures, re-representations, and multiple areas separated by fuzzy borders. Here, we suggest that the organization of the motor cortex, premotor cortex, supplementary motor cortex, frontal eye field, and supplementary eye field can in principle be understood as a best-fit rendering of the motor repertoire onto the two-dimensional cortical sheet in a manner that optimizes local continuity.

Neural encoding of actual and imagined touch within human posterior parietal cortex.

In the human posterior parietal cortex (PPC), single units encode high-dimensional information with partially mixed representations that enable small populations of neurons to encode many variables relevant to movement planning, execution, cognition, and perception. Here, we test whether a PPC neuronal population previously demonstrated to encode visual and motor information is similarly engaged in the somatosensory domain. We recorded neurons within the PPC of a human clinical trial participant during actual touch presentation and during a tactile imagery task. Neurons encoded actual touch at short latency with bilateral receptive fields, organized by body part, and covered all tested regions. The tactile imagery task evoked body part-specific responses that shared a neural substrate with actual touch. Our results are the first neuron-level evidence of touch encoding in human PPC and its cognitive engagement during a tactile imagery task, which may reflect semantic processing, attention, sensory anticipation, or imagined touch.

Preservation of Partially Mixed Selectivity in Human Posterior Parietal Cortex across Changes in Task Context.

Recent studies in posterior parietal cortex (PPC) have found multiple effectors and cognitive strategies represented within a shared neural substrate in a structure termed "partially mixed selectivity" (Zhang et al., 2017). In this study, we examine whether the structure of these representations is preserved across changes in task context and is thus a robust and generalizable property of the neural population. Specifically, we test whether the structure is conserved from an open-loop motor imagery task (training) to a closed-loop cortical control task (online), a change that has led to substantial changes in neural behavior in prior studies in motor cortex. Recording from a 4 × 4 mm electrode array implanted in PPC of a human tetraplegic patient participating in a brain-machine interface (BMI) clinical trial, we studied the representations of imagined/attempted movements of the left/right hand and compare their individual BMI control performance using a one-dimensional cursor control task. We found that the structure of the representations is largely maintained between training and online control. Our results demonstrate for the first time that the structure observed in the context of an open-loop motor imagery task is maintained and accessible in the context of closed-loop BMI control. These results indicate that it is possible to decode the mixed variables found from a small patch of cortex in PPC and use them individually for BMI control. Furthermore, they show that the structure of the mixed representations is maintained and robust across changes in task context.

The human primary somatosensory cortex encodes imagined movement in the absence of sensory information.

Classical systems neuroscience positions primary sensory areas as early feed-forward processing stations for refining incoming sensory information. This view may oversimplify their role given extensive bi-directional connectivity with multimodal cortical and subcortical regions. Here we show that single units in human primary somatosensory cortex encode imagined reaches in a cognitive motor task, but not other sensory-motor variables such as movement plans or imagined arm position. A population reference-frame analysis demonstrates coding relative to the cued starting hand location suggesting that imagined reaching movements are encoded relative to imagined limb position. These results imply a potential role for primary somatosensory cortex in cognitive imagery, engagement during motor production in the absence of sensation or expected sensation, and suggest that somatosensory cortex can provide control signals for future neural prosthetic systems.

Cognition in Sensorimotor Control: Interfacing With the Posterior Parietal Cortex.

Millions of people worldwide are afflicted with paralysis from a disruption of neural pathways between the brain and the muscles. Because their cortical architecture is often preserved, these patients are able to plan movements despite an inability to execute them. In such people, brain machine interfaces have great potential to restore lost function through neuroprosthetic devices, circumventing dysfunctional corticospinal circuitry. These devices have typically derived control signals from the motor cortex (M1) which provides information highly correlated with desired movement trajectories. However, sensorimotor control simultaneously engages multiple cognitive processes such as intent, state estimation, decision making, and the integration of multisensory feedback. As such, cortical association regions upstream of M1 such as the posterior parietal cortex (PPC) that are involved in higher order behaviors such as planning and learning, rather than in encoding movement itself, may enable enhanced, cognitive control of neuroprosthetics, termed cognitive neural prosthetics (CNPs). We illustrate in this review, through a small sampling, the cognitive functions encoded in the PPC and discuss their neural representation in the context of their relevance to motor neuroprosthetics. We aim to highlight through examples a role for cortical signals from the PPC in developing CNPs, and to inspire future avenues for exploration in their research and development.

Intrinsic Variable Learning for Brain-Machine Interface Control by Human Anterior Intraparietal Cortex.

Although animal studies provided significant insights in understanding the neural basis of learning and adaptation, they often cannot dissociate between different learning mechanisms due to the lack of verbal communication. To overcome this limitation, we examined the mechanisms of learning and its limits in a human intracortical brain-machine interface (BMI) paradigm. A tetraplegic participant controlled a 2D computer cursor by modulating single-neuron activity in the anterior intraparietal area (AIP). By perturbing the neuron-to-movement mapping, the participant learned to modulate the activity of the recorded neurons to solve the perturbations by adopting a target re-aiming strategy. However, when no cognitive strategies were adequate to produce correct responses, AIP failed to adapt to perturbations. These findings suggest that learning is constrained by the pre-existing neuronal structure, although it is possible that AIP needs more training time to learn to generate novel activity patterns when cognitive re-adaptation fails to solve the perturbations.

Relationship between unconstrained arm movements and single-neuron firing in the macaque motor cortex.

The activity of single neurons in the monkey motor cortex was studied during semi-naturalistic, unstructured arm movements made spontaneously by the monkey and measured with a high resolution three-dimensional tracking system. We asked how much of the total neuronal variance could be explained by various models of neuronal tuning to movement. On average, tuning to the speed of the hand accounted for 1% of the total variance in neuronal activity, tuning to the direction of the hand in space accounted for 8%, a more complex model of direction tuning, in which the preferred direction of the neuron rotated with the starting position of the arm, accounted for 13%, tuning to the final position of the hand in Cartesian space accounted for 22%, and tuning to the final multijoint posture of the arm accounted for 36%. One interpretation is that motor cortex neurons are significantly tuned to many control parameters important in the animal's repertoire, but that different control parameters are represented in different proportion, perhaps reflecting their prominence in everyday action. The final posture of a movement is an especially prominent control parameter although not the only one. A common mode of action of the monkey arm is to maintain a relatively stable overall posture while making local adjustments in direction during performance of a task. One speculation is that neurons in motor cortex reflect this pattern in which direction tuning predominates in local regions of space and postural tuning predominates over the larger workspace.

Single-Neuron Representation of Memory Strength and Recognition Confidence in Left Human Posterior Parietal Cortex.

The human posterior parietal cortex (PPC) is thought to contribute to memory retrieval, but little is known about its specific role. We recorded single PPC neurons of two human tetraplegic subjects implanted with microelectrode arrays, who performed a recognition memory task. We found two groups of neurons that signaled memory-based choices. Memory-selective neurons preferred either novel or familiar stimuli, scaled their response as a function of confidence, and signaled subjective choices regardless of truth. Confidence-selective neurons signaled confidence regardless of stimulus familiarity. Memory-selective signals appeared 553 ms after stimulus onset, but before action onset. Neurons also encoded spoken numbers, but these number-tuned neurons did not carry recognition signals. Together, this functional separation reveals action-independent coding of declarative memory-based familiarity and confidence of choices in human PPC. These data suggest that, in addition to sensory-motor integration, a function of human PPC is to utilize memory signals to make choices.

Possible origins of the complex topographic organization of motor cortex: reduction of a multidimensional space onto a two-dimensional array.

We propose that some of the features of the topographic organization in motor cortex emerge from a competition among several conflicting mapping requisites. These competing requisites include a somatotopic map of the body, a map of hand location in space, and a partitioning of cortex into regions that emphasize different complex, ethologically relevant movements. No one type of map fully explains the topography; instead, all three influences (and perhaps others untested here) interact to form the topography. A standard algorithm (Kohonen network) was used to generate an artificial motor cortex array that optimized local continuity for these conflicting mapping requisites. The resultant hybrid map contained many features seen in actual motor cortex, including the following: a rough, overlapping somatotopy; a posterior strip in which simpler movements were represented and more somatotopic segregation was observed, and an anterior strip in which more complex, multisegmental movements were represented and the somatotopy was less segregated; a clustering of different complex, multisegmental movements into specific subregions of cortex that resembled the arrangement of subregions found in the monkey; three hand representations arranged on the cortex in a manner similar to the primary motor, dorsal premotor, and ventral premotor hand areas in the monkey; and maps of hand location that approximately matched the maps observed in the monkey.

Rethinking cortical organization: moving away from discrete areas arranged in hierarchies.

One way to understand the topography of the cerebral cortex is that "like attracts like." The cortex is organized to maximize nearest neighbor similarity. This principle can explain the separation of the cortex into discrete areas that emphasize different information domains. It can also explain the maps that form within cortical areas. However, because the cortex is two-dimensional, when a parameter space of much higher dimensionality is reduced onto the cortical sheet while optimizing nearest neighbor relationships, the result may lack an obvious global ordering into separate areas. Instead, the topography may consist of partial gradients, fractures, swirls, regions that resemble separate areas in some ways but not others, and in not a lack of topographic maps but an excess of maps overlaid on each other, no one of which seems to be entirely correct. Like a canvas in a gallery of modern art that no two observers interpret the same way, this lack of obvious ordering of high-dimensional spaces onto the cortex might then result in some scientific controversy over the true organization. In this review, the authors suggest that at least some sectors of the cortex do not have a simple global ordering and are better understood as a result of a reduction of a high-dimensional space onto the cortical sheet. The cortical motor system may be an example of this phenomenon. The authors discuss a model of the lateral motor cortex in which a reduction of many parameters onto a simulated cortical sheet results in a complex topographic pattern that matches the actual monkey motor cortex in surprising detail. Some of the ambiguities of topography and areal boundaries that have plagued the attempt to systematize the lateral motor cortex are explained by the model.

Partially Mixed Selectivity in Human Posterior Parietal Association Cortex.

To clarify the organization of motor representations in posterior parietal cortex, we test how three motor variables (body side, body part, cognitive strategy) are coded in the human anterior intraparietal cortex. All tested movements were encoded, arguing against strict anatomical segregation of effectors. Single units coded for diverse conjunctions of variables, with different dimensions anatomically overlapping. Consistent with recent studies, neurons encoding body parts exhibited mixed selectivity. This mixed selectivity resulted in largely orthogonal coding of body parts, which "functionally segregate" the effector responses despite the high degree of anatomical overlap. Body side and strategy were not coded in a mixed manner as effector determined their organization. Mixed coding of some variables over others, what we term "partially mixed coding," argues that the type of functional encoding depends on the compared dimensions. This structure is advantageous for neuroprosthetics, allowing a single array to decode movements of a large extent of the body.

Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human.

Nonhuman primate and human studies have suggested that populations of neurons in the posterior parietal cortex (PPC) may represent high-level aspects of action planning that can be used to control external devices as part of a brain-machine interface. However, there is no direct neuron-recording evidence that human PPC is involved in action planning, and the suitability of these signals for neuroprosthetic control has not been tested. We recorded neural population activity with arrays of microelectrodes implanted in the PPC of a tetraplegic subject. Motor imagery could be decoded from these neural populations, including imagined goals, trajectories, and types of movement. These findings indicate that the PPC of humans represents high-level, cognitive aspects of action and that the PPC can be a rich source for cognitive control signals for neural prosthetics that assist paralyzed patients.