A brain machine interface framework for exploring proactive control of smart environments.

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

Position- and scale-invariant object-centered spatial localization in monkey frontoparietal cortex dynamically adapts to cognitive demand.

Egocentric encoding is a well-known property of brain areas along the dorsal pathway. Different to previous experiments, which typically only demanded egocentric spatial processing during movement preparation, we designed a task where two male rhesus monkeys memorized an on-the-object target position and then planned a reach to this position after the object re-occurred at variable location with potentially different size. We found allocentric (in addition to egocentric) encoding in the dorsal stream reach planning areas, parietal reach region and dorsal premotor cortex, which is invariant with respect to the position, and, remarkably, also the size of the object. The dynamic adjustment from predominantly allocentric encoding during visual memory to predominantly egocentric during reach planning in the same brain areas and often the same neurons, suggests that the prevailing frame of reference is less a question of brain area or processing stream, but more of the cognitive demands.

Deciding While Acting-Mid-Movement Decisions Are More Strongly Affected by Action Probability than Reward Amount.

When deciding while acting, such as sequentially selecting targets during naturalistic foraging, movement trajectories reveal the dynamics of the unfolding decision process. Ongoing and planned actions may impact decisions in these situations in addition to expected reward outcomes. Here, we test how strongly humans weigh and how fast they integrate individual constituents of expected value, namely the prior probability (PROB) of an action and the prior expected reward amount (AMNT) associated with an action, when deciding based on the combination of both together during an ongoing movement. Unlike other decision-making studies, we focus on PROB and AMNT priors, and not final evidence, in that correct actions were either instructed or could be chosen freely. This means, there was no decision-making under risk. We show that both priors gradually influence movement trajectories already before mid-movement instructions of the correct target and bias free-choice behavior. These effects were consistently stronger for PROB compared with AMNT priors. Participants biased their movements toward a high-PROB target, committed to it faster when instructed or freely chosen, and chose it more frequently even when it was associated with a lower AMNT prior than the alternative option. Despite these differences in effect magnitude, the time course of the effect of both priors on movement direction was highly similar. We conclude that prior action probability, and hence the associated possibility to plan actions accordingly, has higher behavioral relevance than prior action value for decisions that are expressed by adjusting already ongoing movements.

Human and macaque pairs employ different coordination strategies in a transparent decision game.

Many real-world decisions in social contexts are made while observing a partner's actions. To study dynamic interactions during such decisions, we developed a setup where two agents seated face-to-face to engage in game-theoretical tasks on a shared transparent touchscreen display ('transparent games'). We compared human and macaque pairs in a transparent version of the coordination game 'Bach-or-Stravinsky', which entails a conflict about which of two individually-preferred opposing options to choose to achieve coordination. Most human pairs developed coordinated behavior and adopted dynamic turn-taking to equalize the payoffs. All macaque pairs converged on simpler, static coordination. Remarkably, two animals learned to coordinate dynamically after training with a human confederate. This pair selected the faster agent's preferred option, exhibiting turn-taking behavior that was captured by modeling the visibility of the partner's action before one's own movement. Such competitive turn-taking was unlike the prosocial turn-taking in humans, who equally often initiated switches to and from their preferred option. Thus, the dynamic coordination is not restricted to humans but can occur on the background of different social attitudes and cognitive capacities in rhesus monkeys. Overall, our results illustrate how action visibility promotes the emergence and maintenance of coordination when agents can observe and time their mutual actions.

To live with others is to make concessions. You may want to go to the movies tonight, but your partner may prefer the theatre: reaching a mutually desirable goal ­ that is, spending time together ­ requires adjusting your preferences to theirs. Many other social species also make such decisions, in particular monkeys that live in large groups. Conceptually, these interactions are known as coordination games. In such scenarios, two players must coordinate their actions to attain a coveted reward, but they must also resolve a conflict about who gets the larger share. This makes the joint strategy non-trivial, and different pairs of players might resort to different strategies. In the laboratory, coordination games are often tested in settings which do not allow participants to monitor each other's behaviors as they make these complex choices. In real life, however, individuals making a joint decision can often observe each other and receive immediate feedback. In response, Moeller et al. developed a new way to test coordination games that allows more realistic social interactions. In their setup, two participants face each other and use a shared see-through touchscreen to perform a task. This new design was used to test how humans and macaque monkeys solved a simplified version of the 'Bach or Stravinsky' coordination game, which involves choosing between a red and blue target on the screen. Players in a pair had been trained to 'prefer' opposite colors. In this game, collaboration is beneficial (both individuals get a better prize if they choose the same color) but also creates unfairness (the reward is higher for the participant whose 'favorite' color is selected). When paired up, both humans and monkeys learned to collaborate and to go for the same color (or, in some monkey pairs, the same side of the screen). However, only humans took turns selecting red or blue so that players could alternate getting the highest reward. Monkeys usually settled on one color throughout the game, unless they had learned the 'turn-taking' strategy from a human partner; in that case, the color chosen in each trial was typically determined by the monkey who was the faster to move. These experiments show how monkeys and humans use visual information about their partner's actions to coordinate their choices, paving the way for further decision-making studies that accurately reflect how interactions unfold in real life. Moeller et al. expect that this will help to understand how cooperation and competition emerge in these two species, including how direct face-to-face contact, or lack thereof in some aspects of our modern world, shapes our social behavior.

Home-Enclosure-Based Behavioral and Wireless Neural Recording Setup for Unrestrained Rhesus Macaques.

Electrophysiological studies with behaving nonhuman primates often require the separation of animals from their social group as well as partial movement restraint to perform well-controlled experiments. When the research goal per se does not mandate constraining the animals' movements, there are often still experimental needs imposed by tethered data acquisition. Recent technological advances meanwhile allow wireless neurophysiological recordings at high band-width in limited-size enclosures. Here, we demonstrate wireless neural recordings at single unit resolution from unrestrained rhesus macaques while they performed self-paced, structured visuomotor tasks on our custom-built, stand-alone touchscreen system [eXperimental Behavioral Instrument (XBI)] in their home environment. We were able to successfully characterize neural tuning to task parameters, such as visuo-spatial selectivity during movement planning and execution, as expected from existing findings obtained via setup-based neurophysiology recordings. We conclude that when movement restraint and/or a highly controlled, insulated environment are not necessary for scientific reasons, cage-based wireless neural recordings are a viable option. We propose an approach that allows the animals to engage in a self-paced manner with our XBI device, both for fully automatized training and cognitive testing, as well as neural data acquisition in their familiar environment, maintaining auditory and sometimes visual contact with their conspecifics.

Turning decisions into actions.

Are selection and control of action serial processes of separate neural modules? A new study in PLOS Biology argues against this and in favor of an integrated process distributed across multiple brain regions, each contributing in a distinct way.

Universal Guide for Skull Extraction and Custom-Fitting of Implants to Continuous and Discontinuous Skulls.

Intracranial neurophysiological recordings require chronic implants to provide transcranial access to the brain. Especially in larger animals, which participate in experiments over extended periods of time, implants should match the skull curvature to promote osseointegration and avoid tissue and bacterial ingress over time. Proposed CAD methods for designing implants to date have focused on naive animals with continuous and even skull surfaces and calculate Boolean differences between implant and skull surface to fit the implant to the skull curvature. However, custom-fitting by calculating the difference fails, if a discontinuous skull surface needs to be matched. Also, the difference method does not allow designs with constant material thickness along the skull curvature, e.g., to allow fixed screw lengths. We present a universal step-by-step guide for custom-fitting implants which overcomes these limitations. It is suited for unusual skull conditions, like surface discontinuities or irregularities and includes virtual bending as a process to match skull surfaces while maintaining implant thickness. We demonstrate its applicability for a wide spectrum of scenarios, ranging from complex-shaped single-pieced implants to detailed multicomponent implant systems built on even or discontinuous skull. The guide uses only a few software tools and the final virtual product can be manufactured using CNC milling or 3D printing. A detailed description of this process is available on GitHub including step-by-step video instructions suitable for users without any prior knowledge in CAD programming. We report the experience with these implants over several years in 11 rhesus monkeys.

The cone method: Inferring decision times from single-trial 3D movement trajectories in choice behavior.

Ongoing goal-directed movements can be rapidly adjusted following new environmental information, e.g., when chasing pray or foraging. This makes movement trajectories in go-before-you-know decision-making a suitable behavioral readout of the ongoing decision process. Yet, existing methods of movement analysis are often based on statistically comparing two groups of trial-averaged trajectories and are not easily applied to three-dimensional data, preventing them from being applicable to natural free behavior. We developed and tested the cone method to estimate the point of overt commitment (POC) along a single two- or three-dimensional trajectory, i.e., the position where the movement is adjusted towards a newly selected spatial target. In Experiment 1, we established a "ground truth" data set in which the cone method successfully identified the experimentally constrained POCs across a wide range of all but the shallowest adjustment angles. In Experiment 2, we demonstrate the power of the method in a typical decision-making task with expected decision time differences known from previous findings. The POCs identified by cone method matched these expected effects. In both experiments, we compared the cone method's single trial performance with a trial-averaging method and obtained comparable results. We discuss the advantages of the single-trajectory cone method over trial-averaging methods and possible applications beyond the examples presented in this study. The cone method provides a distinct addition to existing tools used to study decisions during ongoing movement behavior, which we consider particularly promising towards studies of non-repetitive free behavior.

The future of General Movement Assessment: The role of computer vision and machine learning - A scoping review.

BACKGROUND: The clinical and scientific value of Prechtl general movement assessment (GMA) has been increasingly recognised, which has extended beyond the detection of cerebral palsy throughout the years. With advancing computer science, a surging interest in developing automated GMA emerges. AIMS: In this scoping review, we focused on video-based approaches, since it remains authentic to the non-intrusive principle of the classic GMA. Specifically, we aimed to provide an overview of recent video-based approaches targeting GMs; identify their techniques for movement detection and classification; examine if the technological solutions conform to the fundamental concepts of GMA; and discuss the challenges of developing automated GMA. METHODS AND PROCEDURES: We performed a systematic search for computer vision-based studies on GMs. OUTCOMES AND RESULTS: We identified 40 peer-reviewed articles, most (n = 30) were published between 2017 and 2020. A wide variety of sensing, tracking, detection, and classification tools for computer vision-based GMA were found. Only a small portion of these studies applied deep learning approaches. A comprehensive comparison between data acquisition and sensing setups across the reviewed studies, highlighting limitations and advantages of each modality in performing automated GMA is provided. CONCLUSIONS AND IMPLICATIONS: A "method-of-choice" for automated GMA does not exist. Besides creating large datasets, understanding the fundamental concepts and prerequisites of GMA is necessary for developing automated solutions. Future research shall look beyond the narrow field of detecting cerebral palsy and open up to the full potential of applying GMA to enable an even broader application.

An Open Resource for Non-human Primate Optogenetics.

Optogenetics has revolutionized neuroscience in small laboratory animals, but its effect on animal models more closely related to humans, such as non-human primates (NHPs), has been mixed. To make evidence-based decisions in primate optogenetics, the scientific community would benefit from a centralized database listing all attempts, successful and unsuccessful, of using optogenetics in the primate brain. We contacted members of the community to ask for their contributions to an open science initiative. As of this writing, 45 laboratories around the world contributed more than 1,000 injection experiments, including precise details regarding their methods and outcomes. Of those entries, more than half had not been published. The resource is free for everyone to consult and contribute to on the Open Science Framework website. Here we review some of the insights from this initial release of the database and discuss methodological considerations to improve the success of optogenetic experiments in NHPs.

Histological assessment of optogenetic tools to study fronto-visual and fronto-parietal cortical networks in the rhesus macaque.

Optogenetics offers unprecedented possibilities to investigate cortical networks. Yet, the number of successful optogenetic applications in non-human primates is still low, and the consequences of opsin expression in the primate brain are not well documented. We assessed histologically if we can target cerebrocortical networks with three common optogenetic constructs (AAV2/5-CaMKIIα-eNpHR3.0-mCherry, -ChR2-eYFP, -C1V1-mCherry). The frontal eye field or the dorsal premotor area of rhesus macaques were virally injected, and the resulting transduction spread, expression specificity, and opsin trafficking into axons projecting to parietal and visual areas were examined. After variable periods (2-24 months), expression was robust for all constructs at the injection sites. The CaMKIIα promoter driven-expression was predominant, but not exclusive, in excitatory neurons. In the case of eNpHR3.0-mCherry and ChR2-eYFP, opsins were present in axonal projections to target areas, in which sparse, retrogradely transduced neurons could also be found. Finally, the intracellular distribution of opsins differed: ChR2-eYFP had almost exclusive membrane localization, while eNpHR3.0-mCherry and C1V1-mCherry showed additional intracellular accumulations, which might affect neuronal survival in the long-term. Results indicate that all three constructs can be used for local neuronal modulation, but axonal stimulation and long-term use require additional considerations of construct selection and verification.

Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex.

System neuroscience of motor cognition regarding the space beyond immediate reach mandates free, yet experimentally controlled movements. We present an experimental environment (Reach Cage) and a versatile visuo-haptic interaction system (MaCaQuE) for investigating goal-directed whole-body movements of unrestrained monkeys. Two rhesus monkeys conducted instructed walk-and-reach movements towards targets flexibly positioned in the cage. We tracked 3D multi-joint arm and head movements using markerless motion capture. Movements show small trial-to-trial variability despite being unrestrained. We wirelessly recorded 192 broad-band neural signals from three cortical sensorimotor areas simultaneously. Single unit activity is selective for different reach and walk-and-reach movements. Walk-and-reach targets could be decoded from premotor and parietal but not motor cortical activity during movement planning. The Reach Cage allows systems-level sensorimotor neuroscience studies with full-body movements in a configurable 3D spatial setting with unrestrained monkeys. We conclude that the primate frontoparietal network encodes reach goals beyond immediate reach during movement planning.

Emergence and suppression of cooperation by action visibility in transparent games.

Real-world agents, humans as well as animals, observe each other during interactions and choose their own actions taking the partners' ongoing behaviour into account. Yet, classical game theory assumes that players act either strictly sequentially or strictly simultaneously without knowing each other's current choices. To account for action visibility and provide a more realistic model of interactions under time constraints, we introduce a new game-theoretic setting called transparent games, where each player has a certain probability of observing the partner's choice before deciding on its own action. By means of evolutionary simulations, we demonstrate that even a small probability of seeing the partner's choice before one's own decision substantially changes the evolutionary successful strategies. Action visibility enhances cooperation in an iterated coordination game, but reduces cooperation in a more competitive iterated Prisoner's Dilemma. In both games, "Win-stay, lose-shift" and "Tit-for-tat" strategies are predominant for moderate transparency, while a "Leader-Follower" strategy emerges for high transparency. Our results have implications for studies of human and animal social behaviour, especially for the analysis of dyadic and group interactions.

Complementary encoding of priors in monkey frontoparietal network supports a dual process of decision-making.

Prior expectations of movement instructions can promote preliminary action planning and influence choices. We investigated how action priors affect action-goal encoding in premotor and parietal cortices and if they bias subsequent free choice. Monkeys planned reaches according to visual cues that indicated relative probabilities of two possible goals. On instructed trials, the reach goal was determined by a secondary cue respecting these probabilities. On rarely interspersed free-choice trials without instruction, both goals offered equal reward. Action priors induced graded free-choice biases and graded frontoparietal motor-goal activity, complementarily in two subclasses of neurons. Down-regulating neurons co-encoded both possible goals and decreased opposite-to-preferred responses with decreasing prior, possibly supporting a process of choice by elimination. Up-regulating neurons showed increased preferred-direction responses with increasing prior, likely supporting a process of computing net likelihood. Action-selection signals emerged earliest in down-regulating neurons of premotor cortex, arguing for an initiation of selection in the frontal lobe.

Comparing Open-Source Toolboxes for Processing and Analysis of Spike and Local Field Potentials Data.

Analysis of spike and local field potential (LFP) data is an essential part of neuroscientific research. Today there exist many open-source toolboxes for spike and LFP data analysis implementing various functionality. Here we aim to provide a practical guidance for neuroscientists in the choice of an open-source toolbox best satisfying their needs. We overview major open-source toolboxes for spike and LFP data analysis as well as toolboxes with tools for connectivity analysis, dimensionality reduction and generalized linear modeling. We focus on comparing toolboxes functionality, statistical and visualization tools, documentation and support quality. To give a better insight, we compare and illustrate functionality of the toolboxes on open-access dataset or simulated data and make corresponding MATLAB scripts publicly available.

Peri-hand space expands beyond reach in the context of walk-and-reach movements.

The brain incorporates sensory information across modalities to be able to interact with our environment. The peripersonal space (PPS), defined by a high level of crossmodal interaction, is centered on the relevant body part, e.g. the hand, but can spatially expand to encompass tools or reach targets during goal-directed behavior. Previous studies considered expansion of the PPS towards goals within immediate or tool-mediated reach, but not the translocation of the body as during walking. Here, we used the crossmodal congruency effect (CCE) to quantify the extension of the PPS and test if PPS can also expand further to include far located walk-and-reach targets accessible only by translocation of the body. We tested for orientation specificity of the hand-centered reference frame, asking if the CCE inverts with inversion of the hand orientation during reach. We show a high CCE with onset of the movement not only towards reach targets but also walk-and-reach targets. When participants must change hand orientation, the CCE decreases, if not vanishes, and does not simply invert. We conclude that the PPS can expand to the action space beyond immediate or tool-mediated reaching distance but is not purely hand-centered with respect to orientation.

Correction: Using imaging photoplethysmography for heart rate estimation in non-human primates.

[This corrects the article DOI: 10.1371/journal.pone.0202581.].

Experimental Testing of Bionic Peripheral Nerve and Muscle Interfaces: Animal Model Considerations.

Introduction: Man-machine interfacing remains the main challenge for accurate and reliable control of bionic prostheses. Implantable electrodes in nerves and muscles may overcome some of the limitations by significantly increasing the interface's reliability and bandwidth. Before human application, experimental preclinical testing is essential to assess chronic in-vivo biocompatibility and functionality. Here, we analyze available animal models, their costs and ethical challenges in special regards to simulating a potentially life-long application in a short period of time and in non-biped animals. Methods: We performed a literature analysis following the PRISMA guidelines including all animal models used to record neural or muscular activity via implantable electrodes, evaluating animal models, group size, duration, origin of publication as well as type of interface. Furthermore, behavioral, ethical, and economic considerations of these models were analyzed. Additionally, we discuss experience and surgical approaches with rat, sheep, and primate models and an approach for international standardized testing. Results: Overall, 343 studies matched the search terms, dominantly originating from the US (55%) and Europe (34%), using mainly small animal models (rat: 40%). Electrode placement was dominantly neural (77%) compared to muscular (23%). Large animal models had a mean duration of 135 ± 87.2 days, with a mean of 5.3 ± 3.4 animals per trial. Small animal models had a mean duration of 85 ± 11.2 days, with a mean of 12.4 ± 1.7 animals. Discussion: Only 37% animal models were by definition chronic tests (>3 months) and thus potentially provide information on long-term performance. Costs for large animals were up to 45 times higher than small animals. However, costs are relatively small compared to complication costs in human long-term applications. Overall, we believe a combination of small animals for preliminary primary electrode testing and large animals to investigate long-term biocompatibility, impedance, and tissue regeneration parameters provides sufficient data to ensure long-term human applications.

Using imaging photoplethysmography for heart rate estimation in non-human primates.

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.