Visual working memory models of delayed estimation do not generalize to whole-report tasks.

Whole-report working memory tasks provide a measure of recall for all stimuli in a trial and afford single-trial analyses that are not possible with single-report delayed estimation tasks. However, most whole-report studies assume that trial stimuli are encoded and reported independently, and they do not consider the relationships between stimuli presented and reported within the same trial. Here, we present the results of two independently conducted whole-report experiments. The first dataset was recorded by Adam, Vogel, and Awh (2017) and required participants to report color and orientation stimuli using a continuous response wheel. We recorded the second dataset, which required participants to report color stimuli using a set of discrete buttons. We found that participants often group their reports by color similarity, contradicting the assumption of independence implicit in most encoding models of working memory. Next, we showed that this behavior was consistent across participants and experiments when reporting color but not orientation, two circular variables often assumed to be equivalent.Finally, we implemented an alternative to independent encoding where stimuli are encoded as a hierarchical Bayesian ensemble and found that this model predicts biases that are not present in either dataset. Our results suggest that assumptions made by both independent and hierarchical ensemble encoding models-which were developed in the context of single-report delayed estimation tasks-do not hold for the whole-report task. This failure to generalize highlights the need to consider variations in task structure when inferring fundamental principles of visual working memory.

Attentional modulation of eye blinking is altered by sex, age, and task structure.

Spontaneous eye blinking is gaining popularity as a proxy for higher cognitive functions, as it is readily modulated by both environmental demands and internal processes. Prior studies were impoverished in sample size, sex representation and age distribution, making it difficult to establish a complete picture of the behavior. Here we present eye-tracking data from a large cohort of normative participants (n=604, 393 F, aged 5-93 years) performing two tasks: one with structured, discrete trials (interleaved pro/anti-saccade task; IPAST) and one with a less structured, continuous organization in which participants watch movies (free-viewing; FV). Sex- and age-based analyses revealed that females had higher blink rates between the ages of 22 and 58 years in the IPAST, and 22 and 34 years in FV. We derived a continuous measure of blink probability to reveal behavioral changes driven by stimulus appearance in both paradigms. In the IPAST, blinks were suppressed near stimulus appearance, particularly on correct anti-saccade trials, which we attribute to the stronger inhibitory control required for anti-saccades compared to pro-saccades. In FV, blink suppression occurred immediately after scene changes, and the effect was sustained on scenes where gaze clustered among participants (indicating engagement of attention). Females were more likely than males to blink during appearance of novel stimuli in both tasks, but only within the age bin of 18-44 years. The consistency of blink patterns in each paradigm endorses blinking as a sensitive index for changes in visual processing and attention, while sex and age differences drive interindividual variability.Significance Statement Eye-tracking is becoming useful as a non-invasive tool for detecting preclinical markers of neurological and psychiatric disease. Blinks are understudied despite being an important supplement to saccade and pupil eye-tracking metrics. The present study is a crucial step in developing a healthy baseline for blink behavior to compare to clinical groups. While many prior blink studies suffered from small sample sizes with relatively low age- and sex-diversity (review by Jongkees & Colzato, 2016), our large cohort of healthy participants has permitted a more detailed analysis of sex and age effects in blink behavior. Furthermore, our analysis techniques are robust to temporal changes in blink probability, greatly clarifying the relationship between blinking, visual processing, and inhibitory control mechanisms on visual tasks.

Parietofrontal oscillations show hand-specific interactions with top-down movement plans.

To generate a hand-specific reach plan, the brain must integrate hand-specific signals with the desired movement strategy. Although various neurophysiology/imaging studies have investigated hand-target interactions in simple reach-to-target tasks, the whole brain timing and distribution of this process remain unclear, especially for more complex, instruction-dependent motor strategies. Previously, we showed that a pro/anti pointing instruction influences magnetoencephalographic (MEG) signals in frontal cortex that then propagate recurrently through parietal cortex (Blohm G, Alikhanian H, Gaetz W, Goltz HC, DeSouza JF, Cheyne DO, Crawford JD. NeuroImage 197: 306-319, 2019). Here, we contrasted left versus right hand pointing in the same task to investigate 1) which cortical regions of interest show hand specificity and 2) which of those areas interact with the instructed motor plan. Eight bilateral areas, the parietooccipital junction (POJ), superior parietooccipital cortex (SPOC), supramarginal gyrus (SMG), medial/anterior interparietal sulcus (mIPS/aIPS), primary somatosensory/motor cortex (S1/M1), and dorsal premotor cortex (PMd), showed hand-specific changes in beta band power, with four of these (M1, S1, SMG, aIPS) showing robust activation before movement onset. M1, SMG, SPOC, and aIPS showed significant interactions between contralateral hand specificity and the instructed motor plan but not with bottom-up target signals. Separate hand/motor signals emerged relatively early and lasted through execution, whereas hand-motor interactions only occurred close to movement onset. Taken together with our previous results, these findings show that instruction-dependent motor plans emerge in frontal cortex and interact recurrently with hand-specific parietofrontal signals before movement onset to produce hand-specific motor behaviors.NEW & NOTEWORTHY The brain must generate different motor signals depending on which hand is used. The distribution and timing of hand use/instructed motor plan integration are not understood at the whole brain level. Using MEG we show that different action planning subnetworks code for hand usage and integrating hand use into a hand-specific motor plan. The timing indicates that frontal cortex first creates a general motor plan and then integrates hand specificity to produce a hand-specific motor plan.

Effector-dependent stochastic reference frame transformations alter decision-making.

Psychophysical, motor control, and modeling studies have revealed that sensorimotor reference frame transformations (RFTs) add variability to transformed signals. For perceptual decision-making, this phenomenon could decrease the fidelity of a decision signal's representation or alternatively improve its processing through stochastic facilitation. We investigated these two hypotheses under various sensorimotor RFT constraints. Participants performed a time-limited, forced-choice motion discrimination task under eight combinations of head roll and/or stimulus rotation while responding either with a saccade or button press. This paradigm, together with the use of a decision model, allowed us to parameterize and correlate perceptual decision behavior with eye-, head-, and shoulder-centered sensory and motor reference frames. Misalignments between sensory and motor reference frames produced systematic changes in reaction time and response accuracy. For some conditions, these changes were consistent with a degradation of motion evidence commensurate with a decrease in stimulus strength in our model framework. Differences in participant performance were explained by a continuum of eye-head-shoulder representations of accumulated motion evidence, with an eye-centered bias during saccades and a shoulder-centered bias during button presses. In addition, we observed evidence for stochastic facilitation during head-rolled conditions (i.e., head roll resulted in faster, more accurate decisions in oblique motion for a given stimulus-response misalignment). We show that perceptual decision-making and stochastic RFTs are inseparable within the present context. We show that by simply rolling one's head, perceptual decision-making is altered in a way that is predicted by stochastic RFTs.

Effects of Simulated Microgravity and Hypergravity Conditions on Arm Movements in Normogravity.

The human sensorimotor control has evolved in the Earth's environment where all movement is influenced by the gravitational force. Changes in this environmental force can severely impact the performance of arm movements which can be detrimental in completing certain tasks such as piloting or controlling complex vehicles. For this reason, subjects that are required to perform such tasks undergo extensive training procedures in order to minimize the chances of failure. We investigated whether local gravity simulation of altered gravitational conditions on the arm would lead to changes in kinematic parameters comparable to the full-body experience of microgravity and hypergravity onboard a parabolic flight. To see if this would be a feasible approach for on-ground training of arm reaching movements in altered gravity conditions we developed a robotic device that was able to apply forces at the wrist in order to simulate micro- or hypergravity conditions for the arm while subjects performed pointing movements on a touch screen. We analyzed and compared the results of several kinematic parameters along with muscle activity using this system with data of the same subjects being fully exposed to microgravity and hypergravity conditions on a parabolic flight. Both in our simulation and in-flight, we observed a significant increase in movement durations in microgravity conditions and increased velocities in hypergravity for upward movements. Additionally, we noted a reduced accuracy of pointing both in-flight and in our simulation. These promising results suggest, that locally simulated altered gravity can elicit similar changes in some movement characteristics for arm reaching movements. This could potentially be exploited as a means of developing devices such as exoskeletons to aid in training individuals prior to undertaking tasks in changed gravitational conditions.

Effects of Local Gravity Compensation on Motor Control During Altered Environmental Gravity.

Our sensorimotor control is well adapted to normogravity environment encountered on Earth and any change in gravity significantly disturbs our movement. In order to produce appropriate motor commands for aimed arm movements such as pointing or reaching, environmental changes have to be taken into account. This adaptation is crucial when performing successful movements during microgravity and hypergravity conditions. To mitigate the effects of changing gravitational levels, such as the changed movement duration and decreased accuracy, we explored the possible beneficial effects of gravity compensation on movement. Local gravity compensation was achieved using a motorized robotic device capable of applying precise forces to the subject's wrist that generated a normogravity equivalent torque at the shoulder joint during periods of microgravity and hypergravity. The efficiency of the local gravity compensation was assessed with an experiment in which participants performed a series of pointing movements toward the target on a screen during a parabolic flight. We compared movement duration, accuracy, movement trajectory, and muscle activations of movements during periods of microgravity and hypergravity with conditions when local gravity compensation was provided. The use of local gravity compensation at the arm mitigated the changes in movement duration, accuracy, and muscle activity. Our results suggest that the use of such an assistive device helps with movements during unfamiliar environmental gravity.

MoVi: A large multi-purpose human motion and video dataset.

Large high-quality datasets of human body shape and kinematics lay the foundation for modelling and simulation approaches in computer vision, computer graphics, and biomechanics. Creating datasets that combine naturalistic recordings with high-accuracy data about ground truth body shape and pose is challenging because different motion recording systems are either optimized for one or the other. We address this issue in our dataset by using different hardware systems to record partially overlapping information and synchronized data that lend themselves to transfer learning. This multimodal dataset contains 9 hours of optical motion capture data, 17 hours of video data from 4 different points of view recorded by stationary and hand-held cameras, and 6.6 hours of inertial measurement units data recorded from 60 female and 30 male actors performing a collection of 21 everyday actions and sports movements. The processed motion capture data is also available as realistic 3D human meshes. We anticipate use of this dataset for research on human pose estimation, action recognition, motion modelling, gait analysis, and body shape reconstruction.

Movement drift in optic ataxia reveals deficits in hand state estimation in oculocentric coordinates.

When vision is removed, limb position has been shown to progressively drift during repetitive arm movements. The posterior parietal cortex (PPC) is known to be involved in the processing of multisensory information, the formation of internal hand estimate, and online motor control. Here, we compared hand position drift between healthy controls and 2 patients with PPC damage to gain insight into the mechanisms underlying movement drift and investigate the possible role of the PPC in this process. To do so, we asked participants to perform back-and-forth movements between 2 targets, in the dark and under different gaze fixation conditions. Each individual participant consistently drifted to the same end position for a given hand and gaze condition. We found that the final drift distance was related to small systematic errors made on the very first trial in the dark, with an approximate 3.5 fold increase in magnitude. Furthermore, PPC damage resulted in greater movement drift in patients when the unseen hand was in the contralesional oculocentric space and also when the target was located in the lower visual field. We conclude that the PPC is involved in the proprioceptive representation of hand position in oculocentric coordinates used for reach planning and motor control. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Confidence in predicted position error explains saccadic decisions during pursuit.

A fundamental problem in motor control is the coordination of complementary movement types to achieve a common goal. As a common example, humans view moving objects through coordinated pursuit and saccadic eye movements. Pursuit is initiated and continuously controlled by retinal image velocity. During pursuit, eye position may lag behind the target. This can be compensated by the discrete execution of a catch-up saccade. The decision to trigger a saccade is influenced by both position and velocity errors, and the timing of saccades can be highly variable. The observed distributions of saccade frequency and trigger time remain poorly understood, and this decision process remains imprecisely quantified. Here, we propose a predictive, probabilistic model explaining the decision to trigger saccades during pursuit to foveate moving targets. In this model, expected position error and its associated uncertainty are predicted through Bayesian inference across noisy, delayed sensory observations (Kalman filtering). This probabilistic prediction is used to estimate the confidence that a saccade is needed (quantified through log-probability ratio), triggering a saccade upon accumulating to a fixed threshold. The model qualitatively explains behavioral observations on the frequency and trigger time distributions of saccades during pursuit over a range of target motion trajectories. Furthermore, this model makes novel predictions that saccade decisions are highly sensitive to uncertainty for small predicted position errors, but this influence diminishes as the magnitude of predicted position error increases. We suggest that this predictive, confidence-based decision-making strategy represents a fundamental principle for the probabilistic neural control of coordinated movements.NEW & NOTEWORTHY This is the first stochastic dynamical systems model of pursuit-saccade coordination accounting for noise and delays in the sensorimotor system. The model uses Bayesian inference to predictively estimate visual motion, triggering saccades when confidence in predicted position error accumulates to a threshold. This model explains saccade frequency and trigger time distributions across target trajectories and makes novel predictions about the influence of sensory uncertainty in saccade decisions during pursuit.

Reaching around obstacles accounts for uncertainty in coordinate transformations.

When reaching to a visual target, humans need to transform the spatial target representation into the coordinate system of their moving arm. It has been shown that increased uncertainty in such coordinate transformations, for instance, when the head is rolled toward one shoulder, leads to higher movement variability and influence movement decisions. However, it is unknown whether the brain incorporates such added variability in planning and executing movements. We designed an obstacle avoidance task in which participants had to reach with or without visual feedback of the hand to a visual target while avoiding collisions with an obstacle. We varied coordinate transformation uncertainty by varying head roll (straight, 30° clockwise, and 30° counterclockwise). In agreement with previous studies, we observed that the reaching variability increased when the head was tilted. Indeed, head roll did not influence the number of collisions during reaching compared with the head-straight condition, but it did systematically change the obstacle avoidance behavior. Participants changed the preferred direction of passing the obstacle and increased the safety margins indicated by stronger movement curvature. These results suggest that the brain takes the added movement variability during head roll into account and compensates for it by adjusting the reaching trajectories.NEW & NOTEWORTHY We show that changing body geometry such as head roll results in compensatory reaching behaviors around obstacles. Specifically, we observed head roll causes changed preferred movement direction and increased trajectory curvature. As has been shown before, head roll increases movement variability due to stochastic coordinate transformations. Thus these results provide evidence that the brain must consider the added movement variability caused by coordinate transformations for accurate reach movements.

A How-to-Model Guide for Neuroscience.

Within neuroscience, models have many roles, including driving hypotheses, making assumptions explicit, synthesizing knowledge, making experimental predictions, and facilitating applications to medicine. While specific modeling techniques are often taught, the process of constructing models for a given phenomenon or question is generally left opaque. Here, informed by guiding many students through modeling exercises at our summer school in CoSMo (Computational Sensory-Motor Neuroscience), we provide a practical 10-step breakdown of the modeling process. This approach makes choices and criteria more explicit and replicable. Experiment design has long been taught in neuroscience; the modeling process should receive the same attention.

Predicted Position Error Triggers Catch-Up Saccades during Sustained Smooth Pursuit.

For humans, visual tracking of moving stimuli often triggers catch-up saccades during smooth pursuit. The switch between these continuous and discrete eye movements is a trade-off between tolerating sustained position error (PE) when no saccade is triggered or a transient loss of vision during the saccade due to saccadic suppression. de Brouwer et al. (2002b) demonstrated that catch-up saccades were less likely to occur when the target re-crosses the fovea within 40-180 ms. To date, there is no mechanistic explanation for how the trigger decision is made by the brain. Recently, we proposed a stochastic decision model for saccade triggering during visual tracking (Coutinho et al., 2018) that relies on a probabilistic estimate of predicted PE (PEpred). Informed by model predictions, we hypothesized that saccade trigger time length and variability will increase when pre-saccadic predicted errors are small or visual uncertainty is high (e.g., for blurred targets). Data collected from human participants performing a double step-ramp task showed that large pre-saccadic PEpred (>10°) produced short saccade trigger times regardless of the level of uncertainty while saccade trigger times preceded by small PEpred (<10°) significantly increased in length and variability, and more so for blurred targets. Our model also predicted increased signal-dependent noise (SDN) as retinal slip (RS) increases; in our data, this resulted in longer saccade trigger times and more smooth trials without saccades. In summary, our data supports our hypothesized predicted error-based decision process for coordinating saccades during smooth pursuit.

Hierarchical recruitment of competition alleviates working memory overload in a frontoparietal model.

The storage limitations of visual working memory have been the subject of intense research interest for several decades, but few studies have systematically investigated the dependence of these limitations on memory load that exceeds our retention abilities. Under this real-world scenario, performance typically declines beyond a critical load among low-performing subjects, a phenomenon known as working memory overload. We used a frontoparietal cortical model to test the hypothesis that high-performing subjects select a manageable number of items for storage, thereby avoiding overload. The model accounts for behavioral and electrophysiological data from high-performing subjects in a parameter regime where competitive encoding in its prefrontal network selects items for storage, interareal projections sustain their representations after stimulus offset, and weak dynamics in its parietal network limit their mutual interference. Violation of these principles accounts for these data among low-performing subjects, implying that poor visual working memory performance reflects poor control over frontoparietal circuitry, making testable predictions for experiments.

Misperception of motion in depth originates from an incomplete transformation of retinal signals.

Depth perception requires the use of an internal model of the eye-head geometry to infer distance from binocular retinal images and extraretinal 3D eye-head information, particularly ocular vergence. Similarly, for motion in depth perception, gaze angle is required to correctly interpret the spatial direction of motion from retinal images; however, it is unknown whether the brain can make adequate use of extraretinal version and vergence information to correctly transform binocular retinal motion into 3D spatial coordinates. Here we tested this hypothesis by asking participants to reconstruct the spatial trajectory of an isolated disparity stimulus moving in depth either peri-foveally or peripherally while participants' gaze was oriented at different vergence and version angles. We found large systematic errors in the perceived motion trajectory that reflected an intermediate reference frame between a purely retinal interpretation of binocular retinal motion (not accounting for veridical vergence and version) and the spatially correct motion. We quantify these errors with a 3D reference frame model accounting for target, eye, and head position upon motion percept encoding. This model could capture the behavior well, revealing that participants tended to underestimate their version by up to 17%, overestimate their vergence by up to 22%, and underestimate the overall change in retinal disparity by up to 64%, and that the use of extraretinal information depended on retinal eccentricity. Since such large perceptual errors are not observed in everyday viewing, we suggest that both monocular retinal cues and binocular extraretinal signals are required for accurate real-world motion in depth perception.

Saccade-induced changes in ocular torsion reveal predictive orientation perception.

Natural orienting of gaze often results in a retinal image that is rotated relative to space due to ocular torsion. However, we perceive neither this rotation nor a moving world despite visual rotational motion on the retina. This perceptual stability is often attributed to the phenomenon known as predictive remapping, but the current remapping literature ignores this torsional component. In addition, studies often simply measure remapping across either space or features (e.g., orientation) but in natural circumstances, both components are bound together for stable perception. One natural circumstance in which the perceptual system must account for the current and future eye orientation to correctly interpret the orientation of external stimuli occurs during movements to or from oblique eye orientations (i.e., eye orientations with both a horizontal and vertical angular component relative to the primary position). Here we took advantage of oblique eye orientation-induced ocular torsion to examine perisaccadic orientation perception. First, we found that orientation perception was largely predicted by the rotated retinal image. Second, we observed a presaccadic remapping of orientation perception consistent with maintaining a stable (but spatially inaccurate) retinocentric perception throughout the saccade. These findings strongly suggest that our seamless perceptual stability relies on retinocentric signals that are predictively remapped in all three ocular dimensions with each saccade.

Anti-saccades predict cognitive functions in older adults and patients with Parkinson's disease.

A major component of cognitive control is the ability to act flexibly in the environment by either behaving automatically or inhibiting an automatic behaviour. The interleaved pro/anti-saccade task measures cognitive control because the task relies on one's abilities to switch flexibly between pro and anti-saccades, and inhibit automatic saccades during anti-saccade trials. Decline in cognitive control occurs during aging or neurological illnesses such as Parkinson's disease (PD), and indicates decline in other cognitive abilities, such as memory. However, little is known about the relationship between cognitive control and other cognitive processes. Here we investigated whether anti-saccade performance can predict decision-making, visual memory, and pop-out and serial visual search performance. We tested 34 younger adults, 22 older adults, and 20 PD patients on four tasks: an interleaved pro/anti-saccade, a spatial visual memory, a decision-making and two types of visual search (pop-out and serial) tasks. Anti-saccade performance was a good predictor of decision-making and visual memory abilities for both older adults and PD patients, while it predicted visual search performance to a larger extent in PD patients. Our results thus demonstrate the suitability of the interleaved pro/anti-saccade task as a cognitive marker of cognitive control in aging and PD populations.

Vibrotactile information improves proprioceptive reaching target localization.

When pointing to parts of our own body (e.g., the opposite index finger), the position of the target is derived from proprioceptive signals. Consistent with the principles of multisensory integration, it has been found that participants better matched the position of their index finger when they also had visual cues about its location. Unlike vision, touch may not provide an additional information about finger position in space, since fingertip tactile information theoretically remains the same irrespective of the postural configuration of the upper limb. However, since tactile and proprioceptive information are ultimately coded within the same population of posterior parietal neurons within high-level spatial representations, we nevertheless hypothesized that additional tactile information could benefit the processing of proprioceptive signals. To investigate the influence of tactile information on proprioceptive localization, we asked 19 participants to reach with the right hand towards the opposite unseen index finger (proprioceptive target). Vibrotactile stimuli were applied to the target index finger prior to movement execution. We found that participants made smaller errors and more consistent reaches following tactile stimulation. These results demonstrate that transient touch provided at the proprioceptive target improves subsequent reaching precision and accuracy. Such improvement was not observed when tactile stimulation was delivered to a distinct body part (the shoulder). This suggests a specific spatial integration of touch and proprioception at the level of high-level cortical body representations, resulting in touch improving position sense.

Weighted integration of short-term memory and sensory signals in the oculomotor system.

Oculomotor behaviors integrate sensory and prior information to overcome sensory-motor delays and noise. After much debate about this process, reliability-based integration has recently been proposed and several models of smooth pursuit now include recurrent Bayesian integration or Kalman filtering. However, there is a lack of behavioral evidence in humans supporting these theoretical predictions. Here, we independently manipulated the reliability of visual and prior information in a smooth pursuit task. Our results show that both smooth pursuit eye velocity and catch-up saccade amplitude were modulated by visual and prior information reliability. We interpret these findings as the continuous reliability-based integration of a short-term memory of target motion with visual information, which support modeling work. Furthermore, we suggest that saccadic and pursuit systems share this short-term memory. We propose that this short-term memory of target motion is quickly built and continuously updated, and constitutes a general building block present in all sensorimotor systems.

Neck muscle spindle noise biases reaches in a multisensory integration task.

Reference frame transformations (RFTs) are crucial components of sensorimotor transformations in the brain. Stochasticity in RFTs has been suggested to add noise to the transformed signal due to variability in transformation parameter estimates (e.g., angle) as well as the stochastic nature of computations in spiking networks of neurons. Here, we varied the RFT angle together with the associated variability and evaluated the behavioral impact in a reaching task that required variability-dependent visual-proprioceptive multisensory integration. Crucially, reaches were performed with the head either straight or rolled 30° to either shoulder, and we also applied neck loads of 0 or 1.8 kg (left or right) in a 3 × 3 design, resulting in different combinations of estimated head roll angle magnitude and variance required in RFTs. A novel three-dimensional stochastic model of multisensory integration across reference frames was fitted to the data and captured our main behavioral findings: 1) neck load biased head angle estimation across all head roll orientations, resulting in systematic shifts in reach errors; 2) increased neck muscle tone led to increased reach variability due to signal-dependent noise; and 3) both head roll and neck load created larger angular errors in reaches to visual targets away from the body compared with reaches toward the body. These results show that noise in muscle spindles and stochasticity in general have a tangible effect on RFTs underlying reach planning. Since RFTs are omnipresent in the brain, our results could have implications for processes as diverse as motor control, decision making, posture/balance control, and perception. NEW & NOTEWORTHY We show that increasing neck muscle tone systematically biases reach movements. A novel three-dimensional multisensory integration across reference frames model captures the data well and provides evidence that the brain must have online knowledge of full-body geometry together with the associated variability to plan reach movements accurately.