Comparing eye-hand coordination between controller-mediated virtual reality, and a real-world object interaction task.

Virtual reality (VR) technology has advanced significantly in recent years, with many potential applications. However, it is unclear how well VR simulations mimic real-world experiences, particularly in terms of eye-hand coordination. This study compares eye-hand coordination from a previously validated real-world object interaction task to the same task re-created in controller-mediated VR. We recorded eye and body movements and segmented participants' gaze data using the movement data. In the real-world condition, participants wore a head-mounted eye tracker and motion capture markers and moved a pasta box into and out of a set of shelves. In the VR condition, participants wore a VR headset and moved a virtual box using handheld controllers. Unsurprisingly, VR participants took longer to complete the task. Before picking up or dropping off the box, participants in the real world visually fixated the box about half a second before their hand arrived at the area of action. This 500-ms minimum fixation time before the hand arrived was preserved in VR. Real-world participants disengaged their eyes from the box almost immediately after their hand initiated or terminated the interaction, but VR participants stayed fixated on the box for much longer after it was picked up or dropped off. We speculate that the limited haptic feedback during object interactions in VR forces users to maintain visual fixation on objects longer than in the real world, altering eye-hand coordination. These findings suggest that current VR technology does not replicate real-world experience in terms of eye-hand coordination.

Decision-making in sensorimotor control.

Skilled sensorimotor interactions with the world result from a series of decision-making processes that determine, on the basis of information extracted during the unfolding sequence of events, which movements to make and when and how to make them. Despite this inherent link between decision-making and sensorimotor control, research into each of these two areas has largely evolved in isolation, and it is only fairly recently that researchers have begun investigating how they interact and, together, influence behaviour. Here, we review recent behavioural, neurophysiological and computational research that highlights the role of decision-making processes in the selection, planning and control of goal-directed movements in humans and nonhuman primates.

Generating accurate 3D gaze vectors using synchronized eye tracking and motion capture.

Assessing gaze behavior during real-world tasks is difficult; dynamic bodies moving through dynamic worlds make gaze analysis difficult. Current approaches involve laborious coding of pupil positions. In settings where motion capture and mobile eye tracking are used concurrently in naturalistic tasks, it is critical that data collection be simple, efficient, and systematic. One solution is to combine eye tracking with motion capture to generate 3D gaze vectors. When combined with tracked or known object locations, 3D gaze vector generation can be automated. Here we use combined eye and motion capture and explore how linear regression models generate accurate 3D gaze vectors. We compare spatial accuracy of models derived from four short calibration routines across three pupil data inputs: the efficacy of calibration routines was assessed, a validation task requiring short fixations on task-relevant locations, and a naturalistic object interaction task to bridge the gap between laboratory and "in the wild" studies. Further, we generated and compared models using spherical and Cartesian coordinate systems and monocular (left or right) or binocular data. All calibration routines performed similarly, with the best performance (i.e., sub-centimeter errors) coming from the naturalistic task trials when the participant is looking at an object in front of them. We found that spherical coordinate systems generate the most accurate gaze vectors with no differences in accuracy when using monocular or binocular data. Overall, we recommend 1-min calibration routines using binocular pupil data combined with a spherical world coordinate system to produce the highest-quality gaze vectors.

Myoelectric prosthesis users and non-disabled individuals wearing a simulated prosthesis exhibit similar compensatory movement strategies.

BACKGROUND: Research studies on upper limb prosthesis function often rely on the use of simulated myoelectric prostheses (attached to and operated by individuals with intact limbs), primarily to increase participant sample size. However, it is not known if these devices elicit the same movement strategies as myoelectric prostheses (operated by individuals with amputation). The objective of this study was to address the question of whether non-disabled individuals using simulated prostheses employ the same compensatory movements (measured by hand and upper body kinematics) as individuals who use actual myoelectric prostheses. METHODS: The upper limb movements of two participant groups were investigated: (1) twelve non-disabled individuals wearing a simulated prosthesis, and (2) three individuals with transradial amputation using their custom-fitted myoelectric devices. Motion capture was used for data collection while participants performed a standardized functional task. Performance metrics, hand movements, and upper body angular kinematics were calculated. For each participant group, these measures were compared to those from a normative baseline dataset. Each deviation from normative movement behaviour, by either participant group, indicated that compensatory movements were used during task performance. RESULTS: Results show that participants using either a simulated or actual myoelectric prosthesis exhibited similar deviations from normative behaviour in phase durations, hand velocities, hand trajectories, number of movement units, grip aperture plateaus, and trunk and shoulder ranges of motion. CONCLUSIONS: This study suggests that the use of a simulated prosthetic device in upper limb research offers a reasonable approximation of compensatory movements employed by a low- to moderately-skilled transradial myoelectric prosthesis user.

Selective Modulation of Early Visual Cortical Activity by Movement Intention.

The primate visual system contains myriad feedback projections from higher- to lower-order cortical areas, an architecture that has been implicated in the top-down modulation of early visual areas during working memory and attention. Here we tested the hypothesis that these feedback projections also modulate early visual cortical activity during the planning of visually guided actions. We show, across three separate human functional magnetic resonance imaging (fMRI) studies involving object-directed movements, that information related to the motor effector to be used (i.e., limb, eye) and action goal to be performed (i.e., grasp, reach) can be selectively decoded-prior to movement-from the retinotopic representation of the target object(s) in early visual cortex. We also find that during the planning of sequential actions involving objects in two different spatial locations, that motor-related information can be decoded from both locations in retinotopic cortex. Together, these findings indicate that movement planning selectively modulates early visual cortical activity patterns in an effector-specific, target-centric, and task-dependent manner. These findings offer a neural account of how motor-relevant target features are enhanced during action planning and suggest a possible role for early visual cortex in instituting a sensorimotor estimate of the visual consequences of movement.

Clarifying frequency-dependent brightness enhancement: delta- and theta-band flicker, not alpha-band flicker, consistently seen as brightest.

Frequency-dependent brightness enhancement, a perceptual illusion in which a flickering light can appear twice as bright as a constant light, has historically been reported to produce maximum effects at a flicker rate within the alpha (8-12 Hz) band (Bartley in J Exp Psychol 23(3):313-319, 1938). Our recent examinations of this phenomenon using brightness discrimination between two flickering stimuli, however, have instead revealed the brightest percepts from theta-band (4-7 Hz) flicker (Bertrand et al. in Sci Rep 8(1):6152, 2018). Two primary questions arise from these seemingly contradictory findings: first, could task differences between these studies have caused recruitment of discrete oscillatory processes? Second, could the reported theta-band flicker enhancement be the result of an aliased alpha rhythm, sequentially sampling two stimulus locations, resulting in an ~ 5 Hz half-alpha rhythm? Here, we investigated these questions with two experiments: one replicating Bartley's (1938) adjustment paradigm, and one containing both Bartley's adjustment task and Bertrand's (2018) discrimination task, but presenting stimuli only sequentially (rather than concurrently). Examination of a range of frequencies (2-12 Hz) revealed the greatest brightness enhancement arising from flicker in the delta- and theta-band across all conditions, regardless of the spatial or temporal configuration of the stimuli. We speculate that these slower rhythms play an integral role in complex visual operations (e.g., a discrimination decision) where the entrainment of the endogenous neural rhythm to matched exogenous rhythmic stimulation promotes more efficient processing of visual information and thus produces perceptual biases as seen in frequency-dependent brightness enhancement.

Humans are sensitive to attention control when predicting others' actions.

Studies of social perception report acute human sensitivity to where another's attention is aimed. Here we ask whether humans are also sensitive to how the other's attention is deployed. Observers viewed videos of actors reaching to targets without knowing that those actors were sometimes choosing to reach to one of the targets (endogenous control) and sometimes being directed to reach to one of the targets (exogenous control). Experiments 1 and 2 showed that observers could respond more rapidly when actors chose where to reach, yet were at chance when guessing whether the reach was chosen or directed. This implicit sensitivity to attention control held when either actor's faces or limbs were masked (experiment 3) and when only the earliest actor's movements were visible (experiment 4). Individual differences in sensitivity to choice correlated with an independent measure of social aptitude. We conclude that humans are sensitive to attention control through an implicit kinematic process linked to empathy. The findings support the hypothesis that social cognition involves the predictive modeling of others' attentional states.

Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks.

This study explores the role that vision plays in sequential object interactions. We used a head-mounted eye tracker and upper-limb motion capture to quantify visual behavior while participants performed two standardized functional tasks. By simultaneously recording eye and motion tracking, we precisely segmented participants' visual data using the movement data, yielding a consistent and highly functionally resolved data set of real-world object-interaction tasks. Our results show that participants spend nearly the full duration of a trial fixating on objects relevant to the task, little time fixating on their own hand when reaching toward an object, and slightly more time-although still very little-fixating on the object in their hand when transporting it. A consistent spatial and temporal pattern of fixations was found across participants. In brief, participants fixate an object to be picked up at least half a second before their hand arrives at the object and stay fixated on the object until they begin to transport it, at which point they shift their fixation directly to the drop-off location of the object, where they stay fixated until the object is successfully released. This pattern provides additional evidence of a common system for the integration of vision and object interaction in humans, and is consistent with theoretical frameworks hypothesizing the distribution of attention to future action targets as part of eye and hand-movement preparation. Our results thus aid the understanding of visual attention allocation during planning of object interactions both inside and outside the field of view.

The sequential encoding of competing action goals involves dynamic restructuring of motor plans in working memory.

Recent neural and behavioral findings provide support for the influential idea that in situations in which multiple action options are presented simultaneously, we prepare action plans for each competing option before deciding between and executing one of those plans. However, in natural, everyday environments, our available action options frequently change from one moment to the next, and there is often uncertainty as to whether additional options will become available before having to select a particular course of action. Here, with the use of a target-directed reaching task, we show that in this situation, the brain specifies a competing action for each new, sequentially presented potential target and that recently formed action plans can be revisited and updated so as to conform with separate, more newly developed, plans. These findings indicate that the brain forms labile motor plans for sequentially arising target options that can be flexibly restructured to accommodate new motor plans.

Counting on the motor system: rapid action planning reveals the format- and magnitude-dependent extraction of numerical quantity.

Symbolic numbers (e.g., "2") acquire their meaning by becoming linked to the core nonsymbolic quantities they represent (e.g., two items). However, the extent to which symbolic and nonsymbolic information converges onto the same internal core representations of quantity remains a point of considerable debate. As nearly all previous work on this topic has employed perceptual tasks requiring the conscious reporting of numerical magnitudes, here we question the extent to which numerical processing via the visual-motor system might shed further light on the fundamental basis of how different number formats are encoded. We show, using a rapid reaching task and a detailed analysis of initial arm trajectories, that there are key differences in how the quantity information extracted from symbolic Arabic numerals and nonsymbolic collections of discrete items are used to guide action planning. In particular, we found that the magnitude derived from discrete dots resulted in movements being biased by an amount directly proportional to the actual quantities presented whereas the magnitude derived from numerals resulted in movements being biased only by the relative (e.g., larger than) quantities presented. In addition, we found that initial motor plans were more sensitive to changes in numerical quantity within small (1-3) than large (5-15) number ranges, irrespective of their format (dots or numerals). In light of previous work, our visual-motor results clearly show that the processing of numerical quantity information is both format and magnitude dependent.

Activity patterns in the category-selective occipitotemporal cortex predict upcoming motor actions.

Converging lines of evidence point to the occipitotemporal cortex (OTC) as a critical structure in visual perception. For instance, human functional magnetic resonance imaging (fMRI) has revealed a modular organisation of object-selective, face-selective, body-selective and scene-selective visual areas in the OTC, and disruptions to the processing within these regions, either in neuropsychological patients or through transcranial magnetic stimulation, can produce category-specific deficits in visual recognition. Here we show, using fMRI and pattern classification methods, that the activity in the OTC also represents how stimuli will be interacted with by the body--a level of processing more traditionally associated with the preparatory activity in sensorimotor circuits of the brain. Combining functional mapping of different OTC areas with a real object-directed delayed movement task, we found that the pre-movement spatial activity patterns across the OTC could be used to predict both the action of an upcoming hand movement (grasping vs. reaching) and the effector (left hand vs. right hand) to be used. Interestingly, we were able to extract this wide range of predictive movement information even though nearly all OTC areas showed either baseline-level or below baseline-level activity prior to action onset. Our characterisation of different OTC areas according to the features of upcoming movements that they could predict also revealed a general gradient of effector-to-action-dependent movement representations along the posterior-anterior OTC axis. These findings suggest that the ventral visual pathway, which is well known to be involved in object recognition and perceptual processing, plays a larger than previously expected role in preparing object-directed hand actions.

Connecting the dots: object connectedness deceives perception but not movement planning.

The perceptual system parses complex scenes into discrete objects. Parsing is also required for planning visually guided movements when more than one potential target is present. To examine whether visual perception and motor planning use the same or different parsing strategies, we used the connectedness illusion, in which observers typically report seeing fewer targets if pairs of targets are connected by short lines. We found that despite this illusion, when observers are asked to make speeded reaches toward targets in such displays, their reaches are unaffected by the presence of the connecting lines. Instead, their movement plans, as revealed by their movement trajectories, are influenced by the number of potential targets irrespective of whether connecting lines are present or not. This suggests that scene parsing for perception depends on mechanisms that are distinct from those that allow observers to plan rapid and efficient target-directed movements in situations with multiple potential targets.

You read my mind: Generating and minimizing intention uncertainty under different social contexts in a two-player online game.

We investigate an ecologically pertinent form of social uncertainty regarding the ability to read another's intentions. We use classic measures (response time, accuracy) and dynamic measures (mouse trajectories) to investigate how people generate or minimize uncertainty regarding their own intentions under different social contexts, and how uncertainty regarding other's intentions affects decision making. Ninety-six participants (N = 48 dyads) completed a two-player online card game, where the goal was to collect cards with a certain feature (e.g., triangles), with participant cursor movements projected to both players. Participants played six games, three cooperatively and three competitively (Social Decision Context). Points were awarded for two decisions: collecting a card matching one's goal (ability to achieve personal goal) and correctly guessing the other player's goal (ability to guess intention). Data revealed: (a) Card scores did not vary with Social Decision Context, (b) Guess scores did vary with Social Decision Context, with more correct guesses when cooperating compared to competing, and (c) Mouse trajectories (durations and mouse distance traveled) decreased when cooperating compared to competing. These results indicate that better guessing during cooperative play is not due to explicit communication (i.e., circling desired cards), but may be due to increased speed and confidence when making decisions in a cooperative context. Additionally, participants could be actively hiding their intention in a competitive context. Thus, social uncertainty when reading another's intentions is both adaptive-affected by the prescribed social context, and automatic-indirectly inferred from the way another moves their mouse when acting with intention. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

"Real-time" obstacle avoidance in the absence of primary visual cortex.

When we reach toward objects, we easily avoid potential obstacles located in the workspace. Previous studies suggest that obstacle avoidance relies on mechanisms in the dorsal visual stream in the posterior parietal cortex. One fundamental question that remains unanswered is where the visual inputs to these dorsal-stream mechanisms are coming from. Here, we provide compelling evidence that these mechanisms can operate in "real-time" without direct input from primary visual cortex (V1). In our first experiment, we used a reaching task to demonstrate that an individual with a dense left visual field hemianopia after damage to V1 remained strikingly sensitive to the position of unseen static obstacles placed in his blind field. Importantly, in a second experiment, we showed that his sensitivity to the same obstacles in his blind field was abolished when a short 2-s delay (without vision) was introduced before reach onset. These findings have far-reaching implications, not only for our understanding of the time constraints under which different visual pathways operate, but also in relation to how these seemingly "primitive" subcortical visual pathways can control complex everyday behavior without recourse to conscious vision.

Spatiotemporal Coupling of Hand and Eye Movements When Using a Myoelectric Prosthetic Hand.

Upper limb prosthesis users have disruptions in hand-eye coordination, with increased fixations towards the hand and less visual allocation for feedforward planning. The purpose of this study was to explore whether improved motor planning, as reflected by eye gaze behaviour, was associated with more efficient hand movement patterns. Able-bodied participants wore a simulated prosthesis while performing a functional object movement task. Motion and eye tracking data were collected to quantify the eye gaze and hand movement during object interaction. The results of this study demonstrated that the latency of the eye to precede the hand at pick-up was correlated with measures of hand function, including hand variability, movement units, and grasp time, but not reach time. During transport and release, longer latency to disengage gaze from the grasped object and look ahead towards the target was correlated to hand kinematics of hand variability, distance travelled, and transport time. In addition, the latency of the eye to disengage the drop-off location was correlated to release time. Together these may point to control issues with opening and closing the prosthetic hand. Overall, increased feedforward fixations towards the target and reduced feedback fixations towards the hand were related to improved measures of hand function. Hence, coordination between eye and hand movements when using a myoelectric prosthesis may prove to be a useful metric to assess motor planning.

One to four, and nothing more: nonconscious parallel individuation of objects during action planning.

Much of the current understanding about the capacity limits on the number of objects that can be simultaneously processed comes from studies of visual short-term memory, attention, and numerical cognition. Consistent reports suggest that, despite large variability in the perceptual tasks administered (e.g., object tracking, counting), a limit of three to four visual items can be independently processed in parallel. In the research reported here, we asked whether this limit also extends to the domain of action planning. Using a unique rapid visuomotor task and a novel analysis of reach trajectories, we demonstrated an upper limit to the number of targets that can be simultaneously encoded for action, a capacity limit that also turns out to be no more than three to four. Our findings suggest that conscious perceptual processing and nonconscious movement planning are constrained by a common underlying mechanism limited by the number of items that can be simultaneously represented.

To use or to move: goal-set modulates priming when grasping real tools.

How we interact with objects depends on what we intend to do with them. In the current work, we show that priming and the kinematics of grasping depend on the goals of grasping, as well as the context in which tasks are presented. We asked participants to grasp familiar kitchen tools in order to either move them, grasp-to-move (GTM), or to demonstrate their common use, grasp-to-use (GTU). When tasks were blocked separately (Experiment 1), we found that priming was only evident for the GTU task. However, when tasks were presented in the same block of trials (Experiment 2), we observed priming for both tasks. Independent of priming, differences in kinematics and reaction times according to task were evident for both Experiments. Longer reaction times for the GTU task indicate more extensive planning, and differences in grasping reflect the characteristics of subsequent actions. Priming of real grasping is determined by task goals as well as task setting, both of which are likely to modulate how object features (affordances) are perceived and influence the planning of future actions.

Visual salience dominates early visuomotor competition in reaching behavior.

In this study, we investigated whether visual salience influences the competition between potential targets during reach planning. Participants initiated rapid pointing movements toward multiple potential targets, with the final target being cued only after the reach was initiated. We manipulated visual salience by varying the luminance of potential targets. Across two separate experiments, we demonstrate that initial reach trajectories are directed toward more salient targets, even when there are twice as many targets (and therefore twice the likelihood of the final target appearing) on the opposite side of space. We also show that this salience bias is time-dependent, as evidenced by the return of spatially averaged reach trajectories when participants were given an additional 500-ms preview of the target display prior to the cue to move. This study shows both when and to what extent task-irrelevant luminance differences affect the planning of reaches to multiple potential targets.

Selective attention to real-world objects drives their emotional appraisal.

Attentional manipulations have been shown to influence subsequent evaluations of objects and images. For example, images used as distractors in a visual search task are subsequently rated more negatively than are target images. One powerful manipulation of attention occurs when we plan and execute movements toward objects in our environment. Here, in two experiments, we show that selective attention to real-world objects subsequently improves emotional appraisal of those objects-an effect we term "target appreciation." Participants were presented with abstract images on three-dimensional objects, and were cued to either reach and grasp one of the two objects, or to respond to the cued object with a keyboard. Images presented on target objects were appraised more positively when compared with novel images. In contrast, images associated with obstacles or distractor objects were not appraised differently than novel images, despite the attentional suppression thought to be required to successfully avoid or ignore these objects. We speculate that this automatic appreciation of the objects of selective attention may be adaptive for organisms acting in complex environments.

Reaching for known unknowns: Rapid reach decisions accurately reflect the future state of dynamic probabilistic information.

Everyday tasks such as catching a ball appear effortless, but in fact require complex interactions and tight temporal coordination between the brain's visual and motor systems. What makes such interceptive actions particularly impressive is the capacity of the brain to account for temporal delays in the central nervous system-a limitation that can be mitigated by making predictions about the environment as well as one's own actions. Here, we wanted to assess how well human participants can plan an upcoming movement based on a dynamic, predictable stimulus that is not the target of action. A central stationary or rotating stimulus determined the probability that each of two potential targets would be the eventual target of a rapid reach-to-touch movement. We examined the extent to which reach movement trajectories convey internal predictions about the future state of dynamic probabilistic information conveyed by the rotating stimulus. We show that movement trajectories reflect the target probabilities determined at movement onset, suggesting that humans rapidly and accurately integrate visuospatial predictions and estimates of their own reaction times to effectively guide action.