Collaborative robots can augment human cognition in regret-sensitive tasks.

Despite theoretical benefits of collaborative robots, disappointing outcomes are well documented by clinical studies, spanning rehabilitation, prostheses, and surgery. Cognitive load theory provides a possible explanation for why humans in the real world are not realizing the benefits of collaborative robots: high cognitive loads may be impeding human performance. Measuring cognitive availability using an electrocardiogram, we ask 25 participants to complete a virtual-reality task alongside an invisible agent that determines optimal performance by iteratively updating the Bellman equation. Three robots assist by providing environmental information relevant to task performance. By enabling the robots to act more autonomously-managing more of their own behavior with fewer instructions from the human-here we show that robots can augment participants' cognitive availability and decision-making. The way in which robots describe and achieve their objective can improve the human's cognitive ability to reason about the task and contribute to human-robot collaboration outcomes. Augmenting human cognition provides a path to improve the efficacy of collaborative robots. By demonstrating how robots can improve human cognition, this work paves the way for improving the cognitive capabilities of first responders, manufacturing workers, surgeons, and other future users of collaborative autonomy systems.

Characterizing Eye Gaze for Assistive Device Control.

Eye gaze tracking is increasingly popular due to improved technology and availability. However, in assistive device control, eye gaze tracking is often limited to discrete control inputs. In this paper, we present a method for collecting both reactionary and control eye gaze signals to build an individualized characterization for eye gaze interface use. Results from a study conducted with motor-impaired participants are presented, offering insights into maximizing the potential of eye gaze for assistive device control. These findings can inform the development of continuous control paradigms using eye gaze.

Colloidal robotics.

Robots have components that work together to accomplish a task. Colloids are particles, usually less than 100 µm, that are small enough that they do not settle out of solution. Colloidal robots are particles capable of functions such as sensing, computation, communication, locomotion and energy management that are all controlled by the particle itself. Their design and synthesis is an emerging area of interdisciplinary research drawing from materials science, colloid science, self-assembly, robophysics and control theory. Many colloidal robot systems approach synthetic versions of biological cells in autonomy and may find ultimate utility in bringing these specialized functions to previously inaccessible locations. This Perspective examines the emerging literature and highlights certain design principles and strategies towards the realization of colloidal robots.

Emergent microrobotic oscillators via asymmetry-induced order.

Spontaneous oscillations on the order of several hertz are the drivers of many crucial processes in nature. From bacterial swimming to mammal gaits, converting static energy inputs into slowly oscillating power is key to the autonomy of organisms across scales. However, the fabrication of slow micrometre-scale oscillators remains a major roadblock towards fully-autonomous microrobots. Here, we study a low-frequency oscillator that emerges from a collective of active microparticles at the air-liquid interface of a hydrogen peroxide drop. Their interactions transduce ambient chemical energy into periodic mechanical motion and on-board electrical currents. Surprisingly, these oscillations persist at larger ensemble sizes only when a particle with modified reactivity is added to intentionally break permutation symmetry. We explain such emergent order through the discovery of a thermodynamic mechanism for asymmetry-induced order. The on-board power harvested from the stabilised oscillations enables the use of electronic components, which we demonstrate by cyclically and synchronously driving a microrobotic arm. This work highlights a new strategy for achieving low-frequency oscillations at the microscale, paving the way for future microrobotic autonomy.

Mechanical computing.

Mechanical mechanisms have been used to process information for millennia, with famous examples ranging from the Antikythera mechanism of the Ancient Greeks to the analytical machines of Charles Babbage. More recently, electronic forms of computation and information processing have overtaken these mechanical forms, owing to better potential for miniaturization and integration. However, several unconventional computing approaches have recently been introduced, which blend ideas of information processing, materials science and robotics. This has raised the possibility of new mechanical computing systems that augment traditional electronic computing by interacting with and adapting to their environment. Here we discuss the use of mechanical mechanisms, and associated nonlinearities, as a means of processing information, with a view towards a framework in which adaptable materials and structures act as a distributed information processing network, even enabling information processing to be viewed as a material property, alongside traditional material properties such as strength and stiffness. We focus on approaches to abstract digital logic in mechanical systems, discuss how these systems differ from traditional electronic computing, and highlight the challenges and opportunities that they present.

A dynamical model for generating synthetic data to quantify active tactile sensing behavior in the rat.

As it becomes possible to simulate increasingly complex neural networks, it becomes correspondingly important to model the sensory information that animals actively acquire: the biomechanics of sensory acquisition directly determines the sensory input and therefore neural processing. Here, we exploit the tractable mechanics of the well-studied rodent vibrissal ("whisker") system to present a model that can simulate the signals acquired by a full sensor array actively sampling the environment. Rodents actively "whisk" ∼60 vibrissae (whiskers) to obtain tactile information, and this system is therefore ideal to study closed-loop sensorimotor processing. The simulation framework presented here, WHISKiT Physics, incorporates realistic morphology of the rat whisker array to predict the time-varying mechanical signals generated at each whisker base during sensory acquisition. Single-whisker dynamics were optimized based on experimental data and then validated against free tip oscillations and dynamic responses to collisions. The model is then extrapolated to include all whiskers in the array, incorporating each whisker's individual geometry. Simulation examples in laboratory and natural environments demonstrate that WHISKiT Physics can predict input signals during various behaviors, currently impossible in the biological animal. In one exemplary use of the model, the results suggest that active whisking increases in-plane whisker bending compared to passive stimulation and that principal component analysis can reveal the relative contributions of whisker identity and mechanics at each whisker base to the vibrissotactile response. These results highlight how interactions between array morphology and individual whisker geometry and dynamics shape the signals that the brain must process.

Autoperforation of two-dimensional materials to generate colloidal state machines capable of locomotion.

A central ambition of the robotics field has been to increasingly miniaturize such systems, with perhaps the ultimate achievement being the synthetic microbe or cell sized machine. To this end, we have introduced and demonstrated prototypes of what we call colloidal state machines (CSMs) as particulate devices capable of integrating sensing, memory, and energy harvesting as well as other functions onto a single particle. One technique that we have introduced for creating CSMs based on 2D materials such as graphene or monolayer MoS2 is "autoperforation", where the nanometer-scale film is fractured around a designed strain field to produce structured particles upon liftoff. While CSMs have been demonstrated with functions such as memory, sensing, and energy harvesting, the property of locomotion has not yet been demonstrated. In this work, we introduce an inversion moulding technique compatible with autoperforation that allows for the patterning of an external catalytic surface that enables locomotion in an accompanying fuel bath. Optimal processing conditions for electroplating a catalytic Pt layer to one side of an autoperforated CSM are elucidated. The self-driven propulsion of the resulting Janus CSM in H2O2 is studied, including the average velocity, as a function of fluid surface tension and H2O2 concentration in the bath. Since machines have to encode for a specific task, this work summarizes efforts to create a microfluidic testbed that allows for CSM designs to be evaluated for the ultimate purpose of navigation through complex fluidic networks, such as the human circulatory system. We introduce two CSM designs that mimic aspects of human immunity to solve search and recruitment tasks in such environments. These results advance CSM design concepts closer to promising applications in medicine and other areas.

Low rattling: A predictive principle for self-organization in active collectives.

Self-organization is frequently observed in active collectives as varied as ant rafts and molecular motor assemblies. General principles describing self-organization away from equilibrium have been challenging to identify. We offer a unifying framework that models the behavior of complex systems as largely random while capturing their configuration-dependent response to external forcing. This allows derivation of a Boltzmann-like principle for understanding and manipulating driven self-organization. We validate our predictions experimentally, with the use of shape-changing robotic active matter, and outline a methodology for controlling collective behavior. Our findings highlight how emergent order depends sensitively on the matching between external patterns of forcing and internal dynamical response properties, pointing toward future approaches for the design and control of active particle mixtures and metamaterials.

Tuning movement for sensing in an uncertain world.

While animals track or search for targets, sensory organs make small unexplained movements on top of the primary task-related motions. While multiple theories for these movements exist-in that they support infotaxis, gain adaptation, spectral whitening, and high-pass filtering-predicted trajectories show poor fit to measured trajectories. We propose a new theory for these movements called energy-constrained proportional betting, where the probability of moving to a location is proportional to an expectation of how informative it will be balanced against the movement's predicted energetic cost. Trajectories generated in this way show good agreement with measured trajectories of fish tracking an object using electrosense, a mammal and an insect localizing an odor source, and a moth tracking a flower using vision. Our theory unifies the metabolic cost of motion with information theory. It predicts sense organ movements in animals and can prescribe sensor motion for robots to enhance performance.

Ergodicity Reveals Assistance and Learning from Physical Human-Robot Interaction.

This paper applies information theoretic principles to the investigation of physical human-robot interaction. Drawing from the study of human perception and neural encoding, information theoretic approaches offer a perspective that enables quantitatively interpreting the body as an information channel, and bodily motion as an information-carrying signal. We show that ergodicity, which can be interpreted as the degree to which a trajectory encodes information about a task, correctly predicts changes due to reduction of a person's existing deficit or the addition of algorithmic assistance. The measure also captures changes from training with robotic assistance. Other common measures for assessment failed to capture at least one of these effects. This information-based interpretation of motion can be applied broadly, in the evaluation and design of human-machine interactions, in learning by demonstration paradigms, or in human motion analysis.

A robot made of robots: Emergent transport and control of a smarticle ensemble.

Robot locomotion is typically generated by coordinated integration of single-purpose components, like actuators, sensors, body segments, and limbs. We posit that certain future robots could self-propel using systems in which a delineation of components and their interactions is not so clear, becoming robust and flexible entities composed of functional components that are redundant and generic and can interact stochastically. Control of such a collective becomes a challenge because synthesis techniques typically assume known input-output relationships. To discover principles by which such future robots can be built and controlled, we study a model robophysical system: planar ensembles of periodically deforming smart, active particles-smarticles. When enclosed, these individually immotile robots could collectively diffuse via stochastic mechanical interactions. We show experimentally and theoretically that directed drift of such a supersmarticle could be achieved via inactivation of individual smarticles and used this phenomenon to generate endogenous phototaxis. By numerically modeling the relationship between smarticle activity and transport, we elucidated the role of smarticle deactivation on supersmarticle dynamics from little data-a single experimental trial. From this mapping, we demonstrate that the supersmarticle could be exogenously steered anywhere in the plane, expanding supersmarticle capabilities while simultaneously enabling decentralized closed-loop control. We suggest that the smarticle model system may aid discovery of principles by which a class of future "stochastic" robots can rely on collective internal mechanical interactions to perform tasks.

Massive increase in visual range preceded the origin of terrestrial vertebrates.

The evolution of terrestrial vertebrates, starting around 385 million years ago, is an iconic moment in evolution that brings to mind images of fish transforming into four-legged animals. Here, we show that this radical change in body shape was preceded by an equally dramatic change in sensory abilities akin to transitioning from seeing over short distances in a dense fog to seeing over long distances on a clear day. Measurements of eye sockets and simulations of their evolution show that eyes nearly tripled in size just before vertebrates began living on land. Computational simulations of these animal's visual ecology show that for viewing objects through water, the increase in eye size provided a negligible increase in performance. However, when viewing objects through air, the increase in eye size provided a large increase in performance. The jump in eye size was, therefore, unlikely to have arisen for seeing through water and instead points to an unexpected hybrid of seeing through air while still primarily inhabiting water. Our results and several anatomical innovations arising at the same time suggest lifestyle similarity to crocodiles. The consequent combination of the increase in eye size and vision through air would have conferred a 1 million-fold increase in the amount of space within which objects could be seen. The "buena vista" hypothesis that our data suggest is that seeing opportunities from afar played a role in the subsequent evolution of fully terrestrial limbs as well as the emergence of elaborated action sequences through planning circuits in the nervous system.

Ergodic Exploration Using Binary Sensing for Nonparametric Shape Estimation.

Current methods to estimate object shape-using either vision or touch-generally depend on high-resolution sensing. Here, we exploit ergodic exploration to demonstrate successful shape estimation when using a low-resolution binary contact sensor. The measurement model is posed as a collision-based tactile measurement, and classification methods are used to discriminate between shape boundary regions in the search space. Posterior likelihood estimates of the measurement model help the system actively seek out regions where the binary sensor is most likely to return informative measurements. Results show successful shape estimation of various objects as well as the ability to identify multiple objects in an environment. Interestingly, it is shown that ergodic exploration utilizes non-contact motion to gather significant information about shape. The algorithm is extended in three dimensions in simulation and we present two dimensional experimental results using the Rethink Baxter robot.

Sensory Agreement Guides Kinetic Energy Optimization of Arm Movements during Object Manipulation.

The laws of physics establish the energetic efficiency of our movements. In some cases, like locomotion, the mechanics of the body dominate in determining the energetically optimal course of action. In other tasks, such as manipulation, energetic costs depend critically upon the variable properties of objects in the environment. Can the brain identify and follow energy-optimal motions when these motions require moving along unfamiliar trajectories? What feedback information is required for such optimal behavior to occur? To answer these questions, we asked participants to move their dominant hand between different positions while holding a virtual mechanical system with complex dynamics (a planar double pendulum). In this task, trajectories of minimum kinetic energy were along curvilinear paths. Our findings demonstrate that participants were capable of finding the energy-optimal paths, but only when provided with veridical visual and haptic information pertaining to the object, lacking which the trajectories were executed along rectilinear paths.

Augmenting sensorimotor control using "goal-aware" vibrotactile stimulation during reaching and manipulation behaviors.

We describe two sets of experiments that examine the ability of vibrotactile encoding of simple position error and combined object states (calculated from an optimal controller) to enhance performance of reaching and manipulation tasks in healthy human adults. The goal of the first experiment (tracking) was to follow a moving target with a cursor on a computer screen. Visual and/or vibrotactile cues were provided in this experiment, and vibrotactile feedback was redundant with visual feedback in that it did not encode any information above and beyond what was already available via vision. After only 10 minutes of practice using vibrotactile feedback to guide performance, subjects tracked the moving target with response latency and movement accuracy values approaching those observed under visually guided reaching. Unlike previous reports on multisensory enhancement, combining vibrotactile and visual feedback of performance errors conferred neither positive nor negative effects on task performance. In the second experiment (balancing), vibrotactile feedback encoded a corrective motor command as a linear combination of object states (derived from a linear-quadratic regulator implementing a trade-off between kinematic and energetic performance) to teach subjects how to balance a simulated inverted pendulum. Here, the tactile feedback signal differed from visual feedback in that it provided information that was not readily available from visual feedback alone. Immediately after applying this novel "goal-aware" vibrotactile feedback, time to failure was improved by a factor of three. Additionally, the effect of vibrotactile training persisted after the feedback was removed. These results suggest that vibrotactile encoding of appropriate combinations of state information may be an effective form of augmented sensory feedback that can be applied, among other purposes, to compensate for lost or compromised proprioception as commonly observed, for example, in stroke survivors.

Local E-optimality Conditions for Trajectory Design to Estimate Parameters in Nonlinear Systems.

This paper develops an optimization method to synthesize trajectories for use in the identification of system parameters. Using widely studied techniques to compute Fisher information based on observations of nonlinear dynamical systems, an infinite-dimensional, projection-based optimization algorithm is formulated to optimize the system trajectory using eigenvalues of the Fisher information matrix as the cost metric. An example of a cart-pendulum simulation demonstrates a significant increase in the Fisher information using the optimized trajectory with decreased parameter variances shown through Monte-Carlo tests and computation of the Cramer-Rao lower bound.

Modeling forces and moments at the base of a rat vibrissa during noncontact whisking and whisking against an object.

During exploratory behavior, rats brush and tap their whiskers against objects, and the mechanical signals so generated constitute the primary sensory variables upon which these animals base their vibrissotactile perception of the world. To date, however, we lack a general dynamic model of the vibrissa that includes the effects of inertia, damping, and collisions. We simulated vibrissal dynamics to compute the time-varying forces and bending moment at the vibrissa base during both noncontact (free-air) whisking and whisking against an object (collision). Results show the following: (1) during noncontact whisking, mechanical signals contain components at both the whisking frequency and also twice the whisking frequency (the latter could code whisking speed); (2) when rats whisk rhythmically against an object, the intrinsic dynamics of the vibrissa can be as large as many of the mechanical effects of the collision, however, the axial force could still generate responses that reliably indicate collision based on thresholding; and (3) whisking velocity will have only a small effect on the transient response generated during a whisker-object collision. Instead, the transient response will depend in large part on how the rat chooses to decelerate its vibrissae after the collision. The model allows experimentalists to estimate error bounds on quasi-static descriptions of vibrissal shape, and its predictions can be used to bound realistic expectations from neurons that code vibrissal sensing. We discuss the implications of these results under the assumption that primary sensory neurons of the trigeminal ganglion are sensitive to various combinations of mechanical signals.

Trajectory Synthesis for Fisher Information Maximization.

Estimation of model parameters in a dynamic system can be significantly improved with the choice of experimental trajectory. For general nonlinear dynamic systems, finding globally "best" trajectories is typically not feasible; however, given an initial estimate of the model parameters and an initial trajectory, we present a continuous-time optimization method that produces a locally optimal trajectory for parameter estimation in the presence of measurement noise. The optimization algorithm is formulated to find system trajectories that improve a norm on the Fisher information matrix (FIM). A double-pendulum cart apparatus is used to numerically and experimentally validate this technique. In simulation, the optimized trajectory increases the minimum eigenvalue of the FIM by three orders of magnitude, compared with the initial trajectory. Experimental results show that this optimized trajectory translates to an order-of-magnitude improvement in the parameter estimate error in practice.