From motor control to team play in simulated humanoid football.

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.

Magnetic control of tokamak plasmas through deep reinforcement learning.

Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable1,2, including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and 'snowflake' configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained 'droplets' on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.

Cyclic voltammetry deposition of copper nanostructure on MWCNTs modified pencil graphite electrode: An ultra-sensitive hydrazine sensor.

In this study, Copper (Cu) nanostructures (CuNS) were electrochemically deposited on a film of multiwall carbon nanotubes (MWCNTs) modified pencil graphite electrode (MWCNTs/PGE) by cyclic voltammetry method to fabricate a CuNS-MWCNTs composite sensor (CuNS-MWCNT/PGE) for hydrazine detection. Scanning electron microscopy (SEM) and Energy-dispersive X-ray spectroscopy (EDX) were used for the characterization of CuNS on the MWCNTs matrix. The composite of CuNS-MWCNTs was characterized with cyclic voltammetry (CV) and electrochemical impedance spectroscopy (EIS). The preliminary studies showed that the proposed sensor have a synergistic electrocatalytic activity for the oxidation of hydrazine in phosphate buffer. The catalytic currents of square wave voltammetry had a linear correlation with the hydrazine concentration in the range of 0.1 to 800µM with a low detection limit of 70nM. Moreover, the amperometric oxidation current exhibited a linear correlation with hydrazine concentration in the concentration range of 50-800µM with the detection limit of 4.3µM. The proposed electrode was used for the determination of hydrazine in real samples and the results were promising. Empirical results also indicated that the sensor had good reproducibility, long-term stability, and the response of the sensor to hydrazine was free from interferences. Moreover, the proposed sensor benefits from simple preparation, low cost, outstanding sensitivity, selectivity, and reproducibility for hydrazine determination.