Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis.

Optimal learning output tracking control (OLOTC) in a model-free manner has received increasing attention in both the intelligent control and the reinforcement learning (RL) communities. Although the model-free tracking control has been achieved via off-policy learning and Q -learning, another popular RL idea of direct policy learning, with its easy-to-implement feature, is still rarely considered. To fill this gap, this article aims to develop a novel model-free policy optimization (PO) algorithm to achieve the OLOTC for unknown linear discrete-time (DT) systems. The iterative control policy is parameterized to directly improve the discounted value function of the augmented system via the gradient-based method. To implement this algorithm in a model-free manner, a model-free two-point policy gradient (PG) algorithm is designed to approximate the gradient of discounted value function by virtue of the sampled states and the reference trajectories. The global convergence of model-free PO algorithm to the optimal value function is demonstrated with the sufficient quantity of samples and proper conditions. Finally, numerical simulation results are provided to validate the effectiveness of the present method.

Epigenetic regulation and factors that influence the effect of iPSCs-derived neural stem/progenitor cells (NS/PCs) in the treatment of spinal cord injury.

Spinal cord injury (SCI) is a severe neurological disorder that causes neurological impairment and disability. Neural stem/progenitor cells (NS/PCs) derived from induced pluripotent stem cells (iPSCs) represent a promising cell therapy strategy for spinal cord regeneration and repair. However, iPSC-derived NS/PCs face many challenges and issues in SCI therapy; one of the most significant challenges is epigenetic regulation and that factors that influence this mechanism. Epigenetics refers to the regulation of gene expression and function by DNA methylation, histone modification, and chromatin structure without changing the DNA sequence. Previous research has shown that epigenetics plays a crucial role in the generation, differentiation, and transplantation of iPSCs, and can influence the quality, safety, and outcome of transplanted cells. In this study, we review the effects of epigenetic regulation and various influencing factors on the role of iPSC-derived NS/PCs in SCI therapy at multiple levels, including epigenetic reprogramming, regulation, and the adaptation of iPSCs during generation, differentiation, and transplantation, as well as the impact of other therapeutic tools (e.g., drugs, electrical stimulation, and scaffolds) on the epigenetic status of transplanted cells. We summarize our main findings and insights in this field and identify future challenges and directions that need to be addressed and explored.

In Situ Synthesis of NH2-MIL-53-Al/PAN Nanofibers for Removal Co(II) through an Electrospinning Process.

In this study, researchers developed a novel composite material called NH2-MIL-53-Al/PAN, which consists of metal-organic frameworks (MOFs) grown on electrospun PAN nanofibers (NFs). The successful formation of the composite was confirmed by X-ray diffraction (XRD) and Fourier transform infrared (FTIR), and the hydrophilicity of NH2-MIL-53-Al/PAN was demonstrated by the water contact angle (WCA). Batch experiments were conducted to investigate the adsorption performance of Co(II) under different conditions. The maximum adsorption capacity reached 58.72 mg/g, and almost 95% of the adsorption was achieved within the first 6 h. The adsorption process was found to be spontaneous and endothermic and followed the pseudo-second-order kinetics and Langmuir models. Chemisorption and molecular layer adsorption are the main mechanisms of adsorption, and X-ray photoelectron spectroscopy (XPS) analysis further reveals that the interaction between the adsorbent and cobalt is a coordination interaction. In this study, NH2-MIL-53-Al was grown in situ on PAN to ensure effective loading of MOFs and prevent agglomeration during the NF mixing process. This approach successfully addressed the challenge of exposing active sites within the embedded MOF crystals. Additionally, it overcame the difficulty of recycling traditional MOF adsorbents. As a result, the exceptional performance of MOF NFs offers a promising solution for the efficient removal of cobalt-containing wastewater.

Novel Discounted Adaptive Critic Control Designs With Accelerated Learning Formulation.

Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, in which the iterative value function sequence possesses an adjustable convergence rate. The different convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. Based on the properties of the given VI scheme, an accelerated learning algorithm with convergence guarantee is presented. Moreover, the implementations of the new VI scheme and its accelerated learning design are elaborated, which involve value function approximation and policy improvement. A nonlinear fourth-order ball-and-beam balancing plant is used to verify the performance of the developed approaches. Compared with the traditional VI, the present discounted iterative adaptive critic designs greatly accelerate the convergence rate of the value function and reduce the computational cost simultaneously.

Event-Triggered Robust Adaptive Dynamic Programming for Multiplayer Stackelberg-Nash Games of Uncertain Nonlinear Systems.

In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is developed to solve a class of multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. Considering the different roles of players in the MSNG, the hierarchical decision-making process is described as the designed value functions for the leader and all followers, which assist to transform the robust control problem of the uncertain nonlinear system into an optimal regulation problem of the nominal system. Then, an online policy iteration algorithm is formulated to solve the derived coupled Hamilton-Jacobi equation. Meanwhile, an event-triggered mechanism is designed to alleviate computational and communication burdens. Moreover, critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control polices for all players, which constitute the Stackelberg-Nash equilibrium of the MSNG. By using Lyapunov's direct method, the stability of the closed-loop uncertain nonlinear system is guaranteed under the ETRADP-based control scheme in the sense of uniform ultimate boundedness. Finally, a numerical simulation is provided to demonstrate the effectiveness of the present ETRADP-based control scheme.

Rapid Adaptation for Active Pantograph Control in High-Speed Railway via Deep Meta Reinforcement Learning.

Active pantograph control is the most promising technique for reducing contact force (CF) fluctuation and improving the train's current collection quality. Existing solutions, however, suffer from two significant limitations: 1) they are incapable of dealing with the various pantograph types, catenary line operating conditions, changing operating speeds, and contingencies well and 2) it is challenging to implement in practical systems due to the lack of rapid adaptability to a new pantograph-catenary system (PCS) operating conditions and environmental disturbances. In this work, we alleviate these problems by developing a revolutionary context-based deep meta-reinforcement learning (CB-DMRL) algorithm. The proposed CB-DMRL algorithm combines Bayesian optimization (BO) with deep reinforcement learning (DRL), allowing the general agent to adapt to new tasks quickly and efficiently. We evaluated the CB-DMRL algorithm's performance on a proven PCS model. The experimental results demonstrate that meta-training DRL policies with latent space swiftly adapt to new operating conditions and unknown perturbations. The meta-agent adapts quickly after two iterations with a high reward, which require only ten spans, approximately equal to 0.5 km of PCS interaction data. Compared with state-of-the-art DRL algorithms and traditional solutions, the proposed method can promptly traverse scenario changes and reduce CF fluctuations, resulting in an excellent performance.

Adaptive dynamic programming-based hierarchical decision-making of non-affine systems.

In this paper, the problem of multiplayer hierarchical decision-making problem for non-affine systems is solved by adaptive dynamic programming. Firstly, the control dynamics are obtained according to the theory of dynamic feedback and combined with the original system dynamics to construct the affine augmented system. Thus, the non-affine multiplayer system is transformed into a general affine form. Then, the hierarchical decision problem is modeled as a Stackelberg game. In the Stackelberg game, the leader makes a decision based on the information of all followers, whereas the followers do not know each other's information and only obtain their optimal control strategy based on the leader's decision. Then, the augmented system is reconstructed by a neural network (NN) using input-output data. Moreover, a single critic NN is used to approximate the value function to obtain the optimal control strategy for each player. An extra term added to the weight update law makes the initial admissible control law no longer needed. According to the Lyapunov theory, the state of the system and the error of the weights of the NN are both uniformly ultimately bounded. Finally, the feasibility and validity of the algorithm are confirmed by simulation.

Synchronization of Delayed Memristor-Based Neural Networks via Pinning Control With Local Information.

In this article, a novel pinning control method, only requiring information from partial nodes, is developed to synchronize drive-response memristor-based neural networks (MNNs) with time delay. An improved mathematical model of MNNs is established to describe the dynamic behaviors of MNNs accurately. In the existing literature, pinning controllers for synchronization of drive-response systems were designed based on information of all nodes, but in some specific situations, the control gains may be very large and challenging to realize in practice. To overcome this problem, a novel pinning control policy is developed to achieve synchronization of delayed MNNs, which depends only on local information of MNNs, for reducing communication and calculation burdens. Furthermore, sufficient conditions for synchronization of delayed MNNs are provided. Finally, numerical simulation and comparative experiments are conducted to verify the effectiveness and superiority of the proposed pinning control method.

Neuro-Optimal Event-Triggered Impulsive Control for Stochastic Systems via ADP.

This article presents a novel neural-network-based optimal event-triggered impulsive control method. First, a novel general-event-based impulsive transition matrix (GITM) is constructed to represent the probability distribution evolving characteristics regarding all system states across the impulsive actions, rather than the prefixed timing sequence. On the foundation of this GITM, the event-triggered impulsive adaptive dynamic programming (ETIADP) algorithm and its high-efficiency version (HEIADP) are developed to deal with the optimization problems for stochastic systems with event-triggered impulsive controls. It is shown that the obtained controller design scheme can reduce the computational and communication burden caused by updating the controller periodically. By analyzing the admissibility, monotonicity, and optimality properties of ETIADP and HEIADP, we further establish the approximation error bound of the neural networks to address the connection between the ideal and neural-network-based realizations of the present methods. It is proven that the iterative value functions of both the ETIADP and HEIADP algorithms fall in a small neighborhood of the optimum as the iteration index increases to infinity. By adopting a novel task synchronization mechanism, the proposed HEIADP algorithm fully utilizes the computing resources of multiprocessor systems (MPSs), while significantly reducing the memory requirement compared to traditional ADP approaches. Finally, we carry out a numerical study to show that the proposed methods can fulfill the desired goals.

Event-triggered adaptive dynamic programming for decentralized tracking control of input constrained unknown nonlinear interconnected systems.

This paper addresses decentralized tracking control (DTC) problems for input constrained unknown nonlinear interconnected systems via event-triggered adaptive dynamic programming. To reconstruct the system dynamics, a neural-network-based local observer is established by using local input-output data and the desired trajectories of all other subsystems. By employing a nonquadratic value function, the DTC problem of the input constrained nonlinear interconnected system is transformed into an optimal control problem. By using the observer-critic architecture, the DTC policy is obtained by solving the local Hamilton-Jacobi-Bellman equation through the local critic neural network, whose weights are tuned by the experience replay technique to relax the persistence of excitation condition. Under the event-triggering mechanism, the DTC policy is updated at the event-triggering instants only. Then, the computational resource and the communication bandwidth are saved. The stability of the closed-loop system is guaranteed by implementing event-triggered DTC policy via Lyapunov's direct method. Finally, simulation examples are provided to demonstrate the effectiveness of the proposed scheme.

Adaptive Dynamic Programming-Based Event-Triggered Robust Control for Multiplayer Nonzero-Sum Games With Unknown Dynamics.

In this article, the event-triggered robust control of unknown multiplayer nonlinear systems with constrained inputs and uncertainties is investigated by using adaptive dynamic programming. To relax the requirement of system dynamics, a neural network-based identifier is constructed by using the system input-output data. Subsequently, by designing a nonquadratic value function, which contains the bounded functions, the system states, and the control inputs of all players, the event-triggered robust stabilization problem is converted into an event-triggered constrained optimal control problem. To obtain the approximate solution of the event-triggered Hamilton-Jacobi (HJ) equation, a critic network for each player is established with a novel weight updating law to relax the persistence of excitation condition based on the experience replay technique. Furthermore, according to the Lyapunov stability theorem, the present event-triggered robust optimal control ensures the multiplayer system to be uniformly ultimately bounded. Finally, two simulation examples are employed to show the effectiveness of the present method.

A Novel Value Iteration Scheme With Adjustable Convergence Rate.

In this article, a novel value iteration scheme is developed with convergence and stability discussions. A relaxation factor is introduced to adjust the convergence rate of the value function sequence. The convergence conditions with respect to the relaxation factor are given. The stability of the closed-loop system using the control policies generated by the present VI algorithm is investigated. Moreover, an integrated VI approach is developed to accelerate and guarantee the convergence by combining the advantages of the present and traditional value iterations. Also, a relaxation function is designed to adaptively make the developed value iteration scheme possess fast convergence property. Finally, the theoretical results and the effectiveness of the present algorithm are validated by numerical examples.

An Efficient Impulsive Adaptive Dynamic Programming Algorithm for Stochastic Systems.

In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a "piece-by-piece" manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all "sizes" of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.

Adaptive Dynamic Programming-Based Cooperative Motion/Force Control for Modular Reconfigurable Manipulators: A Joint Task Assignment Approach.

This article develops a cooperative motion/force control (CMFC) scheme based on adaptive dynamic programming (ADP) for modular reconfigurable manipulators (MRMs) with the joint task assignment approach. By separating terms depending on local variables only, the dynamic model of the entire MRM system can be regarded as a set of joint modules interconnected by coupling torque. In addition, the Jacobian matrix, which reflects the interaction force of the MRM end-effector, can be mapped into each joint. Using this approach, both the motion and force tasks on the end-effector of the entire MRM system can be assigned to each joint module cooperatively. Then, by substituting the actual states of coupled joint modules with their desired ones, the norm-boundedness assumption on the interconnection of joint module can be relaxed. By using the measured input-output data of each joint module, a neural network (NN)-based robust decentralized observer, which guarantees the observation error to be asymptotically stable is established. An improved local value function is constructed for each joint module to reflect the interconnection. Then, the local Hamilton-Jacobi-Bellman equation is solved by constructing a local critic NN with a nested learning structure. Hereafter, the ADP-based CMFC is obtained by the assistance of force feedback compensation. Based on the Lyapunov stability analysis, the closed-loop MRM system is guaranteed to be uniformly ultimately bounded under the present ADP-based CMFC scheme. The simulation on a two-degree of freedom MRM system demonstrates the effectiveness of the present control approach.

Bioinformatics analysis of ferroptosis in spinal cord injury.

Ferroptosis plays a key role in aggravating the progression of spinal cord injury (SCI), but the specific mechanism remains unknown. In this study, we constructed a rat model of T10 SCI using a modified Allen method. We identified 48, 44, and 27 ferroptosis genes that were differentially expressed at 1, 3, and 7 days after SCI induction. Compared with the sham group and other SCI subgroups, the subgroup at 1 day after SCI showed increased expression of the ferroptosis marker acyl-CoA synthetase long-chain family member 4 and the oxidative stress marker malondialdehyde in the injured spinal cord while glutathione in the injured spinal cord was lower. These findings with our bioinformatics results suggested that 1 day after SCI was the important period of ferroptosis progression. Bioinformatics analysis identified the following top ten hub ferroptosis genes in the subgroup at 1 day after SCI: STAT3, JUN, TLR4, ATF3, HMOX1, MAPK1, MAPK9, PTGS2, VEGFA, and RELA. Real-time polymerase chain reaction on rat spinal cord tissue confirmed that STAT3, JUN, TLR4, ATF3, HMOX1, PTGS2, and RELA mRNA levels were up-regulated and VEGFA, MAPK1 and MAPK9 mRNA levels were down-regulated. Ten potential compounds were predicted using the DSigDB database as potential drugs or molecules targeting ferroptosis to repair SCI. We also constructed a ferroptosis-related mRNA-miRNA-lncRNA network in SCI that included 66 lncRNAs, 10 miRNAs, and 12 genes. Our results help further the understanding of the mechanism underlying ferroptosis in SCI.

Distributed Fault Tolerant Consensus Control of Nonlinear Multiagent Systems via Adaptive Dynamic Programming.

This article develops a distributed fault-tolerant consensus control (DFTCC) approach for multiagent systems by using adaptive dynamic programming. By establishing a local fault observer, the potential actuator faults of each agent are estimated. Subsequently, the DFTCC problem is transformed into an optimal consensus control problem by designing a novel local value function for each agent which contains the estimated fault, the consensus errors, and the control laws of the local agent and its neighbors. In order to solve the coupled Hamilton-Jacobi-Bellman equation of each agent, a critic-only structure is established to obtain the approximate local optimal consensus control law of each agent. Moreover, by using Lyapunov's direct method, it is proven that the approximate local optimal consensus control law guarantees the uniform ultimate boundedness of the consensus error of all agents, which means that all following agents with potential actuator faults synchronize to the leader. Finally, two simulation examples are provided to validate the effectiveness of the present DFTCC scheme.

N6-methyladenosine modification: A potential regulatory mechanism in spinal cord injury.

N6-methyladenosine (m6A), an essential post-transcriptional modification in eukaryotes, is closely related to the development of pathological processes in neurological diseases. Notably, spinal cord injury (SCI) is a serious traumatic disease of the central nervous system, with a complex pathological mechanism which is still not completely understood. Recent studies have found that m6A modification levels are changed after SCI, and m6A-related regulators are involved in the changes of the local spinal cord microenvironment after injury. However, research on the role of m6A modification in SCI is still in the early stages. This review discusses the latest progress in the dynamic regulation of m6A modification, including methyltransferases ("writers"), demethylases ("erasers") and m6A -binding proteins ("readers"). And then analyses the pathological mechanism relationship between m6A and the microenvironment after SCI. The biological processes involved included cell death, axon regeneration, and scar formation, which provides new insight for future research on the role of m6A modification in SCI and the clinical transformation of strategies for promoting recovery of spinal cord function.

Explainable Intelligent Fault Diagnosis for Nonlinear Dynamic Systems: From Unsupervised to Supervised Learning.

The increased complexity and intelligence of automation systems require the development of intelligent fault diagnosis (IFD) methodologies. By relying on the concept of a suspected space, this study develops explainable data-driven IFD approaches for nonlinear dynamic systems. More specifically, we parameterize nonlinear systems through a generalized kernel representation for system modeling and the associated fault diagnosis. An important result obtained is a unified form of kernel representations, applicable to both unsupervised and supervised learning. More importantly, through a rigorous theoretical analysis, we discover the existence of a bridge (i.e., a bijective mapping) between some supervised and unsupervised learning-based entities. Notably, the designed IFD approaches achieve the same performance with the use of this bridge. In order to have a better understanding of the results obtained, both unsupervised and supervised neural networks are chosen as the learning tools to identify the generalized kernel representations and design the IFD schemes; an invertible neural network is then employed to build the bridge between them. This article is a perspective article, whose contribution lies in proposing and formalizing the fundamental concepts for explainable intelligent learning methods, contributing to system modeling and data-driven IFD designs for nonlinear dynamic systems.

Event-triggered integral reinforcement learning for nonzero-sum games with asymmetric input saturation.

In this paper, an event-triggered integral reinforcement learning (IRL) algorithm is developed for the nonzero-sum game problem with asymmetric input saturation. First, for each player, a novel non-quadratic value function with a discount factor is designed, and the coupled Hamilton-Jacobi equation that does not require a complete knowledge of the game is derived by using the idea of IRL. Second, the execution of each player is based on the event-triggered mechanism. In the implementation, an adaptive dynamic programming based learning scheme using a single critic neural network (NN) is developed. Experience replay technique is introduced into the classical gradient descent method to tune the weights of the critic NN. The stability of the system and the elimination of Zeno behavior are proved. Finally, simulation experiments verify the effectiveness of the event-triggered IRL algorithm.

Memristive LIF Spiking Neuron Model and Its Application in Morse Code.

The leaky integrate-and-fire (LIF) spiking model can successively mimic the firing patterns and information propagation of a biological neuron. It has been applied in neural networks, cognitive computing, and brain-inspired computing. Due to the resistance variability and the natural storage capacity of the memristor, the LIF spiking model with a memristor (MLIF) is presented in this article to simulate the function and working mode of neurons in biological systems. First, the comparison between the MLIF spiking model and the LIF spiking model is conducted. Second, it is experimentally shown that a single memristor could mimic the function of the integration and filtering of the dendrite and emulate the function of the integration and firing of the soma. Finally, the feasibility of the proposed MLIF spiking model is verified by the generation and recognition of Morse code. The experimental results indicate that the presented MLIF model efficiently performs good biological frequency adaptation, high firing frequency, and rich spiking patterns. A memristor can be used as the dendrite and the soma, and the MLIF spiking model can emulate the axon. The constructed single neuron can efficiently complete the generation and propagation of firing patterns.