Data-Driven H∞ Optimal Output Feedback Control for Linear Discrete-Time Systems Based on Off-Policy Q-Learning.

This article develops two novel output feedback (OPFB) Q -learning algorithms, on-policy Q -learning and off-policy Q -learning, to solve H∞ static OPFB control problem of linear discrete-time (DT) systems. The primary contribution of the proposed algorithms lies in a newly developed OPFB control algorithm form for completely unknown systems. Under the premise of satisfying disturbance attenuation conditions, the conditions for the existence of the optimal OPFB solution are given. The convergence of the proposed Q -learning methods, and the difference and equivalence of two algorithms are rigorously proven. Moreover, considering the effects brought by probing noise for the persistence of excitation (PE), the proposed off-policy Q -learning method has the advantage of being immune to probing noise and avoiding biasedness of solution. Simulation results are presented to verify the effectiveness of the proposed approaches.

Off-Policy Reinforcement Learning for Tracking in Continuous-Time Systems on Two Time Scales.

This article applies a singular perturbation theory to solve an optimal linear quadratic tracker problem for a continuous-time two-time-scale process. Previously, singular perturbation was applied for system regulation. It is shown that the two-time-scale tracking problem can be separated into a linear-quadratic tracker (LQT) problem for the slow system and a linear-quadratic regulator (LQR) problem for the fast system. We prove that the solutions to these two reduced-order control problems can approximate the LQT solution of the original control problem. The reduced-order slow LQT and fast LQR control problems are solved by off-policy integral reinforcement learning (IRL) using only measured data from the system. To test the effectiveness of the proposed method, we use an industrial thickening process as a simulation example and compare our method to a method with the known system model and a method without time-scale separation.

Adaptive Optimal Control for Stochastic Multiplayer Differential Games Using On-Policy and Off-Policy Reinforcement Learning.

Control-theoretic differential games have been used to solve optimal control problems in multiplayer systems. Most existing studies on differential games either assume deterministic dynamics or dynamics corrupted with additive noise. In realistic environments, multidimensional environmental uncertainties often modulate system dynamics in a more complicated fashion. In this article, we study stochastic multiplayer differential games, where the players' dynamics are modulated by randomly time-varying parameters. We first formulate two differential games for systems of general uncertain linear dynamics, including the two-player zero-sum and multiplayer nonzero-sum games. We then show that optimal control policies, which constitute the Nash equilibrium solutions, can be derived from the corresponding Hamiltonian functions. Stability is proven using the Lyapunov type of analysis. In order to solve the stochastic differential games online, we integrate reinforcement learning (RL) and an effective uncertainty sampling method called the multivariate probabilistic collocation method (MPCM). Two learning algorithms, including the on-policy integral RL (IRL) and off-policy IRL, are designed for the formulated games, respectively. We show that the proposed learning algorithms can effectively find the Nash equilibrium solutions for the stochastic multiplayer differential games.

[Epidemiology and new predictors of atrial fibrillation after coronary surgery]. / Epidemiología y nuevos predictores de la fibrilación auricular tras cirugía coronaria.

INTRODUCTION AND OBJECTIVES: Postoperative atrial fibrillation (PAF) is a frequent complication of coronary artery bypass grafting (CABG). Our aims were to study its epidemiology and to identify predictors in everyday clinical practice, while taking into account statin use, extracorporeal circulation, and new biomarkers of inflammation and ventricular stress. METHODS: The study included 102 consecutive patients (65 [9] years, 72% male) who were undergoing CABG. Blood samples were taken the day before surgery to determine baseline levels of C-reactive protein (CRP) and N-terminal probrain natriuretic peptide (NT-proBNP). Details of baseline clinical characteristics, preoperative treatment and surgery were recorded. The end-point was PAF at 30 days. RESULTS: The incidence of PAF was 23% (n=23; 3.2 [2.9] days, range 1-15 days). Its appearance was associated with a longer stay in the intensive care unit (+ 1 day; P=.019), but not with an increased total hospital stay (P=.213). Among patients with PAF, 4.3% had an embolism and 8.6% remained in atrial fibrillation at discharge. Moreover, PAF was associated with a longer duration of ischemia (28.5 [22.3] vs 18.0 [27.9]; P=.045) and a lower statin pretreatment rate (39% vs 66%; P=.022). Multivariate analysis showed that the only factor associated with a higher risk of PAF was the absence of statin pretreatment (odds ratio = 4.31, 95% confidence interval 1.33-13.88; P=.015). There was no association between either extracorporeal circulation or the baseline CRP or NT-proBNP level and an increased risk of PAF. CONCLUSION: In everyday clinical practice, PAF is a frequent complication. Statin pretreatment could have a protective effect against its appearance.

Epidemiología y nuevos predictores de la fibrilación auricular tras cirugía coronaria / Epidemiology and new predictors of atrial fibrillation after coronary surgery

Introducción y objetivos. La fibrilación auricular postoperatoria (FAP) es una complicación frecuente tras la cirugía de revascularización coronaria. Estudiamos su epidemiología y los predictores en la práctica actual, considerando el uso de estatinas, la circulación extracorpórea y los nuevos biomarcadores de inflamación y estrés ventricular. Métodos. Se estudió a 102 pacientes consecutivos (65 ± 9 años, 72% varones) en los que se realizó cirugía coronaria. El día previo se obtuvieron las muestras plasmáticas para medida de proteína C reactiva (PCR) y porción amino-terminal del propéptido natriurético cerebral (NT-proBNP), se recogieron las características clínicas basales y el tratamiento preoperatorio, posteriormente se registraron los datos quirúrgicos y se estudió la aparición de FAP a 30 días. Resultados. La incidencia de FAP fue del 23% (n = 23) (3,2 ± 2,9 días; intervalo, 1-15 días). Su aparición prolongó los cuidados intensivos (mediana + un día; p = 0,019) pero no la estancia hospitalaria total (p = 0,213). Entre los pacientes con FAP, los embolismos y la persistencia en FA en el momento del alta fueron del 4,3 y el 8,6%, respectivamente. La FAP se asoció con un mayor tiempo de isquemia (28,5 ± 22,30 frente a 18,0 ± 27,9 min; p = 0,045) y una menor tasa de estatinas preoperatorias (el 39 frente al 66%; p = 0,022). En el análisis multivariable, sólo la ausencia de estatinas conllevó un mayor riesgo de FAP (odds ratio [OR] = 4,31; intervalo de confianza [IC] del 95%, 1,33-13,88; p = 0,015). El uso de circulación extracorpórea (CEC) y los valores basales de PCR y NT-proBNP no se asociaron con un mayor riesgo. Conclusión. En la práctica actual, la FAP es una complicación frecuente y la administración de estatinas en el preoperatorio podría proteger frente a su aparición (AU)

Introduction and objectives. Postoperative atrial fibrillation (PAF) is a frequent complication of coronary artery bypass grafting (CABG). Our aims were to study its epidemiology and to identify predictors in everyday clinical practice, while taking into account statin use, extracorporeal circulation, and new biomarkers of inflammation and ventricular stress. Methods. The study included 102 consecutive patients (65 [9] years, 72% male) who were undergoing CABG. Blood samples were taken the day before surgery to determine baseline levels of C-reactive protein (CRP) and N-terminal probrain natriuretic peptide (NT-proBNP). Details of baseline clinical characteristics, preoperative treatment and surgery were recorded. The end-point was PAF at 30 days. Results. The incidence of PAF was 23% (n=23; 3.2 [2.9] days, range 1-15 days). Its appearance was associated with a longer stay in the intensive care unit (+ 1 day; P=.019), but not with an increased total hospital stay (P=.213). Among patients with PAF, 4.3% had an embolism and 8.6% remained in atrial fibrillation at discharge. Moreover, PAF was associated with a longer duration of ischemia (28.5 [22.3] vs 18.0 [27.9]; P=.045) and a lower statin pretreatment rate (39% vs 66%; P=.022). Multivariate analysis showed that the only factor associated with a higher risk of PAF was the absence of statin pretreatment (odds ratio = 4.31, 95% confidence interval 1.33­13.88; P=.015). There was no association between either extracorporeal circulation or the baseline CRP or NT-proBNP level and an increased risk of PAF. Conclusion. In everyday clinical practice, PAF is a frequent complication. Statin pretreatment could have a protective effect against its appearance (AU)