A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems.

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Mixed-integer quadratic programming approach for noninvasive estimation of respiratory effort profile during pressure support ventilation.

Information about respiratory mechanics such as resistance, elastance, and muscular pressure is important to mitigate ventilator-induced lung injury. Particularly during pressure support ventilation, the available options to quantify breathing effort and calculate respiratory system mechanics are often invasive or complex. We herein propose a robust and flexible estimation of respiratory effort better than current methods. We developed a method for non-invasively estimating breathing effort using only flow and pressure signals. Mixed-integer quadratic programming (MIQP) was employed, and the binary variables were the switching moments of the respiratory effort waveform. Mathematical constraints, based on ventilation physiology, were set for some variables to restrict feasible solutions. Simulated and patient data were used to verify our method, and the results were compared to an established estimation methodology. Our algorithm successfully estimated the respiratory effort, resistance, and elastance of the respiratory system, resulting in more robust performance and faster solver times than a previously proposed algorithm that used quadratic programming (QP) techniques. In a numerical simulation benchmark, the worst-case errors for resistance and elastance were 25% and 23% for QP versus <0.1% and <0.1% for MIQP, whose solver times were 4.7 s and 0.5 s, respectively. This approach can estimate several breathing effort profiles and identify the respiratory system's mechanical properties in invasively ventilated critically ill patients.