A Review of Safe Reinforcement Learning: Methods, Theories and Applications.

Reinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. Firstly, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as "2H3W". Secondly, we analyze the algorithm and theory progress from the perspectives of answering the "2H3W" problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing major safe RL algorithms at the link.

Lateral flexion of a compliant spine improves motor performance in a bioinspired mouse robot.

A flexible spine is critical to the motion capability of most animals and plays a pivotal role in their agility. Although state-of-the-art legged robots have already achieved very dynamic and agile movement solely relying on their legs, they still exhibit the type of stiff movement that compromises movement efficiency. The integration of a flexible spine thus appears to be a promising approach to improve their agility, especially for small and underactuated quadruped robots that are underpowered because of size limitations. Here, we show that the lateral flexion of a compliant spine can promote both walking speed and maneuver agility for a neurorobotic mouse (NeRmo). We present NeRmo as a biomimetic robotic mouse that mimics the morphology of biological mice and their muscle-tendon actuation system. First, by leveraging the lateral flexion of the compliant spine, NeRmo can greatly increase its static stability in an initially unstable configuration by adjusting its posture. Second, the lateral flexion of the spine can also effectively extend the stride length of a gait and therefore improve the walking speeds of NeRmo. Finally, NeRmo shows agile maneuvers that require both a small turning radius and fast walking speed with the help of the spine. These results advance our understanding of spine-based quadruped locomotion skills and highlight promising design concepts to develop more agile legged robots.

Indoor air pollutants and respiratory symptoms among residents of an informal urban settlement in Uganda: A cross-sectional study.

BACKGROUND: Indoor air pollutants (IAP) and household conditions such as dampness, crowding and chemical exposures have been associated with acute and chronic respiratory infections. In Uganda, literature on the effects of IAP on respiratory outcomes in informal settlements is limited. METHODS: We describe the baseline household characteristics of 284 adults and their children in an informal settlement in Uganda from April to May 2022. We monitored same-day indoor concentrations of particulate matter PM2.5, PM10, Carbon monoxide (CO), relative humidity %, and temperature from 9 am to 2 pm and interviewed caregivers/mothers about their respiratory symptoms and those of their children in the previous 30 days. We employed robust Poisson regressions to evaluate the associations between indoor air indicators and respiratory health symptoms. RESULTS: Approximately 94.7% of households primarily used biomass fuels and 32.7% cooked from inside their dwelling rooms. The median PM2.5, PM10 and CO levels were 49.5 (Interquartile range (IQR) = 31.1,86.2) µg/m3, 73.6 (IQR = 47.3,130.5) µg/m3 and 7.70 (IQR = 4.1,12.5) ppm respectively. Among adults, a 10 unit increase in PM2.5 was associated with cough (Prevalence Ratio (PR) = 3.75, 95%CI 1.15-1.55). Dwelling unit dampness was associated with phlegm (PR = 2.53, 95%CI = 1.39-4.61) and shortness of breath (PR = 1.78, 95% CI 1.23-2.54) while cooking from outside the house was protective against shortness of breath (PR = 0.62, 95% CI = 0.44-0.87). In children, dampness was associated with phlegm (PR = 13.87, 95% CI 3.16-60.91) and cough (PR = 1.62, 95% CI 1.12-2.34) while indoor residual spraying was associated with phlegm (PR = 3.36, 95%CI 1.71-6.61). CONCLUSION: Poor indoor air conditions were associated with respiratory symptoms in adults and children. Efforts to address indoor air pollution should be made to protect adults and children from adverse health effects.

Association between recent COVID-19 diagnosis, depression and anxiety symptoms among slum residents in Kampala, Uganda.

BACKGROUND: Despite the known link between poor living conditions and mental health, there has been little research on the mental health of slum dwellers worldwide. Although the Coronavirus disease 2019 (COVID-19) pandemic has led to an increase in mental health issues, little focus has been given to the impact on slum dwellers. The study aimed to investigate the association between recent COVID-19 diagnosis and the risk of depression and anxiety symptoms among people living in an urban slum in Uganda. METHODS: A cross-sectional study was conducted among 284 adults (at least 18 years of age) in a slum settlement in Kampala, Uganda between April and May 2022. We assessed depression symptoms and anxiety using validated Patient Health Questionnaire (PHQ-9) and Generalized Anxiety Disorder assessment tool (GAD-7) questionnaires respectively. We collected data on sociodemographic characteristics, and self-reported recent COVID-19 diagnosis (in the previous 30 days). Using a modified Poisson regression, adjusted for age, sex, gender and household income, we separately provided prevalence ratios and 95% confidence intervals for the associations between recent COVID-19 diagnosis and depressive and anxiety symptoms. RESULTS: Overall, 33.8% and 13.4% of the participants met the depression and generalized anxiety screening criteria respectively and 11.3% were reportedly diagnosed with COVID-19 in the previous 30 days. People with recent COVID-19 diagnosis were more likely to be depressed (53.1%) than those with no recent diagnosis (31.4%) (p<0.001). Participants who were recently diagnosed with COVID-19 reported higher prevalence of anxiety (34.4%) compared to those with no recent diagnosis of COVID-19 (10.7%) (p = 0.014). After adjusting for confounding, recent diagnosis with COVID-19 was associated with depression (PR = 1.60, 95% CI 1.09-2.34) and anxiety (PR = 2.83, 95% CI 1.50-5.31). CONCLUSION: This study suggests an increased risk of depressive symptoms and GAD in adults following a COVID-19 diagnosis. We recommend additional mental health support for recently diagnosed persons. The long-term of COVID-19 on mental health effects also need to be investigated.

Patient safety in prisons: a multi-method analysis of reported incidents in England.

OBJECTIVES: Prisoners use healthcare services three times more frequently than the general population with poorer health outcomes. Their distinct healthcare needs often pose challenges to safe healthcare provision. This study aimed to characterise patient safety incidents reported in prisons to guide practice improvement and identify health policy priorities.Design: We carried out an exploratory multi-method analysis of anonymised safety incidents from prisons. SETTING: Safety incidents had been reported to the National Reporting and Learning System by prisons in England between April 2018 and March 2019. PARTICIPANTS: Reports were reviewed to identify any unintended or unexpected incident(s) which could have, or did, lead to harm for prisoners receiving healthcare. MAIN OUTCOME MEASURES: Free-text descriptions were examined to identify the type and nature of safety incidents, their outcomes and harm severity. Analysis was contextualised with subject experts through structured workshops to explain relationships between the most common incidents and contributory factors. RESULTS: Of 4112 reports, the most frequently observed incidents were medication-related (n = 1167, 33%), specifically whilst administering medications (n = 626, 54%). Next, were access-related (n = 559,15%), inclusive of delays in patients accessing healthcare professionals (n = 236, 42%) and managing medical appointments (n = 171, 31%). The workshops contextualised incidents involving contributing factors (n = 1529, 28%) into three key themes, namely healthcare access, continuity of care and the balance between prison and healthcare priorities. CONCLUSIONS: This study highlights the importance of improving medication safety and access to healthcare services for prisoners. We recommend staffing level reviews to ensure healthcare appointments are attended, and to review procedures for handling missed appointments, communication during patient transfers and medication prescribing.

Evaluation of a complex intervention for prisoners with common mental health problems, near to and after release: the Engager randomised controlled trial.

BACKGROUND: Many male prisoners have significant mental health problems, including anxiety and depression. High proportions struggle with homelessness and substance misuse. AIMS: This study aims to evaluate whether the Engager intervention improves mental health outcomes following release. METHOD: The design is a parallel randomised superiority trial that was conducted in the North West and South West of England (ISRCTN11707331). Men serving a prison sentence of 2 years or less were individually allocated 1:1 to either the intervention (Engager plus usual care) or usual care alone. Engager included psychological and practical support in prison, on release and for 3-5 months in the community. The primary outcome was the Clinical Outcomes in Routine Evaluation Outcome Measure (CORE-OM), 6 months after release. Primary analysis compared groups based on intention-to-treat (ITT). RESULTS: In total, 280 men were randomised out of the 396 who were potentially eligible and agreed to participate; 105 did not meet the mental health inclusion criteria. There was no mean difference in the ITT complete case analysis between groups (92 in each arm) for change in the CORE-OM score (1.1, 95% CI -1.1 to 3.2, P = 0.325) or secondary analyses. There were no consistent clinically significant between-group differences for secondary outcomes. Full delivery was not achieved, with 77% (108/140) receiving community-based contact. CONCLUSIONS: Engager is the first trial of a collaborative care intervention adapted for prison leavers. The intervention was not shown to be effective using standard outcome measures. Further testing of different support strategies for prison with mental health problems is needed.

Effect of community-level intervention on antenatal care attendance: a quasi-experimental study among postpartum women in Eastern Uganda.

BACKGROUND: Early Initiation of antenatal care (ANC) and at least four visits during pregnancy allow screening and support for a healthy lifestyle and self-care during pregnancy however, community-directed interventions to improve access to these services are rarely explored. OBJECTIVE: To assess the effect of community health worker (CHW) involvement on utilisation of antenatal services during pregnancy in resource-constrained rural settings in Uganda. METHODS: We conducted a quasi-experimental evaluation study among mothers from Eastern Uganda. We used Difference in Differences (DiD) analysis to assess the effect of CHW intervention on ANC attendance. Components of the intervention included community dialogues and empowering CHWs to educate pregnant women about using maternal health services. The primary endpoints were early initiation of ANC and completion of at least 4 ANC visits. RESULTS: Overall, the intervention significantly improved attendance of ≥ 4 ANC visits (DiD = 5.5%). The increase was significant in both intervention and comparison areas (46.2-64.4% vs. 54.1-66.8%, respectively), with slightly greater gains in the intervention area. Other elements that predicted ≥4 ANC attendance besides the intervention were post-primary education (PR1.14, 95%CI 1.02-1.30), higher wealth quintile (PR1.17, 95%CI 1.06-1.30), and early initiation of ANC (PR1.58, 95%CI 1.49-1.68). The intervention did not significantly improve early initiation of ANC (DiD =-1.3%). Instead, early initiation of ANC was associated with higher husband education (PR1.19,95%CI 1.02-1.39), larger household size (PR = 0.81, 95%CI 0.70-0.95), and higher wealth index (PR1.19,95%CI 1.03-1.37). CONCLUSIONS: The CHW intervention improved attendance of at least 4 ANC visits but not early initiation of ANC. There is need to promote CHW-led health education to increase attendance at 4+ ANC visits, but other approaches to promote early initiation are urgently required.

Complicated Diverticulitis With Associated Small Bowel Obstruction: A Case Report.

Diverticulitis is a common disease of the colon with a broad spectrum of presentations. This case outlines a less commonly seen presentation of delayed complicated diverticulitis with abscess formation and associated small bowel obstruction. This report outlines the case of a 58-year-old obese male who initially presented to the emergency department with uncomplicated diverticulitis, was discharged home, and returned two days later with worsening symptoms. A repeat computed tomography (CT) scan demonstrated abscess formation with an associated small bowel obstruction. He underwent midline exploratory laparotomy with adhesiolysis of the portion of obstructed small bowel and resection of the diseased segment with primary anastomosis and diverting loop ileostomy. He had an ileostomy reversal at a later date. Small bowel obstruction is a less observed complication of diverticulitis. This report serves to educate on the presentation of these patients as well as to discuss the potential surgical management of these patients. If this patient had presented without the small bowel obstruction, he may have just needed percutaneous drainage.

Toward Cognitive Navigation: Design and Implementation of a Biologically Inspired Head Direction Cell Network.

As a vital cognitive function of animals, the navigation skill is first built on the accurate perception of the directional heading in the environment. Head direction cells (HDCs), found in the limbic system of animals, are proven to play an important role in identifying the directional heading allocentrically in the horizontal plane, independent of the animal's location and the ambient conditions of the environment. However, practical HDC models that can be implemented in robotic applications are rarely investigated, especially those that are biologically plausible and yet applicable to the real world. In this article, we propose a computational HDC network that is consistent with several neurophysiological findings concerning biological HDCs and then implement it in robotic navigation tasks. The HDC network keeps a representation of the directional heading only relying on the angular velocity as an input. We examine the proposed HDC model in extensive simulations and real-world experiments and demonstrate its excellent performance in terms of accuracy and real-time capability.

Toward robust and scalable deep spiking reinforcement learning.

Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.

The biophysical, molecular, and anatomical landscape of pigeon CRY4: A candidate light-based quantal magnetosensor.

The biophysical and molecular mechanisms that enable animals to detect magnetic fields are unknown. It has been proposed that birds have a light-dependent magnetic compass that relies on the formation of radical pairs within cryptochrome molecules. Using spectroscopic methods, we show that pigeon cryptochrome clCRY4 is photoreduced efficiently and forms long-lived spin-correlated radical pairs via a tetrad of tryptophan residues. We report that clCRY4 is broadly and stably expressed within the retina but enriched at synapses in the outer plexiform layer in a repetitive manner. A proteomic survey for retinal-specific clCRY4 interactors identified molecules that are involved in receptor signaling, including glutamate receptor-interacting protein 2, which colocalizes with clCRY4. Our data support a model whereby clCRY4 acts as an ultraviolet-blue photoreceptor and/or a light-dependent magnetosensor by modulating glutamatergic synapses between horizontal cells and cones.

Multiple adverse outcomes following first discharge from inpatient psychiatric care: a national cohort study.

BACKGROUND: Discharged psychiatric inpatients are at elevated risk of serious adverse outcomes, but no previous study has comprehensively examined an array of multiple risks in a single cohort. METHODS: We used data from the Danish Civil Registration System to delineate a cohort of all individuals born in Denmark in 1967-2000, who were alive and residing in Denmark on their 15th birthday, and who had been discharged from their first inpatient psychiatric episode at age 15 years or older. Each individual in the discharged cohort was matched on age and sex with 25 comparators without a history of psychiatric admission. Data linked to each individual were also obtained from the Psychiatric Central Research Register, Register of Causes of Death, National Patient Register, and the National Crime Register. We used survival analysis techniques to estimate absolute and relative risks of all-cause mortality, suicide, accidental death, homicide victimisation, homicide perpetration, non-fatal self-harm, violent criminality, and hospitalisation following violence, until Dec 31, 2015. FINDINGS: We included 62 922 individuals in the discharged cohort, and 1 573 050 matched comparators. Risks for each of all eight outcomes examined were markedly elevated in the discharged cohort relative to the comparators. Within 10 years of first discharge, the cumulative incidence of death, self-harm, committing a violent crime, or hospitalisation due to interpersonal violence was 32·0% (95% CI 31·6-32·5) in the discharged cohort (37·1% [36·5-37·8] in men and 27·2% [26·7-27·8] in women). Absolute risk of at least one adverse outcome occurring within this timeframe were highest in people diagnosed with a psychoactive substance use disorder at first discharge (cumulative incidence 49·4% [48·4-50·4]), and lowest in those diagnosed with a mood disorder (24·4% [23·6-25·2]). For suicide and non-fatal self-harm, risks were especially high during the first 3 months post-discharge, whereas risks for accidental death, violent criminality, and hospitalisation due to violence were more constant throughout the 10-year follow-up. INTERPRETATION: People discharged from inpatient psychiatric care are at higher risk than the rest of the population for a range of serious fatal and non-fatal adverse outcomes. Improved inter-agency liaison, intensive follow-up immediately after discharge, and longer-term social support are indicated. FUNDING: Medical Research Council, European Research Council, and Wellcome Trust.

Suicide Methods and Specific Types of Accidental Death and Fatal Poisoning Among Discharged Psychiatric Patients: A National Cohort Study.

OBJECTIVE: Persons discharged from inpatient psychiatric units are at greatly elevated risk of dying unnaturally. We conducted a comprehensive examination of specific causes of unnatural death post-discharge in a national register-based cohort. METHOD: A cohort of 1,683,645 Danish residents born 1967-1996 was followed from their 15th birthday until death, emigration, or December 31, 2011, whichever came first. Survival analysis techniques were used to estimate incidence rate ratios (IRRs) comparing risk for persons with and without psychiatric admission history in relation to (a) suicide method, (b) accidental death type, (c) fatal poisoning type, and (d) homicide. RESULTS: More than half (52.5%, n = 711) of all unnatural deaths post-discharge were fatal poisonings, compared with less than a fifth (17.0%, n = 1,012) among persons in the general population not admitted. Just 6.8% (n = 92) of all unnatural deaths post-discharge were due to transport accidents-the most common unnatural death type in the general population (53.4%, n = 3,184). Suicide risk was 32 times higher among discharged patients (IRR 32.3; 95% CI, 29.2-35.8) and was even higher during the first year post-discharge (IRR 70.4; 95% CI, 59.7-83.0). Among the suicide methods examined, relative risk values were significantly larger for intentional self-poisoning (IRR 40.8; 95% CI, 33.9-49.1) than for "violent" suicide methods (IRR 29.4; 95% CI, 26.1-33.2). The greatest relative risk observed was for fatal poisoning (irrespective of intent) by psychotropic medication (IRR 93.7; 95% CI, 62.5-140.5). The highest post-discharge mortality rate was for accidental self-poisoning among persons diagnosed with a psychoactive substance abuse disorder: 290.1 per 100,000 person-years. CONCLUSIONS: Closer liaison between inpatient services and community care, more effective early treatment for comorbid substance abuse, enhanced psychosocial assessment following self-harm, and tighter medication surveillance could decrease risk of unnatural death post-discharge.

Hebbian learning for online prediction, neural recall and classical conditioning of anthropomimetic robot arm motions.

Classical conditioning plays a vital role in learning in every mammal. It is based on unsupervised neural learning embodied in a physical body that is in continuous interaction with the environment. Embedding the hierarchical temporal memory (HTM) in the closed loop of the sensorimotor space of a Myorobotics tendon-driven robotic arm we demonstrate learning, prediction and control of biomimetic body motions. Experiments finally lead to conditioned reactions in natural interaction with a human partner. The HTM is able to learn arm movements generated by interaction with a human partner in a short time. It predicts future positions in different time scales up to seconds in advance. Closing the loop we utilize HTM predictions for motor control. Hereby learned motions are recalled from synaptic connections proactively continuing motion execution. Association, prediction and control are required by the HTM for conditioning according to Pavlov: neutral stimuli get associated with motions, and after learning sensor impulses can trigger single arm lifting motions. Hereby, both the motions and the stimuli are learned from the environment and get associated efficiently. We can demonstrate high biological plausibility as for example even input variations result in similar variations in the action output. The robotic system consisting of biologically-derived hardware and software components utilizes only unsupervised Hebbian learning to act autonomously. Learning is executed in real time, can handle natural variations of human motions and takes morphologically plausible sensor input into account. The setup is fully scalable due to its modularity. Hereby, novel applications for the HTM are opened: it can be used in musculoskeletal robot control scenarios and robots being able to interactively learn from human partners and the environment.

Premature Mortality Among Patients Recently Discharged From Their First Inpatient Psychiatric Treatment.

Importance: Nationwide cohorts provide sufficient statistical power for examining premature, cause-specific mortality in patients recently discharged from inpatient psychiatric services. Objective: To investigate premature mortality in a nationwide cohort of patients recently discharged from inpatient psychiatric treatment at ages 15 to 44 years. Design, Setting, and Participants: This single-cohort design included all persons born in Denmark (N = 1 683 385) between January 1, 1967, and December 31, 1996. Exactly 48 599 of these Danish residents were discharged from an inpatient psychiatric unit or ward on or after their 15th birthday, which took place during this study's observation period from January 1, 1982, through December 31, 2011. This group of patients was followed up beginning on their 15th birthday until their death, emigration, or December 31, 2011, whichever came first. Individuals discharged from inpatient psychiatric care at least once before their 15th birthday (n = 5882) were excluded from the study. All data were obtained from the Danish Civil Registration System, Psychiatric Central Research Register, and Register of Causes of Death. Data analysis took place between February 1, 2016, and December 10, 2016. Main Outcomes and Measures: Incidence rates and incidence rate ratios (IRRs) for all-cause mortality and for an array of unnatural and natural causes of death among patients recently discharged from an inpatient psychiatric unit vs persons not admitted to a psychiatric facility. Primary analysis considered risk within the year of first discharge. Results: Of the 48 599 discharged patients who were included in the study, 25 006 (51.4%) were female, 35 660 (73.4%) were aged 15 to 29 years, and 33 995 (70.0%) had a length of stay of 30 days or less. Compared with persons not admitted, patients discharged had an elevated risk for all-cause mortality within 1 year (IRR, 16.2; 95% CI, 14.5-18.0). The relative risk for unnatural death (IRR, 25.0; 95% CI, 22.0-28.4) was much higher than for natural death (IRR, 8.6; 95% CI, 7.0-10.7). The highest IRR found was for suicide at 66.9 (95% CI, 56.4-79.4), followed by alcohol-related death at 42.0 (95% CI, 26.6-66.1). Among the psychiatric diagnostic categories assessed, psychoactive substance abuse conferred the highest risk for all-cause mortality (IRR, 24.8; 95% CI, 21.0-29.4). Across the array of cause-specific outcomes examined, risk of premature death during the first year after discharge was markedly higher than the risk of death beyond the first year of discharge. Conclusions and Relevance: Clinicians may help protect patients after discharge by serving as a liaison between primary and secondary health services to ensure they are receiving holistic care. Early intervention programs for drug and alcohol misuse could substantially decrease the greatly elevated mortality risk among these patients.

Neuromorphic implementations of neurobiological learning algorithms for spiking neural networks.

The application of biologically inspired methods in design and control has a long tradition in robotics. Unlike previous approaches in this direction, the emerging field of neurorobotics not only mimics biological mechanisms at a relatively high level of abstraction but employs highly realistic simulations of actual biological nervous systems. Even today, carrying out these simulations efficiently at appropriate timescales is challenging. Neuromorphic chip designs specially tailored to this task therefore offer an interesting perspective for neurorobotics. Unlike Von Neumann CPUs, these chips cannot be simply programmed with a standard programming language. Like real brains, their functionality is determined by the structure of neural connectivity and synaptic efficacies. Enabling higher cognitive functions for neurorobotics consequently requires the application of neurobiological learning algorithms to adjust synaptic weights in a biologically plausible way. In this paper, we therefore investigate how to program neuromorphic chips by means of learning. First, we provide an overview over selected neuromorphic chip designs and analyze them in terms of neural computation, communication systems and software infrastructure. On the theoretical side, we review neurobiological learning techniques. Based on this overview, we then examine on-die implementations of these learning algorithms on the considered neuromorphic chips. A final discussion puts the findings of this work into context and highlights how neuromorphic hardware can potentially advance the field of autonomous robot systems. The paper thus gives an in-depth overview of neuromorphic implementations of basic mechanisms of synaptic plasticity which are required to realize advanced cognitive capabilities with spiking neural networks.