Machine Learning for RNA Design: LEARNA.

Machine learning algorithms, and in particular deep learning approaches, have recently garnered attention in the field of molecular biology due to remarkable results. In this chapter, we describe machine learning approaches specifically developed for the design of RNAs, with a focus on the learna_tools Python package, a collection of automated deep reinforcement learning algorithms for secondary structure-based RNA design. We explain the basic concepts of reinforcement learning and its extension, automated reinforcement learning, and outline how these concepts can be successfully applied to the design of RNAs. The chapter is structured to guide through the usage of the different programs with explicit examples, highlighting particular applications of the individual tools.

Energy-aware bio-inspired spiking reinforcement learning system architecture for real-time autonomous edge applications.

Mobile, low-cost, and energy-aware operation of Artificial Intelligence (AI) computations in smart circuits and autonomous robots will play an important role in the next industrial leap in intelligent automation and assistive devices. Neuromorphic hardware with spiking neural network (SNN) architecture utilizes insights from biological phenomena to offer encouraging solutions. Previous studies have proposed reinforcement learning (RL) models for SNN responses in the rat hippocampus to an environment where rewards depend on the context. The scale of these models matches the scope and capacity of small embedded systems in the framework of Internet-of-Bodies (IoB), autonomous sensor nodes, and other edge applications. Addressing energy-efficient artificial learning problems in such systems enables smart micro-systems with edge intelligence. A novel bio-inspired RL system architecture is presented in this work, leading to significant energy consumption benefits without foregoing real-time autonomous processing and accuracy requirements of the context-dependent task. The hardware architecture successfully models features analogous to synaptic tagging, changes in the exploration schemes, synapse saturation, and spatially localized task-based activation observed in the brain. The design has been synthesized, simulated, and tested on Intel MAX10 Field-Programmable Gate Array (FPGA). The problem-based bio-inspired approach to SNN edge architectural design results in 25X reduction in average power compared to the state-of-the-art for a test with real-time context learning and 30 trials. Furthermore, 940x lower energy consumption is achieved due to improvement in the execution time.

Effectiveness of a Digital Health Intervention Leveraging Reinforcement Learning: Results From the Diabetes and Mental Health Adaptive Notification Tracking and Evaluation (DIAMANTE) Randomized Clinical Trial.

BACKGROUND: Digital and mobile health interventions using personalization via reinforcement learning algorithms have the potential to reach large number of people to support physical activity and help manage diabetes and depression in daily life. OBJECTIVE: The Diabetes and Mental Health Adaptive Notification and Tracking Evaluation (DIAMANTE) study tested whether a digital physical activity intervention using personalized text messaging via reinforcement learning algorithms could increase step counts in a diverse, multilingual sample of people with diabetes and depression symptoms. METHODS: From January 2020 to June 2022, participants were recruited from 4 San Francisco, California-based public primary care clinics and through web-based platforms to participate in the 24-week randomized controlled trial. Eligibility criteria included English or Spanish language preference and a documented diagnosis of diabetes and elevated depression symptoms. The trial had 3 arms: a Control group receiving a weekly mood monitoring message, a Random messaging group receiving randomly selected feedback and motivational text messages daily, and an Adaptive messaging group receiving text messages selected by a reinforcement learning algorithm daily. Randomization was performed with a 1:1:1 allocation. The primary outcome, changes in daily step counts, was passively collected via a mobile app. The primary analysis assessed changes in daily step count using a linear mixed-effects model. An a priori subanalysis compared the primary step count outcome within recruitment samples. RESULTS: In total, 168 participants were analyzed, including those with 24% (40/168) Spanish language preference and 37.5% (63/168) from clinic-based recruitment. The results of the linear mixed-effects model indicated that participants in the Adaptive arm cumulatively gained an average of 3.6 steps each day (95% CI 2.45-4.78; P<.001) over the 24-week intervention (average of 608 total steps), whereas both the Control and Random arm participants had significantly decreased rates of change. Postintervention estimates suggest that participants in the Adaptive messaging arm showed a significant step count increase of 19% (606/3197; P<.001), in contrast to 1.6% (59/3698) and 3.9% (136/3480) step count increase in the Random and Control arms, respectively. Intervention effectiveness differences were observed between participants recruited from the San Francisco clinics and those recruited via web-based platforms, with the significant step count trend persisting across both samples for participants in the Adaptive group. CONCLUSIONS: Our study supports the use of reinforcement learning algorithms for personalizing text messaging interventions to increase physical activity in a diverse sample of people with diabetes and depression. It is the first to test this approach in a large, diverse, and multilingual sample. TRIAL REGISTRATION: ClinicalTrials.gov NCT03490253; https://clinicaltrials.gov/study/NCT03490253. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1136/bmjopen-2019-034723.

Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks.

Many important behavior changes are frictionful; they require individuals to expend effort over a long period with little immediate gratification. Here, an artificial intelligence (AI) agent can provide personalized interventions to help individuals stick to their goals. In these settings, the AI agent must personalize rapidly (before the individual disengages) and interpretably, to help us understand the behavioral interventions. In this paper, we introduce Behavior Model Reinforcement Learning (BMRL), a framework in which an AI agent intervenes on the parameters of a Markov Decision Process (MDP) belonging to a boundedly rational human agent. Our formulation of the human decision-maker as a planning agent allows us to attribute undesirable human policies (ones that do not lead to the goal) to their maladapted MDP parameters, such as an extremely low discount factor. Furthermore, we propose a class of tractable human models that captures fundamental behaviors in frictionful tasks. Introducing a notion of MDP equivalence specific to BMRL, we theoretically and empirically show that AI planning with our human models can lead to helpful policies on a wide range of more complex, ground-truth humans.

An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies.

Learning structures that effectively abstract decision policies is key to the flexibility of human intelligence. Previous work has shown that humans use hierarchically structured policies to efficiently navigate complex and dynamic environments. However, the computational processes that support the learning and construction of such policies remain insufficiently understood. To address this question, we tested 1026 human participants, who made over 1 million choices combined, in a decision-making task where they could learn, transfer, and recompose multiple sets of hierarchical policies. We propose a novel algorithmic account for the learning processes underlying observed human behavior. We show that humans rely on compressed policies over states in early learning, which gradually unfold into hierarchical representations via meta-learning and Bayesian inference. Our modeling evidence suggests that these hierarchical policies are structured in a temporally backward, rather than forward, fashion. Taken together, these algorithmic architectures characterize how the interplay between reinforcement learning, policy compression, meta-learning, and working memory supports structured decision-making and compositionality in a resource-rational way.

Deep Reinforcement Learning for personalized diagnostic decision pathways using Electronic Health Records: A comparative study on anemia and Systemic Lupus Erythematosus.

BACKGROUND: Clinical diagnoses are typically made by following a series of steps recommended by guidelines that are authored by colleges of experts. Accordingly, guidelines play a crucial role in rationalizing clinical decisions. However, they suffer from limitations, as they are designed to cover the majority of the population and often fail to account for patients with uncommon conditions. Moreover, their updates are long and expensive, making them unsuitable for emerging diseases and new medical practices. METHODS: Inspired by guidelines, we formulate the task of diagnosis as a sequential decision-making problem and study the use of Deep Reinforcement Learning (DRL) algorithms to learn the optimal sequence of actions to perform in order to obtain a correct diagnosis from Electronic Health Records (EHRs), which we name a diagnostic decision pathway. We apply DRL to synthetic yet realistic EHRs and develop two clinical use cases: Anemia diagnosis, where the decision pathways follow a decision tree schema, and Systemic Lupus Erythematosus (SLE) diagnosis, which follows a weighted criteria score. We particularly evaluate the robustness of our approaches to noise and missing data, as these frequently occur in EHRs. RESULTS: In both use cases, even with imperfect data, our best DRL algorithms exhibit competitive performance compared to traditional classifiers, with the added advantage of progressively generating a pathway to the suggested diagnosis, which can both guide and explain the decision-making process. CONCLUSION: DRL offers the opportunity to learn personalized decision pathways for diagnosis. Our two use cases illustrate the advantages of this approach: they generate step-by-step pathways that are explainable, and their performance is competitive when compared to state-of-the-art methods.

Therapeutic effects of DOX-loaded hydrogel MOF nanocarriers on triple negative breast cancer and derivative design via reinforcement learning.

Triple negative breast cancer (TNBC) is one of the most difficult of all types of breast cancer to treat. TNBC is characterized by the absence of estrogen receptor, progesterone receptor, and human epidermal growth factor receptor 2. The development of effective drugs can help to alleviate the suffering of patients. The novel nickel(II)-based coordination polymer (CP), [Ni2(HL)(O)(H2O)3·H2O] (1) (where H4L=[1,1':2',1''-triphenyl]-3,3'',4',5'-tetracarboxylic acid), was synthesized via solvothermal reaction in this study. The overall structure of CP1 was fully identified by SXRD, Fourier transform infrared spectroscopy and elemental analysis. Using advanced chemical synthesis, we developed Hyaluronic Acid/Carboxymethyl Chitosan-CP1@Doxorubicin (HA/CMCS-CP1@DOX), a nanocarrier system encapsulating doxorubicin (DOX), which was thoroughly characterized using Scanning Electron Microscopy (SEM), Fourier Transform Infrared Spectroscopy (FTIR), and Thermogravimetric Analysis (TGA). These analyses confirmed the integration of doxorubicin and provided data on the nanocarriers' stability and structure. In vitro experiments showed that this system significantly downregulated Tissue Inhibitor of Metalloproteinases-1 (TIMP-1) in triple-negative breast cancer cells and inhibited their proliferation. Molecular docking simulations revealed the biological effects of CP1 are derived from its carboxyl groups. Using reinforcement learning, multiple new derivatives were generated from this compound, displaying excellent biological activities. These findings highlight the potential clinical applications and the innovative capacity of this nanocarrier system in drug development.

Estimating best nanomaterial for energy harvesting through reinforcement learning DQN coupled with fuzzy PROMETHEE under road-based conditions.

Energy harvesters based on nanomaterials are getting more and more popular, but on their way to commercial availability, some crucial issues still need to be solved. The objective of the study is to select an appropriate nanomaterial. Using features of the Reinforcement Deep Q-Network (DQN) in conjunction with Fuzzy PROMETHEE, the proposed model, we present in this work a hybrid fuzzy approach to selecting appropriate materials for a vehicle-environmental-hazardous substance (EHS) combination that operates in roadways and under traffic conditions. The DQN is able to accumulate useful experience of operating in a dynamic traffic environment, accordingly selecting materials that deliver the highest energy output but at the same time bring consideration to factors such as durability, cost, and environmental impact. Fuzzy PROMETHEE allows the participation of human experts during the decision-making process, going beyond the quantitative data typically learned by DQN through the inclusion of qualitative preferences. Instead, this hybrid method unites the strength of individual approaches, as a result providing highly resistant and adjustable material selection to real EHS. The result of the study pointed out materials that can give high energy efficiency with reference to years of service, price, and environmental effects. The proposed model provides 95% accuracy with a computational efficiency of 300 s, and the application of hypothesis and practical testing on the chosen materials showed the high efficiency of the selected materials to harvest energy under fluctuating traffic conditions and proved the concept of a hybrid approach in True Vehicle Environmental High-risk Substance scenarios.

Dopamine transients encode reward prediction errors independent of learning rates.

Biological accounts of reinforcement learning posit that dopamine encodes reward prediction errors (RPEs), which are multiplied by a learning rate to update state or action values. These values are thought to be represented by corticostriatal synaptic weights, which are updated by dopamine-dependent plasticity. This suggests that dopamine release reflects the product of the learning rate and RPE. Here, we characterize dopamine encoding of learning rates in the nucleus accumbens core (NAcc) in a volatile environment. Using a task with semi-observable states offering different rewards, we find that rats adjust how quickly they initiate trials across states using RPEs. Computational modeling and behavioral analyses show that learning rates are higher following state transitions and scale with trial-by-trial changes in beliefs about hidden states, approximating normative Bayesian strategies. Notably, dopamine release in the NAcc encodes RPEs independent of learning rates, suggesting that dopamine-independent mechanisms instantiate dynamic learning rates.

Acquiring musculoskeletal skills with curriculum-based reinforcement learning.

Efficient musculoskeletal simulators and powerful learning algorithms provide computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the inaugural NeurIPS MyoChallenge leverages an approach mirroring human skill learning. Using a novel curriculum learning approach, we trained a recurrent neural network to control a realistic model of the human hand with 39 muscles to rotate two Baoding balls in the palm of the hand. In agreement with data from human subjects, the policy uncovers a small number of kinematic synergies, even though it is not explicitly biased toward low-dimensional solutions. However, selectively inactivating parts of the control signal, we found that more dimensions contribute to the task performance than suggested by traditional synergy analysis. Overall, our work illustrates the emerging possibilities at the interface of musculoskeletal physics engines, reinforcement learning, and neuroscience to advance our understanding of biological motor control.

The flow of reward information through neuronal ensembles in the accumbens.

The nucleus accumbens shell (NAcSh) integrates reward information through diverse and specialized neuronal ensembles, influencing decision-making. By training rats in a probabilistic choice task and recording NAcSh neuronal activity, we found that rats adapt their choices based solely on the presence or absence of a sucrose reward, suggesting they build an internal representation of reward likelihood. We further demonstrate that NAcSh ensembles dynamically process different aspects of reward-guided behavior, with changes in composition and functional connections observed throughout the reinforcement learning process. The NAcSh forms a highly connected network characterized by a heavy-tailed distribution and the presence of neuronal hubs, facilitating efficient information flow. Reward delivery enhances mutual information, indicating increased communication between ensembles and network synchronization, whereas reward omission decreases it. Our findings reveal how reward information flows through dynamic NAcSh ensembles, whose flexible membership adapts as the rat learns to obtain rewards (energy) in an ever-changing environment.

A delay-robust method for enhanced real-time reinforcement learning.

In reinforcement learning, the Markov Decision Process (MDP) framework typically operates under a blocking paradigm, assuming a static environment during the agent's decision-making and stationary agent behavior while the environment executes its actions. This static model often proves inadequate for real-time tasks, as it lacks the flexibility to handle concurrent changes in both the agent's decision-making process and the environment's dynamic responses. Contemporary solutions, such as linear interpolation or state space augmentation, attempt to address the asynchronous nature of delayed states and actions in real-time environments. However, these methods frequently require precise delay measurements and may fail to fully capture the complexities of delay dynamics. However, these methods frequently require precise delay measurements and may fail to fully capture the complexities of delay dynamics. To address these challenges, we introduce a minimal information set that encapsulates concurrent information during agent-environment interactions, serving as the foundation of our real-time decision-making framework. The traditional blocking-mode MDP is then reformulated as a Minimal Information State Markov Decision Process (MISMDP), aligning more closely with the demands of real-time environments. Within this MISMDP framework, we propose the "Minimal information set for Real-time tasks using Actor-Critic" (MRAC), a general approach for addressing delay issues in real-time tasks, supported by a rigorous theoretical analysis of Q-function convergence. Extensive experiments across both discrete and continuous action space environments demonstrate that MRAC outperforms state-of-the-art algorithms, delivering superior performance and generalization in managing delays within real-time tasks.

Sequence Decision Transformer for Adaptive Traffic Signal Control.

Urban traffic congestion poses significant economic and environmental challenges worldwide. To mitigate these issues, Adaptive Traffic Signal Control (ATSC) has emerged as a promising solution. Recent advancements in deep reinforcement learning (DRL) have further enhanced ATSC's capabilities. This paper introduces a novel DRL-based ATSC approach named the Sequence Decision Transformer (SDT), employing DRL enhanced with attention mechanisms and leveraging the robust capabilities of sequence decision models, akin to those used in advanced natural language processing, adapted here to tackle the complexities of urban traffic management. Firstly, the ATSC problem is modeled as a Markov Decision Process (MDP), with the observation space, action space, and reward function carefully defined. Subsequently, we propose SDT, specifically tailored to solve the MDP problem. The SDT model uses a transformer-based architecture with an encoder and decoder in an actor-critic structure. The encoder processes observations and outputs, both encoded data for the decoder, and value estimates for parameter updates. The decoder, as the policy network, outputs the agent's actions. Proximal Policy Optimization (PPO) is used to update the policy network based on historical data, enhancing decision-making in ATSC. This approach significantly reduces training times, effectively manages larger observation spaces, captures dynamic changes in traffic conditions more accurately, and enhances traffic throughput. Finally, the SDT model is trained and evaluated in synthetic scenarios by comparing the number of vehicles, average speed, and queue length against three baselines, including PPO, a DQN tailored for ATSC, and FRAP, a state-of-the-art ATSC algorithm. SDT shows improvements of 26.8%, 150%, and 21.7% over traditional ATSC algorithms, and 18%, 30%, and 15.6% over the FRAP. This research underscores the potential of integrating Large Language Models (LLMs) with DRL for traffic management, offering a promising solution to urban congestion.

Human-to-Robot Handover Based on Reinforcement Learning.

This study explores manipulator control using reinforcement learning, specifically targeting anthropomorphic gripper-equipped robots, with the objective of enhancing the robots' ability to safely exchange diverse objects with humans during human-robot interactions (HRIs). The study integrates an adaptive HRI hand for versatile grasping and incorporates image recognition for efficient object identification and precise coordinate estimation. A tailored reinforcement-learning environment enables the robot to dynamically adapt to diverse scenarios. The effectiveness of this approach is validated through simulations and real-world applications. The HRI hand's adaptability ensures seamless interactions, while image recognition enhances cognitive capabilities. The reinforcement-learning framework enables the robot to learn and refine skills, demonstrated through successful navigation and manipulation in various scenarios. The transition from simulations to real-world applications affirms the practicality of the proposed system, showcasing its robustness and potential for integration into practical robotic platforms. This study contributes to advancing intelligent and adaptable robotic systems for safe and dynamic HRIs.

Deep Reinforcement Learning with Local Attention for Single Agile Optical Satellite Scheduling Problem.

This paper investigates the single agile optical satellite scheduling problem, which has received increasing attention due to the rapid growth in earth observation requirements. Owing to the complicated constraints and considerable solution space of this problem, the conventional exact methods and heuristic methods, which are sensitive to the problem scale, demand high computational expenses. Thus, an efficient approach is demanded to solve this problem, and this paper proposes a deep reinforcement learning algorithm with a local attention mechanism. A mathematical model is first established to describe this problem, which considers a series of complex constraints and takes the profit ratio of completed tasks as the optimization objective. Then, a neural network framework with an encoder-decoder structure is adopted to generate high-quality solutions, and a local attention mechanism is designed to improve the generation of solutions. In addition, an adaptive learning rate strategy is proposed to guide the actor-critic training algorithm to dynamically adjust the learning rate in the training process to enhance the training effectiveness of the proposed network. Finally, extensive experiments verify that the proposed algorithm outperforms the comparison algorithms in terms of solution quality, generalization performance, and computation efficiency.

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze.

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman's correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

Using Reinforcement Learning to Develop a Novel Gait for a Bio-Robotic California Sea Lion.

While researchers have made notable progress in bio-inspired swimming robot development, a persistent challenge lies in creating propulsive gaits tailored to these robotic systems. The California sea lion achieves its robust swimming abilities through a careful coordination of foreflippers and body segments. In this paper, reinforcement learning (RL) was used to develop a novel sea lion foreflipper gait for a bio-robotic swimmer using a numerically modelled computational representation of the robot. This model integration enabled reinforcement learning to develop desired swimming gaits in the challenging underwater domain. The novel RL gait outperformed the characteristic sea lion foreflipper gait in the simulated underwater domain. When applied to the real-world robot, the RL constructed novel gait performed as well as or better than the characteristic sea lion gait in many factors. This work shows the potential for using complimentary bio-robotic and numerical models with reinforcement learning to enable the development of effective gaits and maneuvers for underwater swimming vehicles.

Stable Jumping Control Based on Deep Reinforcement Learning for a Locust-Inspired Robot.

Biologically inspired jumping robots exhibit exceptional movement capabilities and can quickly overcome obstacles. However, the stability and accuracy of jumping movements are significantly compromised by rapid changes in posture. Here, we propose a stable jumping control algorithm for a locust-inspired jumping robot based on deep reinforcement learning. The algorithm utilizes a training framework comprising two neural network modules (actor network and critic network) to enhance training performance. The framework can control jumping by directly mapping the robot's observations (robot position and velocity, obstacle position, target position, etc.) to its joint torques. The control policy increases randomness and exploration by introducing an entropy term to the policy function. Moreover, we designed a stage incentive mechanism to adjust the reward function dynamically, thereby improving the robot's jumping stability and accuracy. We established a locus-inspired jumping robot platform and conducted a series of jumping experiments in simulation. The results indicate that the robot could perform smooth and non-flip jumps, with the error of the distance from the target remaining below 3%. The robot consumed 44.6% less energy to travel the same distance by jumping compared with walking. Additionally, the proposed algorithm exhibited a faster convergence rate and improved convergence effects compared with other classical algorithms.

Method for Bottle Opening with a Dual-Arm Robot.

This paper introduces a novel approach to robotic assistance in bottle opening using the dual-arm robot TIAGo++. The solution enhances accessibility by addressing the needs of individuals with injuries or disabilities who may require help with common manipulation tasks. The aim of this paper is to propose a method involving vision, manipulation, and learning techniques to effectively address the task of bottle opening. The process begins with the acquisition of bottle and cap positions using an RGB-D camera and computer vision. Subsequently, the robot picks the bottle with one gripper and grips the cap with the other, each by planning safe trajectories. Then, the opening procedure is executed via a position and force control scheme that ensures both grippers follow the unscrewing path defined by the cap thread. Within the control loop, force sensor information is employed to control the vertical axis movements, while gripper rotation control is achieved through a Deep Reinforcement Learning (DRL) algorithm trained to determine the optimal angle increments for rotation. The results demonstrate the successful training of the learning agent. The experiments confirm the effectiveness of the proposed method in bottle opening with the TIAGo++ robot, showcasing the practical viability of the approach.

Adaptive Top-K Algorithm for Medical Conversational Diagnostic Model.

With advancements in computing technology and the rapid progress of data science, machine learning has been widely applied in various fields, showing great potential, especially in digital healthcare. In recent years, conversational diagnostic systems have been used to predict diseases through symptom checking. Early systems predicted the likelihood of a single disease by minimizing the number of questions asked. However, doctors typically perform differential diagnoses in real medical practice, considering multiple possible diseases to address diagnostic uncertainty. This requires systems to ask more critical questions to improve diagnostic accuracy. Nevertheless, such systems in acute medical situations need to process information quickly and accurately, but the complexity of differential diagnosis increases the system's computational cost. To improve the efficiency and accuracy of telemedicine diagnostic systems, this study developed an optimized algorithm for the Top-K algorithm. This algorithm dynamically adjusts the number of the most likely diseases and symptoms by real-time monitoring of case progress, optimizing the diagnostic process, enhancing accuracy (99.81%), and increasing the exclusion rate of severe pathologies. Additionally, the Top-K algorithm optimizes the diagnostic model through a policy network loss function, effectively reducing the number of symptoms and diseases processed and improving the system's response speed by 1.3-1.9 times compared to the state-of-the-art differential diagnosis systems.