RESUMEN
Robotic manipulation challenges, such as grasping and object manipulation, have been tackled successfully with the help of deep reinforcement learning systems. We give an overview of the recent advances in deep reinforcement learning algorithms for robotic manipulation tasks in this review. We begin by outlining the fundamental ideas of reinforcement learning and the parts of a reinforcement learning system. The many deep reinforcement learning algorithms, such as value-based methods, policy-based methods, and actor-critic approaches, that have been suggested for robotic manipulation tasks are then covered. We also examine the numerous issues that have arisen when applying these algorithms to robotics tasks, as well as the various solutions that have been put forth to deal with these issues. Finally, we highlight several unsolved research issues and talk about possible future directions for the subject.
RESUMEN
As of 2023, it is estimated that 6.7 million individuals in the United States live with Alzheimer's disease (AD). Prior research indicates that AD disproportionality affects females; females have a greater incidence rate, perform worse on a variety of neuropsychological tasks, and have greater total brain atrophy. Recent research shows that hippocampal functional connectivity differs by sex and may be related to the observed sex differences in AD, and apolipoprotein E (ApoE) ε4 carriers have reduced hippocampal functional connectivity. The purpose of this study was to determine if the ApoE genotype plays a role in the observed sex differences in hippocampal functional connectivity in Alzheimer's disease. The resting state fMRI and T2 MRI of individuals with AD (n = 30, female = 15) and cognitively normal individuals (n = 30, female = 15) from the Alzheimer's Disease Neuroimaging Initiative (ADNI) were analyzed using the functional connectivity toolbox (CONN). Our results demonstrated intrahippocampal functional connectivity differed between those without an ε4 allele and those with at least one ε4 allele in each group. Additionally, intrahippocampal functional connectivity differed only by sex when Alzheimer's participants had at least one ε4 allele. These results improve our current understanding of the role of the interacting relationship between sex, ApoE genotype, and hippocampal function in AD. Understanding these biomarkers may aid in the development of sex-specific interventions for improved AD treatment.
RESUMEN
One of the most critical challenges in using noninvasive brain stimulation (NIBS) techniques for the treatment of psychiatric and neurologic disorders is inter- and intra-individual variability in response to NIBS. Response variations in previous findings suggest that the one-size-fits-all approach does not seem the most appropriate option for enhancing stimulation outcomes. While there is a growing body of evidence for the feasibility and effectiveness of individualized NIBS approaches, the optimal way to achieve this is yet to be determined. Transcranial electrical stimulation (tES) is one of the NIBS techniques showing promising results in modulating treatment outcomes in several psychiatric and neurologic disorders, but it faces the same challenge for individual optimization. With new computational and methodological advances, tES can be integrated with real-time functional magnetic resonance imaging (rtfMRI) to establish closed-loop tES-fMRI for individually optimized neuromodulation. Closed-loop tES-fMRI systems aim to optimize stimulation parameters based on minimizing differences between the model of the current brain state and the desired value to maximize the expected clinical outcome. The methodological space to optimize closed-loop tES fMRI for clinical applications includes (1) stimulation vs. data acquisition timing, (2) fMRI context (task-based or resting-state), (3) inherent brain oscillations, (4) dose-response function, (5) brain target trait and state and (6) optimization algorithm. Closed-loop tES-fMRI technology has several advantages over non-individualized or open-loop systems to reshape the future of neuromodulation with objective optimization in a clinically relevant context such as drug cue reactivity for substance use disorder considering both inter and intra-individual variations. Using multi-level brain and behavior measures as input and desired outcomes to individualize stimulation parameters provides a framework for designing personalized tES protocols in precision psychiatry.
Asunto(s)
Enfermedades del Sistema Nervioso , Estimulación Transcraneal de Corriente Directa , Humanos , Encéfalo , Estimulación Transcraneal de Corriente Directa/métodos , Estimulación Magnética Transcraneal/métodos , Estimulación EléctricaRESUMEN
Recent studies suggest that transcranial electrical stimulation (tES) can be performed during functional magnetic resonance imaging (fMRI). The novel approach of using concurrent tES-fMRI to modulate and measure targeted brain activity/connectivity may provide unique insights into the causal interactions between the brain neural responses and psychiatric/neurologic signs and symptoms, and importantly, guide the development of new treatments. However, tES stimulation parameters to optimally influence the underlying brain activity may vary with respect to phase difference, frequency, intensity, and electrode's montage among individuals. Here, we propose a protocol for closed-loop tES-fMRI to optimize the frequency and phase difference of alternating current stimulation (tACS) for two nodes (frontal and parietal regions) in individual participants. We carefully considered the challenges in an online optimization of tES parameters with concurrent fMRI, specifically in its safety, artifact in fMRI image quality, online evaluation of the tES effect, and parameter optimization method, and we designed the protocol to run an effective study to enhance frontoparietal connectivity and working memory performance with the optimized tACS using closed-loop tES-fMRI. We provide technical details of the protocol, including electrode types, electrolytes, electrode montages, concurrent tES-fMRI hardware, online fMRI processing pipelines, and the optimization algorithm. We confirmed the implementation of this protocol worked successfully with a pilot experiment.
Asunto(s)
Estimulación Transcraneal de Corriente Directa , Artefactos , Encéfalo/diagnóstico por imagen , Encéfalo/fisiología , Mapeo Encefálico/métodos , Humanos , Imagen por Resonancia Magnética/métodos , Estimulación Transcraneal de Corriente Directa/métodosRESUMEN
BACKGROUND: Small hippocampal volume is a prevalent neurostructural abnormality in posttraumatic stress disorder (PTSD). However, whether the hippocampal atrophy is the cause of disease symptoms or a pre-existing risk factor and whether it is a reversible alteration or a permanent trait are unclear. The trait- or state-dependent alteration could also differ among the hippocampal subfields. METHODS: The study examined the longitudinal hippocampal volume changes due to positive emotional training with left amygdala (LA) real-time fMRI neurofeedback (rtfMRI-nf) in combat veterans with PTSD. The participants were trained to increase the neurofeedback signal from LA (experimental group, N = 20) or brain region not involved in emotion processing (control group, N = 9) by recalling a positive autobiographical memory. The pre- and post-training structural MRI brain images were processed with FreeSurfer to evaluate the hippocampal subfield volumes. Hippocampal volumes for healthy controls (N = 43) were also examined to evaluate the baseline abnormality in PTSD. RESULTS: A significant group difference in volume change was found in the left CA1 head region. This region had the most significant volume reduction at the baseline in PTSD. The experimental group showed a significant volume increase, while the control group showed a significant volume decrease in this region. The volume change in the control group negatively correlated with interval days between the scans. LIMITATIONS: A cognitive improvement due to the hippocampal volume increase could not be found with symptom scales. CONCLUSIONS: RtfMRI-nf positive emotional training increased the hippocampus volume among people with PTSD, suggesting that hippocampal atrophy in PTSD is modifiable.