Generalization Enhancement of Visual Reinforcement Learning through Internal States.

Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Salience Interest Option: Temporal abstraction with salience interest functions.

Reinforcement Learning (RL) is a significant machine learning subfield that emphasizes learning actions based on environment to obtain optimal behavior policy. RL agents can make decisions at variable time scales in the form of temporal abstractions, also known as options. The issue of discovering options has seen a considerable research effort. Most notably, the Interest Option Critic (IOC) algorithm first extends the initial set to the interest function, providing a method for learning options specialized to certain state space regions. This approach offers a specific attention mechanism for action selection. Unfortunately, this method still suffers from the classic issues of poor data efficiency and lack of flexibility in RL when learning options end-to-end through backpropagation. This paper proposes a new approach called Salience Interest Option Critic (SIOC), which chooses subsets of existing initiation sets for RL. Specifically, these subsets are not learned by backpropagation, which is slow and tends to overfit, but through particle filters. This approach enables the rapid and flexible identification of critical subsets using only reward feedback. We conducted experiments in discrete and continuous domains, and our proposed method demonstrate higher efficiency and flexibility than other methods. The generated options are more valuable within a single task and exhibited greater interpretability and reusability in multi-task learning scenarios.

A rare non-gadolinium enhancing sarcoma brain metastasis with microenvironment dominated by tumor-associated macrophages.

Brain metastases occur in 1% of sarcoma cases and are associated with a median overall survival of 6 months. We report a rare case of a brain metastasis with unique radiologic and histopathologic features in a patient with low grade fibromyxoid sarcoma (LGFMS) previously treated with immune checkpoint inhibitor (ICI) therapy. The lone metastasis progressed in the midbrain tegmentum over 15 months as a non-enhancing, T2-hyperintense lesion with peripheral diffusion restriction, mimicking a demyelinating lesion. Histopathology of the lesion at autopsy revealed a rich infiltrate of tumor-associated macrophages (TAMs) with highest density at the leading edge of the metastasis, whereas there was a paucity of lymphocytes, suggestive of an immunologically cold environment. Given the important immunosuppressive and tumor-promoting functions of TAMs in gliomas and carcinoma/melanoma brain metastases, this unusual case provides an interesting example of a dense TAM infiltrate in a much rarer sarcoma brain metastasis.