Learning agile soccer skills for a bipedal robot with deep reinforcement learning.

We investigated whether deep reinforcement learning (deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies. We used deep RL to train a humanoid robot to play a simplified one-versus-one soccer game. The resulting agent exhibits robust and dynamic movement skills, such as rapid fall recovery, walking, turning, and kicking, and it transitions between them in a smooth and efficient manner. It also learned to anticipate ball movements and block opponent shots. The agent's tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. Our agent was trained in simulation and transferred to real robots zero-shot. A combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training enabled good-quality transfer. In experiments, the agent walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline.

Magnetic control of tokamak plasmas through deep reinforcement learning.

Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable1,2, including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and 'snowflake' configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained 'droplets' on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.

[Limitations of medicine:diagnosis and therapy of <> back pain]. / Grenzen der Medizin: Diagnostik und Therapie von <> Ruckenschmerzen.

A 34-year-old man from Ethiopia had been suffering from inflammatory lower backpain for five months. The imaging showed multiple osteolytic lesions histologically corresponding to a granulomatous inflammation. Detection of multiresistant Mycobacteria Tuberculosis (MDR-TB) was delayed because of initially negative microbiological findings. During the therapy of MDR-TB, the patient developed different side effects. In this article we focus on the challenge to diagnose and treat MDR-TB also in a resource-rich setting as it is the case for Switzerland.

Biphasic decay kinetics suggest progressive slowing in turnover of latently HIV-1 infected cells during antiretroviral therapy.

BACKGROUND: Mathematical models based on kinetics of HIV-1 plasma viremia after initiation of combination antiretroviral therapy (cART) inferred HIV-infected cells to decay exponentially with constant rates correlated to their strength of virus production. To further define in vivo decay kinetics of HIV-1 infected cells experimentally, we assessed infected cell-classes of distinct viral transcriptional activity in peripheral blood mononuclear cells (PBMC) of five patients during 1 year after initiation of cART RESULTS: In a novel analytical approach patient-matched PCR for unspliced and multiply spliced viral RNAs was combined with limiting dilution analysis at the single cell level. This revealed that HIV-RNA+ PBMC can be stratified into four distinct viral transcriptional classes. Two overlapping cell-classes of high viral transcriptional activity, suggestive of a virion producing phenotype, rapidly declined to undetectable levels. Two cell classes expressing HIV-RNA at low and intermediate levels, presumably insufficient for virus production and occurring at frequencies exceeding those of productively infected cells matched definitions of HIV-latency. These cells persisted during cART. Nevertheless, during the first four weeks of therapy their kinetics resembled that of productively infected cells. CONCLUSION: We have observed biphasic decays of latently HIV-infected cells of low and intermediate viral transcriptional activity with marked decreases in cell numbers shortly after initiation of therapy and complete persistence in later phases. A similar decay pattern was shared by cells with greatly enhanced viral transcriptional activity which showed a certain grade of levelling off before their disappearance. Thus it is conceivable that turnover/decay rates of HIV-infected PBMC may be intrinsically variable. In particular they might be accelerated by HIV-induced activation and reactivation of the viral life cycle and slowed down by the disappearance of such feedback-loops after initiation of cART.

Shifts in cell-associated HIV-1 RNA but not in episomal HIV-1 DNA correlate with new cycles of HIV-1 infection in vivo.

The significance of distinct classes of HIV-1 nucleic acids as correlates of recent HIV-1 replication was assessed in peripheral blood mononuclear cells (PBMC) obtained from 14 patients during 2 weeks of structured interruption of antiretroviral therapy (STI) and 2 weeks of resuming therapy. Levels of HIV RNA in plasma (HIV-RNAplasma) and of unspliced cell-associated HIV-1 RNA (HIV-UsRNAPBMC) were significantly increased as a result of STI, whereas no significant shifts in the levels of 2-LTR episomal HIV-1 DNA (2-LTR circles) and total late HIV-1 reverse transcripts (late-DNA) were observed. Thus, limited viral replication had occurred, which had no effect on the pool size of infected cells in the periphery. Levels of 2-LTR circles did not reflect rapid changes in HIV-1 replication. In contrast, expression of HIV-UsRNAPBMC increased during STI and consequently provides a more sensitive, albeit not absolute cellular marker of ongoing HIV-1 replication.

HIV RNA in plasma rebounds within days during structured treatment interruptions.

OBJECTIVE: To evaluate time to viral rebound in patients undergoing repeated structured treatment interruptions (STI). METHOD: Fourteen chronically HIV-infected patients enrolled in the Swiss-Spanish Intermittent Treatment Trial (SSITT) underwent frequent blood sampling. Patients underwent four cycles of 2-week STI, followed by 8-week retreatment with the identical antiretroviral treatment (HAART) used before STI. At the fifth cycle, treatment was stopped for a longer period. Before each new STI, plasma viral load (VL) had to reach < 50 copies/ml. VL was measured during day 0 (last day on HAART) and on days 4, 8 and 14 during all five STI. RESULTS: During the first cycle, plasma HIV RNA increased to > 50 copies/ml (range, 67-88) in five patients at day 4, in eight patients (> 100 copies/ml) at day 8 and in 12 patients (> 100 copies/ml) at day 14. Cumulative analysis of the frequency of detectable HIV RNA at days 4, 8 and 14 compared with day 0 for all five cycles revealed nine patients with VL > 50 copies/ml [13 of 54 samples tested (24.1%); = 0.14] at day 4, 11 patients [33 of 58 samples tested (56.9%); < 0.0001] at day 8 and 12 patients [53 of 65 samples tested (81.5%); < 0.0001] at day 14. CONCLUSIONS: Significant viral replication can be induced during 1 week STI, and this may increase the risk of the emergence of drug resistance during long-term cycling. Therefore, short-term cycling strategies such as 1-week-on, 1-week-off treatment, although conceptually intriguing, should still be regarded as investigational and should be restricted to rigorously controlled clinical trials ideally involving patients who have never failed treatment before.

Human immunodeficiency virus-specific CD8(+) T-cell responses do not predict viral growth and clearance rates during structured intermittent antiretroviral therapy.

There is a continuing search for better ways to use existing drugs against human immunodeficiency virus (HIV). One idea is to use short therapy interruptions to "autovaccinate" HIV-infected patients. A group of 13 chronically HIV-infected patients enrolled in a trial of such so-called structured treatment interruptions (STIs) were intensively studied with respect to their viral load (VL) and HIV-specific CD8+ T-cell (cytotoxic T-lymphocyte [CTL]) responses. We found that 10 of the 13 patients had plateau VLs after STIs that were lower than their pretreatment VLs. While viral rebound rates became lower over STIs, there were no changes in clearance rates. Although numbers of CTLs did increase over the same time that viral rebounds decreased, there was no correlation between CTL count and either viral rebound rates or clearance rates. Finally, we asked whether absolute numbers of or changes in numbers of CTLs predict plateau VLs after STIs. No measure of CTLs was able to predict plateau VLs. Thus, there was no signature in these data of an important contribution to virological control from HIV-specific CD8+ T lymphocytes.