Faster sorting algorithms discovered using deep reinforcement learning.

Fundamental algorithms such as sorting or hashing are used trillions of times on any given day1. As demand for computation grows, it has become critical for these algorithms to be as performant as possible. Whereas remarkable progress has been achieved in the past2, making further improvements on the efficiency of these routines has proved challenging for both human scientists and computational approaches. Here we show how artificial intelligence can go beyond the current state of the art by discovering hitherto unknown routines. To realize this, we formulated the task of finding a better sorting routine as a single-player game. We then trained a new deep reinforcement learning agent, AlphaDev, to play this game. AlphaDev discovered small sorting algorithms from scratch that outperformed previously known human benchmarks. These algorithms have been integrated into the LLVM standard C++ sort library3. This change to this part of the sort library represents the replacement of a component with an algorithm that has been automatically discovered using reinforcement learning. We also present results in extra domains, showcasing the generality of the approach.

Discovering faster matrix multiplication algorithms with reinforcement learning.

Improving the efficiency of algorithms for fundamental computations can have a widespread impact, as it can affect the overall speed of a large amount of computations. Matrix multiplication is one such primitive task, occurring in many systems-from neural networks to scientific computing routines. The automatic discovery of algorithms using machine learning offers the prospect of reaching beyond human intuition and outperforming the current best human-designed algorithms. However, automating the algorithm discovery procedure is intricate, as the space of possible algorithms is enormous. Here we report a deep reinforcement learning approach based on AlphaZero1 for discovering efficient and provably correct algorithms for the multiplication of arbitrary matrices. Our agent, AlphaTensor, is trained to play a single-player game where the objective is finding tensor decompositions within a finite factor space. AlphaTensor discovered algorithms that outperform the state-of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4 matrices in a finite field, where AlphaTensor's algorithm improves on Strassen's two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago2. We further showcase the flexibility of AlphaTensor through different use-cases: algorithms with state-of-the-art complexity for structured matrix multiplication and improved practical efficiency by optimizing matrix multiplication for runtime on specific hardware. Our results highlight AlphaTensor's ability to accelerate the process of algorithmic discovery on a range of problems, and to optimize for different criteria.

Structural basis of omega-3 fatty acid transport across the blood-brain barrier.

Docosahexaenoic acid is an omega-3 fatty acid that is essential for neurological development and function, and it is supplied to the brain and eyes predominantly from dietary sources1-6. This nutrient is transported across the blood-brain and blood-retina barriers in the form of lysophosphatidylcholine by major facilitator superfamily domain containing 2A (MFSD2A) in a Na+-dependent manner7,8. Here we present the structure of MFSD2A determined using single-particle cryo-electron microscopy, which reveals twelve transmembrane helices that are separated into two pseudosymmetric domains. The transporter is in an inward-facing conformation and features a large amphipathic cavity that contains the Na+-binding site and a bound lysolipid substrate, which we confirmed using native mass spectrometry. Together with our functional analyses and molecular dynamics simulations, this structure reveals details of how MFSD2A interacts with substrates and how Na+-dependent conformational changes allow for the release of these substrates into the membrane through a lateral gate. Our work provides insights into the molecular mechanism by which this atypical major facility superfamily transporter mediates the uptake of lysolipids into the brain, and has the potential to aid in the delivery of neurotherapeutic agents.

Highly accurate protein structure prediction with AlphaFold.

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1-4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence-the structure prediction component of the 'protein folding problem'8-has been an important open research problem for more than 50 years9. Despite recent progress10-14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm.

Mastering Atari, Go, chess and shogi by planning with a learned model.

Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess1 and Go2, where a perfect simulator is available. However, in real-world problems, the dynamics governing the environment are often complex and unknown. Here we present the MuZero algorithm, which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. The MuZero algorithm learns an iterable model that produces predictions relevant to planning: the action-selection policy, the value function and the reward. When evaluated on 57 different Atari games3-the canonical video game environment for testing artificial intelligence techniques, in which model-based planning approaches have historically struggled4-the MuZero algorithm achieved state-of-the-art performance. When evaluated on Go, chess and shogi-canonical environments for high-performance planning-the MuZero algorithm matched, without any knowledge of the game dynamics, the superhuman performance of the AlphaZero algorithm5 that was supplied with the rules of the game.

Improved protein structure prediction using potentials from deep learning.

Protein structure prediction can be used to determine the three-dimensional shape of a protein from its amino acid sequence1. This problem is of fundamental importance as the structure of a protein largely determines its function2; however, protein structures can be difficult to determine experimentally. Considerable progress has recently been made by leveraging genetic information. It is possible to infer which amino acid residues are in contact by analysing covariation in homologous sequences, which aids in the prediction of protein structures3. Here we show that we can train a neural network to make accurate predictions of the distances between pairs of residues, which convey more information about the structure than contact predictions. Using this information, we construct a potential of mean force4 that can accurately describe the shape of a protein. We find that the resulting potential can be optimized by a simple gradient descent algorithm to generate structures without complex sampling procedures. The resulting system, named AlphaFold, achieves high accuracy, even for sequences with fewer homologous sequences. In the recent Critical Assessment of Protein Structure Prediction5 (CASP13)-a blind assessment of the state of the field-AlphaFold created high-accuracy structures (with template modelling (TM) scores6 of 0.7 or higher) for 24 out of 43 free modelling domains, whereas the next best method, which used sampling and contact information, achieved such accuracy for only 14 out of 43 domains. AlphaFold represents a considerable advance in protein-structure prediction. We expect this increased accuracy to enable insights into the function and malfunction of proteins, especially in cases for which no structures for homologous proteins have been experimentally determined7.

Mfsd2a utilizes a flippase mechanism to mediate omega-3 fatty acid lysolipid transport.

Major Facilitator Superfamily Domain containing 2a (Mfsd2a) is a sodium-dependent lysophosphatidylcholine (LPC) transporter expressed at the blood-brain barrier that constitutes the main pathway by which the brain obtains omega-3 fatty acids, such as docosahexanoic acid. Mfsd2a deficiency in humans results in severe microcephaly, underscoring the importance of LPC transport by Mfsd2a for brain development. Biochemical studies and recent cryo-electron microscopy (cryo-EM) structures of Mfsd2a bound to LPC suggest that Mfsd2a transports LPC via an alternating access mechanism between outward-facing and inward-facing conformational states in which the LPC inverts during transport between the outer and inner leaflet of a membrane. However, direct biochemical evidence of flippase activity by Mfsd2a has not been demonstrated and it is not understood how Mfsd2a could invert LPC between the outer and inner leaflet of the membrane in a sodium-dependent manner. Here, we established a unique in vitro assay using recombinant Mfsd2a reconstituted in liposomes that exploits the ability of Mfsd2a to transport lysophosphatidylserine (LPS) coupled with a small molecule LPS binding fluorophore that allowed for monitoring of directional flipping of the LPS headgroup from the outer to the inner liposome membrane. Using this assay, we demonstrate that Mfsd2a flips LPS from the outer to the inner leaflet of a membrane bilayer in a sodium-dependent manner. Furthermore, using cryo-EM structures as guides together with mutagenesis and a cell-based transport assay, we identify amino acid residues important for Mfsd2a activity that likely constitute substrate interaction domains. These studies provide direct biochemical evidence that Mfsd2a functions as a lysolipid flippase.

Author Correction: The mid-developmental transition and the evolution of animal body plans.

An Amendment to this paper has been published and can be accessed via a link at the top of the paper.

Grandmaster level in StarCraft II using multi-agent reinforcement learning.

Many real-world applications require artificial agents to compete and coordinate with other agents in complex environments. As a stepping stone to this goal, the domain of StarCraft has emerged as an important challenge for artificial intelligence research, owing to its iconic and enduring status among the most difficult professional esports and its relevance to the real world in terms of its raw complexity and multi-agent challenges. Over the course of a decade and numerous competitions1-3, the strongest agents have simplified important aspects of the game, utilized superhuman capabilities, or employed hand-crafted sub-systems4. Despite these advantages, no previous agent has come close to matching the overall skill of top StarCraft players. We chose to address the challenge of StarCraft using general-purpose learning methods that are in principle applicable to other complex domains: a multi-agent reinforcement learning algorithm that uses data from both human and agent games within a diverse league of continually adapting strategies and counter-strategies, each represented by deep neural networks5,6. We evaluated our agent, AlphaStar, in the full game of StarCraft II, through a series of online games against human players. AlphaStar was rated at Grandmaster level for all three StarCraft races and above 99.8% of officially ranked human players.

Spns1 is a lysophospholipid transporter mediating lysosomal phospholipid salvage.

The lysosome is central to the degradation of proteins, carbohydrates, and lipids and their salvage back to the cytosol for reutilization. Lysosomal transporters for amino acids, sugars, and cholesterol have been identified, and the metabolic fates of these molecules in the cytoplasm have been elucidated. Remarkably, it is not known whether lysosomal salvage exists for glycerophospholipids, the major constituents of cellular membranes. By using a transport assay screen against orphan lysosomal transporters, we identified the major facilitator superfamily protein Spns1 that is ubiquitously expressed in all tissues as a proton-dependent lysophosphatidylcholine (LPC) and lysophosphatidylethanolamine (LPE) transporter, with LPC and LPE being the lysosomal breakdown products of the most abundant eukaryotic phospholipids, phosphatidylcholine and phosphatidylethanolamine, respectively. Spns1 deficiency in cells, zebrafish embryos, and mouse liver resulted in lysosomal accumulation of LPC and LPE species with pathological consequences on lysosomal function. Flux analysis using stable isotope-labeled phospholipid apolipoprotein E nanodiscs targeted to lysosomes showed that LPC was transported out of lysosomes in an Spns1-dependent manner and re-esterified back into the cytoplasmic pools of phosphatidylcholine. Our findings identify a phospholipid salvage pathway from lysosomes to the cytosol that is dependent on Spns1 and critical for maintaining normal lysosomal function.

Destabilization of ß Cell FIT2 by saturated fatty acids alter lipid droplet numbers and contribute to ER stress and diabetes.

SignificanceWith obesity on the rise, there is a growing appreciation for intracellular lipid droplet (LD) regulation. Here, we show how saturated fatty acids (SFAs) reduce fat storage-inducing transmembrane protein 2 (FIT2)-facilitated, pancreatic ß cell LD biogenesis, which in turn induces ß cell dysfunction and death, leading to diabetes. This mechanism involves direct acylation of FIT2 cysteine residues, which then marks the FIT2 protein for endoplasmic reticulum (ER)-associated degradation. Loss of ß cell FIT2 and LDs reduces insulin secretion, increases intracellular ceramides, stimulates ER stress, and exacerbates diet-induced diabetes in mice. While palmitate and stearate degrade FIT2, unsaturated fatty acids such as palmitoleate and oleate do not, results of which extend to nutrition and diabetes.

Association of Frailty with Healthcare Utilization for Patients over One Year Following Surgical Evaluation.

OBJECTIVE: Characterize the distribution of healthcare utilization associated with pre-operative frailty in the year following evaluation by a surgeon. SUMMARY BACKGROUND DATA: Frailty is associated with increased morbidity, mortality, and costs for surgical patients. However, the total financial burden for frail patients beyond the index surgery and inpatient stay remains unknown. METHODS: Prospective cohort assembled from February 2016 to December 2020 within a multi-hospital integrated healthcare delivery and finance system (IDFS), from patients evaluated with the Risk Analysis Index (RAI) of frailty. Inclusion criteria: age greater than 18, valid RAI, membership in the IDFS Health Plan. Data were stratified by frailty and surgical status. RESULTS: The mean (SD) age was 54.7 (16.1) and 58.2% female of the cohort (n=86,572). For all patients with reimbursement for surgery (n=53,856), frail and very frail patients incurred respective increases of 8% ( P =0.027) and 29% ( P <0.001) on utilization relative to the normal group. Robust patients saw a 52% ( P <0.001) decrease. This pattern was more pronounced in the cohort without surgery (n=32,716). The increase over normal utilization for frail and very frail patients increased to 23% ( P =0.004) and 68% ( P <0.001), respectively. Utilization among robust patients decreased 62% ( P <0.001). Increases among the frail were primarily due to increased inpatient medical and post-acute care services (all P <0.001). CONCLUSIONS: Patient frailty is associated with increased total healthcare utilization, primarily via increased inpatient medical and post-acute care following surgery. Quantifying these frailty-related financial burdens may inform clinical decision making as well as the design of value-based reimbursement strategies.

Association Between Emergency Medical Service Agency Volume and Mortality in Trauma Patients.

OBJECTIVE: The aim of this study was to evaluate the association of annual trauma patient volume on outcomes for emergency medical services (EMS) agencies. BACKGROUND: Regionalization of trauma care saves lives. The underlying concept driving this is a volume-outcome relationship. EMS are the entry point to the trauma system, yet it is unknown if a volume-outcome relationship exists for EMS. METHODS: A retrospective analysis of prospective cohort including 8 trauma centers and 20 EMS air medical and metropolitan ground transport agencies. Patients 18 to 90 years old with injury severity scores ≥9 transported from the scene were included. Patient and agency-level risk-adjusted regression determined the association between EMS agency trauma patient volume and early mortality. RESULTS: A total of 33,511 were included with a median EMS agency volume of 374 patients annually (interquartile range: 90-580). Each 50-patient increase in EMS agency volume was associated with 5% decreased odds of 6-hour mortality (adjusted odds ratio=0.95; 95% CI: 0.92-0.99, P =0.03) and 3% decreased odds of 24-hour mortality (adjusted odds ratio=0.97; 95% CI: 0.95-0.99, P =0.04). Prespecified subgroup analysis showed EMS agency volume was associated with reduced odds of mortality for patients with prehospital shock, requiring prehospital airway placement, undergoing air medical transport, and those with traumatic brain injury. Agency-level analysis demonstrated that high-volume (>374 patients/year) EMS agencies had a significantly lower risk-standardized 6-hour mortality rate than low-volume (<374 patients/year) EMS agencies (1.9% vs 4.8%, P <0.01). CONCLUSIONS: A higher volume of trauma patients transported at the EMS agency level is associated with improved early mortality. Further investigation of this volume-outcome relationship is necessary to leverage quality improvement, benchmarking, and educational initiatives.

Mfsd2a-mediated lysolipid transport is important for renal recovery after acute kidney injury.

Acute kidney injury (AKI) is a global public health concern with high mortality and morbidity. In ischemic-reperfusion injury (IRI), a main cause of AKI, the brush border membrane of S3 proximal tubules (PT) is lost to the tubular lumen. How injured tubules reconstitute lost membrane lipids during renal recovery is not known. Here, we identified Mfsd2a, a sodium-dependent lysophosphatidylcholine (LPC) transporter, to be expressed specifically in the basolateral membrane of S3 PT. Using an in vivo activity probe for Mfsd2a, transport activity was found to be specific to the S3 PT. Mice with haploinsufficiency of Mfsd2a exhibited delayed recovery of renal function after acute IRI, with depressed urine osmolality and elevated levels of histological markers of damage, fibrosis, and inflammation, findings corroborated by transcriptomic analysis. Lipidomics revealed a deficiency in docosahexaenoic acid (DHA) containing phospholipids in Mfsd2a haploinsufficiency. Treatment of Mfsd2a haploinsufficient mice with LPC-DHA improved renal function and reduced markers of injury, fibrosis, and inflammation. Additionally, LPC-DHA treatment restored S3 brush border membrane architecture and normalized DHA-containing phospholipid content. These findings indicate that Mfsd2a-mediated transport of LPC-DHA is limiting for renal recovery after AKI and suggest that LPC-DHA could be a promising dietary supplement for improving recovery following AKI.

The lipid transporter Mfsd2a maintains pulmonary surfactant homeostasis.

Pulmonary surfactant is a lipoprotein complex essential for lung function, and insufficiency or altered surfactant composition is associated with major lung diseases, such as acute respiratory distress syndromes, idiopathic pulmonary fibrosis, and chronic obstructive pulmonary disease. Pulmonary surfactant is primarily composed of phosphatidylcholine (PC) in complex with specialized surfactant proteins and secreted by alveolar type 2 (AT2) cells. Surfactant homeostasis on the alveolar surface is balanced by the rates of synthesis and secretion with reuptake and recycling by AT2 cells, with some degradation by pulmonary macrophages and loss up the bronchial tree. However, whether phospholipid (PL) transporters exist in AT2 cells to mediate reuptake of surfactant PL remains to be identified. Here, we demonstrate that major facilitator superfamily domain containing 2a (Mfsd2a), a sodium-dependent lysophosphatidylcholine (LPC) transporter, is expressed at the apical surface of AT2 cells. A mouse model with inducible AT2 cell-specific deficiency of Mfsd2a exhibited AT2 cell hypertrophy with reduced total surfactant PL levels because of reductions in the most abundant surfactants, PC containing dipalmitic acid, and PC species containing the omega-3 fatty acid docosahexaenoic acid. These changes in surfactant levels and composition were mirrored by similar changes in the AT2 cell lipidome. Mechanistically, direct tracheal instillation of fluorescent LPC and PC probes indicated that Mfsd2a mediates the uptake of LPC generated by pulmonary phospholipase activity in the alveolar space. These studies reveal that Mfsd2a-mediated LPC uptake is quantitatively important in maintaining surfactant homeostasis and identify this lipid transporter as a physiological component of surfactant recycling.

Single-Cell Analysis of Blood-Brain Barrier Response to Pericyte Loss.

RATIONALE: Pericytes are capillary mural cells playing a role in stabilizing newly formed blood vessels during development and tissue repair. Loss of pericytes has been described in several brain disorders, and genetically induced pericyte deficiency in the brain leads to increased macromolecular leakage across the blood-brain barrier (BBB). However, the molecular details of the endothelial response to pericyte deficiency remain elusive. OBJECTIVE: To map the transcriptional changes in brain endothelial cells resulting from lack of pericyte contact at single-cell level and to correlate them with regional heterogeneities in BBB function and vascular phenotype. METHODS AND RESULTS: We reveal transcriptional, morphological, and functional consequences of pericyte absence for brain endothelial cells using a combination of methodologies, including single-cell RNA sequencing, tracer analyses, and immunofluorescent detection of protein expression in pericyte-deficient adult Pdgfbret/ret mice. We find that endothelial cells without pericyte contact retain a general BBB-specific gene expression profile, however, they acquire a venous-shifted molecular pattern and become transformed regarding the expression of numerous growth factors and regulatory proteins. Adult Pdgfbret/ret brains display ongoing angiogenic sprouting without concomitant cell proliferation providing unique insights into the endothelial tip cell transcriptome. We also reveal heterogeneous modes of pericyte-deficient BBB impairment, where hotspot leakage sites display arteriolar-shifted identity and pinpoint putative BBB regulators. By testing the causal involvement of some of these using reverse genetics, we uncover a reinforcing role for angiopoietin 2 at the BBB. CONCLUSIONS: By elucidating the complexity of endothelial response to pericyte deficiency at cellular resolution, our study provides insight into the importance of brain pericytes for endothelial arterio-venous zonation, angiogenic quiescence, and a limited set of BBB functions. The BBB-reinforcing role of ANGPT2 (angiopoietin 2) is paradoxical given its wider role as TIE2 (TEK receptor tyrosine kinase) receptor antagonist and may suggest a unique and context-dependent function of ANGPT2 in the brain.

The Geography of Injuries in Trauma Systems: Using Home as a Proxy for Incident Location.

INTRODUCTION: Effective trauma system organization is crucial to timely access to care and requires accurate understanding of injury and resource locations. Many systems rely on home zip codes to evaluate geographic distribution of injury; however, few studies have evaluated the reliability of home as a proxy for incident location after injury. METHODS: We analyzed data from a multicenter prospective cohort collected from 2017 to 2021. Injured patients with both home and incident zip codes were included. Outcomes included discordance and differential distance between home and incident zip code. Associations of discordance with patient characteristics were determined by logistic regression. We also assessed trauma center catchment areas based on home versus incident zip codes and variation regionally at each center. RESULTS: Fifty thousand one hundred seventy-five patients were included in the analysis. Home and incident zip codes were discordant in 21,635 patients (43.1%). Injuries related to motor vehicles (aOR: 4.76 [95% CI 4.50-5.04]) and younger adults 16-64 (aOR: 2.46 [95% CI 2.28-2.65]) were most likely to be discordant. Additionally, as injury severity score increased, discordance increased. Trauma center catchment area differed up to two-thirds of zip codes when using home versus incident location. Discordance rate, discordant distance, and catchment area overlap between home and incident zip codes all varied significantly by geographic region. CONCLUSIONS: Home location as proxy for injury location should be used with caution and may impact trauma system planning and policy, especially in certain populations. More accurate geolocation data are warranted to further optimize trauma system design.

Mastering the game of Go without human knowledge.

A long-standing goal of artificial intelligence is an algorithm that learns, tabula rasa, superhuman proficiency in challenging domains. Recently, AlphaGo became the first program to defeat a world champion in the game of Go. The tree search in AlphaGo evaluated positions and selected moves using deep neural networks. These neural networks were trained by supervised learning from human expert moves, and by reinforcement learning from self-play. Here we introduce an algorithm based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules. AlphaGo becomes its own teacher: a neural network is trained to predict AlphaGo's own move selections and also the winner of AlphaGo's games. This neural network improves the strength of the tree search, resulting in higher quality move selection and stronger self-play in the next iteration. Starting tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning 100-0 against the previously published, champion-defeating AlphaGo.

Mfsd2b is essential for the sphingosine-1-phosphate export in erythrocytes and platelets.

Sphingosine-1-phosphate (S1P), a potent signalling lipid secreted by red blood cells and platelets, plays numerous biologically significant roles. However, the identity of its long-sought exporter is enigmatic. Here we show that the major facilitator superfamily transporter 2b (Mfsd2b), an orphan transporter, is essential for S1P export from red blood cells and platelets. Comprehensive lipidomic analysis indicates a dramatic and specific accumulation of S1P species in Mfsd2b knockout red blood cells and platelets compared with that of wild-type controls. Consistently, biochemical assays from knockout red blood cells, platelets, and cell lines overexpressing human and mouse Mfsd2b proteins demonstrate that Mfsd2b actively exports S1P. Plasma S1P level in knockout mice is significantly reduced by 42-54% of that of wild-type level, indicating that Mfsd2b pathway contributes approximately half of the plasma S1P pool. The reduction of plasma S1P in knockout mice is insufficient to cause blood vessel leakiness, but it does render the mice more sensitive to anaphylactic shock. Stress-induced erythropoiesis significantly increased plasma S1P levels and knockout mice were sensitive to these treatments. Surprisingly, knockout mice exhibited haemolysis associated with red blood cell stomatocytes, and the haemolytic phenotype was severely increased with signs of membrane fragility under stress erythropoiesis. We show that S1P secretion by Mfsd2b is critical for red blood cell morphology. Our data reveal an unexpected physiological role of red blood cells in sphingolipid metabolism in circulation. These findings open new avenues for investigating the signalling roles of S1P derived from red blood cells and platelets.

Fast reinforcement learning with generalized policy updates.

The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision-making problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of a task can be well approximated as a linear combination of the reward functions of tasks previously solved, we can reduce a reinforcement-learning problem to a simpler linear regression. When this is not the case, the agent can still exploit the task solutions by using them to interact with and learn about the environment. Both strategies considerably reduce the amount of data needed to solve a reinforcement-learning problem.