A reinforcement-based mechanism for discontinuous learning.

Problem-solving and reasoning involve mental exploration and navigation in sparse relational spaces. A physical analogue is spatial navigation in structured environments such as a network of burrows. Recent experiments with mice navigating a labyrinth show a sharp discontinuity during learning, corresponding to a distinct moment of "sudden insight" when mice figure out long, direct paths to the goal. This discontinuity is seemingly at odds with reinforcement learning (RL), which involves a gradual build-up of a value signal during learning. Here, we show that biologically plausible RL rules combined with persistent exploration generically exhibit discontinuous learning. In tree-like structured environments, positive feedback from learning on behavior generates a "reinforcement wave" with a steep profile. The discontinuity occurs when the wave reaches the starting point. By examining the nonlinear dynamics of reinforcement propagation, we establish a quantitative relationship between the learning rule, the agent's exploration biases, and learning speed. Predictions explain existing data and motivate specific experiments to isolate the phenomenon. Additionally, we characterize the exact learning dynamics of various RL rules for a complex sequential task.

Sector search strategies for odor trail tracking.

Ants, mice, and dogs often use surface-bound scent trails to establish navigation routes or to find food and mates, yet their tracking strategies remain poorly understood. Chemotaxis-based strategies cannot explain casting, a characteristic sequence of wide oscillations with increasing amplitude performed upon sustained loss of contact with the trail. We propose that tracking animals have an intrinsic, geometric notion of continuity, allowing them to exploit past contacts with the trail to form an estimate of where it is headed. This estimate and its uncertainty form an angular sector, and the emergent search patterns resemble a "sector search." Reinforcement learning agents trained to execute a sector search recapitulate the various phases of experimentally observed tracking behavior. We use ideas from polymer physics to formulate a statistical description of trails and show that search geometry imposes basic limits on how quickly animals can track trails. By formulating trail tracking as a Bellman-type sequential optimization problem, we quantify the geometric elements of optimal sector search strategy, effectively explaining why and when casting is necessary. We propose a set of experiments to infer how tracking animals acquire, integrate, and respond to past information on the tracked trail. More generally, we define navigational strategies relevant for animals and biomimetic robots and formulate trail tracking as a behavioral paradigm for learning, memory, and planning.

Glider soaring via reinforcement learning in the field.

Soaring birds often rely on ascending thermal plumes (thermals) in the atmosphere as they search for prey or migrate across large distances1-4. The landscape of convective currents is rugged and shifts on timescales of a few minutes as thermals constantly form, disintegrate or are transported away by the wind5,6. How soaring birds find and navigate thermals within this complex landscape is unknown. Reinforcement learning7 provides an appropriate framework in which to identify an effective navigational strategy as a sequence of decisions made in response to environmental cues. Here we use reinforcement learning to train a glider in the field to navigate atmospheric thermals autonomously. We equipped a glider of two-metre wingspan with a flight controller that precisely controlled the bank angle and pitch, modulating these at intervals with the aim of gaining as much lift as possible. A navigational strategy was determined solely from the glider's pooled experiences, collected over several days in the field. The strategy relies on on-board methods to accurately estimate the local vertical wind accelerations and the roll-wise torques on the glider, which serve as navigational cues. We establish the validity of our learned flight policy through field experiments, numerical simulations and estimates of the noise in measurements caused by atmospheric turbulence. Our results highlight the role of vertical wind accelerations and roll-wise torques as effective mechanosensory cues for soaring birds and provide a navigational strategy that is directly applicable to the development of autonomous soaring vehicles.

Epistasis and evolution: recent advances and an outlook for prediction.

As organisms evolve, the effects of mutations change as a result of epistatic interactions with other mutations accumulated along the line of descent. This can lead to shifts in adaptability or robustness that ultimately shape subsequent evolution. Here, we review recent advances in measuring, modeling, and predicting epistasis along evolutionary trajectories, both in microbial cells and single proteins. We focus on simple patterns of global epistasis that emerge in this data, in which the effects of mutations can be predicted by a small number of variables. The emergence of these patterns offers promise for efforts to model epistasis and predict evolution.

Sticky issues in turbulent transport.

We investigate the role of partial stickiness among particles or with a surface for turbulent transport. For the former case, we re-derive known results for the case of the compressible Kraichnan model by using a method based on bi-orthogonality for the expansion of the propagator in terms of left and right eigenvectors. In particular, we show that enforcing the constraints of orthogonality and normalization yields results that were previously obtained by a rigorous, yet possibly less intuitive method. For the latter case, we introduce a general model of transport within the atmospheric boundary layer. As suggested by experimental observations on the transport of atmospheric tracers, both drift and diffusivity scale with the height to the ground. The strength of the drift is parameterized by a velocity V. We use the bi-orthogonality method to show that for V in the range -1 < V < 0 and 0 < V < 1 there is a one-parameter family of boundary conditions that are a priori admissible. Outside of that range, there is a single boundary condition that is admissible. In physical terms, the one-parameter family is parametrized by the degree to which particles stick to the ground.

A lexical approach for identifying behavioural action sequences.

Animals display characteristic behavioural patterns when performing a task, such as the spiraling of a soaring bird or the surge-and-cast of a male moth searching for a female. Identifying such recurring sequences occurring rarely in noisy behavioural data is key to understanding the behavioural response to a distributed stimulus in unrestrained animals. Existing models seek to describe the dynamics of behaviour or segment individual locomotor episodes rather than to identify the rare and transient sequences of locomotor episodes that make up the behavioural response. To fill this gap, we develop a lexical, hierarchical model of behaviour. We designed an unsupervised algorithm called "BASS" to efficiently identify and segment recurring behavioural action sequences transiently occurring in long behavioural recordings. When applied to navigating larval zebrafish, BASS extracts a dictionary of remarkably long, non-Markovian sequences consisting of repeats and mixtures of slow forward and turn bouts. Applied to a novel chemotaxis assay, BASS uncovers chemotactic strategies deployed by zebrafish to avoid aversive cues consisting of sequences of fast large-angle turns and burst swims. In a simulated dataset of soaring gliders climbing thermals, BASS finds the spiraling patterns characteristic of soaring behaviour. In both cases, BASS succeeds in identifying rare action sequences in the behaviour deployed by freely moving animals. BASS can be easily incorporated into the pipelines of existing behavioural analyses across diverse species, and even more broadly used as a generic algorithm for pattern recognition in low-dimensional sequential data.

Effect of percutaneous paravalvular leak closure on hemolysis.

OBJECTIVE: To study the effect of percutaneous paravalvular leak closure on hemolysis. BACKGROUND: Although transcatheter PVL closure reduces heart failure and mortality in symptomatic patients with paravalvular leaks (PVL), little is known about its effect on hemolysis. METHODS: We retrospectively analyzed patients undergoing transcatheter mitral or aortic PVL closure (January 2005-December 2016) at Mayo Clinic. Patients with anemia or abnormal hemolysis markers (LDH, haptoglobin) were included in the analysis. The primary outcome was defined as hemoglobin increase ≥ 1.5 mg/dL, decrease in LDH above median or improvement in haptoglobin. Univariate and multivariate binary logistic regression modeling were used to determine predictors of successful treatment of hemolysis. RESULTS: Final study population included 168 patients (130 [77%] mitral, 38 [23%] aortic PVL). Primary outcome occurred in 70 patients (42%). Hemoglobin increased by 1.74 ± 1.69 mg/dL in patients who reached primary outcome. 57/168 (34%) patients required blood transfusion prior to PVL closure compared to 35/168 (21%) postprocedure. The mean reduction in LDH was 403 U/L. Multivariate regression showed that patients with mechanical valves were more likely to have successful outcome (P = 0.044). CONCLUSION: Percutaneous PVL closure is associated with modest improvement in hemolysis markers, increase in hemoglobin levels and reduction in blood transfusion requirements. This benefit is most significant in patients with mechanical valves.

Learning to soar in turbulent environments.

Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a long-term strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days. The formation of thermals unavoidably generates strong turbulent fluctuations, which constitute an essential element of soaring. Here, we approach soaring flight as a problem of learning to navigate complex, highly fluctuating turbulent environments. We simulate the atmospheric boundary layer by numerical models of turbulent convective flow and combine them with model-free, experience-based, reinforcement learning algorithms to train the gliders. For the learned policies in the regimes of moderate and strong turbulence levels, the glider adopts an increasingly conservative policy as turbulence levels increase, quantifying the degree of risk affordable in turbulent environments. Reinforcement learning uncovers those sensorimotor cues that permit effective control over soaring in turbulent environments.

Transcatheter aortic valve replacement for stenotic bicuspid aortic valves: Systematic review and meta analyses of observational studies.

OBJECTIVE: The aim of this study was to perform a systematic review and meta-analyses of observational studies of transcatheter aortic valve replacement (TAVR) for bicuspid aortic valve stenosis (BcAV). BACKGROUND: TAVR for BcAV stenosis has been associated with an increased incidence of paravalvular leaks, valve malposition, pacemaker placement and all-cause mortality. The conclusions drawn have been limited by small sample sizes. The use of TAVR for BcAV stenosis remains controversial. METHODS: We searched multiple databases from the inception of the databases through September 30, 2016 for studies of TAVR for BcAV stenosis. We included all observational studies with more than one patient and at least 1 month of outcomes. RESULTS: We analyzed 13 observational studies with 758 patients. Meta analyses showed device success rate of 95% [95% confidence interval (CI) 90.2% to 98.5%] and an early safety event in 16.9% [95% CI 12.2% to 22%]. At 30 days, moderate to severe paravalvular leak was seen in 12.2% [95% CI 3.1% to 24.8%] and new pacemaker implantation in 17.9% [95% CI 14.2% to 22%]. All-cause mortality was 3.7% [95% CI 2.1% to 5.6%], which should be viewed in the context of an STS PROM of 5.0%. CONCLUSIONS: This analysis suggests that TAVR for BcAV is not associated with excess mortality. The incidence of paravalvular leaks and pacemaker implant is increased compared to tricuspid aortic valve cohorts undergoing TAVR, and operators should weigh these potential complications against the clinical benefit provided by TAVR for BcAV patients at high risk for surgical valve replacement.

Percutaneous valvuloplasty for severe bioprosthetic tricuspid valve stenosis in the setting of infective endocarditis.

Bioprosthetic tricuspid valve stenosis as a sequela of infective endocarditis is extremely rare. We describe the case of a 29-year-old male patient with a history of intravenous drug use and two previous bioprosthetic tricuspid valve placements who presented with recurrent endocarditis and severe tricuspid stenosis. He was deemed extremely high risk for redo valve replacement surgery. Intracardiac ultrasound-guided balloon valvuloplasty was performed with good clinical outcome. We believe that interventional treatment of prosthetic valvular stenosis in the setting of endocarditis is a reasonable therapeutic choice when open surgical repair is associated with prohibitively high mortality. This can be performed either as destination therapy or as a bridge to valve replacement. The use of intracardiac ultrasound provided additive information to that obtained by transthoracic and transesophageal echocardiography.

Distinct information conveyed to the olfactory bulb by feedforward input from the nose and feedback from the cortex.

Sensory systems are organized hierarchically, but feedback projections frequently disrupt this order. In the olfactory bulb (OB), cortical feedback projections numerically match sensory inputs. To unravel information carried by these two streams, we imaged the activity of olfactory sensory neurons (OSNs) and cortical axons in the mouse OB using calcium indicators, multiphoton microscopy, and diverse olfactory stimuli. Here, we show that odorant mixtures of increasing complexity evoke progressively denser OSN activity, yet cortical feedback activity is of similar sparsity for all stimuli. Also, representations of complex mixtures are similar in OSNs but are decorrelated in cortical axons. While OSN responses to increasing odorant concentrations exhibit a sigmoidal relationship, cortical axonal responses are complex and nonmonotonic, which can be explained by a model with activity-dependent feedback inhibition in the cortex. Our study indicates that early-stage olfactory circuits have access to local feedforward signals and global, efficiently formatted information about odor scenes through cortical feedback.

Inferring sparse structure in genotype-phenotype maps.

Correlation among multiple phenotypes across related individuals may reflect some pattern of shared genetic architecture: individual genetic loci affect multiple phenotypes (an effect known as pleiotropy), creating observable relationships between phenotypes. A natural hypothesis is that pleiotropic effects reflect a relatively small set of common "core" cellular processes: each genetic locus affects one or a few core processes, and these core processes in turn determine the observed phenotypes. Here, we propose a method to infer such structure in genotype-phenotype data. Our approach, sparse structure discovery (SSD) is based on a penalized matrix decomposition designed to identify latent structure that is low-dimensional (many fewer core processes than phenotypes and genetic loci), locus-sparse (each locus affects few core processes), and/or phenotype-sparse (each phenotype is influenced by few core processes). Our use of sparsity as a guide in the matrix decomposition is motivated by the results of a novel empirical test indicating evidence of sparse structure in several recent genotype-phenotype datasets. First, we use synthetic data to show that our SSD approach can accurately recover core processes if each genetic locus affects few core processes or if each phenotype is affected by few core processes. Next, we apply the method to three datasets spanning adaptive mutations in yeast, genotoxin robustness assay in human cell lines, and genetic loci identified from a yeast cross, and evaluate the biological plausibility of the core process identified. More generally, we propose sparsity as a guiding prior for resolving latent structure in empirical genotype-phenotype maps.

Adaptive algorithms for shaping behavior.

Dogs and laboratory mice are commonly trained to perform complex tasks by guiding them through a curriculum of simpler tasks ('shaping'). What are the principles behind effective shaping strategies? Here, we propose a machine learning framework for shaping animal behavior, where an autonomous teacher agent decides its student's task based on the student's transcript of successes and failures on previously assigned tasks. Using autonomous teachers that plan a curriculum in a common sequence learning task, we show that near-optimal shaping algorithms adaptively alternate between simpler and harder tasks to carefully balance reinforcement and extinction. Based on this intuition, we derive an adaptive shaping heuristic with minimal parameters, which we show is near-optimal on the sequence learning task and robustly trains deep reinforcement learning agents on navigation tasks that involve sparse, delayed rewards. Extensions to continuous curricula are explored. Our work provides a starting point towards a general computational framework for shaping animal behavior.

Preservation of biological function despite oxidative modification of the apolipoprotein A-I mimetic peptide 4F.

Myeloperoxidase (MPO)-derived hypochlorous acid induces changes in HDL function via redox modifications at the level of apolipoprotein A-I (apoA-I). As 4F and apoA-I share structural and functional properties, we tested the hypothesis that 4F acts as a reactive substrate for hypochlorous acid (HOCl). 4F reduced the HOCl-mediated oxidation of the fluorescent substrate APF in a concentration-dependent manner (ED(50) ∼ 56 ± 3 µM). This reaction induced changes in the physical properties of 4F. Addition of HOCl to 4F at molar ratios ranging from 1:1 to 3:1 reduced 4F band intensity on SDS-PAGE gels and was accompanied by the formation of a higher molecular weight species. Chromatographic studies showed a reduction in 4F peak area with increasing HOCl and the formation of new products. Mass spectral analyses of collected fractions revealed oxidation of the sole tryptophan (Trp) residue in 4F. 4F was equally susceptible to oxidation in the lipid-free and lipid-bound states. To determine whether Trp oxidation influenced its apoA-I mimetic properties, we monitored effects of HOCl on 4F-mediated lipid binding and ABCA1-dependent cholesterol efflux. Neither property was altered by HOCl. These results suggest that 4F serves as a reactive substrate for HOCl, an antioxidant response that does not influence the lipid binding and cholesterol effluxing capacities of the peptide.

Alternation emerges as a multi-modal strategy for turbulent odor navigation.

Foraging mammals exhibit a familiar yet poorly characterized phenomenon, 'alternation', a pause to sniff in the air preceded by the animal rearing on its hind legs or raising its head. Rodents spontaneously alternate in the presence of airflow, suggesting that alternation serves an important role during plume-tracking. To test this hypothesis, we combine fully resolved simulations of turbulent odor transport and Bellman optimization methods for decision-making under partial observability. We show that an agent trained to minimize search time in a realistic odor plume exhibits extensive alternation together with the characteristic cast-and-surge behavior observed in insects. Alternation is linked with casting and occurs more frequently far downwind of the source, where the likelihood of detecting airborne cues is higher relative to ground cues. Casting and alternation emerge as complementary tools for effective exploration with sparse cues. A model based on marginal value theory captures the interplay between casting, surging, and alternation.

Outcomes of Allograft Medial Patellofemoral Ligament Reconstruction in Children and Adolescents with Hypermobility.

Introduction: Medial patellofemoral ligament (MPFL) reconstruction is used to treat patellofemoral instability either in isolation or in combination with other procedures. Use of allograft can preserve native tissue in children and can be advantageous in patients with connective tissue disorders, including ligamentous laxity. There is limited evidence regarding functional outcomes of allograft MPFL reconstruction in children and adolescents. This study aimed to assess the short to mid-term results of allograft MPFL reconstruction in children with hypermobility at a tertiary pediatric orthopedic center. Materials and Methods: We retrospectively reviewed all children and adolescents who had undergone allograft MPFL reconstruction over 4 years. The primary outcome measure was the validated Kujala score for patellofemoral disorders. The secondary outcome measures included complications such as redislocation of the patella needing revision surgery. Patients with hypermobility were quantified using Beighton criteria. Statistical analysis was performed using Graph Pad Prism (V6). Results: Between 2012 and 2016, the senior author performed 76 allograft MPFL reconstructions in 57 patients. Nineteen patients had bilateral surgery. The mean age was 14 (7-16) years with a female: male ratio of 3:1. The mean Beighton score was 7. Hypermobility was part of a syndrome in ten patients. The mean follow-up was 3 (1-4) years. Nine patients had trochleoplasty and six patients had tibial tubercle osteotomy, in addition to allograft MPFL reconstruction. These fifteen patients, who had additional procedures, were excluded during the analysis of the outcome measures. The mean Kujala score was 89 (80-100). The overall complication rate was 11% (9/76). These included two patella fractures and seven (9%) patients with recurrent instability needing revision surgery. There was no significant difference in complication rates between syndromic and nonsyndromic patients (P = 0.9). Conclusion: Our study shows excellent short to mid-term functional outcomes of allograft MPFL reconstruction in children and adolescents with hypermobility.