Dense Sample Deep Learning.

Deep learning (DL), a variant of the neural network algorithms originally proposed in the 1980s (Rumelhart et al., 1986), has made surprising progress in artificial intelligence (AI), ranging from language translation, protein folding (Jumper et al., 2021), autonomous cars, and, more recently, human-like language models (chatbots). All that seemed intractable until very recently. Despite the growing use of DL networks, little is understood about the learning mechanisms and representations that make these networks effective across such a diverse range of applications. Part of the answer must be the huge scale of the architecture and, of course, the large scale of the data, since not much has changed since 1986. But the nature of deep learned representations remains largely unknown. Unfortunately, training sets with millions or billions of tokens have unknown combinatorics, and networks with millions or billions of hidden units can't easily be visualized and their mechanisms can't be easily revealed. In this letter, we explore these challenges with a large (1.24 million weights VGG) DL in a novel high-density sample task (five unique tokens with more than 500 exemplars per token), which allows us to more carefully follow the emergence of category structure and feature construction. We use various visualization methods for following the emergence of the classification and the development of the coupling of feature detectors and structures that provide a type of graphical bootstrapping. From these results, we harvest some basic observations of the learning dynamics of DL and propose a new theory of complex feature construction based on our results.

Causally informed activity flow models provide mechanistic insight into network-generated cognitive activations.

Brain activity flow models estimate the movement of task-evoked activity over brain connections to help explain network-generated task functionality. Activity flow models have been shown to accurately generate task-evoked brain activations across a wide variety of brain regions and task conditions. However, these models have had limited explanatory power, given known issues with causal interpretations of the standard functional connectivity measures used to parameterize activity flow models. We show here that functional/effective connectivity (FC) measures grounded in causal principles facilitate mechanistic interpretation of activity flow models. We progress from simple to complex FC measures, with each adding algorithmic details reflecting causal principles. This reflects many neuroscientists' preference for reduced FC measure complexity (to minimize assumptions, minimize compute time, and fully comprehend and easily communicate methodological details), which potentially trades off with causal validity. We start with Pearson correlation (the current field standard) to remain maximally relevant to the field, estimating causal validity across a range of FC measures using simulations and empirical fMRI data. Finally, we apply causal-FC-based activity flow modeling to a dorsolateral prefrontal cortex region (DLPFC), demonstrating distributed causal network mechanisms contributing to its strong activation during a working memory task. Notably, this fully distributed model is able to account for DLPFC working memory effects traditionally thought to rely primarily on within-region (i.e., not distributed) recurrent processes. Together, these results reveal the promise of parameterizing activity flow models using causal FC methods to identify network mechanisms underlying cognitive computations in the human brain.

Experimental evidence for a child-to-adolescent switch in human amygdala-prefrontal cortex communication: A cross-sectional pilot study.

Interactions between the amygdala and prefrontal cortex are fundamental to human emotion. Despite the central role of frontoamygdala communication in adult emotional learning and regulation, little is known about how top-down control emerges during human development. In the present cross-sectional pilot study, we experimentally manipulated prefrontal engagement to test its effects on the amygdala during development. Inducing dorsal anterior cingulate cortex (dACC) activation resulted in developmentally-opposite effects on amygdala reactivity during childhood versus adolescence, such that dACC activation was followed by increased amygdala reactivity in childhood but reduced amygdala reactivity in adolescence. Bayesian network analyses revealed an age-related switch between childhood and adolescence in the nature of amygdala connectivity with the dACC and ventromedial PFC (vmPFC). Whereas adolescence was marked by information flow from dACC and vmPFC to amygdala (consistent with that observed in adults), the reverse information flow, from the amygdala to dACC and vmPFC, was dominant in childhood. The age-related switch in information flow suggests a potential shift from bottom-up co-excitatory to top-down regulatory frontoamygdala connectivity and may indicate a profound change in the circuitry supporting maturation of emotional behavior. These findings provide novel insight into the developmental construction of amygdala-cortical connections and implications for the ways in which childhood experiences may influence subsequent prefrontal function.

The Stochastic Delta Rule: Faster and More Accurate Deep Learning Through Adaptive Weight Noise.

Multilayer neural networks have led to remarkable performance on many kinds of benchmark tasks in text, speech, and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (e.g., saddle points, colinearity, feature discovery) is called Dropout. The Dropout algorithm removes hidden units according to a binomial random variable with probability p prior to each update, creating random "shocks" to the network that are averaged over updates (thus creating weight sharing). In this letter, we reestablish an older parameter search method and show that Dropout is a special case of this more general model, stochastic delta rule (SDR), published originally in 1990. Unlike Dropout, SDR redefines each weight in the network as a random variable with mean µwij and standard deviation σwij. Each weight random variable is sampled on each forward activation, consequently creating an exponential number of potential networks with shared weights (accumulated in the mean values). Both parameters are updated according to prediction error, thus resulting in weight noise injections that reflect a local history of prediction error and local model averaging. SDR therefore implements a more sensitive local gradient-dependent simulated annealing per weight converging in the limit to a Bayes optimal network. We run tests on standard benchmarks (CIFAR and ImageNet) using a modified version of DenseNet and show that SDR outperforms standard Dropout in top-5 validation error by approximately 13% with DenseNet-BC 121 on ImageNet and find various validation error improvements in smaller networks. We also show that SDR reaches the same accuracy that Dropout attains in 100 epochs in as few as 40 epochs, as well as improvements in training error by as much as 80%.

Back to the future: The return of cognitive functionalism.

The claims that learning systems must build causal models and provide explanations of their inferences are not new, and advocate a cognitive functionalism for artificial intelligence. This view conflates the relationships between implicit and explicit knowledge representation. We present recent evidence that neural networks do engage in model building, which is implicit, and cannot be dissociated from the learning process.

Maraba virus as a potent oncolytic vaccine vector.

The rhabdovirus Maraba has recently been characterized as a potent oncolytic virus. In the present study, we engineered an attenuated Maraba strain, defined as MG1, to express a melanoma-associated tumor antigen. Its ability to mount an antitumor immunity was evaluated in tumor-free and melanoma tumor-bearing mice. Alone, the MG1 vaccine appeared insufficient to prime detectable adaptive immunity against the tumor antigen. However, when used as a boosting vector in a heterologous prime-boost regimen, MG1 vaccine rapidly generated strong antigen-specific T-cell immune responses. Once applied for treating syngeneic murine melanoma tumors, our oncolytic prime-boost vaccination protocol involving Maraba MG1 dramatically extended median survival and allowed complete remission in more than 20% of the animals treated. This work describes Maraba virus MG1 as a potent vaccine vector for cancer immunotherapy displaying both oncolytic activity and a remarkable ability to boost adaptive antitumor immunity.

Brain network response underlying decisions about abstract reinforcers.

Decision making studies typically use tasks that involve concrete action-outcome contingencies, in which subjects do something and get something. No studies have addressed decision making involving abstract reinforcers, where there are no action-outcome contingencies and choices are entirely hypothetical. The present study examines these kinds of choices, as well as whether the same biases that exist for concrete reinforcer decisions, specifically framing effects, also apply during abstract reinforcer decisions. We use both General Linear Model as well as Bayes network connectivity analysis using the Independent Multi-sample Greedy Equivalence Search (IMaGES) algorithm to examine network response underlying choices for abstract reinforcers under positive and negative framing. We find for the first time that abstract reinforcer decisions activate the same network of brain regions as concrete reinforcer decisions, including the striatum, insula, anterior cingulate, and VMPFC, results that are further supported via comparison to a meta-analysis of decision making studies. Positive and negative framing activated different parts of this network, with stronger activation in VMPFC during negative framing and in DLPFC during positive, suggesting different decision making pathways depending on frame. These results were further clarified using connectivity analysis, which revealed stronger connections between anterior cingulate, insula, and accumbens during negative framing compared to positive. Taken together, these results suggest that not only do abstract reinforcer decisions rely on the same brain substrates as concrete reinforcers, but that the response underlying framing effects on abstract reinforcers also resemble those for concrete reinforcers, specifically increased limbic system connectivity during negative frames.

Modeling activation and effective connectivity of VWFA in same script bilinguals.

Previous neuroimaging research revealed a small area in the inferior occipito-temporal cortex (VWFA), which seems to be involved in recognition of written words. The specialized response of the VWFA to words could result from repeated exposure to print in the course of functional fine-tuning of the brain. Research with bilingual speakers holds promise in helping to reveal response properties of the VWFA by assessing its sensitivity to language proficiency, word-form similarity, and meaning overlap across two languages. Using fMRI, we compared VWFA activity for cognate and homograph prime-target pairs in a group of fluent Spanish-English speakers. Cognates share form and meaning in two languages, while homographs only share form. Relative to baseline, the VWFA showed repetition suppression to pairs of homographs, but not to pairs of cognates, suggesting that this area is sensitive to word meaning. The different response to cognates and homographs was only observed when English was the prime language and Spanish was the target language. To help explain this result we compared patterns of effective connectivity between the VWFA and other parts of the reading network implicated in semantic and phonological processing. Our neural models showed that English targets engaged a direct ventral route from the VWFA to the frontal lobe and Spanish targets engaged an indirect dorsal route. Considering that frontal cortex has been implicated in semantic processing, a direct connection to this area could signal a fast and automatic access to meaning and would facilitate early semantic influences in visual word recognition.

Thrombotic responses of endothelial outgrowth cells to protein-coated surfaces.

There is significant clinical need for viable small-diameter vascular grafts. While there are many graft biomaterials in development, few have been clinically successful. Evaluation of grafts with a clinically relevant model is needed to drive development. This work examined extracellular matrix coatings on the thrombotic phenotype of endothelial outgrowth cells (EOCs). EOCs were tested on flat plates and tubular grafts. Flat plate studies examined collagen I, collagen IV, fibronectin and α-elastin coatings. EOCs attached or proliferated more readily on collagen I and fibronectin surfaces as determined by total DNA. The production of activated protein C (APC) by EOCs was also dependent on the surface coating, with collagen I and fibronectin displaying a higher activity than both collagen IV and α-elastin on flat plate studies. Based on these results, only collagen I and fibronectin coatings were tested on expanded polytetrafluoroethylene (ePTFE) in the ex vivo model. Tubular samples showed significantly greater tissue factor pathway inhibitor gene expression on collagen I than on fibronectin. Platelet adhesion was not significantly different, but EOCs on collagen I produced significantly lower APC than on fibronectin, suggesting that differences exist between the flat plate and tubular cultures. Overall, while the hemostatic phenotype of EOCs displayed some differences, cell responses were largely independent of the matrix coating. EOCs adhered strongly to both fibronectin- and collagen-I-coated ePTFE grafts under ex vivo (100 ml/min) flow conditions suggesting the usefulness of this clinically relevant cell source, testing modality, and shunt model for future work examining biomaterials and cell conditioning before implantation.

Antithrombotic effect of antisense factor XI oligonucleotide treatment in primates.

OBJECTIVE: During coagulation, factor IX (FIX) is activated by 2 distinct mechanisms mediated by the active proteases of either FVIIa or FXIa. Both coagulation factors may contribute to thrombosis; FXI, however, plays only a limited role in the arrest of bleeding. Therefore, therapeutic targeting of FXI may produce an antithrombotic effect with relatively low hemostatic risk. APPROACH AND RESULTS: We have reported that reducing FXI levels with FXI antisense oligonucleotides produces antithrombotic activity in mice, and that administration of FXI antisense oligonucleotides to primates decreases circulating FXI levels and activity in a dose-dependent and time-dependent manner. Here, we evaluated the relationship between FXI plasma levels and thrombogenicity in an established baboon model of thrombosis and hemostasis. In previous studies with this model, antibody-induced inhibition of FXI produced potent antithrombotic effects. In the present article, antisense oligonucleotides-mediated reduction of FXI plasma levels by ≥ 50% resulted in a demonstrable and sustained antithrombotic effect without an increased risk of bleeding. CONCLUSIONS: These results indicate that reducing FXI levels using antisense oligonucleotides is a promising alternative to direct FXI inhibition, and that targeting FXI may be potentially safer than conventional antithrombotic therapies that can markedly impair primary hemostasis.

Dynamic changes in the medial temporal lobe during incidental learning of object-location associations.

The role of the medial temporal lobe (MTL) in associative memory encoding has been the focus of many memory experiments. However, there has been surprisingly little investigation of whether the contributions of different MTL subregions (amygdala, hippocampus [HPC], parahippocampal [PHc], perirhinal cortex [PRc], and temporal polar cortex [TPc]) shift across multiple presentations during associative encoding. We examined this issue using event-related functional magnetic resonance imaging and a multivoxel pattern classification analysis. Subjects performed a visual search task, becoming faster with practice to locate objects whose locations were held constant across trials. The classification analysis implicated right HPC and amygdala early in the task when the speed-up from trial to trial was greatest. The same analysis implicated right PRc and TPc late in learning when speed-up was minimal. These results suggest that associative encoding relies on complex patterns of neural activity in MTL that cannot be expressed by simple increases or decreases of blood oxygenation level-dependent signal during learning. Involvement of MTL subregions during encoding of object-location associations depends on the nature of the learning phase. Right HPC and amygdala support active integration of object and location information, while right PRc and TPc are involved when object and spatial representations become unitized into a single representation.

Capture of circulatory endothelial progenitor cells and accelerated re-endothelialization of a bio-engineered stent in human ex vivo shunt and rabbit denudation model.

AIMS: The Genous™ Bio-engineered R™ stent (GS) aims to promote vascular healing by capture of circulatory endothelial progenitor cells (EPCs) to the surface of the stent struts, resulting in accelerated re-endothelialization. Here, we assessed the function of the GS in comparison to bare-metal stent (BMS), when exposed to the human and animal circulation. METHODS AND RESULTS: First, 15 patients undergoing coronary angiography received an extracorporeal femoral arteriovenous (AV) shunt containing BMS and GS. Macroscopical mural thrombi were observed in BMS, whereas GS remained visibly clean. Confocal and scanning electron microscopic (SEM) analysis of GS showed an increase in strut coverage. Quantitative polymerase chain reaction (qPCR) analysis of captured cells on the GS demonstrated increased expression of endothelial markers KDR/VEGFR2 and E-selectin, and a decrease in pro-thrombogenic markers tissue factor pathway inhibitor and plasminogen activator inhibitor-1 compared with BMS. Secondly, a similar primate AV shunt model was used to validate these findings and occlusion of BMS was observed, while GS remained patent, as demonstrated by live imaging of indium-labelled platelets. Thirdly, in an in vitro cell-capture assay, GS struts showed increased coverage by EPCs, whereas monocyte coverage remained similar to BMS. Finally, the assessment of re-endothelialization was studied in a rabbit denudation model. Twenty animals received BMS and GS in the aorta and iliac arteries for 7 days. Scanning electron microscopic analysis showed a trend towards increased strut coverage, confirmed by qPCR analysis revealing increased levels of endothelial markers (Tie2, CD34, PCD31, and P-selectin) in GS. CONCLUSION: In this proof-of-concept study, we have demonstrated that the bio-engineered EPC-capture stent, Genous™ R™ stent, is effective in EPC capture, resulting in accelerated re-endothelialization and reduced thrombogenicity.

Phylogenetic analysis of the alfalfa weevil complex (Coleoptera: Curculionidae) in North America.

The Eastern, Western, and Egyptian strains of alfalfa weevil are pests introduced to North America on three separate occasions, now they share partially overlapping geographic ranges, covering most of the continental United States. Behavior, susceptibility to parasites, and subtle morphological differences separate the strains. The difficulty in differentiating among these strains morphologically has led to the application of molecular phylogeny approaches including restriction fragment-length polymorphism characterization and sequencing of mitochondrial genes. While valuable for strain identification, this approach cannot identify interstrain hybrids because mitochondrial markers are maternally inherited. The work reported here extends previous findings by comparing over 7 Kb of sequence from two mitochondrial and four nuclear loci to increase the resolution of molecular phylogeny for these weevils. The related clover leaf weevil, also an occasional pest of alfalfa, was included in the analysis because the molecular phylogeny of this weevil has not been examined to date. Analysis of nuclear loci indicate that the clover weevil is a distinct species. Furthermore, while the three alfalfa weevil strains are separable based on mitochondrial sequence data they cannot be separated using nuclearloci suggesting that they are all recently diverged members of the same species. These data refine the relationships among these strains and may find application in design of better control strategies.

A role for factor XIIa-mediated factor XI activation in thrombus formation in vivo.

Mice lacking factor XII (fXII) or factor XI (fXI) are resistant to experimentally-induced thrombosis, suggesting fXIIa activation of fXI contributes to thrombus formation in vivo. It is not clear whether this reaction has relevance for thrombosis in pri mates. In 2 carotid artery injury models (FeCl(3) and Rose Bengal/laser), fXII-deficient mice are more resistant to thrombosis than fXI- or factor IX (fIX)-deficient mice, raising the possibility that fXII and fXI function in distinct pathways. Antibody 14E11 binds fXI from a variety of mammals and interferes with fXI activation by fXIIa in vitro. In mice, 14E11 prevented arterial occlusion induced by FeCl(3) to a similar degree to total fXI deficiency. 14E11 also had a modest beneficial effect in a tissue factor-induced pulmonary embolism model, indicating fXI and fXII contribute to thrombus formation even when factor VIIa/tissue factor initiates thrombosis. In baboons, 14E11 reduced platelet-rich thrombus growth in collagen-coated grafts inserted into an arteriovenous shunt. These data support the hypothesis that fXIIa-mediated fXI activation contributes to thrombus formation in rodents and primates. Since fXII deficiency does not impair hemostasis, targeted inhibition of fXI activation by fXIIa may be a useful antithrombotic strategy associated with a low risk of bleeding complications.

Latent functional connectivity underlying multiple brain states.

Functional connectivity (FC) studies have predominantly focused on resting state, where ongoing dynamics are thought to reflect the brain's intrinsic network architecture, which is thought to be broadly relevant because it persists across brain states (i.e., is state-general). However, it is unknown whether resting state is the optimal state for measuring intrinsic FC. We propose that latent FC, reflecting shared connectivity patterns across many brain states, better captures state-general intrinsic FC relative to measures derived from resting state alone. We estimated latent FC independently for each connection using leave-one-task-out factor analysis in seven highly distinct task states (24 conditions) and resting state using fMRI data from the Human Connectome Project. Compared with resting-state connectivity, latent FC improves generalization to held-out brain states, better explaining patterns of connectivity and task-evoked activation. We also found that latent connectivity improved prediction of behavior outside the scanner, indexed by the general intelligence factor (g). Our results suggest that FC patterns shared across many brain states, rather than just resting state, better reflect state-general connectivity. This affirms the notion of "intrinsic" brain network architecture as a set of connectivity properties persistent across brain states, providing an updated conceptual and mathematical framework of intrinsic connectivity as a latent factor.

High-resolution imaging of the fusiform face area (FFA) using multivariate non-linear classifiers shows diagnosticity for non-face categories.

Does the "fusiform face area" (FFA) code only for faces? This question continues to elude the neuroimaging field due to at least two kinds of problems: first, the relatively low spatial resolution of fMRI in which the FFA was defined and second, the potential bias inherent in prevailing statistical methods for analyzing the actual diagnosticity of cortical tissue. Using high-resolution (1 mm × 1 mm × 1 mm) imaging data of the fusiform face area (FFA) from 4 subjects who had categorized images as 'animal', 'car', 'face', or 'sculpture', we used multivariate linear and non-linear classifiers to decode the resultant voxel patterns. Prior to identifying the appropriate classifier we performed exploratory analysis to determine the nature of the distributions over classes and the voxel intensity pattern structure between classes. The FFA was visualized using non-metric multidimensional scaling revealing "string-like" sequences of voxels, which appeared in small non-contiguous clusters of categories, intertwined with other categories. Since this analysis suggested that feature space was highly non-linear, we trained various statistical classifiers on the class-conditional distributions (labelled) and separated the four categories with 100% reliability (over replications) and generalized to out-of-sample cases with high significance (up to 50%; p<.000001, chance=25%). The increased noise inherent in high-resolution neuroimaging data relative to standard resolution resisted any further gains in category performance above ~60% (with FACE category often having the highest bias per category) even coupled with various feature extraction/selection methods. A sensitivity/diagnosticity analysis for each classifier per voxel showed: (1) reliable (with S.E.<3%) sensitivity present throughout the FFA for all 4 categories, and (2) showed multi-selectivity; that is, many voxels were selective for more than one category with some high diagnosticity but at submaximal intensity. This work is clearly consistent with the characterization of the FFA as a distributed, object-heterogeneous similarity structure and bolsters the view that the FFA response to "FACE" stimuli in standard resolution may be primarily due to a linear bias, which has resulted from an averaging artefact.

Multi-subject search correctly identifies causal connections and most causal directions in the DCM models of the Smith et al. simulation study.

Smith et al. report a large study of the accuracy of 38 search procedures for recovering effective connections in simulations of DCM models under 28 different conditions. Their results are disappointing: no method reliably finds and directs connections without large false negatives, large false positives, or both. Using multiple subject inputs, we apply a previously published search algorithm, IMaGES, and novel orientation algorithms, LOFS, in tandem to all of the simulations of DCM models described by Smith et al. (2011). We find that the procedures accurately identify effective connections in almost all of the conditions that Smith et al. simulated and, in most conditions, direct causal connections with precision greater than 90% and recall greater than 80%.