SPD Manifold Deep Metric Learning for Image Set Classification.

By characterizing each image set as a nonsingular covariance matrix on the symmetric positive definite (SPD) manifold, the approaches of visual content classification with image sets have made impressive progress. However, the key challenge of unhelpfully large intraclass variability and interclass similarity of representations remains open to date. Although, several recent studies have mitigated the two problems by jointly learning the embedding mapping and the similarity metric on the original SPD manifold, their inherent shallow and linear feature transformation mechanism are not powerful enough to capture useful geometric features, especially in complex scenarios. To this end, this article explores a novel approach, termed SPD manifold deep metric learning (SMDML), for image set classification. Specifically, SMDML first selects a prevailing SPD manifold neural network (SPDNet) as the backbone (encoder) to derive an SPD matrix nonlinear representation. To counteract the degradation of structural information during multistage feature embedding, we construct a Riemannian decoder at the end of the encoder, trained by a reconstruction error term (RT), to induce the generated low-dimensional feature manifold of the hidden layer to capture the pivotal information about the visual data describing the imaged scene. We demonstrate through theory and experiments that it is feasible to replace the Riemannian metric with Euclidean distance in RT. Then, the ReCov layer is introduced into the established Riemannian network to regularize the local statistical information within each input feature matrix, which enhances the effectiveness of the learning process. The theoretical analysis of the activation function used in the ReCov layer in terms of continuity and conditions for generating positive definite matrices is beneficial for network design. Inspired by the fact that the single cross-entropy loss used for training is unable to effectively parse the geometric distribution of the deep representations, we finally endow the suggested model with a novel metric learning regularization term. By explicitly incorporating the encoding and processing of the data variations into the network learning process, this term can not only derive a powerful Riemannian representation but also train an effective classifier. The experimental results show the superiority of the proposed approach on three typical visual classification tasks.

Shared structural features of Miro binding control mitochondrial homeostasis.

Miro proteins are universally conserved mitochondrial calcium-binding GTPases that regulate a multitude of mitochondrial processes, including transport, clearance, and lipid trafficking. The exact role of Miro in these functions is unclear but involves binding to a variety of client proteins. How this binding is operated at the molecular level and whether and how it is important for mitochondrial health, however, remains unknown. Here, we show that known Miro interactors-namely, CENPF, Trak, and MYO19-all use a similar short motif to bind the same structural element: a highly conserved hydrophobic pocket in the first calcium-binding domain of Miro. Using these Miro-binding motifs, we identified direct interactors de novo, including MTFR1/2/1L, the lipid transporters Mdm34 and VPS13D, and the ubiquitin E3-ligase Parkin. Given the shared binding mechanism of these functionally diverse clients and its conservation across eukaryotes, we propose that Miro is a universal mitochondrial adaptor coordinating mitochondrial health.

Importance Weighted Structure Learning for Scene Graph Generation.

Scene graph generation is a structured prediction task aiming to explicitly model objects and their relationships via constructing a visually-grounded scene graph for an input image. Currently, the message passing neural network based mean field variational Bayesian methodology is the ubiquitous solution for such a task, in which the variational inference objective is often assumed to be the classical evidence lower bound. However, the variational approximation inferred from such loose objective generally underestimates the underlying posterior, which often leads to inferior generation performance. In this paper, we propose a novel importance weighted structure learning method aiming to approximate the underlying log-partition function with a tighter importance weighted lower bound, which is computed from multiple samples drawn from a reparameterizable Gumbel-Softmax sampler. A generic entropic mirror descent algorithm is applied to solve the resulting constrained variational inference task. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.

Mitochondrial regulation of local supply of energy in neurons.

Brain computation is metabolically expensive and requires the supply of significant amounts of energy. Mitochondria are highly specialized organelles whose main function is to generate cellular energy. Due to their complex morphologies, neurons are especially dependent on a set of tools necessary to regulate mitochondrial function locally in order to match energy provision with local demands. By regulating mitochondrial transport, neurons control the local availability of mitochondrial mass in response to changes in synaptic activity. Neurons also modulate mitochondrial dynamics locally to adjust metabolic efficiency with energetic demand. Additionally, neurons remove inefficient mitochondria through mitophagy. Neurons coordinate these processes through signalling pathways that couple energetic expenditure with energy availability. When these mechanisms fail, neurons can no longer support brain function giving rise to neuropathological states like metabolic syndromes or neurodegeneration.

Constrained Structure Learning for Scene Graph Generation.

As a structured prediction task, scene graph generation aims to build a visually-grounded scene graph to explicitly model objects and their relationships in an input image. Currently, the mean field variational Bayesian framework is the de facto methodology used by the existing methods, in which the unconstrained inference step is often implemented by a message passing neural network. However, such formulation fails to explore other inference strategies, and largely ignores the more general constrained optimization models. In this paper, we present a constrained structure learning method, for which an explicit constrained variational inference objective is proposed. Instead of applying the ubiquitous message-passing strategy, a generic constrained optimization method - entropic mirror descent - is utilized to solve the constrained variational inference step. We validate the proposed generic model on various popular scene graph generation benchmarks and show that it outperforms the state-of-the-art methods.

Fast Self-Guided Multi-View Subspace Clustering.

Multi-view subspace clustering is an important topic in cluster analysis. Its aim is to utilize the complementary information conveyed by multiple views of objects to be clustered. Recently, view-shared anchor learning based multi-view clustering methods have been developed to speed up the learning of common data representation. Although widely applied to large-scale scenarios, most of the existing approaches are still faced with two limitations. First, they do not pay sufficient consideration on the negative impact caused by certain noisy views with unclear clustering structures. Second, many of them only focus on the multi-view consistency, yet are incapable of capturing the cross-view diversity. As a result, the learned complementary features may be inaccurate and adversely affect clustering performance. To solve these two challenging issues, we propose a Fast Self-guided Multi-view Subspace Clustering (FSMSC) algorithm which skillfully integrates the view-shared anchor learning and global-guided-local self-guidance learning into a unified model. Such an integration is inspired by the observation that the view with clean clustering structures will play a more crucial role in grouping the clusters when the features of all views are concatenated. Specifically, we first learn a locally-consistent data representation shared by all views in the local learning module, then we learn a globally-discriminative data representation from multi-view concatenated features in the global learning module. Afterwards, a feature selection matrix constrained by the l2,1 -norm is designed to construct a guidance from global learning to local learning. In this way, the multi-view consistent and diverse information can be simultaneously utilized and the negative impact caused by noisy views can be overcame to some extent. Extensive experiments on different datasets demonstrate the effectiveness of our proposed fast self-guided learning model, and its promising performance compared to both, the state-of-the-art non-deep and deep multi-view clustering algorithms. The code of this paper is available at https://github.com/chenzhe207/FSMSC.

Discriminative Dictionary Pair Learning With Scale-Constrained Structured Representation for Image Classification.

The dictionary pair learning (DPL) model aims to design a synthesis dictionary and an analysis dictionary to accomplish the goal of rapid sample encoding. In this article, we propose a novel structured representation learning algorithm based on the DPL for image classification. It is referred to as discriminative DPL with scale-constrained structured representation (DPL-SCSR). The proposed DPL-SCSR utilizes the binary label matrix of dictionary atoms to project the representation into the corresponding label space of the training samples. By imposing a non-negative constraint, the learned representation adaptively approximates a block-diagonal structure. This innovative transformation is also capable of controlling the scale of the block-diagonal representation by enforcing the sum of within-class coefficients of each sample to 1, which means that the dictionary atoms of each class compete to represent the samples from the same class. This implies that the requirement of similarity preservation is considered from the perspective of the constraint on the sum of coefficients. More importantly, the DPL-SCSR does not need to design a classifier in the representation space as the label matrix of the dictionary can also be used as an efficient linear classifier. Finally, the DPL-SCSR imposes the l2,p -norm on the analysis dictionary to make the process of feature extraction more interpretable. The DPL-SCSR seamlessly incorporates the scale-constrained structured representation learning, within-class similarity preservation of representation, and the linear classifier into one regularization term, which dramatically reduces the complexity of training and parameter tuning. The experimental results on several popular image classification datasets show that our DPL-SCSR can deliver superior performance compared with the state-of-the-art (SOTA) dictionary learning methods. The MATLAB code of this article is available at https://github.com/chenzhe207/DPL-SCSR.

Towards Robust Visual Object Tracking with Independent Target-Agnostic Detection and Effective Siamese Cross-Task Interaction.

Advanced Siamese visual object tracking architectures are jointly trained using pair-wise input images to perform target classification and bounding box regression. They have achieved promising results in recent benchmarks and competitions. However, the existing methods suffer from two limitations: First, though the Siamese structure can estimate the target state in an instance frame, provided the target appearance does not deviate too much from the template, the detection of the target in an image cannot be guaranteed in the presence of severe appearance variations. Second, despite the classification and regression tasks sharing the same output from the backbone network, their specific modules and loss functions are invariably designed independently, without promoting any interaction. Yet, in a general tracking task, the centre classification and bounding box regression tasks are collaboratively working to estimate the final target location. To address the above issues, it is essential to perform target-agnostic detection so as to promote cross-task interactions in a Siamese-based tracking framework. In this work, we endow a novel network with a target-agnostic object detection module to complement the direct target inference, and to avoid or minimise the misalignment of the key cues of potential template-instance matches. To unify the multi-task learning formulation, we develop a cross-task interaction module to ensure consistent supervision of the classification and regression branches, improving the synergy of different branches. To eliminate potential inconsistencies that may arise within a multi-task architecture, we assign adaptive labels, rather than fixed hard labels, to supervise the network training more effectively. The experimental results obtained on several benchmarks, i.e., OTB100, UAV123, VOT2018, VOT2019, and LaSOT, demonstrate the effectiveness of the advanced target detection module, as well as the cross-task interaction, exhibiting superior tracking performance as compared with the state-of-the-art tracking methods.

Neural Belief Propagation for Scene Graph Generation.

Scene graph generation aims to interpret an input image by explicitly modelling the objects contained therein and their relationships. In existing methods the problem is predominantly solved by message passing neural network models. Unfortunately, in such models, the variational distributions generally ignore the structural dependencies among the output variables, and most of the scoring functions only consider pairwise dependencies. This can lead to inconsistent interpretations. In this article, we propose a novel neural belief propagation method seeking to replace the traditional mean field approximation with a structural Bethe approximation. To find a better bias-variance trade-off, higher-order dependencies among three or more output variables are also incorporated into the relevant scoring function. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.

LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images.

Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods.

U-SPDNet: An SPD manifold learning-based neural network for visual classification.

With the development of neural networking techniques, several architectures for symmetric positive definite (SPD) matrix learning have recently been put forward in the computer vision and pattern recognition (CV&PR) community for mining fine-grained geometric features. However, the degradation of structural information during multi-stage feature transformation limits their capacity. To cope with this issue, this paper develops a U-shaped neural network on the SPD manifolds (U-SPDNet) for visual classification. The designed U-SPDNet contains two subsystems, one of which is a shrinking path (encoder) making up of a prevailing SPD manifold neural network (SPDNet (Huang and Van Gool, 2017)) for capturing compact representations from the input data. Another is a constructed symmetric expanding path (decoder) to upsample the encoded features, trained by a reconstruction error term. With this design, the degradation problem will be gradually alleviated during training. To enhance the representational capacity of U-SPDNet, we also append skip connections from encoder to decoder, realized by manifold-valued geometric operations, namely Riemannian barycenter and Riemannian optimization. On the MDSD, Virus, FPHA, and UAV-Human datasets, the accuracy achieved by our method is respectively 6.92%, 8.67%, 1.57%, and 1.08% higher than SPDNet, certifying its effectiveness.

CPInformer for Efficient and Robust Compound-Protein Interaction Prediction.

Recently, deep learning has become the mainstream methodology for Compound-Protein Interaction (CPI) prediction. However, the existing compound-protein feature extraction methods have some issues that limit their performance. First, graph networks are widely used for structural compound feature extraction, but the chemical properties of a compound depend on functional groups rather than graphic structure. Besides, the existing methods lack capabilities in extracting rich and discriminative protein features. Last, the compound-protein features are usually simply combined for CPI prediction, without considering information redundancy and effective feature mining. To address the above issues, we propose a novel CPInformer method. Specifically, we extract heterogeneous compound features, including structural graph features and functional class fingerprints, to reduce prediction errors caused by similar structural compounds. Then, we combine local and global features using dense connections to obtain multi-scale protein features. Last, we apply ProbSparse self-attention to protein features, under the guidance of compound features, to eliminate information redundancy, and to improve the accuracy of CPInformer. More importantly, the proposed method identifies the activated local regions that link a CPI, providing a good visualisation for the CPI state. The results obtained on five benchmarks demonstrate the merits and superiority of CPInformer over the state-of-the-art approaches.

A Theoretical Insight Into the Effect of Loss Function for Deep Semantic-Preserving Learning.

Good generalization performance is the fundamental goal of any machine learning algorithm. Using the uniform stability concept, this article theoretically proves that the choice of loss function impacts the generalization performance of a trained deep neural network (DNN). The adopted stability-based framework provides an effective tool for comparing the generalization error bound with respect to the utilized loss function. The main result of our analysis is that using an effective loss function makes stochastic gradient descent more stable which consequently leads to the tighter generalization error bound, and so better generalization performance. To validate our analysis, we study learning problems in which the classes are semantically correlated. To capture this semantic similarity of neighboring classes, we adopt the well-known semantics-preserving learning framework, namely label distribution learning (LDL). We propose two novel loss functions for the LDL framework and theoretically show that they provide stronger stability than the other widely used loss functions adopted for training DNNs. The experimental results on three applications with semantically correlated classes, including facial age estimation, head pose estimation, and image esthetic assessment, validate the theoretical insights gained by our analysis and demonstrate the usefulness of the proposed loss functions in practical applications.

Deep Order-Preserving Learning With Adaptive Optimal Transport Distance.

We consider a framework for taking into consideration the relative importance (ordinality) of object labels in the process of learning a label predictor function. The commonly used loss functions are not well matched to this problem, as they exhibit deficiencies in capturing natural correlations of the labels and the corresponding data. We propose to incorporate such correlations into our learning algorithm using an optimal transport formulation. Our approach is to learn the ground metric, which is partly involved in forming the optimal transport distance, by leveraging ordinality as a general form of side information in its formulation. Based on this idea, we then develop a novel loss function for training deep neural networks. A highly efficient alternating learning method is then devised to alternatively optimise the ground metric and the deep model in an end-to-end learning manner. This scheme allows us to adaptively adjust the shape of the ground metric, and consequently the shape of the loss function for each application. We back up our approach by theoretical analysis and verify the performance of our proposed scheme by applying it to two learning tasks, i.e. chronological age estimation from the face and image aesthetic assessment. The numerical results on several benchmark datasets demonstrate the superiority of the proposed algorithm.

NPT-Loss: Demystifying Face Recognition Losses With Nearest Proxies Triplet.

Face recognition (FR) using deep convolutional neural networks (DCNNs) has seen remarkable success in recent years. One key ingredient of DCNN-based FR is the design of a loss function that ensures discrimination between various identities. The state-of-the-art (SOTA) solutions utilise normalised Softmax loss with additive and/or multiplicative margins. Despite being popular and effective, these losses are justified only intuitively with little theoretical explanations. In this work, we show that under the LogSumExp (LSE) approximation, the SOTA Softmax losses become equivalent to a proxy-triplet loss that focuses on nearest-neighbour negative proxies only. This motivates us to propose a variant of the proxy-triplet loss, entitled Nearest Proxies Triplet (NPT) loss, which unlike SOTA solutions, converges for a wider range of hyper-parameters and offers flexibility in proxy selection and thus outperforms SOTA techniques. We generalise many SOTA losses into a single framework and give theoretical justifications for the assertion that minimising the proposed loss ensures a minimum separability between all identities. We also show that the proposed loss has an implicit mechanism of hard-sample mining. We conduct extensive experiments using various DCNN architectures on a number of FR benchmarks to demonstrate the efficacy of the proposed scheme over SOTA methods.

SARS-CoV-2 triggers pericyte-mediated cerebral capillary constriction.

The SARS-CoV-2 receptor, ACE2, is found on pericytes, contractile cells enwrapping capillaries that regulate brain, heart and kidney blood flow. ACE2 converts vasoconstricting angiotensin II into vasodilating angiotensin-(1-7). In brain slices from hamster, which has an ACE2 sequence similar to human ACE2, angiotensin II evoked a small pericyte-mediated capillary constriction via AT1 receptors, but evoked a large constriction when the SARS-CoV-2 receptor binding domain (RBD, original Wuhan variant) was present. A mutated non-binding RBD did not potentiate constriction. A similar RBD-potentiated capillary constriction occurred in human cortical slices, and was evoked in hamster brain slices by pseudotyped virions expressing SARS-CoV-2 spike protein. This constriction reflects an RBD-induced decrease in the conversion of angiotensin II to angiotensin-(1-7) mediated by removal of ACE2 from the cell surface membrane and was mimicked by blocking ACE2. The clinically used drug losartan inhibited the RBD-potentiated constriction. Thus, AT1 receptor blockers could be protective in COVID-19 by preventing pericyte-mediated blood flow reductions in the brain, and perhaps the heart and kidney.

Mitochondrial Ca2+ uniporter haploinsufficiency enhances long-term potentiation at hippocampal mossy fibre synapses.

Long-term changes in synaptic strength form the basis of learning and memory. These changes rely upon energy-demanding mechanisms, which are regulated by local Ca2+ signalling. Mitochondria are optimised for providing energy and buffering Ca2+. However, our understanding of the role of mitochondria in regulating synaptic plasticity is incomplete. Here, we have used optical and electrophysiological techniques in cultured hippocampal neurons and ex vivo hippocampal slices from mice with haploinsufficiency of the mitochondrial Ca2+ uniporter (MCU+/-) to address whether reducing mitochondrial Ca2+ uptake alters synaptic transmission and plasticity. We found that cultured MCU+/- hippocampal neurons have impaired Ca2+ clearance, and consequently enhanced synaptic vesicle fusion at presynapses occupied by mitochondria. Furthermore, long-term potentiation (LTP) at mossy fibre (MF) synapses, a process which is dependent on presynaptic Ca2+ accumulation, is enhanced in MCU+/- slices. Our results reveal a previously unrecognised role for mitochondria in regulating presynaptic plasticity of a major excitatory pathway involved in learning and memory.

Phosphorylation of neuroligin-2 by PKA regulates its cell surface abundance and synaptic stabilization.

The trans-synaptic adhesion molecule neuroligin-2 (NL2) is essential for the development and function of inhibitory synapses. NL2 recruits the postsynaptic scaffold protein gephyrin, which, in turn, stabilizes γ-aminobutyric acid type A receptors (GABAARs) in the postsynaptic domain. Thus, the amount of NL2 at the synapse can control synaptic GABAAR concentration to tune inhibitory neurotransmission efficacy. Here, using biochemistry, imaging, single-particle tracking, and electrophysiology, we uncovered a key role for cAMP-dependent protein kinase (PKA) in the synaptic stabilization of NL2. We found that PKA-mediated phosphorylation of NL2 at Ser714 caused its dispersal from the synapse and reduced NL2 surface amounts, leading to a loss of synaptic GABAARs. Conversely, enhancing the stability of NL2 at synapses by abolishing PKA-mediated phosphorylation led to increased inhibitory signaling. Thus, PKA plays a key role in regulating NL2 function and GABA-mediated synaptic inhibition.

RAgE: Robust Age Estimation Through Subject Anchoring with Consistency Regularisation.

Modern facial age estimation systems can achieve high accuracy when training and test datasets are identically distributed and captured under similar conditions. However, domain shifts in data, encountered in practice, lead to a sharp drop in accuracy of most existing age estimation algorithms. In this work, we propose a novel method, namely RAgE, to improve the robustness and reduce the uncertainty of age estimates by leveraging unlabelled data through a subject anchoring strategy and a novel consistency regularisation term. First, we propose an similarity-preserving pseudo-labelling algorithm by which the model generates pseudo-labels for a cohort of unlabelled images belonging to the same subject, while taking into account the similarity among age labels. In order to improve the robustness of the system, a consistency regularisation term is then used to simultaneously encourage the model to produce invariant outputs for the images in the cohort with respect to an anchor image. We propose a novel consistency regularisation term the noise-tolerant property of which effectively mitigates the so-called confirmation bias caused by incorrect pseudo-labels. Experiments on multiple benchmark ageing datasets demonstrate substantial improvements over the state-of-the-art methods and robustness to confounding external factors, including subject's head pose, illumination variation and appearance of expression in the face image.

The non-adrenergic imidazoline-1 receptor protein nischarin is a key regulator of astrocyte glutamate uptake.

Astrocytic GLT-1 is the main glutamate transporter involved in glutamate buffering in the brain, pivotal for glutamate removal at excitatory synapses to terminate neurotransmission and for preventing excitotoxicity. We show here that the surface expression and function of GLT-1 can be rapidly modulated through the interaction of its N-terminus with the nonadrenergic imidazoline-1 receptor protein, Nischarin. The phox domain of Nischarin is critical for interaction and internalization of surface GLT-1. Using live super-resolution imaging, we found that glutamate accelerated Nischarin-GLT-1 internalization into endosomal structures. The surface GLT-1 level increased in Nischarin knockout astrocytes, and this correlated with a significant increase in transporter uptake current. In addition, Nischarin knockout in astrocytes is neuroprotective against glutamate excitotoxicity. These data provide new molecular insights into regulation of GLT-1 surface level and function and suggest new drug targets for the treatment of neurological disorders.