Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints.

Momentum technique has recently emerged as an effective strategy in accelerating convergence of gradient descent (GD) methods and exhibits improved performance in deep learning as well as regularized learning. Typical momentum examples include Nesterov's accelerated gradient (NAG) and heavy-ball (HB) methods. However, so far, almost all the acceleration analyses are only limited to NAG, and a few investigations about the acceleration of HB are reported. In this article, we address the convergence about the last iterate of HB in nonsmooth optimizations with constraints, which we name individual convergence. This question is significant in machine learning, where the constraints are required to impose on the learning structure and the individual output is needed to effectively guarantee this structure while keeping an optimal rate of convergence. Specifically, we prove that HB achieves an individual convergence rate of O(1/√t) , where t is the number of iterations. This indicates that both of the two momentum methods can accelerate the individual convergence of basic GD to be optimal. Even for the convergence of averaged iterates, our result avoids the disadvantages of the previous work in restricting the optimization problem to be unconstrained as well as limiting the performed number of iterations to be predefined. The novelty of convergence analysis presented in this article provides a clear understanding of how the HB momentum can accelerate the individual convergence and reveals more insights about the similarities and differences in getting the averaging and individual convergence rates. The derived optimal individual convergence is extended to regularized and stochastic settings, in which an individual solution can be produced by the projection-based operation. In contrast to the averaged output, the sparsity can be reduced remarkably without sacrificing the theoretical optimal rates. Several real experiments demonstrate the performance of HB momentum strategy.

Classifying the tracing difficulty of 3D neuron image blocks based on deep learning.

Quickly and accurately tracing neuronal morphologies in large-scale volumetric microscopy data is a very challenging task. Most automatic algorithms for tracing multi-neuron in a whole brain are designed under the Ultra-Tracer framework, which begins the tracing of a neuron from its soma and traces all signals via a block-by-block strategy. Some neuron image blocks are easy for tracing and their automatic reconstructions are very accurate, and some others are difficult and their automatic reconstructions are inaccurate or incomplete. The former are called low Tracing Difficulty Blocks (low-TDBs), while the latter are called high Tracing Difficulty Blocks (high-TDBs). We design a model named 3D-SSM to classify the tracing difficulty of 3D neuron image blocks, which is based on 3D Residual neural Network (3D-ResNet), Fully Connected Neural Network (FCNN) and Long Short-Term Memory network (LSTM). 3D-SSM contains three modules: Structure Feature Extraction (SFE), Sequence Information Extraction (SIE) and Model Fusion (MF). SFE utilizes a 3D-ResNet and a FCNN to extract two kinds of features in 3D image blocks and their corresponding automatic reconstruction blocks. SIE uses two LSTMs to learn sequence information hidden in 3D image blocks. MF adopts a concatenation operation and a FCNN to combine outputs from SIE. 3D-SSM can be used as a stop condition of an automatic tracing algorithm in the Ultra-Tracer framework. With its help, neuronal signals in low-TDBs can be traced by the automatic algorithm and in high-TDBs may be reconstructed by annotators. 12732 training samples and 5342 test samples are constructed on neuron images of a whole mouse brain. The 3D-SSM achieves classification accuracy rates 87.04% on the training set and 84.07% on the test set. Furthermore, the trained 3D-SSM is tested on samples from another whole mouse brain and its accuracy rate is 83.21%.

Exploring highly reliable substructures in auto-reconstructions of a neuron.

The digital reconstruction of a neuron is the most direct and effective way to investigate its morphology. Many automatic neuron tracing methods have been proposed, but without manual check it is difficult to know whether a reconstruction or which substructure in a reconstruction is accurate. For a neuron's reconstructions generated by multiple automatic tracing methods with different principles or models, their common substructures are highly reliable and named individual motifs. In this work, we propose a Vaa3D-based method called Lamotif to explore individual motifs in automatic reconstructions of a neuron. Lamotif utilizes the local alignment algorithm in BlastNeuron to extract local alignment pairs between a specified objective reconstruction and multiple reference reconstructions, and combines these pairs to generate individual motifs on the objective reconstruction. The proposed Lamotif is evaluated on reconstructions of 163 multiple species neurons, which are generated by four state-of-the-art tracing methods. Experimental results show that individual motifs are almost on corresponding gold standard reconstructions and have much higher precision rate than objective reconstructions themselves. Furthermore, an objective reconstruction is mostly quite accurate if its individual motifs have high recall rate. Individual motifs contain common geometry substructures in multiple reconstructions, and can be used to select some accurate substructures from a reconstruction or some accurate reconstructions from automatic reconstruction dataset of different neurons.

A study of the interaction of cationic dyes with gold nanostructures.

The interaction of methylene blue and crystal violet dyes with a range of gold nanoparticles (AuNPs), gold nanoclusters and gold/silver nanoclusters is reported. It is found that 20 nm citrate-capped AuNPs have strong interactions with these two dyes that result in red-shifted absorption peaks in their electronic absorption spectra. Transmission electron microscopy and dynamic light scattering measurements show that this can be attributed to these AuNPs combining into large agglomerates. Eventually, precipitation is observed. The agglomeration process is triggered when the dye reaches or exceeds a threshold concentration and then does not stop until all the AuNPs have agglomerated into large particles and precipitated. Calculations suggest that the threshold concentration corresponds to having sufficient dye molecules to form a monolayer on the surface of AuNPs. We also observe similar red-shifting in the absorption peaks of the electronic absorption spectra of 11-50 nm citrate-capped AuNPs formed by both single step and seeded growth methods. No such interactions were observed in the UV-vis spectra of the dyes with Tris-capped AuNPs, gold nanoclusters or gold/silver nanoclusters.

Continuous Single-Phase Synthesis of [Au25(Cys)18] Nanoclusters and their Photobactericidal Enhancement.

Thiolate-gold nanoclusters have various applications. However, most of the synthesis methods require prolonged synthesis times from several hours to days. In the present study, we report a rapid synthesis method for [Au25(Cys)18] nanoclusters and their application for photobactericidal enhancement. For [Au25(Cys)18] synthesis, we employed a tube-in-tube membrane reactor using CO as a reducing agent at elevated temperatures. This approach allows continuous generation of high-quality [Au25(Cys)18] within 3 min. Photobactericidal tests against Staphylococcus aureus showed that crystal violet-treated polymer did not have photobactericidal activity, but addition of [Au25(Cys)18] in the treated polymer demonstrated a potent photobactericidal activity at a low white light flux, resulting in >4.29 log reduction in viable bacteria numbers. Steady-state and time-resolved photoluminescence spectroscopies demonstrated that after light irradiation, photoexcited electrons in crystal violet flowed to [Au25(Cys)18] in the silicone, suggesting that redox reaction from [Au25(Cys)18] enhanced the photobactericidal activity. Stability tests revealed that leaching of crystal violet and [Au25(Cys)18] from the treated silicone was negligible and cyclic testing showed that the silicone maintained a strong photobactericidal activity after repeated use.

Photobactericidal activity activated by thiolated gold nanoclusters at low flux levels of white light.

The emergence of antibiotic resistant bacteria is a major threat to the practice of modern medicine. Photobactericidal agents have obtained significant attention as promising candidates to kill bacteria, and they have been extensively studied. However, to obtain photobactericidal activity, an intense white light source or UV-activation is usually required. Here we report a photobactericidal polymer containing crystal violet (CV) and thiolated gold nanocluster ([Au25(Cys)18]) activated at a low flux levels of white light. It was shown that the polymer encapsulated with CV do not have photobactericidal activity under white light illumination of an average 312 lux. However, encapsulation of [Au25(Cys)18] and CV into the polymer activates potent photobactericidal activity. The study of the photobactericidal mechanism shows that additional encapsulation of [Au25(Cys)18] into the CV treated polymer promotes redox reactions through generation of alternative electron transfer pathways, while it reduces photochemical reaction type-ІІ pathways resulting in promotion of hydrogen peroxide (H2O2) production.

A Modular Millifluidic Platform for the Synthesis of Iron Oxide Nanoparticles with Control over Dissolved Gas and Flow Configuration.

Gas-liquid reactions are poorly explored in the context of nanomaterials synthesis, despite evidence of significant effects of dissolved gas on nanoparticle properties. This applies to the aqueous synthesis of iron oxide nanoparticles, where gaseous reactants can influence reaction rate, particle size and crystal structure. Conventional batch reactors offer poor control of gas-liquid mass transfer due to lack of control on the gas-liquid interface and are often unsafe when used at high pressure. This work describes the design of a modular flow platform for the water-based synthesis of iron oxide nanoparticles through the oxidative hydrolysis of Fe2+ salts, targeting magnetic hyperthermia applications. Four different reactor systems were designed through the assembly of two modular units, allowing control over the type of gas dissolved in the solution, as well as the flow pattern within the reactor (single-phase and liquid-liquid two-phase flow). The two modular units consisted of a coiled millireactor and a tube-in-tube gas-liquid contactor. The straightforward pressurization of the system allows control over the concentration of gas dissolved in the reactive solution and the ability to operate the reactor at a temperature above the solvent boiling point. The variables controlled in the flow system (temperature, flow pattern and dissolved gaseous reactants) allowed full conversion of the iron precursor to magnetite/maghemite nanocrystals in just 3 min, as compared to several hours normally employed in batch. The single-phase configuration of the flow platform allowed the synthesis of particles with sizes between 26.5 nm (in the presence of carbon monoxide) and 34 nm. On the other hand, the liquid-liquid two-phase flow reactor showed possible evidence of interfacial absorption, leading to particles with different morphology compared to their batch counterpart. When exposed to an alternating magnetic field, the particles produced by the four flow systems showed ILP (intrinsic loss parameter) values between 1.2 and 2.7 nHm2/kg. Scale up by a factor of 5 of one of the configurations was also demonstrated. The scaled-up system led to the synthesis of nanoparticles of equivalent quality to those produced with the small-scale reactor system. The equivalence between the two systems is supported by a simple analysis of the transport phenomena in the small and large-scale setups.

Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence.

Many well-known first-order gradient methods have been extended to cope with large-scale composite problems, which often arise as a regularized empirical risk minimization in machine learning. However, their optimal convergence is attained only in terms of the weighted average of past iterative solutions. How to make the individual convergence of stochastic gradient descent (SGD) optimal, especially for strongly convex problems has now become a challenging problem in the machine learning community. On the other hand, Nesterov's recent weighted averaging strategy succeeds in achieving the optimal individual convergence of dual averaging (DA) but it fails in the basic mirror descent (MD). In this paper, a new primal averaging (PA) gradient operation step is presented, in which the gradient evaluation is imposed on the weighted average of all past iterative solutions. We prove that simply modifying the gradient operation step in MD by PA strategy suffices to recover the optimal individual rate for general convex problems. Along this line, the optimal individual rate of convergence for strongly convex problems can also be achieved by imposing the strong convexity on the gradient operation step. Furthermore, we extend PA-MD to solve regularized nonsmooth learning problems in the stochastic setting, which reveals that PA strategy is a simple yet effective extra step toward the optimal individual convergence of SGD. Several real experiments on sparse learning and SVM problems verify the correctness of our theoretical analysis.

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization.

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this article, the convergence of individual iterates of projected subgradient (PSG) methods for nonsmooth convex optimization problems is theoretically studied based on Nesterov's extrapolation, which we name individual convergence. We prove that Nesterov's extrapolation has the strength to make the individual convergence of PSG optimal for nonsmooth problems. In light of this consideration, a direct modification of the subgradient evaluation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step toward the open question about stochastic gradient descent (SGD) posed by Shamir. Furthermore, we give an extension of the derived algorithms to solve regularized learning tasks with nonsmooth losses in stochastic settings. Compared with other state-of-the-art nonsmooth methods, the derived algorithms can serve as an alternative to the basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence. Typically, our method is applicable as an efficient tool for solving large-scale l1 -regularized hinge-loss learning problems. Several comparison experiments demonstrate that our individual output not only achieves an optimal convergence rate but also guarantees better sparsity than the averaged solution.

A rugged, self-sterilizing antimicrobial copper coating on ultra-high molecular weight polyethylene: a preliminary study on the feasibility of an antimicrobial prosthetic joint material.

We report here for the first time how the combination of a precursor solution and low temperature (170 °C) aerosol assisted chemical vapour deposition were used to bond a copper coating to ultra-high molecular weight polyethylene (UHMWPE) and promote robustness. This metallic thin film remained intact on the UHMWPE substrate after the Scotch tape test and showed notable wear-resistance after 10 cycles of sand paper-abrasion. Antimicrobial assays against both Escherichia coli and Staphylococcus aureus revealed potent dark bactericidal activity with 99.999% reduction in bacterial number within 15 minutes. These results suggest that the modified UHMWPE could be a potential candidate for antimicrobial plastics and in the long term may find application in prosthetic joint applications.

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems.

The truncated regular -loss support vector machine can eliminate the excessive number of support vectors (SVs); thus, it has significant advantages in robustness and scalability. However, in this paper, we discover that the associated state-of-the-art solvers, such as difference convex algorithm and concave-convex procedure, not only have limited sparsity promoting property for general truncated losses especially the -loss but also have poor scalability for large-scale problems. To circumvent these drawbacks, we present a general multistage scheme with explicit interpretation regarding SVs as well as outliers. In particular, we solve the general nonconvex truncated loss minimization through a sequence of associated convex subproblems, in which the outliers are removed in advance. The proposed algorithm can be regarded as a structural optimization attempt carefully considering sparsity imposed by the nonconvex truncated losses. We show that this general multistage algorithm offers sufficient sparsity especially for the truncated -loss. To further improve the scalability, we propose a linear multistep algorithm by employing a single iteration of coordinate descent to monotonically decrease the objective function at each stage and a kernel algorithm by using the Karush-Kuhn-Tucker conditions to cheaply find most part of the outliers for the next stage. Comparison experiments demonstrate that our methods have superiority in sparsity as well as efficiency in scalability.

Stochastic learning via optimizing the variational inequalities.

A wide variety of learning problems can be posed in the framework of convex optimization. Many efficient algorithms have been developed based on solving the induced optimization problems. However, there exists a gap between the theoretically unbeatable convergence rate and the practically efficient learning speed. In this paper, we use the variational inequality (VI) convergence to describe the learning speed. To this end, we avoid the hard concept of regret in online learning and directly discuss the stochastic learning algorithms. We first cast the regularized learning problem as a VI. Then, we present a stochastic version of alternating direction method of multipliers (ADMMs) to solve the induced VI. We define a new VI-criterion to measure the convergence of stochastic algorithms. While the rate of convergence for any iterative algorithms to solve nonsmooth convex optimization problems cannot be better than O(1/√t), the proposed stochastic ADMM (SADMM) is proved to have an O(1/t) VI-convergence rate for the l1-regularized hinge loss problems without strong convexity and smoothness. The derived VI-convergence results also support the viewpoint that the standard online analysis is too loose to analyze the stochastic setting properly. The experiments demonstrate that SADMM has almost the same performance as the state-of-the-art stochastic learning algorithms but its O(1/t) VI-convergence rate is capable of tightly characterizing the real learning speed.

Pt-based core-shell nanocatalysts with enhanced activity and stability for CO oxidation.

This communication describes the synthesis of Pt@CeO2 core-shell catalysts for the application of highly efficient CO oxidation, where the 50% CO conversion temperature is lower than 200 °C. Pt@CeO2 is thermally stable as no deactivation occurs during the 70 h reaction, and the morphology is unchanged even after 700 °C thermal treatment.

Sintering-resistant Ni-based reforming catalysts obtained via the nanoconfinement effect.

This communication describes the design and synthesis of anti-sintering and -coke nickel phyllosilicate (PS) nanotubes (Ni/PSn) for hydrogen production via reforming reactions. The introduction of nickel particles in PS nanotubes could effectively maintain the Ni size and increase the resistance of metal particles for carbon deposition.

A Ni@ZrO2 nanocomposite for ethanol steam reforming: enhanced stability via strong metal-oxide interaction.

This communication describes the synthesis of a nanocomposite Ni@ZrO2 catalyst with enhanced metal-support interaction by introducing metal nanoparticles into the framework of the oxide support. The catalyst shows high catalytic activity and stability for hydrogen production via steam reforming of ethanol.

Superior reactivity of skeletal Ni-based catalysts for low-temperature steam reforming to produce CO-free hydrogen.

This paper describes the utilization of skeletal Ni-based catalysts for steam reforming of ethanol to produce CO-free hydrogen, which could be superior in the application of fuel cells. Assistant metals play different roles in the reaction; Pt and Cu suppress the methanation and enhance H(2) production, while Co promotes the methanation.

On the origin of reactivity of steam reforming of ethylene glycol on supported Ni catalysts.

This paper describes a strategy for producing hydrogen via steam reforming of ethylene glycol over supported nickel catalysts. Nickel plays a crucial role in conversion of ethylene glycol and production of hydrogen, while oxide supports affect product distribution of carbonaceous species. A plausible reaction pathway is proposed based on our results and the literature.

Integrated analysis of multiple data sources reveals modular structure of biological networks.

It has been a challenging task to integrate high-throughput data into investigations of the systematic and dynamic organization of biological networks. Here, we presented a simple hierarchical clustering algorithm that goes a long way to achieve this aim. Our method effectively reveals the modular structure of the yeast protein-protein interaction network and distinguishes protein complexes from functional modules by integrating high-throughput protein-protein interaction data with the added subcellular localization and expression profile data. Furthermore, we take advantage of the detected modules to provide a reliably functional context for the uncharacterized components within modules. On the other hand, the integration of various protein-protein association information makes our method robust to false-positives, especially for derived protein complexes. More importantly, this simple method can be extended naturally to other types of data fusion and provides a framework for the study of more comprehensive properties of the biological network and other forms of complex networks.

Posterior probability support vector machines for unbalanced data.

This paper proposes a complete framework of posterior probability support vector machines (PPSVMs) for weighted training samples using modified concepts of risks, linear separability, margin, and optimal hyperplane. Within this framework, a new optimization problem for unbalanced classification problems is formulated and a new concept of support vectors established. Furthermore, a soft PPSVM with an interpretable parameter v is obtained which is similar to the v-SVM developed by Schölkopf et al., and an empirical method for determining the posterior probability is proposed as a new approach to determine v. The main advantage of an PPSVM classifier lies in that fact that it is closer to the Bayes optimal without knowing the distributions. To validate the proposed method, two synthetic classification examples are used to illustrate the logical correctness of PPSVMs and their relationship to regular SVMs and Bayesian methods. Several other classification experiments are conducted to demonstrate that the performance of PPSVMs is better than regular SVMs in some cases. Compared with fuzzy support vector machines (FSVMs), the proposed PPSVM is a natural and an analytical extension of regular SVMs based on the statistical learning theory.