Scalable and Practical Natural Gradient for Large-Scale Deep Learning.

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose scalable and practical natural gradient descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4 percent in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9 percent with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.

Auto-spermatophore extrusion in male crickets.

The reproductive cycle of the male cricket consists of the mating stage and the sexually refractory stage. The latter is further divided into the first refractory stage (RS1) from spermatophore extrusion in copulation to spermatophore preparation after copulation, and the second refractory stage (RS2) from spermatophore preparation to recommencement of a calling song. RS2 is time-fixed and unaffected by the female or by stress, hence RS2 is assumed to be controlled by the reproductive timer. Previously, we suggested that the timer is located in the terminal abdominal ganglion (TAG), because functional inactivation of the TAG by local cooling lengthened RS2 in proportion to cooling time. To obtain further evidence of timer localization and to examine the operation of the timer in dissected animals, we investigated the characteristics of auto-spermatophore extrusion, a phenomenon in which males eject the mature spermatophore themselves without any prior courtship. The occurrence of auto-spermatophore extrusion was 100% in dissected males with the TAG separated, compared to 1.7% in intact males. The time interval (SPaSE) between spermatophore preparation and auto-spermatophore extrusion was comparable to RS2 measured by the calling song. Spike recording from a genital motor neurone in the separated TAG indicated that burst discharge associated with auto-spermatophore extrusion occurred with a SPaSE comparable to RS2. Other efferent neurones, some of which were identified as dorsal unpaired median (DUM) neurones, showed a time-dependent spike frequency increase during SPaSE. These results strengthen our previous conclusion that the reproductive timer is located within the TAG, and demonstrate that the timer functions normally even when the TAG is separated from the central nervous system.