Correlations of Cross-Entropy Loss in Machine Learning.

Cross-entropy loss is crucial in training many deep neural networks. In this context, we show a number of novel and strong correlations among various related divergence functions. In particular, we demonstrate that, in some circumstances, (a) cross-entropy is almost perfectly correlated with the little-known triangular divergence, and (b) cross-entropy is strongly correlated with the Euclidean distance over the logits from which the softmax is derived. The consequences of these observations are as follows. First, triangular divergence may be used as a cheaper alternative to cross-entropy. Second, logits can be used as features in a Euclidean space which is strongly synergistic with the classification process. This justifies the use of Euclidean distance over logits as a measure of similarity, in cases where the network is trained using softmax and cross-entropy. We establish these correlations via empirical observation, supported by a mathematical explanation encompassing a number of strongly related divergence functions.

A New Regularization for Deep Learning-Based Segmentation of Images with Fine Structures and Low Contrast.

Deep learning methods have achieved outstanding results in many image processing and computer vision tasks, such as image segmentation. However, they usually do not consider spatial dependencies among pixels/voxels in the image. To obtain better results, some methods have been proposed to apply classic spatial regularization, such as total variation, into deep learning models. However, for some challenging images, especially those with fine structures and low contrast, classical regularizations are not suitable. We derived a new regularization to improve the connectivity of segmentation results and make it applicable to deep learning. Our experimental results show that for both deep learning methods and unsupervised methods, the proposed method can improve performance by increasing connectivity and dealing with low contrast and, therefore, enhance segmentation results.

Deep Classification with Linearity-Enhanced Logits to Softmax Function.

Recently, there has been a rapid increase in deep classification tasks, such as image recognition and target detection. As one of the most crucial components in Convolutional Neural Network (CNN) architectures, softmax arguably encourages CNN to achieve better performance in image recognition. Under this scheme, we present a conceptually intuitive learning objection function: Orthogonal-Softmax. The primary property of the loss function is to use a linear approximation model that is designed by Gram-Schmidt orthogonalization. Firstly, compared with the traditional softmax and Taylor-Softmax, Orthogonal-Softmax has a stronger relationship through orthogonal polynomials expansion. Secondly, a new loss function is advanced to acquire highly discriminative features for classification tasks. At last, we present a linear softmax loss to further promote the intra-class compactness and inter-class discrepancy simultaneously. The results of the widespread experimental discussion on four benchmark datasets manifest the validity of the presented method. Besides, we want to explore the non-ground truth samples in the future.

Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning.

A new trend in deep learning, represented by Mutual Information Neural Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is emerging. In this trend, similarity functions and Estimated Mutual Information (EMI) are used as learning and objective functions. Coincidentally, EMI is essentially the same as Semantic Mutual Information (SeMI) proposed by the author 30 years ago. This paper first reviews the evolutionary histories of semantic information measures and learning functions. Then, it briefly introduces the author's semantic information G theory with the rate-fidelity function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to multi-label learning, the maximum Mutual Information (MI) classification, and mixture models. Then it discusses how we should understand the relationship between SeMI and Shannon's MI, two generalized entropies (fuzzy entropy and coverage entropy), Autoencoders, Gibbs distributions, and partition functions from the perspective of the R(G) function or the G theory. An important conclusion is that mixture models and Restricted Boltzmann Machines converge because SeMI is maximized, and Shannon's MI is minimized, making information efficiency G/R close to 1. A potential opportunity is to simplify deep learning by using Gaussian channel mixture models for pre-training deep neural networks' latent layers without considering gradients. It also discusses how the SeMI measure is used as the reward function (reflecting purposiveness) for reinforcement learning. The G theory helps interpret deep learning but is far from enough. Combining semantic information theory and deep learning will accelerate their development.

Automatic classification of nerve discharge rhythms based on sparse auto-encoder and time series feature.

BACKGROUND: Nerve discharge is the carrier of information transmission, which can reveal the basic rules of various nerve activities. Recognition of the nerve discharge rhythm is the key to correctly understand the dynamic behavior of the nervous system. The previous methods for the nerve discharge recognition almost depended on the traditional statistical features, and the nonlinear dynamical features of the discharge activity. The artificial extraction and the empirical judgment of the features were required for the recognition. Thus, these methods suffered from subjective factors and were not conducive to the identification of a large number of discharge rhythms. RESULTS: The ability of automatic feature extraction along with the development of the neural network has been greatly improved. In this paper, an effective discharge rhythm classification model based on sparse auto-encoder was proposed. The sparse auto-encoder was used to construct the feature learning network. The simulated discharge data from the Chay model and its variants were taken as the input of the network, and the fused features, including the network learning features, covariance and approximate entropy of nerve discharge, were classified by Softmax. The results showed that the accuracy of the classification on the testing data was 87.5%, which could provide more accurate classification results. Compared with other methods for the identification of nerve discharge types, this method could extract the characteristics of nerve discharge rhythm automatically without artificial design, and show a higher accuracy. CONCLUSIONS: The sparse auto-encoder, even neural network has not been used to classify the basic nerve discharge from neither biological experiment data nor model simulation data. The automatic classification method of nerve discharge rhythm based on the sparse auto-encoder in this paper reduced the subjectivity and misjudgment of the artificial feature extraction, saved the time for the comparison with the traditional method, and improved the intelligence of the classification of discharge types. It could further help us to recognize and identify the nerve discharge activities in a new way.

Secure tumor classification by shallow neural network using homomorphic encryption.

BACKGROUND: Disclosure of patients' genetic information in the process of applying machine learning techniques for tumor classification hinders the privacy of personal information. Homomorphic Encryption (HE), which supports operations between encrypted data, can be used as one of the tools to perform such computation without information leakage, but it brings great challenges for directly applying general machine learning algorithms due to the limitations of operations supported by HE. In particular, non-polynomial activation functions, including softmax functions, are difficult to implement with HE and require a suitable approximation method to minimize the loss of accuracy. In the secure genome analysis competition called iDASH 2020, it is presented as a competition task that a multi-label tumor classification method that predicts the class of samples based on genetic information using HE. METHODS: We develop a secure multi-label tumor classification method using HE to ensure privacy during all the computations of the model inference process. Our solution is based on a 1-layer neural network with the softmax activation function model and uses the approximate HE scheme. We present an approximation method that enables softmax activation in the model using HE and a technique for efficiently encoding data to reduce computational costs. In addition, we propose a HE-friendly data filtering method to reduce the size of large-scale genetic data. RESULTS: We aim to analyze the dataset from The Cancer Genome Atlas (TCGA) dataset, which consists of 3,622 samples from 11 types of cancers, genetic features from 25,128 genes. Our preprocessing method reduces the number of genes to 4,096 or less and achieves a microAUC value of 0.9882 (85% accuracy) with a 1-layer shallow neural network. Using our model, we successfully compute the tumor classification inference steps on the encrypted test data in 3.75 minutes. As a result of exceptionally high microAUC values, our solution was awarded co-first place in iDASH 2020 Track 1: "Secure multi-label Tumor classification using Homomorphic Encryption". CONCLUSIONS: Our solution is the first result of implementing a neural network model with softmax activation using HE. Also, HE optimization methods presented in this work enable machine learning implementation using HE or other challenging HE applications.

A high-performance SNP panel developed by machine-learning approaches for characterizing genetic differences of Southern and Northern Han Chinese, Korean, and Japanese individuals.

Population stratification analyses targeting genetically closely related East Asians have revealed that distinguishable differentiation exists between Han Chinese, Korean, and Japanese individuals, as well as between southern (S-) and northern (N-) Han Chinese. Previous studies offer a number of choices for ancestry informative single nucleotide polymorphisms (AISNPs) to discriminate East-Asian populations. In this study, we collected and examined the efficiency of 1185 AISNPs using frequency and genotype data from various publicly available databases. With the aim to perform fine-scale classification of S-Han, N-Han, Korean, and Japanese subjects, machine-learning methods (Softmax and Random Forest) were used to screen a panel of highly informative AISNPs and to develop a superior classification model. Stepwise classification was implemented to increase and balance the discrimination in the process of AISNP selection, first discriminating Han, Korean, and Japanese individuals, and then characterizing stratification between S-Han and N-Han. The final 272-AISNP panel is an alternative optimization of various previous works, which promises reliable and >90% accuracy in classification of the four East-Asian groups. This AISNP panel and the machine-learning model could be a useful and superior choice in medical genome-wide association studies and in forensic investigations for unknown suspect identity.

A simple neural vector space model for medical concept normalization using concept embeddings.

OBJECTIVE: Medical concept normalization (MCN), the task of linking textual mentions to concepts in an ontology, provides a solution to unify different ways of referring to the same concept. In this paper, we present a simple neural MCN model that takes mentions as input and directly predicts concepts. MATERIALS AND METHODS: We evaluate our proposed model on clinical datasets from ShARe/CLEF eHealth 2013 shared task and 2019 n2c2/OHNLP shared task track 3. Our neural MCN model consists of an encoder, and a normalized temperature-scaled softmax (NT-softmax) layer that maximizes the cosine similarity score of matching the mention to the correct concept. We adopt SAPBERT as the encoder and initialize the weights in the NT-softmax layer with pre-computed concept embeddings from SAPBERT. RESULTS: Our proposed neural model achieves competitive performance on ShARe/CLEF 2013 and establishes a new state-of-the-art on 2019-n2c2-MCN. Yet this model is simpler than most prior work: it requires no complex pipelines, no hand-crafted rules, and no preprocessing, making it simpler to apply in new settings. DISCUSSION: Analyses of our proposed model show that the NT-softmax is better than the conventional softmax on the MCN task, and both the CUI-less threshold parameter and the initialization of the weight vectors in the NT-softmax layer contribute to the improvements. CONCLUSION: We propose a simple neural model for clinical MCN, an one-step approach with simpler inference and more effective performance than prior work. Our analyses demonstrate future work on MCN may require more effort on unseen concepts.

A Performance Improvement Strategy for Concrete Damage Detection Using Stacking Ensemble Learning of Multiple Semantic Segmentation Networks.

Semantic segmentation network-based methods can detect concrete damage at the pixel level. However, the performance of a single semantic segmentation network is often limited. To improve the concrete damage detection performance of a semantic segmentation network, a stacking ensemble learning-based concrete crack detection method using multiple semantic segmentation networks is proposed. To realize this method, a database including 500 images and their labels with concrete crack and spalling is built and divided into training and testing sets. At first, the training and prediction of five semantic segmentation networks (FCN-8s, SegNet, U-Net, PSPNet and DeepLabv3+) are respectively implemented on the built training set according to a five-fold cross-validation principle, where 80% of the training images are used in the training process, and 20% training images are reserved. Then, in predicting the results of reserved training images from trained semantic segmentation networks, the class labels of all pixels are collected, and then four softmax regression-based ensemble learning models are trained using the collected class labels and their true classification labels. The trained ensemble learning models are applied to regressed testing results of semantic segmentation network models. Compared with the best single semantic segmentation network, the best ensemble learning model provides performance improvement of 0.21% PA, 0.54% MPA, 3.66% MIoU, and 0.12% FWIoU, respectively. The study results show that the stacking ensemble learning strategy can indeed improve concrete damage detection performance through ensemble learning of multiple semantic segmentation networks.

Automatic Stones Classification through a CNN-Based Approach.

This paper presents an automatic recognition system for classifying stones belonging to different Calabrian quarries (Southern Italy). The tool for stone recognition has been developed in the SILPI project (acronym of "Sistema per l'Identificazione di Lapidei Per Immagini"), financed by POR Calabria FESR-FSE 2014-2020. Our study is based on the Convolutional Neural Network (CNNs) that is used in literature for many different tasks such as speech recognition, neural language processing, bioinformatics, image classification and much more. In particular, we propose a two-stage hybrid approach based on the use of a model of Deep Learning (DL), in our case the CNN, in the first stage and a model of Machine Learning (ML) in the second one. In this work, we discuss a possible solution to stones classification which uses a CNN for the feature extraction phase and the Softmax or Multinomial Logistic Regression (MLR), Support Vector Machine (SVM), k-Nearest Neighbors (kNN), Random Forest (RF) and Gaussian Naive Bayes (GNB) ML techniques in order to perform the classification phase basing our study on the approach called Transfer Learning (TL). We show the image acquisition process in order to collect adequate information for creating an opportune database of the stone typologies present in the Calabrian quarries, also performing the identification of quarries in the considered region. Finally, we show a comparison of different DL and ML combinations in our Two-Stage Hybrid Model solution.

Adversarial Multiscale Feature Learning Framework for Overlapping Chromosome Segmentation.

Chromosome karyotype analysis is of great clinical importance in the diagnosis and treatment of diseases. Since manual analysis is highly time and effort consuming, computer-assisted automatic chromosome karyotype analysis based on images is routinely used to improve the efficiency and accuracy of the analysis. However, the strip-shaped chromosomes easily overlap each other when imaged, significantly affecting the accuracy of the subsequent analysis and hindering the development of chromosome analysis instruments. In this paper, we present an adversarial, multiscale feature learning framework to improve the accuracy and adaptability of overlapping chromosome segmentation. We first adopt the nested U-shaped network with dense skip connections as the generator to explore the optimal representation of the chromosome images by exploiting multiscale features. Then we use the conditional generative adversarial network (cGAN) to generate images similar to the original ones; the training stability of the network is enhanced by applying the least-square GAN objective. Finally, we replace the common cross-entropy loss with the advanced Lovász-Softmax loss to improve the model's optimization and accelerate the model's convergence. Comparing with the established algorithms, the performance of our framework is proven superior by using public datasets in eight evaluation criteria, showing its great potential in overlapping chromosome segmentation.

Residual and Past Discrete Tsallis and Renyi Extropy with an Application to Softmax Function.

In this paper, based on the discrete lifetime distribution, the residual and past of the Tsallis and Renyi extropy are introduced as new measures of information. Moreover, some of their properties and their relation to other measures are discussed. Furthermore, an example of a uniform distribution of the obtained models is given. Moreover, the softmax function can be used as a discrete probability distribution function with a unity sum. Thus, applying those measures to the softmax function for simulated and real data is demonstrated. Besides, for real data, the softmax data are fit to a convenient ARIMA model.

A Non-Contact Fault Diagnosis Method for Bearings and Gears Based on Generalized Matrix Norm Sparse Filtering.

Fault diagnosis of mechanical equipment is mainly based on the contact measurement and analysis of vibration signals. In some special working conditions, the non-contact fault diagnosis method represented by the measurement of acoustic signals can make up for the lack of contact testing. However, its engineering application value is greatly restricted due to the low signal-to-noise ratio (SNR) of the acoustic signal. To solve this deficiency, a novel fault diagnosis method based on the generalized matrix norm sparse filtering (GMNSF) is proposed in this paper. Specially, the generalized matrix norm is introduced into the sparse filtering to seek the optimal sparse feature distribution to overcome the defect of low SNR of acoustic signals. Firstly, the collected acoustic signals are randomly overlapped to form the sample fragment data set. Then, three constraints are imposed on the multi-period data set by the GMNSF model to extract the sparse features in the sample. Finally, softmax is used to as a classifier to categorize different fault types. The diagnostic performance of the proposed method is verified by the bearing and planetary gear datasets. Results show that the GMNSF model has good feature extraction ability performance and anti-noise ability than other traditional methods.

Application of Generalized Composite Multiscale Lempel-Ziv Complexity in Identifying Wind Turbine Gearbox Faults.

Wind turbine gearboxes operate in harsh environments; therefore, the resulting gear vibration signal has characteristics of strong nonlinearity, is non-stationary, and has a low signal-to-noise ratio, which indicates that it is difficult to identify wind turbine gearbox faults effectively by the traditional methods. To solve this problem, this paper proposes a new fault diagnosis method for wind turbine gearboxes based on generalized composite multiscale Lempel-Ziv complexity (GCMLZC). Within the proposed method, an effective technique named multiscale morphological-hat convolution operator (MHCO) is firstly presented to remove the noise interference information of the original gear vibration signal. Then, the GCMLZC of the filtered signal was calculated to extract gear fault features. Finally, the extracted fault features were input into softmax classifier for automatically identifying different health conditions of wind turbine gearboxes. The effectiveness of the proposed method was validated by the experimental and engineering data analysis. The results of the analysis indicate that the proposed method can identify accurately different gear health conditions. Moreover, the identification accuracy of the proposed method is higher than that of traditional multiscale Lempel-Ziv complexity (MLZC) and several representative multiscale entropies (e.g., multiscale dispersion entropy (MDE), multiscale permutation entropy (MPE) and multiscale sample entropy (MSE)).

Self-adapting extraction of cropland phenological transitions of rotation agroecosystems using dynamically fused NDVI images.

Remote sensing can be used to monitor cropland phenological characteristics; however, tradeoffs between the spatial and temporal resolutions of cloudless satellite images limit the accuracy of their retrieval. In this study, an improved enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) was applied to human-dominated Xiong'an New Area to develop a self-adapting algorithm automating the extraction of main phenological transition points (greenup, maturity, senescence, and dormancy). The analyses of cropland phenological characteristics were performed utilizing the Softmax classification method. By examining three different phases of fusion images, it was found that the improved ESTARFM was more accurate than the original ESTARFM (correlation coefficient > 0.76; relative root mean square error < 0.25; structural similarity index > 0.79). The reconstructed normalized difference vegetation indexes were consistent with that acquired by the Moderate Resolution Imaging Spectroradiometer (average discrepancy: 0.1136, median absolute deviation: 0.0110). The greenup, maturity, senescence, and dormancy points were monitored in 5-day resolution and 50-day length on a 30-m grid scale, and their average day of year (DOY) were 67, 119, 127, and 166 for wheat; 173, 224, 232, and 283 for single-season corn; and 189, 227, 232, and 285 for rotation corn, respectively. The corresponding median absolute deviations were 2, 3, 2, and 2 days for wheat; 2, 5, 3, and 4 days for single-season corn; and 2, 5, 2, and 2 days for rotation corn, respectively, while all coefficients of variation did not exceed 6%. The proposed self-adapting approach can be used for identifying the planting modes at grid level in rotation agroecosystems and cropland phenological dynamics on a global or regional scale.

iRSpot-DTS: Predict recombination spots by incorporating the dinucleotide-based spare-cross covariance information into Chou's pseudo components.

Meiotic recombination plays an important role in the process of genetic evolution. Previous researches have shown that the recombination rates provide important information about the mechanism of recombination study. However, at present, most methods ignore the hidden correlation and spatial autocorrelation of the DNA sequence. In this study, we proposed a predictor called iRSpot-DTS to identify hot/cold spots based on the benchmark datasets. We proposed a feature extraction method called dinucleotide-based spatial autocorrelation(DSA) which can incorporate the original DNA properties and spatial information of DNA sequence. Then it used t-SNE method to remove the noise which outperformed PCA. Finally, we used SAE softmax classifier to do classification which is based on networks and can get more hidden information of DNA sequence, our iRSpot-DTS achieved remarkable performance. Jackknife cross validation tests were done on two benchmark datasets. We achieved state-of-the-art results with 96.61% overall accuracy(OA), 93.16% Matthews correlation coefficient (MCC) and over 95% in Sn and Sp which are the best in this state.

Leveraging Uncertainties in Softmax Decision-Making Models for Low-Power IoT Devices.

Internet of Things (IoT) devices bring us rich sensor data, such as images capturing the environment. One prominent approach to understanding and utilizing such data is image classification which can be effectively solved by deep learning (DL). Combined with cross-entropy loss, softmax has been widely used for classification problems, despite its limitations. Many efforts have been made to enhance the performance of softmax decision-making models. However, they require complex computations and/or re-training the model, which is computationally prohibited on low-power IoT devices. In this paper, we propose a light-weight framework to enhance the performance of softmax decision-making models for DL. The proposed framework operates with a pre-trained DL model using softmax, without requiring any modification to the model. First, it computes the level of uncertainty as to the model's prediction, with which misclassified samples are detected. Then, it makes a probabilistic control decision to enhance the decision performance of the given model. We validated the proposed framework by conducting an experiment for IoT car control. The proposed model successfully reduced the control decision errors by up to 96.77% compared to the given DL model, and that suggests the feasibility of building DL-based IoT applications with high accuracy and low complexity.

Fault Detection and Classification in MMC-HVDC Systems Using Learning Methods.

In this paper, we explore learning methods to improve the performance of the open-circuit fault diagnosis of modular multilevel converters (MMCs). Two deep learning methods, namely, convolutional neural networks (CNN) and auto encoder based deep neural networks (AE-based DNN), as well as stand-alone SoftMax classifier are explored for the detection and classification of faults of MMC-based high voltage direct current converter (MMC-HVDC). Only AC-side three-phase current and the upper and lower bridges' currents of the MMCs are used directly in our proposed approaches without any explicit feature extraction or feature subset selection. The two-terminal MMC-HVDC system is implemented in Power Systems Computer-Aided Design/Electromagnetic Transients including DC (PSCAD/EMTDC) to verify and compare our methods. The simulation results indicate CNN, AE-based DNN, and SoftMax classifier can detect and classify faults with high detection accuracy and classification accuracy. Compared with CNN and AE-based DNN, the SoftMax classifier performed better in detection and classification accuracy as well as testing speed. The detection accuracy of AE-based DNN is a little better than CNN, while CNN needs less training time than the AE-based DNN and SoftMax classifier.

A Sparse Autoencoder and Softmax Regression Based Diagnosis Method for the Attachment on the Blades of Marine Current Turbine.

The development and application of marine current energy are attracting more and more attention around the world. Due to the hardness of its working environment, it is important and difficult to study the fault diagnosis of a marine current generation system. In this paper, an underwater image is chosen as the fault-diagnosing signal, after different sensors are compared. This paper proposes a diagnosis method based on the sparse autoencoder (SA) and softmax regression (SR). The SA is used to extract the features and SR is used to classify them. Images are used to monitor whether the blade is attached by benthos and to determine its corresponding degree of attachment. Compared with other methods, the experiment results show that the proposed method can diagnose the blade attachment with higher accuracy.

Prediction of pH Value by Multi-Classification in the Weizhou Island Area.

Ocean acidification is changing the chemical environment on which marine life depends. It causes a decrease in seawater pH and changes the water quality parameters of seawater. Changes in water quality parameters may affect pH, a key indicator for assessing ocean acidification. Therefore, it is particularly important to study the correlation between pH and various water quality parameters. In this paper, several water quality parameters with potential correlation with pH are investigated, and multiple linear regression, softmax regression, and support vector machine are used to perform multi-classification. Most importantly, experimental data were collected from Weizhou Island, China. The classification results show that the pH has a strong correlation with salinity, temperature, and dissolved oxygen. The prediction accuracy of the classification is good, and the correlation with dissolved oxygen is the most significant. The prediction accuracies of the three methods for multi-classifiers based on the above three factors reach 87.01%, 87.77%, and 89.04%, respectively.