FlexSleepTransformer: a transformer-based sleep staging model with flexible input channel configurations.

Clinical sleep diagnosis traditionally relies on polysomnography (PSG) and expert manual classification of sleep stages. Recent advancements in deep learning have shown promise in automating sleep stage classification using a single PSG channel. However, variations in PSG acquisition devices and environments mean that the number of PSG channels can differ across sleep centers. To integrate a sleep staging method into clinical practice effectively, it must accommodate a flexible number of PSG channels. In this paper, we proposed FlexSleepTransformer, a transformer-based model designed to handle varying number of input channels, making it adaptable to diverse sleep staging datasets. We evaluated FlexSleepTransformer using two distinct datasets: the public SleepEDF-78 dataset and the local SleepUHS dataset. Notably, FlexSleepTransformer is the first model capable of simultaneously training on datasets with differing number of PSG channels. Our experiments showed that FlexSleepTransformer trained on both datasets together achieved 98% of the accuracy compared to models trained on each dataset individually. Furthermore, it outperformed models trained exclusively on one dataset when tested on the other dataset. Additionally, FlexSleepTransformer surpassed state-of-the-art CNN and RNN-based models on both datasets. Due to its adaptability with varying channels numbers, FlexSleepTransformer holds significant potential for clinical adoption, especially when trained with data from a wide range of sleep centers.

A deep learning approach for line-level Amharic Braille image recognition.

Braille, the most popular tactile-based writing system, uses patterns of raised dots arranged in cells to inscribe characters for visually impaired persons. Amharic is Ethiopia's official working language, spoken by more than 100 million people. To bridge the written communication gap between persons with and without eyesight, multiple Optical braille recognition systems for various language scripts have been developed utilizing both statistical and deep learning approaches. However, the need for half-character identification and character segmentation has complicated these systems, particularly in the Amharic script, where each character is represented by two braille cells. To address these challenges, this study proposed deep learning model that combines a CNN and a BiLSTM network with CTC. The model was trained with 1,800 line images with 32 × 256 and 48 × 256 dimensions, and validated with 200 line images and evaluated using Character Error Rate. The best-trained model had a CER of 7.81% on test data with a 48 × 256 image dimension. These findings demonstrate that the proposed sequence-to-sequence learning method is a viable Optical Braille Recognition (OBR) solution that does not necessitate extensive image pre and post processing. Inaddition, we have made the first Amharic braille line-image data set available for free to researchers via the link: https://github.com/Ne-UoG-git/Am-Br-line-image.github.io .

Prediction of mine water quality by the Seq2Seq model based on attention mechanism.

In recent years, as China's industrialization level has advanced, the issue of environmental pollution, particularly mine water pollution, has become increasingly severe. Water quality prediction is a fundamental aspect of water resource protection and a critical approach to addressing the water resource crisis. For improvement in water quality prediction, this research first analyzes the characteristics of mine water quality changes and provides a brief overview of water quality prediction. Subsequently, the Long Short-Term Memory and Sequence to Sequence (Seq2Seq) models, derived from Artificial Neural Networks, are introduced. The Seq2Seq water quality prediction model is implemented, incorporating the attention mechanism. Experimental validation confirms the effectiveness of the proposed model. The results demonstrate that the attention mechanism-based Seq2Seq model accurately predicts parameters such as pH value, Dissolved Oxygen, ammonia nitrogen, and Chemical Oxygen Demand, exhibiting a high degree of consistency with actual results. They play a vital role in assessing the health of the water and its ability to support aquatic life. The change of these indicators can reflect the degree and type of water pollution. Moreover, the Seq2Seq + attention model stands out with the lowest predicted Root Mean Square Error of 0.309. Notably, in comparison to the traditional Seq2Seq model, the incorporation of attention mechanisms in the Seq2Seq model results in a substantial 2.94 reduction in Mean Absolute Error. This research on the Seq2Seq water quality prediction model with attention mechanism provides valuable insights and references for future endeavors in water quality prediction.

Co-Mutations and Possible Variation Tendency of the Spike RBD and Membrane Protein in SARS-CoV-2 by Machine Learning.

Since the onset of the coronavirus disease 2019 (COVID-19) pandemic, SARS-CoV-2 variants capable of breakthrough infections have attracted global attention. These variants have significant mutations in the receptor-binding domain (RBD) of the spike protein and the membrane (M) protein, which may imply an enhanced ability to evade immune responses. In this study, an examination of co-mutations within the spike RBD and their potential correlation with mutations in the M protein was conducted. The EVmutation method was utilized to analyze the distribution of the mutations to elucidate the relationship between the mutations in the spike RBD and the alterations in the M protein. Additionally, the Sequence-to-Sequence Transformer Model (S2STM) was employed to establish mapping between the amino acid sequences of the spike RBD and M proteins, offering a novel and efficient approach for streamlined sequence analysis and the exploration of their interrelationship. Certain mutations in the spike RBD, G339D-S373P-S375F and Q493R-Q498R-Y505, are associated with a heightened propensity for inducing mutations at specific sites within the M protein, especially sites 3 and 19/63. These results shed light on the concept of mutational synergy between the spike RBD and M proteins, illuminating a potential mechanism that could be driving the evolution of SARS-CoV-2.

Customized deep learning based Turkish automatic speech recognition system supported by language model.

Background: In today's world, numerous applications integral to various facets of daily life include automatic speech recognition methods. Thus, the development of a successful automatic speech recognition system can significantly augment the convenience of people's daily routines. While many automatic speech recognition systems have been established for widely spoken languages like English, there has been insufficient progress in developing such systems for less common languages such as Turkish. Moreover, due to its agglutinative structure, designing a speech recognition system for Turkish presents greater challenges compared to other language groups. Therefore, our study focused on proposing deep learning models for automatic speech recognition in Turkish, complemented by the integration of a language model. Methods: In our study, deep learning models were formulated by incorporating convolutional neural networks, gated recurrent units, long short-term memories, and transformer layers. The Zemberek library was employed to craft the language model to improve system performance. Furthermore, the Bayesian optimization method was applied to fine-tune the hyper-parameters of the deep learning models. To evaluate the model's performance, standard metrics widely used in automatic speech recognition systems, specifically word error rate and character error rate scores, were employed. Results: Upon reviewing the experimental results, it becomes evident that when optimal hyper-parameters are applied to models developed with various layers, the scores are as follows: Without the use of a language model, the Turkish Microphone Speech Corpus dataset yields scores of 22.2 -word error rate and 14.05-character error rate, while the Turkish Speech Corpus dataset results in scores of 11.5 -word error rate and 4.15 character error rate. Upon incorporating the language model, notable improvements were observed. Specifically, for the Turkish Microphone Speech Corpus dataset, the word error rate score decreased to 9.85, and the character error rate score lowered to 5.35. Similarly, the word error rate score improved to 8.4, and the character error rate score decreased to 2.7 for the Turkish Speech Corpus dataset. These results demonstrate that our model outperforms the studies found in the existing literature.

A Sequential End-to-End Neonatal Sleep Staging Model with Squeeze and Excitation Blocks and Sequential Multi-Scale Convolution Neural Networks.

Automatic sleep staging offers a quick and objective assessment for quantitatively interpreting sleep stages in neonates. However, most of the existing studies either do not encompass any temporal information, or simply apply neural networks to exploit temporal information at the expense of high computational overhead and modeling ambiguity. This limits the application of these methods to multiple scenarios. In this paper, a sequential end-to-end sleep staging model, SeqEESleepNet, which is competent for parallelly processing sequential epochs and has a fast training rate to adapt to different scenarios, is proposed. SeqEESleepNet consists of a sequence epoch generation (SEG) module, a sequential multi-scale convolution neural network (SMSCNN) and squeeze and excitation (SE) blocks. The SEG module expands independent epochs into sequential signals, enabling the model to learn the temporal information between sleep stages. SMSCNN is a multi-scale convolution neural network that can extract both multi-scale features and temporal information from the signal. Subsequently, the followed SE block can reassign the weights of features through mapping and pooling. Experimental results exhibit that in a clinical dataset, the proposed method outperforms the state-of-the-art approaches, achieving an overall accuracy, F1-score, and Kappa coefficient of 71.8%, 71.8%, and 0.684 on a three-class classification task with a single channel EEG signal. Based on our overall results, we believe the proposed method could pave the way for convenient multi-scenario neonatal sleep staging methods.

Human Activity Prediction Based on Forecasted IMU Activity Signals by Sequence-to-Sequence Deep Neural Networks.

Human Activity Recognition (HAR) has gained significant attention due to its broad range of applications, such as healthcare, industrial work safety, activity assistance, and driver monitoring. Most prior HAR systems are based on recorded sensor data (i.e., past information) recognizing human activities. In fact, HAR works based on future sensor data to predict human activities are rare. Human Activity Prediction (HAP) can benefit in multiple applications, such as fall detection or exercise routines, to prevent injuries. This work presents a novel HAP system based on forecasted activity data of Inertial Measurement Units (IMU). Our HAP system consists of a deep learning forecaster of IMU activity signals and a deep learning classifier to recognize future activities. Our deep learning forecaster model is based on a Sequence-to-Sequence structure with attention and positional encoding layers. Then, a pre-trained deep learning Bi-LSTM classifier is used to classify future activities based on the forecasted IMU data. We have tested our HAP system for five daily activities with two tri-axial IMU sensors. The forecasted signals show an average correlation of 91.6% to the actual measured signals of the five activities. The proposed HAP system achieves an average accuracy of 97.96% in predicting future activities.

Exploring a multi-output temporal convolutional network driven encoder-decoder framework for ammonia nitrogen forecasting.

Artificial neural networks exhibit significant advantages in terms of learning capability and generalizability, and have been increasingly applied in water quality prediction. Through learning a compressed representation of the input data, the Encoder-Decoder (ED) structure not only could remove noise and redundancies, but also could efficiently capture the complex nonlinear relationships of meteorological and water quality factors. The novelty of this study lies in proposing a multi-output Temporal Convolutional Network based ED model (TCN-ED) to make ammonia nitrogen forecasts for the first time. The contribution of our study is indebted to systematically assessing the significance of combining the ED structure with advanced neural networks for making accurate and reliable water quality forecasts. The water quality gauge station located at Haihong village of an island in Shanghai City of China constituted the case study. The model input contained one hourly water quality factor and hourly meteorological factors of 32 observed stations, where each factor was traced back to the previous 24 h and each meteorological factor of 32 gauge stations was aggregated into one areal average factor. A total of 13,128 hourly water quality and meteorological data were divided into two datasets corresponding to model training and testing stages. The Long Short-Term Memory based ED (LSTM-ED), LSTM and TCN models were constructed for comparison purposes. The results demonstrated that the developed TCN-ED model can succeed in mimicking the complex dependence between ammonia nitrogen and water quality and meteorological factors, and provide more accurate ammonia nitrogen forecasts (1- up to 6-h-ahead) than the LSTM-ED, LSTM and TCN models. The TCN-ED model, in general, achieved higher accuracy, stability and reliability compared with the other models. Consequently, the improvement can facilitate river water quality forecasting and early warning, as well as benefit water pollution prevention in the interest of river environmental restoration and sustainability.

Sequence-to-sequence pretraining for a less-resourced Slovenian language.

Introduction: Large pretrained language models have recently conquered the area of natural language processing. As an alternative to predominant masked language modeling introduced in BERT, the T5 model has introduced a more general training objective, namely sequence to sequence transformation, which more naturally fits text generation tasks. The monolingual variants of T5 models have been limited to well-resourced languages, while the massively multilingual T5 model supports 101 languages. Methods: We trained two different-sized T5-type sequence-to-sequence models for morphologically rich Slovene language with much fewer resources. We analyzed the behavior of new models on 11 tasks, eight classification ones (named entity recognition, sentiment classification, lemmatization, two question answering tasks, two natural language inference tasks, and a coreference resolution task), and three text generation tasks (text simplification and two summarization tasks on different datasets). We compared the new SloT5 models with the multilingual mT5 model, multilingual mBART-50 model, and with four encoder BERT-like models: multilingual BERT, multilingual XLM-RoBERTa, trilingual Croatian-Slovene-English BERT, and monolingual Slovene RoBERTa model. Results: Concerning the classification tasks, the SloT5 models mostly lag behind the monolingual Slovene SloBERTa model. However, these models are helpful for generative tasks and provide several useful results. In general, the size of models matters, and currently, there is not enough training data for Slovene for successful pretraining of large models. Discussion: While the results are obtained on Slovene, we believe that they may generalize to other less-resourced languages, where such models will be built. We make the training and evaluation code, as well as the trained models, publicly available.

Natural Language Processing for Covid-19 Consulting System.

The world was taken aback when the Covid-19 pandemic hit in 2019. Ever since precautions have been taken to prevent the spreading or mutating of the virus, but the virus still keeps spreading and mutating. Scientists predict that the virus is going to stay for a long time but with reduced effectiveness. Recognizing the symptoms of the virus is essential in order to provide proper treatment for the virus. Visiting hospitals for consultation becomes quite difficult when people are supposed to maintain social distancing. Recently neural network generative models have shown impressive abilities in developing chatbots. However, using these neural network generative models that lack the required Covid specific knowledge to develop a Covid consulting system makes them difficult to be scaled. In order to bridge the gap between patients and a limited number of doctors we have proposed a Covid consulting agent by integrating the medical knowledge of Covid-19 with the neural network generative models. This system will automatically scan patient's dialogues seeking for a consultation to recognize the symptoms for Covid-19. The transformer and pretrained systems of BERT-GPT and GPT were fine-tuned CovidDialog-English dataset to generate responses for Covid-19 which were doctor-like and clinically meaningful to further solve the problem of the surging demand for medical consultations compared to the limited number of medical professionals. The results are evaluated and compared using multiple evaluation metrics which are NIST-n, perplexity, BLEU-n, METEOR, Entropy-n and Dist-n. In this paper, we also hope to prove that the results obtained from the automated dialogue systems were significantly similar to human evaluation. Furthermore, the evaluation shows that state-of-the-art BERT-GPT performs better.

Deep learning for dense Z-spectra reconstruction from CEST images at sparse frequency offsets.

A direct way to reduce scan time for chemical exchange saturation transfer (CEST)-magnetic resonance imaging (MRI) is to reduce the number of CEST images acquired in experiments. In some scenarios, a sufficient number of CEST images acquired in experiments was needed to estimate parameters for quantitative analysis, and this prolonged the scan time. For that, we aim to develop a general deep-learning framework to reconstruct dense CEST Z-spectra from experimentally acquired images at sparse frequency offsets so as to reduce the number of experimentally acquired CEST images and achieve scan time reduction. The main innovation works are outlined as follows: (1) a general sequence-to-sequence (seq2seq) framework is proposed to reconstruct dense CEST Z-spectra from experimentally acquired images at sparse frequency offsets; (2) we create a training set from wide-ranging simulated Z-spectra instead of experimentally acquired CEST data, overcoming the limitation of the time and labor consumption in manual annotation; (3) a new seq2seq network that is capable of utilizing information from both short-range and long-range is developed to improve reconstruction ability. One of our intentions is to establish a simple and efficient framework, i.e., traditional seq2seq can solve the reconstruction task and obtain satisfactory results. In addition, we propose a new seq2seq network that includes the short- and long-range ability to boost dense CEST Z-spectra reconstruction. The experimental results demonstrate that the considered seq2seq models can accurately reconstruct dense CEST images from experimentally acquired images at 11 frequency offsets so as to reduce the scan time by at least 2/3, and our new seq2seq network contributes to competitive advantage.

Attention-Based Sequence-to-Sequence Model for Time Series Imputation.

Time series data are usually characterized by having missing values, high dimensionality, and large data volume. To solve the problem of high-dimensional time series with missing values, this paper proposes an attention-based sequence-to-sequence model to imputation missing values in time series (ASSM), which is a sequence-to-sequence model based on the combination of feature learning and data computation. The model consists of two parts, encoder and decoder. The encoder part is a BIGRU recurrent neural network and incorporates a self-attentive mechanism to make the model more capable of handling long-range time series; The decoder part is a GRU recurrent neural network and incorporates a cross-attentive mechanism into associate with the encoder part. The relationship weights between the generated sequences in the decoder part and the known sequences in the encoder part are calculated to achieve the purpose of focusing on the sequences with a high degree of correlation. In this paper, we conduct comparison experiments with four evaluation metrics and six models on four real datasets. The experimental results show that the model proposed in this paper outperforms the six comparative missing value interpolation algorithms.

Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language.

Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain-computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder-decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation.

Carbon price forecasting: a novel deep learning approach.

Carbon emission trading market promotes carbon emission reduction effectively. Accurate carbon price forecasting is crucial for relevant policy makers and investors. However, due to the non-linearity, uncertainty, and complexity of carbon prices, the current predication models fail to predict carbon prices accurately. In this paper, an advanced deep neural network model named TCN-Seq2Seq is proposed to forecast carbon prices. The novelty of the proposed model focuses on the "sequence to sequence" layout to learn temporal data dependencies using only fully convolutional layers. Being provided with parallel training for fewer parameters, TCN-Seq2Seq forecasting model is more suitable for small carbon price dataset in few-shot learning way. Qualitatively and quantitatively, we find that the proposed framework consistently and significantly outperforms traditional statistical forecasting models and state-of-the-art deep learning prediction model with respect to predictive ability and robustness. Particularly, our proposed model achieves forecasting accuracy with the highest DA value (0.9697), the lowest MAPE value (0.0027), and the lowest RMSE value (0.0149), showing superior prediction performance compared with the traditional statistical forecasting models. The accuracy of carbon price forecasting gives insight to policy makers and carbon market investors.

An Analytical Investigation of Anomaly Detection Methods Based on Sequence to Sequence Model in Satellite Power Subsystem.

The satellite power subsystem is responsible for all power supply in a satellite, and is an important component of it. The system's performance has a direct impact on the operations of other systems as well as the satellite's lifespan. Sequence to sequence (seq2seq) learning has recently advanced, gaining even more power in evaluating complicated and large-scale data. The potential of the seq2seq model in detecting anomalies in the satellite power subsystem is investigated in this work. A seq2seq-based scheme is given, with a thorough comparison of different neural-network cell types and levels of data smoothness. Three specific approaches were created to evaluate the seq2seq model performance, taking into account the unsupervised learning mechanism. The findings reveal that a CNN-based seq2seq with attention model under suitable data-smoothing conditions has a better ability to detect anomalies in the satellite power subsystem.

Wearable Sensor-Based Human Activity Recognition with Transformer Model.

Computing devices that can recognize various human activities or movements can be used to assist people in healthcare, sports, or human-robot interaction. Readily available data for this purpose can be obtained from the accelerometer and the gyroscope built into everyday smartphones. Effective classification of real-time activity data is, therefore, actively pursued using various machine learning methods. In this study, the transformer model, a deep learning neural network model developed primarily for the natural language processing and vision tasks, was adapted for a time-series analysis of motion signals. The self-attention mechanism inherent in the transformer, which expresses individual dependencies between signal values within a time series, can match the performance of state-of-the-art convolutional neural networks with long short-term memory. The performance of the proposed adapted transformer method was tested on the largest available public dataset of smartphone motion sensor data covering a wide range of activities, and obtained an average identification accuracy of 99.2% as compared with 89.67% achieved on the same data by a conventional machine learning method. The results suggest the expected future relevance of the transformer model for human activity recognition.

Basecalling Using Joint Raw and Event Nanopore Data Sequence-to-Sequence Processing.

Third-generation DNA sequencers provided by Oxford Nanopore Technologies (ONT) produce a series of samples of an electrical current in the nanopore. Such a time series is used to detect the sequence of nucleotides. The task of translation of current values into nucleotide symbols is called basecalling. Various solutions for basecalling have already been proposed. The earlier ones were based on Hidden Markov Models, but the best ones use neural networks or other machine learning models. Unfortunately, achieved accuracy scores are still lower than competitive sequencing techniques, like Illumina's. Basecallers differ in the input data type-currently, most of them work on a raw data straight from the sequencer (time series of current). Still, the approach of using event data is also explored. Event data is obtained by preprocessing of raw data and dividing it into segments described by several features computed from raw data values within each segment. We propose a novel basecaller that uses joint processing of raw and event data. We define basecalling as a sequence-to-sequence translation, and we use a machine learning model based on an encoder-decoder architecture of recurrent neural networks. Our model incorporates twin encoders and an attention mechanism. We tested our solution on simulated and real datasets. We compare the full model accuracy results with its components: processing only raw or event data. We compare our solution with the existing ONT basecaller-Guppy. Results of numerical experiments show that joint raw and event data processing provides better basecalling accuracy than processing each data type separately. We implement an application called Ravvent, freely available under MIT licence.

Markup: A Web-Based Annotation Tool Powered by Active Learning.

Across various domains, such as health and social care, law, news, and social media, there are increasing quantities of unstructured texts being produced. These potential data sources often contain rich information that could be used for domain-specific and research purposes. However, the unstructured nature of free-text data poses a significant challenge for its utilisation due to the necessity of substantial manual intervention from domain-experts to label embedded information. Annotation tools can assist with this process by providing functionality that enables the accurate capture and transformation of unstructured texts into structured annotations, which can be used individually, or as part of larger Natural Language Processing (NLP) pipelines. We present Markup (https://www.getmarkup.com/) an open-source, web-based annotation tool that is undergoing continued development for use across all domains. Markup incorporates NLP and Active Learning (AL) technologies to enable rapid and accurate annotation using custom user configurations, predictive annotation suggestions, and automated mapping suggestions to both domain-specific ontologies, such as the Unified Medical Language System (UMLS), and custom, user-defined ontologies. We demonstrate a real-world use case of how Markup has been used in a healthcare setting to annotate structured information from unstructured clinic letters, where captured annotations were used to build and test NLP applications.

Video captioning with stacked attention and semantic hard pull.

Video captioning, i.e., the task of generating captions from video sequences creates a bridge between the Natural Language Processing and Computer Vision domains of computer science. The task of generating a semantically accurate description of a video is quite complex. Considering the complexity, of the problem, the results obtained in recent research works are praiseworthy. However, there is plenty of scope for further investigation. This paper addresses this scope and proposes a novel solution. Most video captioning models comprise two sequential/recurrent layers-one as a video-to-context encoder and the other as a context-to-caption decoder. This paper proposes a novel architecture, namely Semantically Sensible Video Captioning (SSVC) which modifies the context generation mechanism by using two novel approaches-"stacked attention" and "spatial hard pull". As there are no exclusive metrics for evaluating video captioning models, we emphasize both quantitative and qualitative analysis of our model. Hence, we have used the BLEU scoring metric for quantitative analysis and have proposed a human evaluation metric for qualitative analysis, namely the Semantic Sensibility (SS) scoring metric. SS Score overcomes the shortcomings of common automated scoring metrics. This paper reports that the use of the aforementioned novelties improves the performance of state-of-the-art architectures.

Augmenting Paraphrase Generation with Syntax Information Using Graph Convolutional Networks.

Paraphrase generation is an important yet challenging task in natural language processing. Neural network-based approaches have achieved remarkable success in sequence-to-sequence learning. Previous paraphrase generation work generally ignores syntactic information regardless of its availability, with the assumption that neural nets could learn such linguistic knowledge implicitly. In this work, we make an endeavor to probe into the efficacy of explicit syntactic information for the task of paraphrase generation. Syntactic information can appear in the form of dependency trees, which could be easily acquired from off-the-shelf syntactic parsers. Such tree structures could be conveniently encoded via graph convolutional networks to obtain more meaningful sentence representations, which could improve generated paraphrases. Through extensive experiments on four paraphrase datasets with different sizes and genres, we demonstrate the utility of syntactic information in neural paraphrase generation under the framework of sequence-to-sequence modeling. Specifically, our graph convolutional network-enhanced models consistently outperform their syntax-agnostic counterparts using multiple evaluation metrics.