Musical Training Facilitates Exogenous Temporal Attention via Delta Phase Entrainment within a Sensorimotor Network.

Temporal orienting of attention plays an important role in our day-to-day lives and can use timing information from exogenous or endogenous sources. Yet, it is unclear what neural mechanisms give rise to temporal attention, and it is debated whether both exogenous and endogenous forms of temporal attention share a common neural source. Here, older adult nonmusicians (N = 47, 24 female) were randomized to undergo 8 weeks of either rhythm training, which places demands on exogenous temporal attention, or word search training as a control. The goal was to assess (1) the neural basis of exogenous temporal attention and (2) whether training-induced improvements in exogenous temporal attention can transfer to enhanced endogenous temporal attention abilities, thereby providing support for a common neural mechanism of temporal attention. Before and after training, exogenous temporal attention was assessed using a rhythmic synchronization paradigm, whereas endogenous temporal attention was evaluated via a temporally cued visual discrimination task. Results showed that rhythm training improved performance on the exogenous temporal attention task, which was associated with increased intertrial coherence within the δ (1-4 Hz) band as assessed by EEG recordings. Source localization revealed increased δ-band intertrial coherence arose from a sensorimotor network, including premotor cortex, anterior cingulate cortex, postcentral gyrus, and the inferior parietal lobule. Despite these improvements in exogenous temporal attention, such benefits were not transferred to endogenous attentional ability. These results support the notion that exogenous and endogenous temporal attention uses independent neural sources, with exogenous temporal attention relying on the precise timing of δ band oscillations within a sensorimotor network.SIGNIFICANCE STATEMENT Allocating attention to specific points in time is known as temporal attention, and may arise from external (exogenous) or internal (endogenous) sources. Despite its importance to our daily lives, it is unclear how the brain gives rise to temporal attention and whether exogenous- or endogenous-based sources for temporal attention rely on shared brain regions. Here, we demonstrate that musical rhythm training improves exogenous temporal attention, which was associated with more consistent timing of neural activity in sensory and motor processing brain regions. However, these benefits did not extend to endogenous temporal attention, indicating that temporal attention relies on different brain regions depending on the source of timing information.

Dynamic alpha power modulations and slow negative potentials track natural shifts of spatio-temporal attention.

Alpha power modulations and slow negative potentials have previously been associated with anticipatory processes in spatial and temporal top-down attention. In typical experimental designs, however, neural responses triggered by transient stimulus onsets can interfere with attention-driven activity patterns and our interpretation of such. Here, we investigated these signatures of spatio-temporal attention in a dynamic paradigm free from potentially confounding stimulus-driven activity using electroencephalography. Participants attended the cued side of a bilateral stimulus rotation and mentally counted how often one of two remembered sample orientations (i.e., the target) was displayed while ignoring the uncued side and non-target orientation. Afterwards, participants performed a delayed match-to-sample task, in which they indicated if the orientation of a probe stimulus matched the corresponding sample orientation (previously target or non-target). We observed dynamic alpha power reductions and slow negative waves around task-relevant points in space and time (i.e., onset of the target orientation in the cued hemifield) over posterior electrodes contralateral to the locus of attention. In contrast to static alpha power lateralization, these dynamic signatures correlated with subsequent memory performance (primarily detriments for matching probes of the non-target orientation), suggesting a preferential allocation of attention to task-relevant locations and time points at the expense of reduced resources and impaired performance for information outside the current focus of attention. Our findings suggest that humans can naturally and dynamically focus their attention at relevant points in space and time and that such spatio-temporal attention shifts can be reflected by dynamic alpha power modulations and slow negative potentials.

Temporal attention networks for biomedical hypothesis generation.

OBJECTIVES: Hypothesis Generation (HG) is a task that aims to uncover hidden associations between disjoint scientific terms, which influences innovations in prevention, treatment, and overall public health. Several recent studies strive to use Recurrent Neural Network (RNN) to learn evolutional embeddings for HG. However, the complex spatiotemporal dependencies of term-pair relations will be difficult to depict due to the inherent recurrent structure. This paper aims to accurately model the temporal evolution of term-pair relations using only attention mechanisms, for capturing crucial information on inferring the future connectivities. METHODS: This paper proposes a Temporal Attention Networks (TAN) to produce powerful spatiotemporal embeddings for Biomedical Hypothesis Generation. Specifically, we formulate HG problem as a future connectivity prediction task in a temporal attributed graph. Our TAN develops a Temporal Spatial Attention Module (TSAM) to establish temporal dependencies of node-pair (term-pair) embeddings between any two time-steps for smoothing spatiotemporal node-pair embeddings. Meanwhile, a Temporal Difference Attention Module (TDAM) is proposed to sharpen temporal differences of spatiotemporal embeddings for highlighting the historical changes of node-pair relations. As such, TAN can adaptively calibrate spatiotemporal embeddings by considering both continuity and difference of node-pair embeddings. RESULTS: Three real-world biomedical term relationship datasets are constructed from PubMed papers. TAN significantly outperforms the best baseline with 12.03%, 4.59 and 2.34% Micro-F1 Score improvement in Immunotherapy, Virology and Neurology, respectively. Extensive experiments demonstrate that TAN can model complex spatiotemporal dependencies of term-pairs for explicitly capturing the temporal evolution of relation, significantly outperforming existing state-of-the-art methods. CONCLUSION: We proposed a novel TAN to learn spatiotemporal embeddings based on pure attention mechanisms for HG. TAN learns the evolution of relationships by modeling both the continuity and difference of temporal term-pair embeddings. The important spatiotemporal dependencies of term-pair relations are extracted based solely on attention mechanism for generating hypotheses.

Confidence for intrusion errors during the attentional blink depends on target-defining features.

Research surrounding the attentional blink phenomenon - a deficit in responding to the second of two temporally proximal stimuli when presented 150-500 ms after the first - has used a wide variety of target-defining and response features of stimuli. The typical U-shape curve for absolute performance is robust, surviving across most stimulus features, and therefore changes in performance are discussed as dynamics in an attentional system that are nonspecific a stimulus type. However, the patterns of errors participants make might not show the same robustness, and participants' confidences in these errors might differ - potentially suggesting the involvement of different attentional or perceptual mechanisms. The present research is a comparison of error patterns and confidence in those errors when letter target stimuli are defined by either the color of the letter, the presence of a surrounding annulus, or the color of the annulus. Across three experiments, we show that participants erroneously report stimuli that are further away from T2 and they are similarly confident in specifically their post-target errors as their correct responses when annuli define targets, but not when color of the letter defines targets. Experiment 3 provides some evidence to suggest that this error pattern and associated confidence is time-dependent when the color of the annulus defines the target, but not when the color of the letter defines the target. These results raise questions concerning the nature of the errors and possibly the mechanisms of the attentional blink phenomenon itself.

Dissociation between temporal attention and Consciousness: Unconscious temporal cue induces temporal expectation effect.

The debate over the independence of attention and consciousness is ongoing. Prior studies have established that invisible spatial cues can direct attention. However, our exploration extends beyond spatial dimensions to temporal information as a potent guide for attention. A intriguing question arises: Can unconscious temporal cues trigger attentional orienting? To investigate, we employed a modified reaction-time task in Experiments 1 and 2, using Gabor stimuli or human facial stimuli as temporal cues rendered invisible through continuous flash suppression. We aimed to uncover temporal expectation effects (TE effects) without conscious awareness. Moreover, Experiments 3 and 4 probed the boundaries of this unconscious processing, assessing whether conscious temporal cues could modulate TE effects. Our results confirm that invisible temporal cues can induce TE effects, and these effects can be overridden by conscious temporal cues. This dissociation between temporal attention and consciousness provide a new perspective on our understanding of their relationship.

CenterADNet: Infrared Video Target Detection Based on Central Point Regression.

Infrared video target detection is a fundamental technology within infrared warning and tracking systems. In long-distance infrared remote sensing images, targets often manifest as circular spots or even single points. Due to the weak and similar characteristics of the target to the background noise, the intelligent detection of these targets is extremely complex. Existing deep learning-based methods are affected by the downsampling of image features by convolutional neural networks, causing the features of small targets to almost disappear. So, we propose a new infrared video weak-target detection network based on central point regression. We focus on suppressing the image background by fusing the different features between consecutive frames with the original image features to eliminate the background's influence. We also employ high-resolution feature preservation and incorporate a spatial-temporal attention module into the network to capture as many target features as possible and improve detection accuracy. Our method achieves superior results on the infrared image weak aircraft target detection dataset proposed by the National University of Defense Technology, as well as on the simulated dataset generated based on real-world observation. This demonstrates the efficiency of our approach for detecting weak point targets in infrared continuous images.

AST3DRNet: Attention-Based Spatio-Temporal 3D Residual Neural Networks for Traffic Congestion Prediction.

Traffic congestion prediction has become an indispensable component of an intelligent transport system. However, one limitation of the existing methods is that they treat the effects of spatio-temporal correlations on traffic prediction as invariable during modeling spatio-temporal features, which results in inadequate modeling. In this paper, we propose an attention-based spatio-temporal 3D residual neural network, named AST3DRNet, to directly forecast the congestion levels of road networks in a city. AST3DRNet combines a 3D residual network and a self-attention mechanism together to efficiently model the spatial and temporal information of traffic congestion data. Specifically, by stacking 3D residual units and 3D convolution, we proposed a 3D convolution module that can simultaneously capture various spatio-temporal correlations. Furthermore, a novel spatio-temporal attention module is proposed to explicitly model the different contributions of spatio-temporal correlations in both spatial and temporal dimensions through the self-attention mechanism. Extensive experiments are conducted on a real-world traffic congestion dataset in Kunming, and the results demonstrate that AST3DRNet outperforms the baselines in short-term (5/10/15 min) traffic congestion predictions with an average accuracy improvement of 59.05%, 64.69%, and 48.22%, respectively.

Human Action Recognition and Note Recognition: A Deep Learning Approach Using STA-GCN.

Human action recognition (HAR) is growing in machine learning with a wide range of applications. One challenging aspect of HAR is recognizing human actions while playing music, further complicated by the need to recognize the musical notes being played. This paper proposes a deep learning-based method for simultaneous HAR and musical note recognition in music performances. We conducted experiments on Morin khuur performances, a traditional Mongolian instrument. The proposed method consists of two stages. First, we created a new dataset of Morin khuur performances. We used motion capture systems and depth sensors to collect data that includes hand keypoints, instrument segmentation information, and detailed movement information. We then analyzed RGB images, depth images, and motion data to determine which type of data provides the most valuable features for recognizing actions and notes in music performances. The second stage utilizes a Spatial Temporal Attention Graph Convolutional Network (STA-GCN) to recognize musical notes as continuous gestures. The STA-GCN model is designed to learn the relationships between hand keypoints and instrument segmentation information, which are crucial for accurate recognition. Evaluation on our dataset demonstrates that our model outperforms the traditional ST-GCN model, achieving an accuracy of 81.4%.

Load Prediction in Double-Channel Residual Self-Attention Temporal Convolutional Network with Weight Adaptive Updating in Cloud Computing.

When resource demand increases and decreases rapidly, container clusters in the cloud environment need to respond to the number of containers in a timely manner to ensure service quality. Resource load prediction is a prominent challenge issue with the widespread adoption of cloud computing. A novel cloud computing load prediction method has been proposed, the Double-channel residual Self-attention Temporal convolutional Network with Weight adaptive updating (DSTNW), in order to make the response of the container cluster more rapid and accurate. A Double-channel Temporal Convolution Network model (DTN) has been developed to capture long-term sequence dependencies and enhance feature extraction capabilities when the model handles long load sequences. Double-channel dilated causal convolution has been adopted to replace the single-channel dilated causal convolution in the DTN. A residual temporal self-attention mechanism (SM) has been proposed to improve the performance of the network and focus on features with significant contributions from the DTN. DTN and SM jointly constitute a dual-channel residual self-attention temporal convolutional network (DSTN). In addition, by evaluating the accuracy aspects of single and stacked DSTNs, an adaptive weight strategy has been proposed to assign corresponding weights for the single and stacked DSTNs, respectively. The experimental results highlight that the developed method has outstanding prediction performance for cloud computing in comparison with some state-of-the-art methods. The proposed method achieved an average improvement of 24.16% and 30.48% on the Container dataset and Google dataset, respectively.

Spatial Attention Tunes Temporal Processing in Early Visual Cortex by Speeding and Slowing Alpha Oscillations.

The perception of dynamic visual stimuli relies on two apparently conflicting perceptual mechanisms: rapid visual input must sometimes be integrated into unitary percepts but at other times must be segregated or parsed into separate objects or events. Though they have opposite effects on our perceptual experience, the deployment of spatial attention benefits both operations. Little is known about the neural mechanisms underlying this impact of spatial attention on temporal perception. Here, we record magnetoencephalography (MEG) in male and female humans to demonstrate that the deployment of spatial attention for the purpose of segregating or integrating visual stimuli impacts prestimulus oscillatory activity in retinotopic visual brain areas where the attended location is represented. Alpha band oscillations contralateral to an attended location are therefore faster than ipsilateral oscillations when stimuli appearing at this location will need to be segregated, but slower in expectation of the need for integration, consistent with the idea that α frequency is linked to perceptual sampling rate. These results demonstrate a novel interaction between temporal visual processing and the allocation of attention in space.SIGNIFICANCE STATEMENT Our environment is dynamic and visual input therefore varies over time. To make sense of continuously changing information, our visual system balances two complementary processes: temporal segregation in order to identify changes, and temporal integration to identify consistencies in time. When we know that a circumstance requires use of one or the other of these operations, we are able to prepare for this, and this preparation can be tracked in oscillatory brain activity. Here, we show how this preparation for temporal processing can be focused spatially. When we expect to integrate or segregate visual stimuli that will appear at a specific location, oscillatory brain activity changes in visual areas responsible for the representation of that location. In this way, spatial and temporal mechanisms interact to support adaptive, efficient perception.

The Spatiotemporal Link of Temporal Expectations: Contextual Temporal Expectation Is Independent of Spatial Attention.

Temporal expectation is the ability to construct predictions regarding the timing of events, based on previously experienced temporal regularities of different types. For example, cue-based expectations are constructed when a cue validly indicates when a target is expected to occur. However, in the absence of such cues, expectations can be constructed based on contextual temporal information, including the onset distribution of the event and recent prior experiences, both providing implicit probabilistic information regarding the timing of the event. It was previously suggested that cue-based temporal expectation is exerted via synchronization of spatially specific neural activity at a predictable time of a target, within receptive fields corresponding to the expected location of the target. Here, we tested whether the same theoretical model holds for contextual temporal effects. Participants (n = 40, 25 females) performed a speeded spatial-cuing detection task with two-thirds valid spatial cues. The hazard-rate function of the target was modulated by varying the foreperiod-the interval between the spatial cue and the target-among trials and was manipulated between groups by changing the interval distribution. Reaction times were analyzed using both frequentist and Bayesian generalized linear mixed models, accounting for hazard and sequential effects. Results showed that the effects of contextual temporal structures on reaction times were independent of spatial attention. This suggests that the spatiotemporal mechanisms, thought to account for cue-based expectation, cannot explain other sources of temporal expectations. We conclude that expectations based on contextual structures have different characteristics than cue-based temporal expectation, suggesting reliance on distinct neural mechanisms.SIGNIFICANCE STATEMENT Temporal expectation is the ability to predict an event onset based on temporal regularities. A neurophysiological model suggested that temporal expectation relies on the synchronization of spatially specific neurons whose receptive fields represent the attended location. This model predicts that temporal expectation would be evident solely within the locus of spatial attention. Existing evidence supported this model for expectation based on associations between a temporal cue and a target, but here we show that it cannot account for temporal expectation that is based on contextual information, that is, the distribution of intervals and recent priors. These findings reveal the existence of different predictive mechanisms for cued and contextual temporal predictions, with the former depending on spatial attention and the latter nonspatially specific.

Prediction of Pollutant Concentration Based on Spatial-Temporal Attention, ResNet and ConvLSTM.

Accurate and reliable prediction of air pollutant concentrations is important for rational avoidance of air pollution events and government policy responses. However, due to the mobility and dynamics of pollution sources, meteorological conditions, and transformation processes, pollutant concentration predictions are characterized by great uncertainty and instability, making it difficult for existing prediction models to effectively extract spatial and temporal correlations. In this paper, a powerful pollutant prediction model (STA-ResConvLSTM) is proposed to achieve accurate prediction of pollutant concentrations. The model consists of a deep learning network model based on a residual neural network (ResNet), a spatial-temporal attention mechanism, and a convolutional long short-term memory neural network (ConvLSTM). The spatial-temporal attention mechanism is embedded in each residual unit of the ResNet to form a new residual neural network with the spatial-temporal attention mechanism (STA-ResNet). Deep extraction of spatial-temporal distribution features of pollutant concentrations and meteorological data from several cities is carried out using STA-ResNet. Its output is used as an input to the ConvLSTM, which is further analyzed to extract preliminary spatial-temporal distribution features extracted from the STA-ResNet. The model realizes the spatial-temporal correlation of the extracted feature sequences to accurately predict pollutant concentrations in the future. In addition, experimental studies on urban agglomerations around Long Beijing show that the prediction model outperforms various popular baseline models in terms of accuracy and stability. For the single-step prediction task, the proposed pollutant concentration prediction model performs well, exhibiting a root-mean-square error (RMSE) of 9.82. Furthermore, even for the pollutant prediction task of 1 to 48 h, we performed a multi-step prediction and achieved a satisfactory performance, being able to achieve an average RMSE value of 13.49.

A dynamic 1/f noise protocol to assess visual attention without biasing perceptual processing.

Psychophysical paradigms measure visual attention via localized test items to which observers must react or whose features have to be discriminated. These items, however, potentially interfere with the intended measurement, as they bias observers' spatial and temporal attention to their location and presentation time. Furthermore, visual sensitivity for conventional test items naturally decreases with retinal eccentricity, which prevents direct comparison of central and peripheral attention assessments. We developed a stimulus that overcomes these limitations. A brief oriented discrimination signal is seamlessly embedded into a continuously changing 1/f noise field, such that observers cannot anticipate potential test locations or times. Using our new protocol, we demonstrate that local orientation discrimination accuracy for 1/f filtered signals is largely independent of retinal eccentricity. Moreover, we show that items present in the visual field indeed shape the distribution of visual attention, suggesting that classical studies investigating the spatiotemporal dynamics of visual attention via localized test items may have obtained a biased measure. We recommend our protocol as an efficient method to evaluate the behavioral and neurophysiological correlates of attentional orienting across space and time.

Differences in Rhythmic Neural Activity Supporting the Temporal and Spatial Cueing of Attention.

The neural processes serving the orienting of attention toward goal-relevant stimuli are generally examined with informative cues that direct visual attention to a spatial location. However, cues predicting the temporal emergence of an object are also known to be effective in attentional orienting but are implemented less often. Differences in the neural oscillatory dynamics supporting these divergent types of attentional orienting have only rarely been examined. In this study, we utilized magnetoencephalography and an adapted Posner cueing task to investigate the spectral specificity of neural oscillations underlying these different types of attentional orienting (i.e., spatial vs. temporal). We found a spectral dissociation of attentional cueing, such that alpha (10-16 Hz) oscillations were central to spatial orienting and theta (3-6 Hz) oscillations were critical to temporal orienting. Specifically, we observed robust decreases in alpha power during spatial orienting in key attention areas (i.e., lateral occipital, posterior cingulate, and hippocampus), along with strong theta increases during temporal orienting in the primary visual cortex. These results suggest that the oscillatory dynamics supporting attentional orienting are spectrally and anatomically specific, such that spatial orienting is served by stronger alpha oscillations in attention regions, whereas temporal orienting is associated with stronger theta responses in visual sensory regions.

A Graph Neural Network with Spatio-Temporal Attention for Multi-Sources Time Series Data: An Application to Frost Forecast.

Frost forecast is an important issue in climate research because of its economic impact on several industries. In this study, we propose GRAST-Frost, a graph neural network (GNN) with spatio-temporal architecture, which is used to predict minimum temperatures and the incidence of frost. We developed an IoT platform capable of acquiring weather data from an experimental site, and in addition, data were collected from 10 weather stations in close proximity to the aforementioned site. The model considers spatial and temporal relations while processing multiple time series simultaneously. Performing predictions of 6, 12, 24, and 48 h in advance, this model outperforms classical time series forecasting methods, including linear and nonlinear machine learning methods, simple deep learning architectures, and nongraph deep learning models. In addition, we show that our model significantly improves on the current state of the art of frost forecasting methods.

Occluded Pedestrian-Attribute Recognition for Video Sensors Using Group Sparsity.

Pedestrians are often obstructed by other objects or people in real-world vision sensors. These obstacles make pedestrian-attribute recognition (PAR) difficult; hence, occlusion processing for visual sensing is a key issue in PAR. To address this problem, we first formulate the identification of non-occluded frames as temporal attention based on the sparsity of a crowded video. In other words, a model for PAR is guided to prevent paying attention to the occluded frame. However, we deduced that this approach cannot include a correlation between attributes when occlusion occurs. For example, "boots" and "shoe color" cannot be recognized simultaneously when the foot is invisible. To address the uncorrelated attention issue, we propose a novel temporal-attention module based on group sparsity. Group sparsity is applied across attention weights in correlated attributes. Accordingly, physically-adjacent pedestrian attributes are grouped, and the attention weights of a group are forced to focus on the same frames. Experimental results indicate that the proposed method achieved 1.18% and 6.21% higher F1-scores than the advanced baseline method on the occlusion samples in DukeMTMC-VideoReID and MARS video-based PAR datasets, respectively.

Lane Mark Detection with Pre-Aligned Spatial-Temporal Attention.

Lane mark detection plays an important role in autonomous driving under structural environments. Many deep learning-based lane mark detection methods have been put forward in recent years. However, most of current methods limit their solutions within one single image and do not make use of the de facto successive image input during the driving scene, which may lead to inferior performance in some challenging scenarios such as occlusion, shadows, and lane mark degradation. To address the issue, we propose a novel lane mark detection network which takes pre-aligned multiple successive frames as inputs to produce more stable predictions. A Spatial-Temporal Attention Module (STAM) is designed in the network to adaptively aggregate the feature information of history frames to the current frame. Various structure of the STAM is also studied to ensure the best performance. Experiments on Tusimple and ApolloScape datasets show that our method can effectively improve lane mark detection and achieve state-of-the-art performance.

Attention Limitations in the Detection and Identification of Alarms in Close Temporal Proximity.

OBJECTIVE: The aim of this study was to establish the effects of simultaneous and asynchronous masking on the detection and identification of visual and auditory alarms in close temporal proximity. BACKGROUND: In complex and highly coupled systems, malfunctions can trigger numerous alarms within a short period of time. During such alarm floods, operators may fail to detect and identify alarms due to asynchronous and simultaneous masking. To date, the effects of masking on detection and identification have been studied almost exclusively for two alarms during single-task performance. This research examines 1) how masking affects alarm detection and identification in multitask environments and 2) whether those effects increase as a function of the number of alarms. METHOD: Two experiments were conducted using a simulation of a drone-based package delivery service. Participants were required to ensure package delivery and respond to visual and auditory alarms associated with eight drones. The alarms were presented at various stimulus onset asynchronies (SOAs). The dependent measures included alarm detection rate, identification accuracy, and response time. RESULTS: Masking was observed intramodally and cross-modally for visual and auditory alarms. The SOAs at which asynchronous masking occurred were longer than reported in basic research on masking. The effects of asynchronous and, even more so, simultaneous masking became stronger as the number of alarms increased. CONCLUSION: Masking can lead to breakdowns in the detection and identification of alarms in close temporal proximity in complex data-rich domains. APPLICATION: The findings from this research provide guidance for the design of alarm systems.

Selection of Essential Neural Activity Timesteps for Intracortical Brain-Computer Interface Based on Recurrent Neural Network.

Intracortical brain-computer interfaces (iBCIs) translate neural activity into control commands, thereby allowing paralyzed persons to control devices via their brain signals. Recurrent neural networks (RNNs) are widely used as neural decoders because they can learn neural response dynamics from continuous neural activity. Nevertheless, excessively long or short input neural activity for an RNN may decrease its decoding performance. Based on the temporal attention module exploiting relations in features over time, we propose a temporal attention-aware timestep selection (TTS) method that improves the interpretability of the salience of each timestep in an input neural activity. Furthermore, TTS determines the appropriate input neural activity length for accurate neural decoding. Experimental results show that the proposed TTS efficiently selects 28 essential timesteps for RNN-based neural decoders, outperforming state-of-the-art neural decoders on two nonhuman primate datasets (R2=0.76±0.05 for monkey Indy and CC=0.91±0.01 for monkey N). In addition, it reduces the computation time for offline training (reducing 5-12%) and online prediction (reducing 16-18%). When visualizing the attention mechanism in TTS, the preparatory neural activity is consecutively highlighted during arm movement, and the most recent neural activity is highlighted during the resting state in nonhuman primates. Selecting only a few essential timesteps for an RNN-based neural decoder provides sufficient decoding performance and requires only a short computation time.

Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information.

In recent times, as interest in stress control has increased, many studies on stress recognition have been conducted. Several studies have been based on physiological signals, but the disadvantage of this strategy is that it requires physiological-signal-acquisition devices. Another strategy employs facial-image-based stress-recognition methods, which do not require devices, but predominantly use handcrafted features. However, such features have low discriminating power. We propose a deep-learning-based stress-recognition method using facial images to address these challenges. Given that deep-learning methods require extensive data, we constructed a large-capacity image database for stress recognition. Furthermore, we used temporal attention, which assigns a high weight to frames that are highly related to stress, as well as spatial attention, which assigns a high weight to regions that are highly related to stress. By adding a network that inputs the facial landmark information closely related to stress, we supplemented the network that receives only facial images as the input. Experimental results on our newly constructed database indicated that the proposed method outperforms contemporary deep-learning-based recognition methods.