Earthquake alerting based on spatial geodetic data by spatiotemporal information transformation learning.

Alerting for imminent earthquakes is particularly challenging due to the high nonlinearity and nonstationarity of geodynamical phenomena. In this study, based on spatiotemporal information (STI) transformation for high-dimensional real-time data, we developed a model-free framework, i.e., real-time spatiotemporal information transformation learning (RSIT), for extending the nonlinear and nonstationary time series. Specifically, by transforming high-dimensional information of the global navigation satellite system into one-dimensional dynamics via the STI strategy, RSIT efficiently utilizes two criteria of the transformed one-dimensional dynamics, i.e., unpredictability and instability. Such two criteria contemporaneously signal a potential critical transition of the geodynamical system, thereby providing early-warning signals of possible upcoming earthquakes. RSIT explores both the spatial and temporal dynamics of real-world data on the basis of a solid theoretical background in nonlinear dynamics and delay-embedding theory. The effectiveness of RSIT was demonstrated on geodynamical data of recent earthquakes from a number of regions across at least 4 y and through further comparison with existing methods.

Satellite-Based Global Sea Surface Oxygen Mapping and Interpretation with Spatiotemporal Machine Learning.

The assessment of dissolved oxygen (DO) concentration at the sea surface is essential for comprehending the global ocean oxygen cycle and associated environmental and biochemical processes as it serves as the primary site for photosynthesis and sea-air exchange. However, limited comprehensive measurements and imprecise numerical simulations have impeded the study of global sea surface DO and its relationship with environmental challenges. This paper presents a novel spatiotemporal information embedding machine-learning framework that provides explanatory insights into the underlying driving mechanisms. By integrating extensive in situ data and high-resolution satellite data, the proposed framework successfully generated high-resolution (0.25° × 0.25°) estimates of DO concentration with exceptional accuracy (R2 = 0.95, RMSE = 11.95 µmol/kg, and test number = 2805) for near-global sea surface areas from 2010 to 2018, uncertainty estimated to be ±13.02 µmol/kg. The resulting sea surface DO data set exhibits precise spatial distribution and reveals compelling correlations with prominent marine phenomena and environmental stressors. Leveraging its interpretability, our model further revealed the key influence of marine factors on surface DO and their implications for environmental issues. The presented machine-learning framework offers an improved DO data set with higher resolution, facilitating the exploration of oceanic DO variability, deoxygenation phenomena, and their potential consequences for environments.

Adaptively Lightweight Spatiotemporal Information-Extraction-Operator-Based DL Method for Aero-Engine RUL Prediction.

Accurate prediction of machine RUL plays a crucial role in reducing human casualties and economic losses, which is of significance. The ability to handle spatiotemporal information contributes to improving the prediction performance of machine RUL. However, most existing models for spatiotemporal information processing are not only complex in structure but also lack adaptive feature extraction capabilities. Therefore, a lightweight operator with adaptive spatiotemporal information extraction ability named Involution GRU (Inv-GRU) is proposed for aero-engine RUL prediction. Involution, the adaptive feature extraction operator, is replaced by the information connection in the gated recurrent unit to achieve adaptively spatiotemporal information extraction and reduce the parameters. Thus, Inv-GRU can well extract the degradation information of the aero-engine. Then, for the RUL prediction task, the Inv-GRU-based deep learning (DL) framework is firstly constructed, where features extracted by Inv-GRU and several human-made features are separately processed to generate health indicators (HIs) from multi-raw data of aero-engines. Finally, fully connected layers are adopted to reduce the dimension and regress RUL based on the generated HIs. By applying the Inv-GRU-based DL framework to the Commercial Modular Aero Propulsion System Simulation (C-MAPSS) datasets, successful predictions of aero-engines RUL have been achieved. Quantitative comparative experiments have demonstrated the advantage of the proposed method over other approaches in terms of both RUL prediction accuracy and computational burden.

Enhancing UAV Detection in Surveillance Camera Videos through Spatiotemporal Information and Optical Flow.

The growing intelligence and prevalence of drones have led to an increase in their disorderly and illicit usage, posing substantial risks to aviation and public safety. This paper focuses on addressing the issue of drone detection through surveillance cameras. Drone targets in images possess distinctive characteristics, including small size, weak energy, low contrast, and limited and varying features, rendering precise detection a challenging task. To overcome these challenges, we propose a novel detection method that extends the input of YOLOv5s to a continuous sequence of images and inter-frame optical flow, emulating the visual mechanisms employed by humans. By incorporating the image sequence as input, our model can leverage both temporal and spatial information, extracting more features of small and weak targets through the integration of spatiotemporal data. This integration augments the accuracy and robustness of drone detection. Furthermore, the inclusion of optical flow enables the model to directly perceive the motion information of drone targets across consecutive frames, enhancing its ability to extract and utilize features from dynamic objects. Comparative experiments demonstrate that our proposed method of extended input significantly enhances the network's capability to detect small moving targets, showcasing competitive performance in terms of accuracy and speed. Specifically, our method achieves a final average precision of 86.87%, representing a noteworthy 11.49% improvement over the baseline, and the speed remains above 30 frames per second. Additionally, our approach is adaptable to other detection models with different backbones, providing valuable insights for domains such as Urban Air Mobility and autonomous driving.

An Accurate and Convenient Method of Vehicle Spatiotemporal Distribution Recognition Based on Computer Vision.

The Convenient and accurate identification of the traffic load of passing vehicles is of great significance to bridge health monitoring. The existing identification approaches often require prior environment knowledge to determine the location of the vehicle load, i.e., prior information of the road, which is inconvenient in practice and therefore limits its application. Moreover, camera disturbance usually reduces the measurement accuracy in case of long-term monitoring. In this study, a novel approach to identify the spatiotemporal information of passing vehicles is proposed based on computer vision. The position relationship between the camera and the passing vehicle is established, and then the location of the passing vehicle can be calculated by setting the camera shooting point as the origin. Since the angle information of the camera is pre-determined, the identification result is robust to camera disturbance. Lab-scale test and field measurement have been conducted to validate the reliability and accuracy of the proposed method.

Spatiotemporal Transformer Neural Network for Time-Series Forecasting.

Predicting high-dimensional short-term time-series is a difficult task due to the lack of sufficient information and the curse of dimensionality. To overcome these problems, this study proposes a novel spatiotemporal transformer neural network (STNN) for efficient prediction of short-term time-series with three major features. Firstly, the STNN can accurately and robustly predict a high-dimensional short-term time-series in a multi-step-ahead manner by exploiting high-dimensional/spatial information based on the spatiotemporal information (STI) transformation equation. Secondly, the continuous attention mechanism makes the prediction results more accurate than those of previous studies. Thirdly, we developed continuous spatial self-attention, temporal self-attention, and transformation attention mechanisms to create a bridge between effective spatial information and future temporal evolution information. Fourthly, we show that the STNN model can reconstruct the phase space of the dynamical system, which is explored in the time-series prediction. The experimental results demonstrate that the STNN significantly outperforms the existing methods on various benchmarks and real-world systems in the multi-step-ahead prediction of a short-term time-series.

Single-Cell Imaging of m6 A Modified RNA Using m6 A-Specific In Situ Hybridization Mediated Proximity Ligation Assay (m6 AISH-PLA).

N6 -methyladenosine (m6 A) modification-the most prevalent mammalian RNA internal modification-plays key regulatory roles in mRNA metabolism. Current approaches for m6 A modified RNA analysis limit at bulk-population level, resulting in a loss of spatiotemporal and cell-to-cell variability information. Here we proposed a m6 A-specific in situ hybridization mediated proximity ligation assay (m6 AISH-PLA) for cellular imaging of m6 A RNA, allowing to identify m6 A modification at specific location in RNAs and image m6 A RNA with single-cell and single-molecule resolution. Using m6 AISH-PLA, we investigated the m6 A level and subcellular location of HSP70 RNA103-m6 A in response to heat shock stress, and found an increased m6 A modified ratio and an increased distribution ratio in cytoplasm under heat shock. m6 AISH-PLA can serve in the study of m6 A RNA in single cells for deciphering epitranscriptomic mechanisms and assisting clinical diagnosis.

Multi-organ spatiotemporal information aware model for sepsis mortality prediction.

BACKGROUND: Sepsis is a syndrome involving multi-organ dysfunction, and the mortality in sepsis patients correlates with the number of lesioned organs. Precise prognosis models play a pivotal role in enabling healthcare practitioners to administer timely and accurate interventions for sepsis, thereby augmenting patient outcomes. Nevertheless, the majority of available models consider the overall physiological attributes of patients, overlooking the asynchronous spatiotemporal interactions among multiple organ systems. These constraints hinder a full application of such models, particularly when dealing with limited clinical data. To surmount these challenges, a comprehensive model, denoted as recurrent Graph Attention Network-multi Gated Recurrent Unit (rGAT-mGRU), was proposed. Taking into account the intricate spatiotemporal interactions among multiple organ systems, the model predicted in-hospital mortality of sepsis using data collected within the 48-hour period post-diagnosis. MATERIAL AND METHODS: Multiple parallel GRU sub-models were formulated to investigate the temporal physiological variations of single organ systems. Meanwhile, a GAT structure featuring a memory unit was constructed to capture spatiotemporal connections among multi-organ systems. Additionally, an attention-injection mechanism was employed to govern the data flowing within the network pertaining to multi-organ systems. The proposed model underwent training and testing using a dataset of 10,181 sepsis cases extracted from the Medical Information Mart for Intensive Care III (MIMIC-III) database. To evaluate the model's superiority, it was compared with the existing common baseline models. Furthermore, ablation experiments were designed to elucidate the rationale and robustness of the proposed model. RESULTS: Compared with the baseline models for predicting mortality of sepsis, the rGAT-mGRU model demonstrated the largest area under the receiver operating characteristic curve (AUROC) of 0.8777 ± 0.0039 and the maximum area under the precision-recall curve (AUPRC) of 0.5818 ± 0.0071, with sensitivity of 0.8358 ± 0.0302 and specificity of 0.7727 ± 0.0229, respectively. The proposed model was capable of delineating the varying contribution of the involved organ systems at distinct moments, as specifically illustrated by the attention weights. Furthermore, it exhibited consistent performance even in the face of limited clinical data. CONCLUSION: The rGAT-mGRU model has the potential to indicate sepsis prognosis by extracting the dynamic spatiotemporal interplay information inherent in multi-organ systems during critical diseases, thereby providing clinicians with auxiliary decision-making support.

Exploring the potential of the sit-to-stand test for self-assessment of physical condition in advanced knee osteoarthritis patients using computer vision.

Introduction: Knee osteoarthritis (KOA) is a prevalent condition often associated with a decline in patients' physical function. Objective self-assessment of physical conditions poses challenges for many advanced KOA patients. To address this, we explored the potential of a computer vision method to facilitate home-based physical function self-assessments. Methods: We developed and validated a simple at-home artificial intelligence approach to recognize joint stiffness levels and physical function in individuals with advanced KOA. One hundred and four knee osteoarthritis (KOA) patients were enrolled, and we employed the WOMAC score to evaluate their physical function and joint stiffness. Subsequently, patients independently recorded videos of five sit-to-stand tests in a home setting. Leveraging the AlphaPose and VideoPose algorithms, we extracted time-series data from these videos, capturing three-dimensional spatiotemporal information reflecting changes in key joint angles over time. To deepen our study, we conducted a quantitative analysis using the discrete wavelet transform (DWT), resulting in two wavelet coefficients: the approximation coefficients (cA) and the detail coefficients (cD). Results: Our analysis specifically focused on four crucial joint angles: "the right hip," "right knee," "left hip," and "left knee." Qualitative analysis revealed distinctions in the time-series data related to functional limitations and stiffness among patients with varying levels of KOA. In quantitative analysis, we observed variations in the cA among advanced KOA patients with different levels of physical function and joint stiffness. Furthermore, there were no significant differences in the cD between advanced KOA patients, demonstrating different levels of physical function and joint stiffness. It suggests that the primary difference in overall movement patterns lies in the varying degrees of joint stiffness and physical function among advanced KOA patients. Discussion: Our method, designed to be low-cost and user-friendly, effectively captures spatiotemporal information distinctions among advanced KOA patients with varying stiffness levels and functional limitations utilizing smartphones. This study provides compelling evidence for the potential of our approach in enabling self-assessment of physical condition in individuals with advanced knee osteoarthritis.

Light-emitting probes for in situ sensing of plant information.

Monitoring plant physiological information for gaining a comprehensive understanding of plant growth and stress responses contributes to safeguarding plant health. Light-emitting probes - in terms of small-molecule, nanomaterials-based, and genetically protein-based probes - can be introduced into plants through foliar and root treatment or genetic transformation. These probes offer exciting opportunities for sensitive and in situ monitoring of dynamic plant chemical information - for example, reactive oxygen species (ROS), calcium ions, phytohormones - with spatiotemporal resolution. In this review we explore the sensing mechanisms of these light-emitting probes and their applications in monitoring various chemical information in plants in situ. These probes can be used as part of a sentinel plant approach to provide stress warning in the field or to explore plant signaling pathways.

Construction of chub mackerel (Scomber japonicus) fishing ground prediction model in the northwestern Pacific Ocean based on deep learning and marine environmental variables.

Accurate prediction of the central fishing grounds of chub mackerel is substantial for assessing and managing marine fishery resources. Based on the high-seas chub mackerel fishery statistics and multi-factor ocean remote-sensing environmental data in the Northwest Pacific Ocean from 2014 to 2021, this article applied the gravity center of the fishing grounds, 2DCNN, and 3DCNN models to analyze the spatial and temporal variability of the chub mackerel catches and fishing grounds. Results:1) the primary fishing season of chub mackerel fishery was April-November which catches were mainly concentrated in 39°âˆ¼43°N, 149°âˆ¼154°E. 2) Since 2019, the annual gravity center of the fishing grounds has continued to move northeastward; the monthly gravity center has prominent seasonal migratory characteristics. 3) 3DCNN model was better than the 2DCNN model. 4) For 3DCNN, the model prioritized learning information on the most easily distinguishable ocean remote-sensing environmental variables in different classifications.

Multi-scale and attention enhanced graph convolution network for skeleton-based violence action recognition.

Graph convolution networks (GCNs) have been widely used in the field of skeleton-based human action recognition. However, it is still difficult to improve recognition performance and reduce parameter complexity. In this paper, a novel multi-scale attention spatiotemporal GCN (MSA-STGCN) is proposed for human violence action recognition by learning spatiotemporal features from four different skeleton modality variants. Firstly, the original joint data are preprocessed to obtain joint position, bone vector, joint motion and bone motion datas as inputs of recognition framework. Then, a spatial multi-scale graph convolution network based on the attention mechanism is constructed to obtain the spatial features from joint nodes, while a temporal graph convolution network in the form of hybrid dilation convolution is designed to enlarge the receptive field of the feature map and capture multi-scale context information. Finally, the specific relationship in the different skeleton data is explored by fusing the information of multi-stream related to human joints and bones. To evaluate the performance of the proposed MSA-STGCN, a skeleton violence action dataset: Filtered NTU RGB+D was constructed based on NTU RGB+D120. We conducted experiments on constructed Filtered NTU RGB+D and Kinetics Skeleton 400 datasets to verify the performance of the proposed recognition framework. The proposed method achieves an accuracy of 95.3% on the Filtered NTU RGB+D with the parameters 1.21M, and an accuracy of 36.2% (Top-1) and 58.5% (Top-5) on the Kinetics Skeleton 400, respectively. The experimental results on these two skeleton datasets show that the proposed recognition framework can effectively recognize violence actions without adding parameters.

A Comparison of Particulate Exposure Levels during Taxi, Bus, and Metro Commuting among Four Chinese Megacities.

Exposure to inhalable particulate matter pollution is a hazard to human health. Many studies have examined the in-transit particulate matter pollution across multiple travel modes. However, limited information is available on the comparison of in-transit exposure among cities that experience different climates and weather patterns. This study aimed to examine the variations in in-cabin particle concentrations during taxi, bus, and metro commutes among four megacities located in the inland and coastal areas of China. To this end, we employed a portable monitoring approach to measure in-transit particle concentrations and the corresponding transit conditions using spatiotemporal information. The results highlighted significant differences in in-cabin particle concentrations among the four cities, indicating that PM concentrations varied in an ascending order of, and the ratios of different-sized particle concentrations varied in a descending order of CS, SZ, GZ, and WH. Variations in in-cabin particle concentrations during bus and metro transits between cities were mainly positively associated with urban background particle concentrations. Unlike those in bus and metro transit, in-cabin PM concentrations in taxi transit were negatively associated with urban precipitation and wind speed. The variations in particle concentrations during the trip were significantly associated with passenger density, posture, the in-cabin location of investigators, and window condition, some of which showed interactive effects. Our findings suggest that improving the urban background environment is essential for reducing particulate pollution in public transport microenvironments. Moreover, optimizing the scheduling of buses and the distribution of bus stops might contribute to mitigating the in-cabin exposure levels in transit. With reference to our methods and insights, policymakers and other researchers may further explore in-transit exposure to particle pollution in different cities.

SAUN: Stack attention U-Net for left ventricle segmentation from cardiac cine magnetic resonance imaging.

PURPOSE: Quantification of left ventricular (LV) volume, ejection fraction and myocardial mass from multi-slice multi-phase cine MRI requires accurate segmentation of the LV in many images. We propose a stack attention-based convolutional neural network (CNN) approach for fully automatic segmentation from short-axis cine MR images. METHODS: To extract the relevant spatiotemporal image features, we introduce two kinds of stack methods, spatial stack model and temporal stack model, combining the target image with its neighboring images as the input of a CNN. A stack attention mechanism is proposed to weigh neighboring image slices in order to extract the relevant features using the target image as a guide. Based on stack attention and standard U-Net, a novel Stack Attention U-Net (SAUN) is proposed and trained to perform the semantic segmentation task. A loss function combining cross-entropy and Dice is used to train SAUN. The performance of the proposed method was evaluated on an internal and a public dataset using technical metrics including Dice, Hausdorff distance (HD), and mean contour distance (MCD), as well as clinical parameters, including left ventricular ejection fraction (LVEF) and myocardial mass (LVM). In addition, the results of SAUN were compared to previously presented CNN methods, including U-Net and SegNet. RESULTS: The spatial stack attention model resulted in better segmentation results than the temporal stack model. On the internal dataset comprising of 167 post-myocardial infarction patients and 57 healthy volunteers, our method achieved a mean Dice of 0.91, HD of 3.37 mm, and MCD of 1.08 mm. Evaluation on the publicly available ACDC dataset demonstrated good generalization performance, yielding a Dice of 0.92, HD of 9.4 mm, and MCD of 0.74 mm on end-diastolic images, and a Dice of 0.89, HD of 7.1 mm and MCD of 1.03 mm on end-systolic images. The Pearson correlation coefficient of LVEF and LVM between automatically and manually derived results were higher than 0.98 in both datasets. CONCLUSION: We developed a CNN with a stack attention mechanism to automatically segment the LV chamber and myocardium from the multi-slice short-axis cine MRI. The experimental results demonstrate that the proposed approach exceeds existing state-of-the-art segmentation methods and verify its potential clinical applicability.

Spatiotemporal Bayesian Regularization for Cardiac Strain Imaging: Simulation and In Vivo Results.

Cardiac strain imaging (CSI) plays a critical role in the detection of myocardial motion abnormalities. Displacement estimation is an important processing step to ensure the accuracy and precision of derived strain tensors. In this paper, we propose and implement Spatiotemporal Bayesian regularization (STBR) algorithms for two-dimensional (2-D) normalized cross-correlation (NCC) based multi-level block matching along with incorporation into a Lagrangian cardiac strain estimation framework. Assuming smooth temporal variation over a short span of time, the proposed STBR algorithm performs displacement estimation using at least four consecutive ultrasound radio-frequency (RF) frames by iteratively regularizing 2-D NCC matrices using information from a local spatiotemporal neighborhood in a Bayesian sense. Two STBR schemes are proposed to construct Bayesian likelihood functions termed as Spatial then Temporal Bayesian (STBR-1) and simultaneous Spatiotemporal Bayesian (STBR-2). Radial and longitudinal strain estimated from a finite-element-analysis (FEA) model of realistic canine myocardial deformation were utilized to quantify strain bias, normalized strain error and total temporal relative error (TTR). Statistical analysis with one-way analysis of variance (ANOVA) showed that all Bayesian regularization methods significantly outperform NCC with lower bias and errors (p < 0.001). However, there was no significant difference among Bayesian methods. For example, mean longitudinal TTR for NCC, SBR, STBR-1 and STBR-2 were 25.41%, 9.27%, 10.38% and 10.13% respectively An in vivo feasibility study using RF data from ten healthy mice hearts were used to compare the elastographic signal-to-noise ratio (SNR e ) calculated using stochastic analysis. STBR-2 had the highest expected SNR e both for radial and longitudinal strain. The mean expected SNR e values for accumulated radial strain for NCC, SBR, STBR-1 and STBR-2 were 5.03, 9.43, 9.42 and 10.58, respectively. Overall results suggest that STBR improves CSI in vivo.

Oxide-Based Electrolyte-Gated Transistors for Spatiotemporal Information Processing.

Spiking neural networks (SNNs) sharing large similarity with biological nervous systems are promising to process spatiotemporal information and can provide highly time- and energy-efficient computational paradigms for the Internet-of-Things and edge computing. Nonvolatile electrolyte-gated transistors (EGTs) provide prominent analog switching performance, the most critical feature of synaptic element, and have been recently demonstrated as a promising synaptic device. However, high performance, large-scale EGT arrays, and EGT application for spatiotemporal information processing in an SNN are yet to be demonstrated. Here, an oxide-based EGT employing amorphous Nb2 O5 and Lix SiO2 is introduced as the channel and electrolyte gate materials, respectively, and integrated into a 32 × 32 EGT array. The engineered EGTs show a quasi-linear update, good endurance (106 ) and retention, a high switching speed of 100 ns, ultralow readout conductance (<100 nS), and ultralow areal switching energy density (20 fJ µm-2 ). The prominent analog switching performance is leveraged for hardware implementation of an SNN with the capability of spatiotemporal information processing, where spike sequences with different timings are able to be efficiently learned and recognized by the EGT array. Finally, this EGT-based spatiotemporal information processing is deployed to detect moving orientation in a tactile sensing system. These results provide an insight into oxide-based EGT devices for energy-efficient neuromorphic computing to support edge application.

Spatiotemporal Information Processing Emulated by Multiterminal Neuro-Transistor Networks.

All external sensory stimuli produce a spatiotemporal pattern of action potentials, which is transmitted to the biological neural system to be processed. The relative timing of synaptic spikes from different presynaptic neurons represents the features of the stimuli. A fundamental prerequisite in cortical information processing is the discrimination of different spatiotemporal input sequences. Here, capacitively coupled multiterminal oxide-based neuro-transistors are proposed for spatiotemporal information processing, mimicking the dendritic discriminability of different spatiotemporal input sequences. The experimental results demonstrate that such multiterminal neuromorphic devices can act as spatiotemporal information processing compartments for fundamental cortical computation. Also, as an example of spatiotemporal information processing, sound location functionality of the human brain is also emulated by constructing a simple artificial neural network based on such oxide-based multiterminal neuro-transistors.

Decoding Rich Spatial Information with High Temporal Resolution.

New research suggests that magnetoencephalography (MEG) contains rich spatial information for decoding neural states. Even small differences in the angle of neighbouring dipoles generate subtle, but statistically separable field patterns. This implies MEG (and electroencephalography: EEG) is ideal for decoding neural states with high-temporal resolution in the human brain.

Spatiotemporal characteristics of synaptic EPSP summation on the dendritic trees of hippocampal CA1 pyramidal neurons as revealed by laser uncaging stimulation.

Synaptic strength is modified by the temporal coincidence of synaptic inputs without back-propagating action potentials (BPAPs) in CA1 pyramidal neurons. In order to clarify the interactive mechanisms of associative long-term potentiation (LTP) without BPAPs, local paired stimuli were applied to the dendrites using high-speed laser uncaging stimulation equipment. When the spatial distance between the paired stimuli was <10 micrometer, nonlinear amplification in excitatory postsynaptic potential summation was observed. In the time window from -20 to 20 ms, supralinear amplification was observed. Supralinear amplification was modulated by antagonist of voltage-gated Na(+)/Ca(2+) channels and NMDA-type glutamate receptors. These results are closely related to the spatiotemporal-characteristics of associative LTP without BPAPs. This study proposes an essential aspect of dendritic information processing.