Classification of Alzheimer's disease: application of a transfer learning deep Q-network method.

Early diagnosis is crucial to slowing the progression of Alzheimer's disease (AD), so it is urgent to find an effective diagnostic method for AD. This study intended to investigate whether the transfer learning approach of deep Q-network (DQN) could effectively distinguish AD patients using local metrics of resting-state functional magnetic resonance imaging (rs-fMRI) as features. This study included 1310 subjects from the Consortium for Reliability and Reproducibility (CoRR) and 50 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) GO/2. The amplitude of low-frequency fluctuation (ALFF), fractional ALFF (fALFF) and percent amplitude of fluctuation (PerAF) were extracted as features using the Power 264 atlas. Based on gender bias in AD, we searched for transferable similar parts between the CoRR feature matrix and the ADNI feature matrix, resulting in the CoRR similar feature matrix served as the source domain and the ADNI similar feature matrix served as the target domain. A DQN classifier was pre-trained in the source domain and transferred to the target domain. Finally, the transferred DQN classifier was used to classify AD and healthy controls (HC). A permutation test was performed. The DQN transfer learning achieved a classification accuracy of 86.66% (p < 0.01), recall of 83.33% and precision of 83.33%. The findings suggested that the transfer learning approach using DQN could be an effective way to distinguish AD from HC. It also revealed the potential value of local brain activity in AD clinical diagnosis.

Deep learning-based energy prediction and tangent search remora optimization-based secure multi-path data communication mechanism in WSN.

Wireless Sensor Network (WSN) has been exploited in numerous regions which can be hardly accessed by humans. However, it is essential to convey the information accumulated by the sensing devices or nodes to the Base Station (BS) for further processing. Multipath routing protocols are found to address these challenges and provide reliable communication. This paper aims to find an optimal path to the gateway with minimum energy consumption and reduced error rate while meeting the end-to-end delay requirements. In this research, an effective multipath routing based on energy prediction and hybrid optimization is developed. Here, a Deep Q-Network (DQN) is applied to predict the energy, and the process is augmented by the usage of a proposed Tangent Search Remora Optimization (TSRO) algorithm. Further, the multipath routing is executed using the TSRO algorithm, considering a fitness function formulated using various factors, like residual energy, distance, throughput, reliability, trust factors, predicted energy, Link Life Time (LLT), delay, and traffic intensity. The devised TSRO-routing is scrutinized for its competence based on trust, throughput, energy, distance, and delay and has achieved superior values of energy of 0.402 J, throughput at 25.056Mbps, trust at 84.975, and minimal distance of 29.964 m, and delay of 0.750 ms.

Deep Reinforcement Learning-Based Resource Management in Maritime Communication Systems.

With the growing maritime economy, ensuring the quality of communication for maritime users has become imperative. The maritime communication system based on nearshore base stations enhances the communication rate of maritime users through dynamic resource allocation. A virtual queue-based deep reinforcement learning beam allocation scheme is proposed in this paper, aiming to maximize the communication rate. More particularly, to reduce the complexity of resource management, we employ a grid-based method to discretize the maritime environment. For the combinatorial optimization problem of grid and beam allocation under unknown channel state information, we model it as a sequential decision process of resource allocation. The nearshore base station is modeled as a learning agent, continuously interacting with the environment to optimize beam allocation schemes using deep reinforcement learning techniques. Furthermore, we guarantee that grids with poor channel state information can be serviced through the virtual queue method. Finally, the simulation results provided show that our proposed beam allocation scheme is beneficial in terms of increasing the communication rate.

An adaptive testing item selection strategy via a deep reinforcement learning approach.

Computerized adaptive testing (CAT) aims to present items that statistically optimize the assessment process by considering the examinee's responses and estimated trait levels. Recent developments in reinforcement learning and deep neural networks provide CAT with the potential to select items that utilize more information across all the items on the remaining tests, rather than just focusing on the next several items to be selected. In this study, we reformulate CAT under the reinforcement learning framework and propose a new item selection strategy based on the deep Q-network (DQN) method. Through simulated and empirical studies, we demonstrate how to monitor the training process to obtain the optimal Q-networks, and we compare the accuracy of the DQN-based item selection strategy with that of five traditional strategies-maximum Fisher information, Fisher information weighted by likelihood, Kullback‒Leibler information weighted by likelihood, maximum posterior weighted information, and maximum expected information-on both simulated and real item banks and responses. We further investigate how sample size and the distribution of the trait levels of the examinees used in training affect DQN performance. The results show that DQN achieves lower RMSE and MAE values than traditional strategies under simulated and real banks and responses in most conditions. Suggestions for the use of DQN-based strategies are provided, as well as their code.

Recognition of Hand Gestures Based on EMG Signals with Deep and Double-Deep Q-Networks.

In recent years, hand gesture recognition (HGR) technologies that use electromyography (EMG) signals have been of considerable interest in developing human-machine interfaces. Most state-of-the-art HGR approaches are based mainly on supervised machine learning (ML). However, the use of reinforcement learning (RL) techniques to classify EMGs is still a new and open research topic. Methods based on RL have some advantages such as promising classification performance and online learning from the user's experience. In this work, we propose a user-specific HGR system based on an RL-based agent that learns to characterize EMG signals from five different hand gestures using Deep Q-network (DQN) and Double-Deep Q-Network (Double-DQN) algorithms. Both methods use a feed-forward artificial neural network (ANN) for the representation of the agent policy. We also performed additional tests by adding a long-short-term memory (LSTM) layer to the ANN to analyze and compare its performance. We performed experiments using training, validation, and test sets from our public dataset, EMG-EPN-612. The final accuracy results demonstrate that the best model was DQN without LSTM, obtaining classification and recognition accuracies of up to 90.37%±10.7% and 82.52%±10.9%, respectively. The results obtained in this work demonstrate that RL methods such as DQN and Double-DQN can obtain promising results for classification and recognition problems based on EMG signals.

Table-Balancing Cooperative Robot Based on Deep Reinforcement Learning.

Reinforcement learning is one of the artificial intelligence methods that enable robots to judge and operate situations on their own by learning to perform tasks. Previous reinforcement learning research has mainly focused on tasks performed by individual robots; however, everyday tasks, such as balancing tables, often require cooperation between two individuals to avoid injury when moving. In this research, we propose a deep reinforcement learning-based technique for robots to perform a table-balancing task in cooperation with a human. The cooperative robot proposed in this paper recognizes human behavior to balance the table. This recognition is achieved by utilizing the robot's camera to take an image of the state of the table, then the table-balance action is performed afterward. Deep Q-network (DQN) is a deep reinforcement learning technology applied to cooperative robots. As a result of learning table balancing, on average, the cooperative robot showed a 90% optimal policy convergence rate in 20 runs of training with optimal hyperparameters applied to DQN-based techniques. In the H/W experiment, the trained DQN-based robot achieved an operation precision of 90%, thus verifying its excellent performance.

Energy-Efficient Resource Allocation Based on Deep Q-Network in V2V Communications.

Recently, with the development of autonomous driving technology, vehicle-to-everything (V2X) communication technology that provides a wireless connection between vehicles, pedestrians, and roadside base stations has gained significant attention. Vehicle-to-vehicle (V2V) communication should provide low-latency and highly reliable services through direct communication between vehicles, improving safety. In particular, as the number of vehicles increases, efficient radio resource management becomes more important. In this paper, we propose a deep reinforcement learning (DRL)-based decentralized resource allocation scheme in the V2X communication network in which the radio resources are shared between the V2V and vehicle-to-infrastructure (V2I) networks. Here, a deep Q-network (DQN) is utilized to find the resource blocks and transmit power of vehicles in the V2V network to maximize the sum rate of the V2I and V2V links while reducing the power consumption and latency of V2V links. The DQN also uses the channel state information, the signal-to-interference-plus-noise ratio (SINR) of V2I and V2V links, and the latency constraints of vehicles to find the optimal resource allocation scheme. The proposed DQN-based resource allocation scheme ensures energy-efficient transmissions that satisfy the latency constraints for V2V links while reducing the interference of the V2V network to the V2I network. We evaluate the performance of the proposed scheme in terms of the sum rate of the V2X network, the average power consumption of V2V links, and the average outage probability of V2V links using a case study in Manhattan with nine blocks of 3GPP TR 36.885. The simulation results show that the proposed scheme greatly reduces the transmit power of V2V links when compared to the conventional reinforcement learning-based resource allocation scheme without sacrificing the sum rate of the V2X network or the outage probability of V2V links.

Deep-Q-Network-Based Packet Scheduling in an IoT Environment.

With the advent of the Internet of Things (IoT) era, a wide array of wireless sensors supporting the IoT have proliferated. As key elements for enabling the IoT, wireless sensor nodes require minimal energy consumption and low device complexity. In particular, energy-efficient resource scheduling is critical in maintaining a network of wireless sensor nodes, since the energy-intensive processing of wireless sensor nodes and their interactions is too complicated to control. In this study, we present a practical deep Q-network (DQN)-based packet scheduling algorithm that coordinates the transmissions of multiple IoT devices. The scheduling algorithm dynamically adjusts the connection interval (CI) and the number of packets transmitted by each node within the interval. Through various experiments, we verify how effectively the proposed scheduler improves energy efficiency and handles the time-varying nature of the network environment. Moreover, we attempt to gain insight into the optimized packet scheduler by analyzing the policy of the DQN scheduler. The experimental results show that the proposed scheduling algorithm can further prolong a network's lifetime in a dynamic network environment in comparison with that in other alternative schemes while ensuring the quality of service (QoS).

Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems.

The rapid development of mobile communication services in recent years has resulted in a scarcity of spectrum resources. This paper addresses the problem of multi-dimensional resource allocation in cognitive radio systems. Deep reinforcement learning (DRL) combines deep learning and reinforcement learning to enable agents to solve complex problems. In this study, we propose a training approach based on DRL to design a strategy for secondary users in the communication system to share the spectrum and control their transmission power. The neural networks are constructed using the Deep Q-Network and Deep Recurrent Q-Network structures. The results of the conducted simulation experiments demonstrate that the proposed method can effectively improve the user's reward and reduce collisions. In terms of reward, the proposed method outperforms opportunistic multichannel ALOHA by about 10% and about 30% for the single SU scenario and the multi-SU scenario, respectively. Furthermore, we explore the complexity of the algorithm and the influence of parameters in the DRL algorithm on the training.

Delay-Aware and Link-Quality-Aware Geographical Routing Protocol for UANET via Dueling Deep Q-Network.

In an unmanned aerial vehicles ad hoc network (UANET), UAVs communicate with each other to accomplish intricate tasks collaboratively and cooperatively. However, the high mobility of UAVs, the variable link quality, and heavy traffic loads can lead to difficulties in finding an optimal communication path. We proposed a delay-aware and link-quality-aware geographical routing protocol for a UANET via the dueling deep Q-network (DLGR-2DQ) to address these problems. Firstly, the link quality was not only related to the physical layer metric, the signal-to-noise ratio, which was influenced by path loss and Doppler shifts, but also the expected transmission count of the data link layer. In addition, we also considered the total waiting time of packets in the candidate forwarding node in order to decrease the end-to-end delay. Then, we modeled the packet-forwarding process as a Markov decision process. We crafted an appropriate reward function that utilized the penalty value for each additional hop, total waiting time, and link quality to accelerate the learning of the dueling DQN algorithm. Finally, the simulation results illustrated that our proposed routing protocol outperformed others in terms of the packet delivery ratio and the average end-to-end delay.

Enhancing Short Track Speed Skating Performance through Improved DDQN Tactical Decision Model.

This paper studies the tactical decision-making model of short track speed skating based on deep reinforcement learning, so as to improve the competitive performance of corresponding short track speed skaters. Short track speed skating, a traditional discipline in the Winter Olympics since its establishment in 1988, has consistently garnered attention. As artificial intelligence continues to advance, the utilization of deep learning methods to enhance athletes' tactical decision-making capabilities has become increasingly prevalent. Traditional tactical decision techniques often rely on the experience and knowledge of coaches and video analysis methods that require a lot of time and effort. Consequently, this study proposes a scientific simulation environment for short track speed skating, that accurately simulates the physical attributes of the venue, the physiological fitness of the athletes, and the rules of the competition. The Double Deep Q-Network (DDQN) model is enhanced and utilized, with improvements to the reward function and the distinct description of four tactics. This enables agents to learn optimal tactical decisions in various competitive states with a simulation environment. Experimental results demonstrate that this approach effectively enhances the competition performance and physiological fitness allocation of short track speed skaters.

A Dynamic Task Allocation Framework in Mobile Crowd Sensing with D3QN.

With the coverage of sensor-rich smart devices (smartphones, iPads, etc.), combined with the need to collect large amounts of data, mobile crowd sensing (MCS) has gradually attracted the attention of academics in recent years. MCS is a new and promising model for mass perception and computational data collection. The main function is to recruit a large group of participants with mobile devices to perform sensing tasks in a given area. Task assignment is an important research topic in MCS systems, which aims to efficiently assign sensing tasks to recruited workers. Previous studies have focused on greedy or heuristic approaches, whereas the MCS task allocation problem is usually an NP-hard optimisation problem due to various resource and quality constraints, and traditional greedy or heuristic approaches usually suffer from performance loss to some extent. In addition, the platform-centric task allocation model usually considers the interests of the platform and ignores the feelings of other participants, to the detriment of the platform's development. Therefore, in this paper, deep reinforcement learning methods are used to find more efficient task assignment solutions, and a weighted approach is adopted to optimise multiple objectives. Specifically, we use a double deep Q network (D3QN) based on the dueling architecture to solve the task allocation problem. Since the maximum travel distance of the workers, the reward value, and the random arrival and time sensitivity of the sensing tasks are considered, this is a dynamic task allocation problem under multiple constraints. For dynamic problems, traditional heuristics (eg, pso, genetics) are often difficult to solve from a modeling and practical perspective. Reinforcement learning can obtain sub-optimal or optimal solutions in a limited time by means of sequential decision-making. Finally, we compare the proposed D3QN-based solution with the standard baseline solution, and experiments show that it outperforms the baseline solution in terms of platform profit, task completion rate, etc., the utility and attractiveness of the platform are enhanced.

Prehensile and Non-Prehensile Robotic Pick-and-Place of Objects in Clutter Using Deep Reinforcement Learning.

In this study, we develop a framework for an intelligent and self-supervised industrial pick-and-place operation for cluttered environments. Our target is to have the agent learn to perform prehensile and non-prehensile robotic manipulations to improve the efficiency and throughput of the pick-and-place task. To achieve this target, we specify the problem as a Markov decision process (MDP) and deploy a deep reinforcement learning (RL) temporal difference model-free algorithm known as the deep Q-network (DQN). We consider three actions in our MDP; one is 'grasping' from the prehensile manipulation category and the other two are 'left-slide' and 'right-slide' from the non-prehensile manipulation category. Our DQN is composed of three fully convolutional networks (FCN) based on the memory-efficient architecture of DenseNet-121 which are trained together without causing any bottleneck situations. Each FCN corresponds to each discrete action and outputs a pixel-wise map of affordances for the relevant action. Rewards are allocated after every forward pass and backpropagation is carried out for weight tuning in the corresponding FCN. In this manner, non-prehensile manipulations are learnt which can, in turn, lead to possible successful prehensile manipulations in the near future and vice versa, thus increasing the efficiency and throughput of the pick-and-place task. The Results section shows performance comparisons of our approach to a baseline deep learning approach and a ResNet architecture-based approach, along with very promising test results at varying clutter densities across a range of complex scenario test cases.

Slicing Resource Allocation Based on Dueling DQN for eMBB and URLLC Hybrid Services in Heterogeneous Integrated Networks.

In 5G/B5G communication systems, network slicing is utilized to tackle the problem of the allocation of network resources for diverse services with changing demands. We proposed an algorithm that prioritizes the characteristic requirements of two different services and tackles the problem of allocation and scheduling of resources in the hybrid services system with eMBB and URLLC. Firstly, the resource allocation and scheduling are modeled, subject to the rate and delay constraints of both services. Secondly, the purpose of adopting a dueling deep Q network (Dueling DQN) is to approach the formulated non-convex optimization problem innovatively, in which a resource scheduling mechanism and the ϵ-greedy strategy were utilized to select the optimal resource allocation action. Moreover, the reward-clipping mechanism is introduced to enhance the training stability of Dueling DQN. Meanwhile, we choose a suitable bandwidth allocation resolution to increase flexibility in resource allocation. Finally, the simulations indicate that the proposed Dueling DQN algorithm has excellent performance in terms of quality of experience (QoE), spectrum efficiency (SE) and network utility, and the scheduling mechanism makes the performance much more stable. In contrast with Q-learning, DQN as well as Double DQN, the proposed algorithm based on Dueling DQN improves the network utility by 11%, 8% and 2%, respectively.

Mobility-Aware Resource Allocation in IoRT Network for Post-Disaster Communications with Parameterized Reinforcement Learning.

Natural disasters, including earthquakes, floods, landslides, tsunamis, wildfires, and hurricanes, have become more common in recent years due to rapid climate change. For Post-Disaster Management (PDM), authorities deploy various types of user equipment (UE) for the search and rescue operation, for example, search and rescue robots, drones, medical robots, smartphones, etc., via the Internet of Robotic Things (IoRT) supported by cellular 4G/LTE/5G and beyond or other wireless technologies. For uninterrupted communication services, movable and deployable resource units (MDRUs) have been utilized where the base stations are damaged due to the disaster. In addition, power optimization of the networks by satisfying the quality of service (QoS) of each UE is a crucial challenge because of the electricity crisis after the disaster. In order to optimize the energy efficiency, UE throughput, and serving cell (SC) throughput by considering the stationary as well as movable UE without knowing the environmental priori knowledge in MDRUs aided two-tier heterogeneous networks (HetsNets) of IoRT, the optimization problem has been formulated based on emitting power allocation and user association combinedly in this article. This optimization problem is nonconvex and NP-hard where parameterized (discrete: user association and continuous: power allocation) action space is deployed. The new model-free hybrid action space-based algorithm called multi-pass deep Q network (MP-DQN) is developed to optimize this complex problem. Simulations results demonstrate that the proposed MP-DQN outperforms the parameterized deep Q network (P-DQN) approach, which is well known for solving parameterized action space, DQN, as well as traditional algorithms in terms of reward, average energy efficiency, UE throughput, and SC throughput for motionless as well as moveable UE.

A multi-robot deep Q-learning framework for priority-based sanitization of railway stations.

Sanitizing railway stations is a relevant issue, primarily due to the recent evolution of the Covid-19 pandemic. In this work, we propose a multi-robot approach to sanitize railway stations based on a distributed Deep Q-Learning technique. The proposed framework relies on anonymous data from existing WiFi networks to dynamically estimate crowded areas within the station and to develop a heatmap of prioritized areas to be sanitized. Such heatmap is then provided to a team of cleaning robots - each endowed with a robot-specific convolutional neural network - that learn how to effectively cooperate and sanitize the station's areas according to the associated priorities. The proposed approach is evaluated in a realistic simulation scenario provided by the Italian largest railways station: Roma Termini. In this setting, we consider different case studies to assess how the approach scales with the number of robots and how the trained system performs with a real dataset retrieved from a one-day data recording of the station's WiFi network.

Deep reinforcement learning-based approach for rumor influence minimization in social networks.

Spreading malicious rumors on social networks such as Facebook, Twitter, and WeChat can trigger political conflicts, sway public opinion, and cause social disruption. A rumor can spread rapidly across a network and can be difficult to control once it has gained traction.Rumor influence minimization (RIM) is a central problem in information diffusion and network theory that involves finding ways to minimize rumor spread within a social network. Existing research on the RIM problem has focused on blocking the actions of influential users who can drive rumor propagation. These traditional static solutions do not adequately capture the dynamics and characteristics of rumor evolution from a global perspective. A deep reinforcement learning strategy that takes into account a wide range of factors may be an effective way of addressing the RIM challenge. This study introduces the dynamic rumor influence minimization (DRIM) problem, a step-by-step discrete time optimization method for controlling rumors. In addition, we provide a dynamic rumor-blocking approach, namely RLDB, based on deep reinforcement learning. First, a static rumor propagation model (SRPM) and a dynamic rumor propagation model (DRPM) based on of independent cascade patterns are presented. The primary benefit of the DPRM is that it can dynamically adjust the probability matrix according to the number of individuals affected by rumors in a social network, thereby improving the accuracy of rumor propagation simulation. Second, the RLDB strategy identifies the users to block in order to minimize rumor influence by observing the dynamics of user states and social network architectures. Finally, we assess the blocking model using four real-world datasets with different sizes. The experimental results demonstrate the superiority of the proposed approach on heuristics such as out-degree(OD), betweenness centrality(BC), and PageRank(PR).

Hand Gesture Recognition Using EMG-IMU Signals and Deep Q-Networks.

Hand gesture recognition systems (HGR) based on electromyography signals (EMGs) and inertial measurement unit signals (IMUs) have been studied for different applications in recent years. Most commonly, cutting-edge HGR methods are based on supervised machine learning methods. However, the potential benefits of reinforcement learning (RL) techniques have shown that these techniques could be a viable option for classifying EMGs. Methods based on RL have several advantages such as promising classification performance and online learning from experience. In this work, we developed an HGR system made up of the following stages: pre-processing, feature extraction, classification, and post-processing. For the classification stage, we built an RL-based agent capable of learning to classify and recognize eleven hand gestures-five static and six dynamic-using a deep Q-network (DQN) algorithm based on EMG and IMU information. The proposed system uses a feed-forward artificial neural network (ANN) for the representation of the agent policy. We carried out the same experiments with two different types of sensors to compare their performance, which are the Myo armband sensor and the G-force sensor. We performed experiments using training, validation, and test set distributions, and the results were evaluated for user-specific HGR models. The final accuracy results demonstrated that the best model was able to reach up to 97.50%±1.13% and 88.15%±2.84% for the classification and recognition, respectively, with regard to static gestures, and 98.95%±0.62% and 90.47%±4.57% for the classification and recognition, respectively, with regard to dynamic gestures with the Myo armband sensor. The results obtained in this work demonstrated that RL methods such as the DQN are capable of learning a policy from online experience to classify and recognize static and dynamic gestures using EMG and IMU signals.

Securing Dynamic Service Function Chain Orchestration in EC-IoT Using Federated Learning.

Dynamic service orchestration is becoming more and more necessary as IoT and edge computing technologies continue to advance due to the flexibility and diversity of services. With the surge in the number of edge devices and the increase in data volume of IoT scenarios, there are higher requirements for the transmission security of privacy information from each edge device and the processing efficiency of SFC orchestration. This paper proposes a kind of dynamic SFC orchestration security algorithm applicable to EC-IoT scenarios based on the federated learning framework, combined with a block coordinated descent approach and the quadratic penalty algorithm to achieve communication efficiency and data privacy protection. A deep reinforcement learning algorithm is used to simultaneously adapt the SFC orchestration method in order to dynamically observe environmental changes and decrease end-to-end delay. The experimental results show that compared with the existing dynamic SFC orchestration algorithms, the proposed algorithm can achieve better convergence and latency performance under the condition of privacy protection; the overall latency is reduced by about 33%, and the overall convergence speed is improved by about 9%, which not only achieves the security of data privacy protection of edge computing nodes, but also meets the requirements of dynamic SFC orchestration.

Powering UAV with Deep Q-Network for Air Quality Tracking.

Tracking the source of air pollution plumes and monitoring the air quality during emergency events in real-time is crucial to support decision-makers in making an appropriate evacuation plan. Internet of Things (IoT) based air quality tracking and monitoring platforms have used stationary sensors around the environment. However, fixed IoT sensors may not be enough to monitor the air quality in a vast area during emergency situations. Therefore, many applications consider utilizing Unmanned Aerial Vehicles (UAVs) to monitor the air pollution plumes environment. However, finding an unhealthy location in a vast area requires a long navigation time. For time efficiency, we employ deep reinforcement learning (Deep RL) to assist UAVs to find air pollution plumes in an equal-sized grid space. The proposed Deep Q-network (DQN) based UAV Pollution Tracking (DUPT) is utilized to guide the multi-navigation direction of the UAV to find the pollution plumes' location in a vast area within a short duration of time. Indeed, we deployed a long short-term memory (LSTM) combined with Q-network to suggest a particular navigation pattern producing minimal time consumption. The proposed DUPT is evaluated and validated using an air pollution environment generated by a well-known Gaussian distribution and kriging interpolation. The evaluation and comparison results are carefully presented and analyzed. The experiment results show that our proposed DUPT solution can rapidly identify the unhealthy polluted area and spends around 28% of the total time of the existing solution.