Intelligent Task Dispatching and Scheduling Using a Deep Q-Network in a Cluster Edge Computing System.

Recently, intelligent IoT applications based on artificial intelligence (AI) have been deployed with mobile edge computing (MEC). Intelligent IoT applications demand more computing resources and lower service latencies for AI tasks in dynamic MEC environments. Thus, in this paper, considering the resource scalability and resource optimization of edge computing, an intelligent task dispatching model using a deep Q-network, which can efficiently use the computing resource of edge nodes is proposed to maximize the computation ability of the cluster edge system, which consists of multiple edge nodes. The cluster edge system can be implemented with the Kubernetes technology. The objective of the proposed model is to minimize the average response time of tasks offloaded to the edge computing system and optimize the resource allocation for computing the offloaded tasks. For this, we first formulate the optimization problem of resource allocation as a Markov decision process (MDP) and adopt a deep reinforcement learning technology to solve this problem. Thus, the proposed intelligent task dispatching model is designed based on a deep Q-network (DQN) algorithm to update the task dispatching policy. The simulation results show that the proposed model archives a better convergence performanc in terms of the average completion time of all offloaded tasks, than existing task dispatching methods, such as the Random Method, Least Load Method and Round-Robin Method, and has a better task completion rate than the existing task dispatching method when using the same resources as the cluster edge system.

Compound Context-Aware Bayesian Inference Scheme for Smart IoT Environment.

The objective of smart cities is to improve the quality of life for citizens by using Information and Communication Technology (ICT). The smart IoT environment consists of multiple sensor devices that continuously produce a large amount of data. In the IoT system, accurate inference from multi-sensor data is imperative to make a correct decision. Sensor data are often imprecise, resulting in low-quality inference results and wrong decisions. Correspondingly, single-context data are insufficient for making an accurate decision. In this paper, a novel compound context-aware scheme is proposed based on Bayesian inference to achieve accurate fusion and inference from the sensory data. In the proposed scheme, multi-sensor data are fused based on the relation and contexts of sensor data whether they are dependent or not on each other. Extensive computer simulations show that the proposed technique significantly improves the inference accuracy when it is compared to the other two representative Bayesian inference techniques.

A Dual-Connectivity Mobility Link Service for Producer Mobility in the Named Data Networking.

With the exponential growth of Cyber-Physical Systems (CPSs) technologies, the Internet of Things (IoT) infrastructure has evolved from built-in static infrastructure to a flexible structure applicable to various mobile environments. In this Internet of Mobile Things (IoMT) environment, each IoT device could operate simultaneously as a provider and consumer of information, and could provide new services through the exchange of such information. Named Data Networking (NDN), which could request data by content name rather than location (IP address), is suitable for such mobile IoT environments. However, in the current Named Data Networking (NDN) specification, producer mobility is one of the major problems in need of remedy. Previously proposed schemes for producer mobility use an anchor to hide the producer's movement from consumers. As a result, they require a special anchor node and a signaling procedure to track the current locations of contents. A few anchorless schemes have also been proposed, but they still require mobility signaling and all NDN routers on the signaling path must understand the meaning of the signaling. We therefore propose an anchorless producer mobility scheme for the NDN. This scheme uses a dual-connectivity strategy that can be expressed as a soft handover. Whenever a producer changes its NDN Access Router (NAR), the new mobility link service located on the mobile producer's old NDN face repairs the old link so that the connectivity with the pNAR can be maintained for a while. The old NDN face is removed after the new location information on the contents of the producer is disseminated over the NDN network by the Named-data Link State Routing Protocol (NLSR) routing protocol at the nNAR. The new mobility link service decouples connection and transaction to hide the collapse of the link. Therefore, the NDN's mobility procedure could be simplified as the handover is defined as transaction completion as opposed to a breakdown of links. The proposed scheme prevents the routing information from being abruptly outdated due to producer mobility. Our simulation results show seamless handover when the producer changes its default access router.

Federated Reinforcement Learning for Training Control Policies on Multiple IoT Devices.

Reinforcement learning has recently been studied in various fields and also used to optimally control IoT devices supporting the expansion of Internet connection beyond the usual standard devices. In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own IoT devices of the same type but with slightly different dynamics. For such multiple IoT devices, there is no guarantee that an agent who interacts only with one IoT device and learns the optimal control policy will also control another IoT device well. Therefore, we may need to apply independent reinforcement learning to each IoT device individually, which requires a costly or time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent IoT device shares their learning experience (i.e., the gradient of loss function) with each other, and transfers a mature policy model parameters into other agents. They accelerate its learning process by using mature parameters. We incorporate the actor-critic proximal policy optimization (Actor-Critic PPO) algorithm into each agent in the proposed collaborative architecture and propose an efficient procedure for the gradient sharing and the model transfer. Using multiple rotary inverted pendulum devices interconnected via a network switch, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple IoT devices and that the learning speed can be faster if more agents are involved.

Sortation Control Using Multi-Agent Deep Reinforcement Learning in N-Grid Sortation System.

Intralogistics is a technology that optimizes, integrates, automates, and manages the logistics flow of goods within a logistics transportation and sortation center. As the demand for parcel transportation increases, many sortation systems have been developed. In general, the goal of sortation systems is to route (or sort) parcels correctly and quickly. We design an n-grid sortation system that can be flexibly deployed and used at intralogistics warehouse and develop a collaborative multi-agent reinforcement learning (RL) algorithm to control the behavior of emitters or sorters in the system. We present two types of RL agents, emission agents and routing agents, and they are trained to achieve the given sortation goals together. For the verification of the proposed system and algorithm, we implement them in a full-fledged cyber-physical system simulator and describe the RL agents' learning performance. From the learning results, we present that the well-trained collaborative RL agents can optimize their performance effectively. In particular, the routing agents finally learn to route the parcels through their optimal paths, while the emission agents finally learn to balance the inflow and outflow of parcels.

An Authentication and Key Management Mechanism for Resource Constrained Devices in IEEE 802.11-based IoT Access Networks.

Many Internet of Things (IoT) services utilize an IoT access network to connect small devices with remote servers. They can share an access network with standard communication technology, such as IEEE 802.11ah. However, an authentication and key management (AKM) mechanism for resource constrained IoT devices using IEEE 802.11ah has not been proposed as yet. We therefore propose a new AKM mechanism for an IoT access network, which is based on IEEE 802.11 key management with the IEEE 802.1X authentication mechanism. The proposed AKM mechanism does not require any pre-configured security information between the access network domain and the IoT service domain. It considers the resource constraints of IoT devices, allowing IoT devices to delegate the burden of AKM processes to a powerful agent. The agent has sufficient power to support various authentication methods for the access point, and it performs cryptographic functions for the IoT devices. Performance analysis shows that the proposed mechanism greatly reduces computation costs, network costs, and memory usage of the resource-constrained IoT device as compared to the existing IEEE 802.11 Key Management with the IEEE 802.1X authentication mechanism.

A socially aware routing based on local contact information in delay-tolerant networks.

In delay-tolerant networks, network topology changes dynamically and there is no guarantee of continuous connectivity between any two nodes. These features make DTN routing one of important research issues, and the application of social network metrics has led to the design of recent DTN routing schemes. In this paper, we propose an efficient routing scheme by using a node's local contact history and social network metrics. Each node first chooses a proper relay node based on the closeness to the destination node. A locally computed betweenness centrality is additionally utilized to enhance the routing efficiency. Through intensive simulation, we finally demonstrate that our algorithm performs efficiently compared to the existing epidemic or friendship routing scheme.

Strangeness-driven exploration in multi-agent reinforcement learning.

In this study, a novel exploration method for centralized training and decentralized execution (CTDE)-based multi-agent reinforcement learning (MARL) is introduced. The method uses the concept of strangeness, which is determined by evaluating (1) the level of the unfamiliarity of the observations an agent encounters and (2) the level of the unfamiliarity of the entire state the agents visit. An exploration bonus, which is derived from the concept of strangeness, is combined with the extrinsic reward obtained from the environment to form a mixed reward, which is then used for training CTDE-based MARL algorithms. Additionally, a separate action-value function is also proposed to prevent the high exploration bonus from overwhelming the sensitivity to extrinsic rewards during MARL training. This separate function is used to design the behavioral policy for generating transitions. The proposed method is not much affected by stochastic transitions commonly observed in MARL tasks and improves the stability of CTDE-based MARL algorithms when used with an exploration method. By providing didactic examples and demonstrating the substantial performance improvement of our proposed exploration method in CTDE-based MARL algorithms, we illustrate the advantages of our approach. These evaluations highlight how our method outperforms state-of-the-art MARL baselines on challenging tasks within the StarCraft II micromanagement benchmark, underscoring its effectiveness in improving MARL.

A target coverage scheduling scheme based on genetic algorithms in directional sensor networks.

As a promising tool for monitoring the physical world, directional sensor networks (DSNs) consisting of a large number of directional sensors are attracting increasing attention. As directional sensors in DSNs have limited battery power and restricted angles of sensing range, maximizing the network lifetime while monitoring all the targets in a given area remains a challenge. A major technique to conserve the energy of directional sensors is to use a node wake-up scheduling protocol by which some sensors remain active to provide sensing services, while the others are inactive to conserve their energy. In this paper, we first address a Maximum Set Covers for DSNs (MSCD) problem, which is known to be NP-complete, and present a greedy algorithm-based target coverage scheduling scheme that can solve this problem by heuristics. This scheme is used as a baseline for comparison. We then propose a target coverage scheduling scheme based on a genetic algorithm that can find the optimal cover sets to extend the network lifetime while monitoring all targets by the evolutionary global search technique. To verify and evaluate these schemes, we conducted simulations and showed that the schemes can contribute to extending the network lifetime. Simulation results indicated that the genetic algorithm-based scheduling scheme had better performance than the greedy algorithm-based scheme in terms of maximizing network lifetime.

Phosphatidylserine and phosphatidylcholine-containing liposomes inhibit amyloid beta and interferon-gamma-induced microglial activation.

There is increasing evidence that microglial activation is one of the major pathogenic factors for Alzheimer's disease (AD) and the inhibition of the inflammatory activation of the microglia thus appears to be neuroprotective and a potentially useful treatment for AD. Phospholipids such as phosphatidylserine (PS) and phosphatidylcholine (PC) have been reported to modulate the immune function of phagocytes. In addition, PS has been reported to be a nootropics that can be used as nonprescription memory or cognitive enhancers. We therefore evaluated the effects of liposomes, which comprise both PS and PC (PS/PC liposomes), on the microglial production of tumor necrosis factor-alpha (TNF-alpha), nitric oxide (NO), and superoxide (*O(2)-) induced by amyloid beta (Abeta) and interferon-gamma (IFN-gamma). Pretreatment of microglia with PS/PC liposomes considerably inhibited the TNF-alpha, NO and *O(2)- production induced by Abeta/IFN-gamma. These results suggest that PS/PC liposomes have both neuroprotective and antioxidative properties through the inhibition of microglial activation, thus supporting the nootropic and antidementia effect of PS.

Phospholipids modulate superoxide and nitric oxide production by lipopolysaccharide and phorbol 12-myristate-13-acetate-activated microglia.

Microglial activation and inflammatory processes have been implicated in the pathogenesis of a number of neurodegenerative disorders. Recently, peroxynitrite (ONOO(-)), the reaction product of superoxide (O(2)(-)) and nitric oxide (NO) both of which can be generated by activated microglia, has been demonstrated to act as a major mediator in the neurotoxicity induced by activated microglia. On the other hand, phospholipids such as phosphatidylserine (PS) and phosphatidylcholine (PC) have been reported to modulate the immune function of phagocytes. We therefore evaluated the effects of liposomes which comprise both PS and PC (PS/PC liposomes) or PC only (PC liposomes) regarding the production of both O(2)(-) and NO by lipopolysaccharide (LPS)/phorbol 12-myristate-13-acetate (PMA)-activated microglia using electron spin resonance (ESR) spin trap technique with a DEPMPO and Griess reaction, respectively. Pretreatment with PS/PC liposomes or PC liposomes considerably inhibited the signal intensity of O(2)(-) adduct associated with LPS/PMA-activated microglia in a dose-dependent manner. In addition, pretreatment with PS/PC liposomes also significantly reduced LPS/PMA-induced microglial NO production. In contrast, pretreatment with PC liposomes had no effect on the NO production. These results indicate that PS/PC liposomes can inhibit the microglial production of both NO and O(2)(-), and thus presumably prevent a subsequent formation of ONOO(-). Therefore, PS/PC liposomes appear to have both neuroprotective and anti-oxidative properties through the inhibition of microglial activation.

Nox4 as the major catalytic component of an endothelial NAD(P)H oxidase.

BACKGROUND: Recent evidence has suggested that reactive oxygen species are important signaling molecules in vascular cells and play a pivotal role in the development of vascular diseases. The activity of NAD(P)H oxidase has been identified as the major source of reactive oxygen species in vascular endothelial cells. However, the precise molecular structure and the mechanism of activation of the oxidase have remained poorly understood. METHODS AND RESULTS: Here, we investigated the molecular identities and the superoxide-producing activity of endothelial NAD(P)H oxidase. We found that Nox4, a homologue of gp91phox/Nox2, was abundantly expressed in endothelial cells. The expression of Nox4 in endothelial cells markedly exceeded that of other Nox proteins, including gp91phox/Nox2, and was affected by cell growth. Using electron spin resonance and chemiluminescence, we measured the superoxide production and found that the endothelial membranes had an NAD(P)H-dependent superoxide-producing activity comparable to that of the neutrophil membranes, whereas the activity was not enhanced by the 2 recombinant proteins p47phox and p67phox, in contrast to that of the neutrophil membranes. Downregulation of Nox4 by an antisense oligonucleotide reduced superoxide production in endothelial cells in vivo and in vitro. CONCLUSIONS: These findings suggest that Nox4 may function as the major catalytic component of an endothelial NAD(P)H oxidase.

A kinetic study of 3-chlorophenol enhanced hydroxyl radical generation during ozonation.

Hydroxyl (OH) radical is proposed as an important factor in the ozonation of water. In the present study, the enhancing effect of 3-chlorophenol on OH radical generation was mathematically evaluated using electron spin resonance (ESR)/spin-trapping technique. OH radical was trapped with a 5,5-dimethyl-1-pyrroline-N-oxide (DMPO) as a stable adduct, DMPO-OH. The initial velocity of DMPO-OH generation in ozonated water containing 3-chlorophenol was quantitatively measured using a combined system of ESR spectroscopy with stopped-flow apparatus which was controlled by home-made software. The initial velocity of DMPO-OH generation increased as a function of the concentration of ozone and the more effectively of 3-chlorophenol concentration. The relation among ozone concentration, amount of 3-chlorophenol and the initial velocity (nu(0)) of DMPO-OH generation was mathematically analyzed and the following equation was obtained, nu(0) (10(-6)M/s)=[9.7 x [3-chlorophenol (10(-9)M)] + 0.0005]exp(57 x [ozone (10(-9)M)]). The equation fitted very well with the experimental results, and the correlation coefficient was larger than 0.99. The equation for the enhancing effect by 3-chlorophenol should provide useful information to optimize the condition in ozone treatment process of water containing phenolic pollutants.

Aripiprazole inhibits superoxide generation from phorbol-myristate-acetate (PMA)-stimulated microglia in vitro: implication for antioxidative psychotropic actions via microglia.

Altered antioxidant status has been implicated in schizophrenia. Microglia, major sources of free radicals such as superoxide (•O(2)(-)), play crucial roles in various brain pathologies. Recent postmortem and imaging studies have indicated microglial activation in the brain of schizophrenic patients. We previously demonstrated that atypical antipsychotics including aripiprazole significantly inhibited the release of nitric oxide and proinflammatory cytokines from interferon-γ-stimulated microglia in vitro. Antioxidative effects of antipsychotics via modulating microglial superoxide generation have never been reported. Therefore, we herein investigated the effects of antipsychotics on the •O(2)(-) generation from phorbol-myristate-acetate (PMA)-stimulated rodent microglia by the electron spin resonance (ESR) spectroscopy and also examined the intracellular mechanism by intracellular Ca(2+) imaging and immunostaining. Neuronal damage induced by microglial activation was also investigated by the co-culture experiment. Among various antipsychotics, only aripiprazole inhibited the •O(2)(-) generation from PMA-stimulated microglia. Aripiprazole proved to inhibit the •O(2)(-) generation through the cascade of protein kinase C (PKC) activation, intracellular Ca(2+) regulation and NADPH oxidase activation via cytosolic p47(phox) translocation to the plasma/phagosomal membranes. Formation of neuritic beading, induced by PMA-stimulated microglia, was attenuated by pretreatment of aripiprazole. D2R antagonism has long been considered as the primary therapeutic action for schizophrenia. Aripiprazole with D2R partial agonism is effective like other antipsychotics with fewer side effects, while aripiprazole's therapeutic mechanism itself remains unclear. Our results imply that aripiprazole may have psychotropic effects by reducing the microglial oxidative reactions and following neuronal reactions, which puts forward a novel therapeutic hypothesis in schizophrenia research.