Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

This paper investigates a Q-learning scheme for the optimal consensus control of discrete-time multiagent systems. The Q-learning algorithm is conducted by reinforcement learning (RL) using system data instead of system dynamics information. In the multiagent systems, the agents are interacted with each other and at least one agent can communicate with the leader directly, which is described by an algebraic graph structure. The objective is to make all the agents achieve synchronization with leader and make the performance indices reach Nash equilibrium. On one hand, the solutions of the optimal consensus control for multiagent systems are acquired by solving the coupled Hamilton–Jacobi–Bellman (HJB) equation. However, it is difficult to get analytical solutions directly of the discrete-time HJB equation. On the other hand, accurate mathematical models of most systems in real world are hard to be obtained. To overcome these difficulties, Q-learning algorithm is developed using system data rather than the accurate system model. We formulate performance index and corresponding Bellman equation of each agent i. Then, the Q-function Bellman equation is acquired on the basis of Q-function. Policy iteration is adopted to calculate the optimal control iteratively, and least square (LS) method is employed to motivate the implementation process. Stability analysis of proposed Q-learning algorithm for multiagent systems by policy iteration is given. Two simulation examples are experimented to verify the effectiveness of the proposed scheme.     

Data-driven control of consensus tracking for discrete-time multi-agent systems

This paper investigates the consensus tracking problem of leader-follower multi-agent systems. Different from most existing works, dynamics of all the agents are assumed completely unknown, whereas some input-output data about the agents are available. It is well known from the Willems et al. Fundamental Lemma that when inputs of a linear time-invariant (LTI) system are persistently exciting, all possible trajectories of the system can be represented in terms of a finite set of measured input-output data. Building on this idea, the present paper proposes a purely data-driven distributed consensus control policy which allows all the follower agents to track the leader agent’s trajectory. It is shown that for a linear discrete-time multi-agent system, the corresponding controller can be designed to ensure the global synchronization with local data. Even if the data are corrupted by noises, the proposed approach is still applicable under certain conditions. Numerical examples corroborate the practical merits of the theoretical results.     

Stochastic leader-following consensus of discrete-time nonlinear multi-agent systems with multiplicative noises

This paper studies the stochastic leader-following consensus problem of discrete-time nonlinear multi-agent systems (MASs) with multiplicative noises. The measurement information obtained from agents’ neighbors is inevitably affected by communication uncertainties, where the multiplicative noise is one of the important communication uncertainties. Multiplicative noises together with the intrinsic nonlinear dynamics bring more difficulties in the consensus control design under the leader-following topology. To solve the problem, the parameter-dependent Lyapunov functions are constructed to analyze the consensus control of first-order and second-order MASs, respectively. Some sufficient conditions, explicitly related to control gains, intensity of multiplicative noises and the Lipschitz constant regarding nonlinear functions, are established for reaching the mean square (m.s.) and almost sure (a.s.) leader-following consensus. Specifically, the obtained conditions are some scalar inequalities, which are more convenient in engineering application. Numerical simulations are conducted to validate the theoretical results.     

Explicit condition for consensus of third-order discrete-time multi-agent systems without accelerated velocity measurements

This paper investigates the consensus problem for third-order discrete-time multi-agent systems in directed networks. For the case when each agent can only receive the information of position and velocity from its neighbors, necessary and sufficient conditions for consensus have been proposed. In contrast to the preceding work, we not only present the exact consensus value, but also illustrate the influence of scaling parameters and nonzero eigenvalues of the involved Laplacian matrix on consensus. Two numerical examples are given to demonstrate the effectiveness of the obtained results.     

Event-based bipartite consensus of second-order discrete-time multi-agent systems with matrix-weighted signed networks

This article researches the bipartite consensus for discrete-time second-order multi-agent systems on matrix-weighted signed networks, which can describe the inter-dependencies of multidimensional states among states. So as to save limited communication resources, based on the matrix-weighted combined measurements of the position and velocity states, a matrix-weighted event-triggered control algorithm is designed. With the help of the stability theory, variable transformation and the inequality technique, the bipartite consensus conditions which are based on coupling gains, discrete interval, the parameters in the event-triggered rule and communication topology are obtained. Furthermore, the conditions to avoid the controller updating in each discrete-time are supplied. At last, a simulation example is offered to demonstrate the theoretical results.     

Leader-following consensus of nonlinear discrete-time multi-agent systems with limited communication channel capacity

This paper considers the problem of the leader-following consensus of generally nonlinear discrete-time multi-agent systems with limited communication channel capacity over directed fixed communication networks. The leader agent and all follower agents are with multi-dimensional nonlinear dynamics. We propose a novel kind of consensus algorithm for each follower agent based on dynamic encoding and decoding algorithms and conduct a rigorous analysis for consensus convergence. It is proved that under the consensus algorithm designed, the leader-following consensus is achievable and the quantizers equipped for the multi-agent systems can never be saturated. Furthermore, we give the explicit forms of the data transmission rate for the connected communication channel. By properly designing the system parameters according to restriction conditions, we can ensure the consensus and communication efficiency with merely one bit information exchanging between each pair of adjacent agents per step. Finally, simulation example is presented to verify the validity of results obtained.     

Stochastic consensus of discrete-time second-order multi-agent systems with measurement noises and time delays

We study the consensus control of discrete-time second-order multi-agents systems with time delays and multiplicative noises, where the consensus protocol is designed by both the local relative position measurements and each agent’s absolute velocity. Due to the existence of time delays and multiplicative noises, the classical methods for deterministic models with time delays cannot work. In this paper, we apply stochastic stability theorem of discrete-time stochastic delay equations to find some explicit sufficient conditions for both mean square and almost sure consensus. It is proven that for any given noise intensities and time delays, the second-order multi-agent consensus can be achieved by choosing appropriate control gains in the relative position measurement and absolute velocity, respectively. Numerical simulation is given to demonstrate the effectiveness of the proposed protocols as well as the theoretical results.     

Formation consensus for discrete-time heterogeneous multi-agent systems with link failures and actuator/sensor faults

In this paper, a distributed control protocol is presented for discrete-time heterogeneous multi-agent systems in order to achieve formation consensus against link failures and actuator/sensor faults under fixed and switching topologies. A model equivalent method is proposed to deal with the heterogeneous system consists of arbitrary order systems with different parameters. Based on graph theory and Lyapunov theory, stability conditions to solve formation consensus problem are developed for the underlying heterogeneous systems with communication link failures. In order to tolerate actuator/sensor faults, a distributed adaptive controller is proposed based on fault compensation. The desired control is designed by linear matrix inequality approach together with cone complementarity linearisation algorithm. After applying the new control scheme to heterogeneous systems under the directed topologies with link failures and faults, the resulting closed-loop heterogeneous system is validated to be stable. The effectiveness of the new formation consensus control strategy and its robustness are verified by simulations.     

Couple-group consensus for multi-agent networks of agents with discrete-time second-order dynamics

This paper considers the couple-group consensus problem for multi-agent networks with fixed and directed communication topology, where all agents are described by discrete-time second-order dynamics. Consensus protocol is designed such that some agents in a network reach a consistent value, while other agents reach another consistent value. The convergence of the system matrix is discussed based on the tools from matrix theory. An algebraic condition is established to guarantee couple-group consensus. Moreover, for a given communication topology, a theorem is derived on how to select proper control parameters and sampling period for couple-group consensus to be reached. Finally, simulation examples are presented to validate the effectiveness of the theoretical results.     

Matrix-weighted consensus for multi-agent systems with sampled-data

This paper researches the consensus issue for multi-agent systems on matrix-weighted directed fixed and undirected switching network topologies by sampled data control method which saves resources and is more practical. Using the sampled information, the distributed control laws are designed under two network topologies, respectively. Under directed fixed network topology, the consensus conditions based on the sampling period and the eigenvalues of Laplacian matrix are deduced by matrix theory and analysis theory. Under undirected switching network topology, by using Lyapunov stability theory, the consensus conditions based on the sampling period and switched network topologies are built. Lastly, two simulation examples are offered to verify the validity of the obtained results.     

Event-based discrete-time multi-agent consensus over signed digraphs with communication delays

The ability to ensure the desired performance of the cooperative-antagonistic multi-agent networks (MANs) in the presence of communication constraints is an important task in many applications of real systems. In this paper, under the proposed event-triggered condition (ETC), different types of consensus are obtained under different network topology. We concentrates on the event-based bipartite consensus. It is shown that under the proposed ETC (i) the addressed cooperative-antagonistic network with arbitrary communication delays reaches bipartite consensus provided that the network is balanced; (ii) the network model reaches zero if the network is unbalanced. Further, to avoid the continuously verifying the triggering condition, a self-triggered algorithm is proposed for realizing the bipartite consensus of the network model. A numerical example is given to illustrate the effectiveness of the theoretical results.     

Sliding-mode consensus algorithms for disturbed second-order multi-agent systems

In this paper, both leaderless and leader-follower consensus problems for a class of disturbed second-order multi-agent systems are studied. Based on integral sliding-mode control, sliding-mode consensus protocols are proposed for leaderless and leader-follower multi-agent systems with disturbances, respectively. Firstly, for leaderless second-order multi-agent systems, a sliding-mode consensus protocol is proposed to make the agents achieve asymptotic consensus. Secondly, for leader-follower second-order multi-agent systems, a finite-time sliding-mode consensus protocol is designed to make the agents achieve consensus in finite time. Both kinds of consensus protocols inherit the anti-disturbance performance and robustness of sliding-mode control and require less communication information. Finally, two numerical simulations are given for leaderless and leader-follower second-order multi-agent systems to validate the efficiency of the proposed consensus protocols.     

Fixed-time bipartite consensus of nonlinear multi-agent systems using event-triggered control design

This paper focuses on designing a leader-following event-triggered control scheme for a category of multi-agent systems with nonlinear dynamics and signed graph topology. First, an event-triggered controller is proposed for each agent to achieve fixed-time bipartite consensus. Then, it is shown that the Zeno-behavior is rejected in the proposed algorithm. To avoid intensive chattering due to the discontinuous controller, the control protocol is improved by estimating the sign function. Moreover, a triggering function is proposed which avoids continuous communication in the event-based strategy. Finally, numerical simulations are given to show the accuracy of the theoretical results.     

A model-free distributed cooperative frequency control strategy for MT-HVDC systems using reinforcement learning method

The performance of the existing frequency control strategies for MT-HVDC systems relies on the accuracy of the mathematical models. However, it is hard to obtain the exact mathematical models of MT-HVDC systems. To deal with this challenge, this paper presents a distributed cooperative frequency control strategy for MT-HVDC systems by using a reinforcement learning method, i.e., value iteration algorithm. Specifically, the proposed control strategy is data driven and hence model free. Besides, the proposed control strategy is distributed in the sense that each AC area only requires the local and neighboring information, rather than the information of all the AC areas. Moreover, the proposed control strategy makes the connected AC areas compensate load disturbances together by sharing their power reserves via HVDC grids, which greatly decreases the operation costs of MT-HVDC systems. The performance of the proposed control strategy is validated by cases study.     

Event-Triggered privacy-preserving average consensus for continuous-time multi-agent network systems

This paper deals with the privacy-preserving average consensus problem for continuous-time multi-agent network systems (MANSs) based on the event-triggered strategy. A novel event-triggered privacy-preserving consensus algorithm is designed to achieve the average consensus of MANSs while avoiding the disclosure of the agents’ initial states. Different from the approaches incorporating stochastic noises, an output mask function in the proposed algorithm is developed to make initial state of each agent indiscernible by the others. Particularly, under the output mask function, all agents can exactly tend to the average value of initial states rather than the mean square value. Under the proposed algorithm, detailed theoretical proof about average consensus and privacy of the MANSs are conducted. Moreover, the proposed algorithm is extended to nonlinear continuous-time MANSs, and the corresponding results are also derived. A numerical simulation eventually is performed to demonstrate the validity of our results.     

Dual ML-ADHDP method for heterogeneous discrete-time nonlinear multi-agent systems with unknown dynamics and time delay

This paper develops a new dual ML-ADHDP method to solve the optimal consensus problem (OCP) of a class of heterogeneous discrete-time nonlinear multi-agent systems (MASs) with unknown dynamics and time delay. A hierarchical and distributed control strategy is used to transform the original problem into nonlinear model reference adaptive control (MRAC) problems and an OCP of virtual linear MASs. For the nonlinear MRAC problems, a new multi-layer action-dependent heuristic dynamic programming (ML-ADHDP) method is developed to overcome the unknown dynamics and neural network estimation errors, which has higher control accuracy. In order to solve the OCP of virtual linear MASs and improve the convergence speed, a new multi-layer performance index is proposed. Then the ML-ADHDP method is used to solve the coupled Hamiltonian–Jacobi–Bellman equation and obtain the optimal virtual control. Theoretical analysis proves that the original MASs can achieve Nash equilibrium, and simulation results show that the developed dual ML-ADHDP method ensures better convergence speed and higher control accuracy of original MASs.     

Predefined-time practical consensus for multi-agent systems via event-triggered control

This paper studies the predefined-time practical consensus problem for multiple single-integrator systems through event-triggered control. A new kind of time-varying functions is firstly proposed. Then, new event-triggered control inputs as well as triggering conditions are designed on the basis of the time-varying function and the local broadcasted states. In particular, the control scheme is fully-distributed because no global information of the system and the communication topology is needed. Furthermore, the consensus analysis is presented based on a sufficient condition for predefined-time practical stability. It illustrates that practical consensus can be ensured with a completely pre-specified time. Besides, the exclusion of Zeno behavior at all the time instants is addressed. Numerical results verify the validity of the obtained control method.     

Consensus control for multi-agent systems with distributed parameter models via iterative learning algorithm

This paper deals with the problem of iterative learning control algorithm for a class of multi-agent systems with distributed parameter models. And the considered distributed parameter models are governed by the parabolic or hyperbolic partial differential equations. Based on the framework of network topologies, a consensus-based iterative learning control protocol is proposed by using the nearest neighbor knowledge. When the iterative learning control law is applied to the systems, the consensus errors between any two agents on L² space are bounded, and furthermore, the consensus errors on L² space can converge to zero as the iteration index tends to infinity in the absence of initial errors. Simulation examples illustrate the effectiveness of the proposed method.     

Distributed optimal consensus protocol for high-order integrator-type multi-agent systems

Optimal consensus control of high-order multi-agent systems (MASs) modeled by multiple integrator-type dynamics is studied. A fully distributed optimal control protocol that achieves the specific consensus behavior is designed for MASs with linear dynamics, where topology-dependent conditions are removed. Further, a distributed consensus protocol for high-order nonlinear MASs with one-sided Lipschitz continuity is presented using the optimization approach, and the optimal solution can be obtained by solving a standard algebraic Riccati equation. Some numerical examples are finally provided to illustrate the effectiveness of the presented approaches.