Multi-Agent Target Defense Differential Game: A Hierarchical Recognition-Allocation-Execution Learning Approach

This research considers an incomplete information multi-agent target defense game (TDG) where multiple defenders protect a target against heterogeneous adversaries. The defenders need to distinguish among the adversaries first and then intercept those who pose a threat to the target. Hierarchical reinforcement learning (HRL) approaches provide an efficient solution for multi-step tasks, and thus show potential in solving the TDG problem. In this article, a novel hierarchical recognition-allocation-execution reinforcement learning (HRAE-RL) approach is proposed. The HRAE-RL, based on a goal-directed HRL framework, is composed of a first-level intention recognizer (IR), a second-level target allocator (TA), and a third-level path planner (PP). In IR, an RL-based compensation structure is proposed to generate stable outputs, which is significant for the lower-level training, and this structure greatly improves data efficiency. The TA assigns an appropriate adversary as the target of each defender. Differential game theory is utilized as an expert during the training process to predict the optimal interception point, based on which the assignment is obtained. The PP takes advantage of multi-agent deep deterministic policy gradient to generate cooperative policies for both sides. Simulation results show the superiority and explainability of HRAE-RL. During execution, the task success rate is not less than 88.47% in various scenarios. In a simple scenario where the optimal solution can be explained analytically, the difference between the average intercept distance generated by HRAE-RL and the optimal value does not exceed 4.77%.