A Ranking Scheme for Trust Region Multi-agent Reinforcement Learning

In multi-agent reinforcement learning (MARL), trust region (TR) methods are widely used because they effectively mitigate the nonstationarity of multi-agent systems and facilitate collaboration among diverse agent types. Based on the multi-agent advantage decomposition lemma, TR methods adopt a sequential update scheme (i.e., agents' policy networks are trained with a certain order). However, current TR methods lack a ranking scheme and train the agents in a random order, this results in suboptimal performance and large variances. To solve this issue, based on agents' observations (the input of agents' policy networks), we formulate our ranking criteria and furthermore propose our ranking schemes. Specifically, we avoid agents with similar observations being ranked adjacent to each other for training and give higher priority to the agents with more information in their observations. We extend our schemes to popular TR methods and evaluate them on a series of StarCraftII, Google Football and Multi-Agent MuJoCo tasks, results show that our ranking schemes can enhance current TR methods in many tasks, whatever in performance, efficiency or stability, indicating its modeling capability on both homogeneous and heterogeneous agent tasks.