Multi-Agent Reinforcement Learning Based Cooperative Caching With Low Entropy Communications in Fog-RANs

In this paper, we investigate a cooperative edge caching problem in the fog radio access networks (F-RANs). In order to obtain the globally optimal caching strategy that minimizes the content transmission delay and maximizes communication efficiency, we propose a multi-agent reinforcement learning based cooperative caching policy with low entropy communications. First, we propose a double deep Q network (DDQN) based caching policy by taking into account the non-deterministic polynomial hard (NP-hard) aspect of this cooperative caching optimization problem. Then, we extend the state transition model of Markov Decision Process (MDP) under the single agent system into the Stochastic Game (SG) one under the multi-agent system. By employing the DDQN in each agent, the agents can learn and make the global decision for caching. For utilizing the cooperation resources of fog access points (F-APs), the interaction of information is introduced to exchange the historical cache records of cooperative F-APs. However, the information in the interaction may require lower entropy in the fiber link. Therefore, the information entropy is largely reduced to improve the communication efficiency by quantifying the information. Finally, due to the non-computable gradient of information entropy, we apply a pseudo gradient descent method to approximate the gradient descent in the local model. Simulation results show that our policy achieves better performance in terms of reducing the transmission delay and improving the cooperation among F-APs compared to the benchmark policies. Additionally, it is demonstrated that the proposed policy improves communication efficiency without compromising the performance of cooperative caching.