Knowledge-Guided Deep Reinforcement Learning for Timely State Estimation in Wireless Embodied Networks

Remote state estimation of numerous functional components is essential for the reliable and real-time operation of modern embodied intelligent agents. In this paper, we focus on solving the optimal resource allocation problem in such a large-scale embodied network with massive connected sensing devices and limited wireless spectrum resources for achieving high-quality remote state estimation. First, we derive the structural properties of the optimal resource allocation policy, including the no-waste channel structure of the optimal action and the monotonicity of the optimal value function. Next, we propose the Knowledge-Guided deep reinforcement learning (KG-DRL) algorithm by leveraging the derived structural knowledge to guide the training of the actor and critic neural networks (NNs). Specifically, a structure-guided trajectory for facilitating the actor NN learning is generated by selecting the action based on the no-waste channel structure, and the critic NN is guided to be monotonic by introducing the monotonicity violation penalty term. Numerical results illustrate that the proposed KG-DRL algorithm outperforms the benchmark DRL algorithms by about 20%-35% in terms of the estimation mean-square error (MSE) and the total power consumption, while saving the required convergence episodes by approximately 35%.