A deep reinforcement learning state representation method that integrates compression and iterative external memory networks

Deep reinforcement learning (DRL) faces short-term sequence modeling difficulties and long-term credit assignment problems (such as reward sparsity and memory challenges caused by delays) when extracting spatiotemporal features from high-dimensional pixel inputs. We propose a deep reinforcement learning framework based on compressed iterative external memory networks. The core contribution of this work is a compression-and-iterative external memory framework for visual reinforcement learning, in which compact spatiotemporal encoding, memory-efficient storage, and multi-hop memory reasoning are jointly designed for long-horizon decision-making. Specifically, the method first compresses stacked image observations into low-dimensional spatiotemporal latent queries using 3D CNNs and RoPE-enhanced self-attention, and then performs iterative multi-hop reasoning over a fixed-size external memory module with two-stage memory compression. Experiments were conducted in the Atari (discrete action space) and MuJoCo (continuous action space) benchmark environments, and the results show that the proposed method significantly improves sample efficiency and agent performance, especially in tasks requiring spatiotemporal perception and long-term planning.