TCN-SAC: Enhancing the Soft Actor-Critic Algorithm with Temporal Convolutional Networks

In the past few years, deep reinforcement learning (DRL) has garnered increasing attention and found widespread use in fields such as robot control, gaming, and autonomous driving. Among model-free reinforcement learning algorithms, the Soft Actor-Critic (SAC) algorithm stands out due to its exceptional exploration ability and high data efficiency, making it a popular choice in both research and practical applications. However, the traditional SAC algorithm lacks the ability to model temporal dependencies in historical state data. Moreover, the current improvements using Long Short-Term Memory (LSTM) as a solution to this limitation still face challenges, including low training efficiency and instability training. In this paper, the TCN-SAC algorithm is proposed, which employs Temporal Convolutional Networks (TCNs) as the backbone for the Actor network. This proposed algorithm models the temporal dependencies in historical states while offering advantages such as high parallelism, flexible receptive fields, and stable training. Additionally, comparative experiments conducted in the Pendulum-v1 environment demonstrate that the proposed TCN-SAC outperforms the baseline, highlighting the superiority of this algorithm.