Optimal Control of Unknown Nonlinear Systems via a Multimodel-Based Multistep Reinforcement Learning Framework

Optimal control of unknown nonlinear systems is challenging due to the absence of the underlying dynamics model. Reinforcement learning (RL) has become an effective framework for such control by optimizing policies with measurement data of the system. However, RL often suffers from: 1) low sample efficiency due to the absence of domain knowledge (e.g., dynamics model); and 2) low learning efficiency due to slow propagation of the reward signal. Model-based RL methods and multistep RL methods were proposed to address these two challenges, respectively. It is natural to combine these two methods to take advantage of both benefits. However, this combination requires the learned single one-step dynamics model, commonly used in model-based settings, to perform multistep prediction in a recursive manner, leading to the error accumulation (also known as compounding error) problem. This work presents a multimodel dynamics learning framework to address the error accumulation challenge for general model-based multistep RL methods by circumventing the recursive prediction. We also present a specific multimodel-based multistep RL algorithm and validate it on benchmark nonlinear systems. It is shown that the multimodel framework improves multistep prediction of dynamics and that the presented multimodel-based multistep RL mostly outperforms model-free, single-model, and one-step counterparts, respectively. We also discuss the limitations of this work and potential future work.