COPR: Continual Human Preference Learning via Optimal Policy Regularization

The growing integration of Large Language Models (LLMs) into real-world applications underscores the critical need for continual alignment with evolving human preferences. Reinforcement Learning from Human Feedback (RLHF) has shown success in improving the alignment of LLMs, but its rigid, multi-stage process presents significant limitations for continual learning (CL) scenarios, where models need to adapt incrementally without catastrophic forgetting. Existing methods, such as Direct Preference Optimization (DPO), offer potential for offline preference learning but exhibit challenges like increased preference gap amplification and reduced model diversity, which can lead to preference collapse. In practical settings, LLMs continuously interact with diverse user feedback across tasks and domains. The inability of current approaches to efficiently incorporate incremental human preferences without retraining or significant computational overhead limits their scalability and adaptability. Addressing these gaps, our study introduces a novel framework, Continual Optimal Policy Regularization (COPR), that ensures robust and flexible continual alignment while preserving historical knowledge and optimizing performance in new preference tasks.