When Size Matters: Balancing Storage and Performance in Continual Learning

With the growing scale and application of large language models (LLMs), continual learning (CL) has become increasingly important for adapting to new tasks without forgetting previously acquired knowledge. However, conventional CL approaches often suffer from inefficiencies issues when applied to large models. Parameter-Efficient Fine-Tuning (PEFT), particularly methods like LoRA, offers a promising alternative by isolating task-specific knowledge through lightweight modules. Despite their advantages, PEFT-based CL methods face challenges in storage and computational efficiency as the number of tasks grows, especially when high-capacity modules are required for large datasets. To address these issues, we propose Binary Parameter-Efficient Fine-Tuning for Continual Learning (BP-CL), a novel method that combines the advantages of LoRA and binarized compression. BP-CL binarizes fine-tuned LoRA modules to significantly reduce storage costs while maintaining or even improving task performance. Additionally, a lightweight task allocation module enables accurate task routing with only a few exemplar samples, supporting dynamic task expansion without retraining other existing modules. We evaluate BP-CL on two widely-used benchmarks: SuperNI and Long Sequence Benchmark. Experimental results demonstrate that BP-CL not only achieves state-of-the-art performance but also offers a highly efficient solution for scalable and storage-friendly PEFT-based CL.