A Personalized Sparse LoRA Fine-Tuning Framework under Federated Learning

Large language models (LLMs) achieve strong performance but rely on centralized training with massive highquality data, raising critical privacy concerns. Federated learning (FL) provides a natural solution; however, directly applying FL to LLM fine-tuning incurs prohibitive computational and communication costs. Parameter-efficient fine-tuning (PEFT) alleviates this issue, yet existing approaches struggle with heterogeneous client data and noise amplification in low-rank adaptation. To address these challenges, we propose PSFed-LoRA, a personalized sparse LoRA framework for federated LLM fine-tuning. At the core of PSFed-LoRA is the decoupling of LoRA ranks into shared and personalized components, combined with a hierarchical lowrank adaptation scheme and a dynamic sparsification mechanism. This design mitigates cross-client interference, reduces redundant updates, and enables an effective trade-off between global generalization and local personalization under heterogeneous data distributions. Extensive experiments on language understanding benchmarks with heterogeneous settings show that PSFed-LoRA consistently outperforms strong baselines, achieving higher accuracy while reducing communication cost by over 50%. Moreover, our method demonstrates improved robustness to data heterogeneity and noise in the low-rank space.