Rethinking Adversarial Robustness Distillation VIA Strength-Dependent Adaptive Regularization

Despite the progress achieved by existing adversarial distillation (AD) approaches, most mainstream models suffer from inadequate adversarial robustness, due to the challenges of fixed attack strength and unreliable teacher guidance. In this paper, we propose a novel Strength-Dependent Adaptive Regularization (SDAR) paradigm to reinforce the function of adversarial distillation with strength-adaptive adversarial attack (SAA) and multi-dimensional knowledge distillation (MKD). Different from the traditional adversarial training (AT) methods, the proposed SAA scheme dynamically assigns an adaptive and efficient attack strength for each instance, which aims to facilitate smoother classification boundaries. By incorporating dynamic strength coefficients, a comprehensive MKD strategy is designed to fully explore the valuable context information and narrow distribution discrepancies across teacher-student domains. Particularly, our SDAR paradigm can seamlessly integrate with the current AD frameworks, further enhancing the adversarial robustness of deep learning models. Extensive experiments on multiple benchmark datasets consistently demonstrate the superiority of SDAR over state-of-the-art baselines.