Adversarial Robust Distillation Method Based on Intensity Correlation Regularization Learning

This research introduces an Adversarial Robust Distillation (ARD) method based on Intensity Correlation Regularization Learning (ICRL) to address the limitations of existing ARD approaches, which are hindered by insufficient and unreliable guidance from the teacher network and fixed attack strengths. The proposed method comprises two key modules: multidimensional knowledge distillation and dynamic adjustment of attack intensity. Multidimensional knowledge distillation effectively addresses distributional discrepancies caused by inadequate or unreliable teacher network guidance by incorporating instance-level and class-level knowledge distillation across teacher and student logits, as well as introspective self-distillation within the student network. To enable adaptive updates of attack strength based on the improved robustness of the student network, an efficient intensity dynamic adjustment algorithm is designed to dynamically select and assign the appropriate attack intensities for each instance. Additionally, ICRL applies regularization to the introspective self-distillation dimension from the attack strength perspective, adaptively normalizing the student's introspective loss and mitigating the impact of extremely adversarial perturbation instances. Extensive experimental results on the CIFAR-10 and CIFAR-100 datasets demonstrated that this method functions as a universal plugin for most mainstream ARD frameworks, and significantly enhances the resilience of baseline methods against multistep attacks. In particular, with the current state-of-the-art baseline method named AdaAD, when using a ResNet-18 student network, AdaAD-ICRL achieves improvements of 2.06 and 2.11 percentage points in adversarial robust accuracy against Projected Gradient Descent (PGD)-10 attacks, validating the compatibility and effectiveness of ICRL within existing frameworks.