Leveraging Neural Architecture Search for improved downstream-agnostic adversarial attack

Self-supervised pre-trained models like SimCLR are widely used as encoders to extract image features without task-specific training. However, the downstream-agnostic adversarial attacks on pre-trained models have raised critical security concerns, as a single and uniform adversarial perturbation can deceive all downstream tasks. But the impact of model architectures on the generation of adversarial attacks is still an unexplored question. Thus, we introduce a novel downstream-agnostic adversarial attack framework empowered by Neural Architecture Search (NAS). Specifically, our approach constructs adversarial perturbations using a U-Net generator, whose skip connections are optimized via NAS. To efficiently explore the vast search space of 2¹⁶ possible architectures, we also integrate a Genetic Algorithm guided by early-stage training metrics. To the best of our knowledge, this is the first work to incorporate NAS into the design of generative adversarial attacks. Extensive experiments on 14 pre-trained models demonstrate that our method achieves superior performance.