Cross-Modal Mixup Enhance Foundation Model Adaptation for Few-Shot Learning

In few-shot learning, adapting foundation models has emerged as a powerful approach to bridge the gap between limited task-specific data and the rich knowledge encoded in pre-trained models. Existing adaptation methods based on CLIP primarily focus on improving the network architecture to enhance the learning ability, which has been proven to cause overfitting and reduce generalizability, but overlook the influence of data augmentation. In this work, we first analyze the differences between our work and previous work from the perspective of probability theory. Then, we concentrate on data augmentation and propose innovative Cross-Modal Mixup methods designed to enhance the model's cross-modal reasoning and classification capabilities, as well as its overall generalization. Specifically, based on dimensions match or not, the Cross-Modal Mixup methods comprise two forms: Cross Mixup (CM) for data augmentation with features of the same dimension, and Cross Expand Mixup (CEM) with features of different dimensions. Extensive experiments have demonstrated the effectiveness of our CM and CEM.