A multi-task learning model with adversarial data augmentation for classification of fine-grained images

A large set of training samples is a prerequisite to effectively learn deep neural networks for image classification. When the labeled samples are scarce, it often fails to produce a promising model. Data augmentation is a widely used technique to overcome the issue, which enlarges the training samples with label invariant transformations, e.g., rotation, flip and random crop etc. However, the diversity of images generated by standard data augmentation is quite limited and thus the final improvement on classification accuracy is not much, especially for fine-grained classification problem. In this paper, we propose a two-stage generative adversarial network, namely Fine-grained Conditional Adversarial Network (F-CGAN), which can produce class-dependent synthetic images with fine-grained details. Moreover, to leverage the synthetic images for fine-grained classification, we develop a multi-task learning classifier, which categorizes training images and synthetic images simultaneously. Experimental results on CUB Birds and Stanford Dogs data sets show that the proposed method indeed improves the classification accuracy.