Part-based fine-grained bird image retrieval respecting species correlation

Most of the existing works on fine-grained bird image categorization and retrieval focus on finding similar images from the same species and often give little importance to inter-species similarity. In this paper, we devise a new fine-grained retrieval task that searches similar instances from different species. To this end, we propose a two-step strategy. In the first step, we search for visually similar parts to a query image using a deep convolutional neural network (CNN). To improve the quality of the retrieved candidates, we incorporate structural cues into the CNN using a novel part-pooling layer. In the second step, we re-rank the retrieved candidates improving the species diversity. We achieve this by formulating a novel ranking function that balances between the similarity of the candidates to the queried parts, while decreasing the similarity to the query species. We provide experiments on the benchmark CUB200 dataset and demonstrate clear benefits of our schemes.