Capturing Cross-Modal Semantics by Generating Comments for Image-Text Contents

Recent studies on multi-modal learning have benefited much from the development of Vision-Language Pre-training (VLP) approaches, which are believed to be able to bridge the semantic gap between the visual and linguistic modalities. Despite the notable successes, the current VLP methodologies mainly focus on the semantic overlap of the visual and textual inputs, and thus the actual cross-modal information complementarity tends to be ignored. Consequently, this paper defines a new VLP task of generating natural-language comments from the given Image-Text contents, named as Cross-Modal Information Complementarity based Comment Generation (CroMIC-CMT), so as to guide the models to capture the cross-modal semantics. A generic architecture is also presented for the proposed task, with formalized components for the bidirectional multi-modal encoding and autoregressive text generation. Furthermore, to validate the effectiveness of CroMIC-CMT as a pre-training method, it is evaluated and compared on the downstream tasks. The experimental results show that the proposed VLP task brings a new perspective on multi-modal semantic learning, and can be taken as a potential pre-training paradigm to address the downstream problems ¹The source code of this work is uploaded (here).