Universal Scene Graph Generation via Semantic Feature Alignment

Scene Graph Generation (SGG) offers a valuable structured representation for various computer vision applications. With advancements in visual and textual alignment, researchers are exploring both open-vocabulary object detection to address unseen object categories and open-vocabulary relation detection to manage unseen relation categories. To address real-world applications, we propose a new universal scene graph generation framework (U-SGG) that handles both unseen object categories and unseen relation categories. Our method detects unseen objects by leveraging the aligned representations of images and text. To effectively address unseen relations, we build the relation prediction head using a Visual Language Transformer, which takes into consideration not only the visual and textual features, but also the attribute and spatial features of the objects. To alleviate the problem of insufficient quantity of training data in SGG tasks, we propose to parse image captions into relation triples, thereby enriching the categories of both object and relation. Using image caption data with diverse object and relation categories, we successfully achieve universal scene graph generation. Comprehensive experimental results on the Visual Genome benchmark demonstrate the effectiveness and superiority of the proposed method.