Hierarchical Prompt Learning for Scene Graph Generation

Scene Graph Generation (SGG) delivers structured knowledge representing complex scenes, which is applied in many computer vision fields. However, existing SGG models falter in predicting novel and informative predicates, undermining their applicability for higher-level visual tasks. Drawing inspiration from the success of prompt learning in zero-shot knowledge transfer, we propose a prompt-learning-based method to address novel and informative predicate learning challenges in SGG. Specifically, we perform a comprehensive analysis of three basic prompts in SGG, considering their computational efficiency and learning ability. Subsequently, we build upon these basic prompts to construct a Hierarchical Prompt (HP) learning method to enhance informative predicate learning. HP utilizes the composition of basic prompts constrained to progressively narrowed class groups and encourages the corresponding prompts to focus on the learning of increasingly informative predicates. HP is a plug-and-play solution applicable to various models. Extensive evaluations on SGG benchmarks demonstrate the excellent ability of HP to improve the performance of informative predicates across different baselines. We also introduce a novel predicate generalization task with a new benchmark. Experiments on it demonstrate the superiority of HP in base-to-novel predicate generalization.