Enhancing Compositional Reasoning in Multimodal Large Language Models

Recent studies have highlighted the shortage in compositional reasoning capabilities within Vision-Language Models (VLMs) such as CLIP and SigLIP. While these deficiencies have been documented in VLMs, their transmission to Multimodal Large Language Models (MLLMs)—which frequently rely on VLM-derived visual encoders—has not been thoroughly explored. This paper systematically investigates the compositional reasoning deficiencies in MLLMs and demonstrates that these models inherit the limitations of their underlying visual architectures. To address this challenge, we propose a vision-language compositionality enhancement strategy aimed at improving compositional reasoning while maintaining the MLLM’s overall multimodal understanding and reasoning capabilities. Our approach integrates two key innovations: (1) adopting multi-layer visual features as inputs of MLLMs to enhance fine-grained visual information, which is essential for effective compositional reasoning, and (2) a contrastive learning (CL) stage intended to improve the compositional reasoning abilities of MLLMs. Extensive experimental evaluations demonstrate that our auxiliary training strategy significantly enhances the performance of LLaVA-1.5-7B on compositional reasoning benchmarks, achieving performance parity with GPT-4V on specific tasks. Importantly, this strategy not only preserves but also improves performance across general multimodal benchmarks, highlighting its dual efficacy in both specialization and generalization.