GFMLLM: Enhance multi-modal large language model for global and fine-grained visual spatial perception

In the past, Multimodal Large Language Models (MLLMs) with visual spatial perceptual capabilities can only handle detection task and perform coarse-grained perceptions through language responses. To achieve fine-grained outputs (e.g., masks), additional task-specific expert models and loss terms are required, which inevitably increases model size, adds training overhead and implicitly weakens the scalability of MLLM. To this end, we present GFMLLM, a multimodal large language model that performs global and fine-grained visual perception without the assistance of expert models. It leverages a unified framework to produce spatially detailed feedback in linguistic form. Technically, we propose a unified textual response paradigm, called the perception text matrix, which utilizes a dense query mechanism of MLLM and intuitively reflects the detailed spatial distribution of target objects within an image. By further processing the information encoded in the matrix, it can be naturally extended to referring expression understanding (REC) task, or collaborated with expert model to handle the more fine-grained referring expression segmentation (RES) task. Moreover, to address the high generation latency caused by the long text of the perception text matrix, we propose a token aggregation technique that significantly reduces the inference latency of GFMLLM. Experimental results in public datasets validate the effectiveness of our approach.