Fine-Grained Attention Enhancement for Mitigating Hallucinations in LVLMs

Large vision-language models (LVLMs) have achieved impressive performance on tasks such as visual question answering and image captioning, yet they still suffer from hallucinations, producing descriptions that conflict with the visual input. Prior efforts to mitigate hallucinations often rely on data-centric strategies or specialized fine-tuning, which require large-scale annotations or costly retraining. More recent training-free methods adjust attention at the modality level but largely ignore fine-grained object–region grounding. In this work, we analyze hallucinations through the lens of cross-modal attention and show that, during decoding, LVLMs tend to collapse onto a few visually global tokens, while progressive encoding compresses visual evidence into global tokens that are easily overridden by language priors. To address this, we propose a training-free framework that (i) leverages CLIP patch embeddings and early-layer attention maps to reweight decoder cross-attention in an object-centric, fine-grained manner, and (ii) introduces an auxiliary decoding branch with masked global tokens for contrastive decoding, effectively reducing hallucinations without additional training.