How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images

Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. Our work, for the first time, systematically validates inadequate visual grounding in clinically relevant image regions as one of the key contributing factors for medical MLLMs' under-performance. We note that this finding is specific to medical image analysis. In contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images.

We show the visual grounding issues in state-of-the-art medical MLLMs that they often suffer from inadequate visual grounding—they fail to accurately localize and interpret image regions that are clinically relevant to the question (subfig. a). In contrast, when applied to natural images, MLLMs are capable of grounding their predictions in the correct image regions (subfig. b). This limitation contributes to medical MLLMs' suboptimal performance on medical VQA tasks.

Existing Med-VQA datasets are not well-suited for systematically studying visual grounding (subfig. a): it's difficult to disentangle whether model errors arise from inadequate semantic grounding (i.e., the model is unable to determine what to look for) or from inadequate visual grounding (i.e., the model is unable to localize the relevant image region even when it knows what to look for). To disentangle visual grounding from semantic grounding, we design VGMED (subfig. b), a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs.

Medical MLLMs demonstrate suboptimal visual grounding when applied to medical images. Analysis using our proposed VGMED dataset shows that all evaluated medical MLLMs exhibit substantial weaker alignment between their attention distributions and ground-truth annotations on medical images compared to natural scene images (from MS COCO).

This failure mode contributed to their underperformance in zero-shot medical image understanding. Further comparison with LLaVA-v1.5 on natural images reinforces this observation: medical MLLMs show significantly lower alignment with annotated regions, highlighting deficiencies in visual grounding for medical image analysis.

We proposed Visual Grounding Refinement (VGRefine): a two-step inference-time method to improve visual grounding in medical MLLMs.

• Step I (Attention Triage): We aggregate attention from the model's most visually sensitive heads and suppress low-confident attention, obtaining a binary mask.
• Step II (Attention Knockout): We use this mask to refine the model's attention distribution, improving its focus on relevant regions during inference.

Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models.