To Agree or To Be Right?
The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

Abstract

Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and CSI (Clinical Safety Index), a unified safety score inspired by FMEA that combines grounding, autonomy, and calibration via a geometric mean. No model in our study excels on both axes simultaneously, suggesting that current training paradigms may implicitly trade off one safety property for the other.

Proposed Metrics

Mapping FMEA to VLM evaluation

Grounding × autonomy × calibration. Each factor is floored at 0.01 so the geometric mean stays defined for models with zero resistance. R is the fraction of correctly answered questions where the model holds its answer across all three pressure types.

Figure 2. Overview of the evaluation pipeline. Medical VQA items from three benchmarks are presented to six VLMs under baseline and adversarial social pressure conditions. Model outputs are assessed using L-VASE, CCS, and CSI.

Results

Main results across three medical VQA benchmarks

L-VASE: hallucination score. R: sycophancy resistance rate. CCS: confidence-calibrated sycophancy. CSI: Clinical Safety Index. Bold is best, underline is second best, red is worst, per column.

Resistance by pressure type

Resistance rate (%) by pressure type, with mean baseline confidence. Higher resistance is safer. Bold is best, underline is second best, red is worst, within each dataset block.

Figure 3. Distribution of the Clinical Safety Index (CSI) across all evaluated models and datasets. CSI enforces the principle that failure on any single safety axis renders a system clinically unsafe.

Why VASE Needed Reformulating

VASE applies a softmax to contrastive differences of probability vectors. Subtracting one probability vector from another produces negative entries, which are not valid in probability space. Measured on 15,187 token-level vectors from LLaVA-1.5, 98.6% contained at least one negative entry, with a mean of 46.2% of total probability mass being negative. For LLaVA-Med, 92.3% of 9,291 vectors were affected. This is not a rare edge case. L-VASE avoids it by operating on raw logits, where subtraction is mathematically valid and a single softmax yields a proper distribution.

Figure 4. Side-by-side comparison of VASE (left) and our proposed L-VASE (right). VASE applies softmax to contrastive differences of probability vectors, producing invalid negatives and losing roughly 46% of probability mass. L-VASE operates in logit space with a single principled softmax, yielding 100% valid distributions. Bar charts are schematic.

Models Evaluated

All models evaluated in their public 7 to 8B configurations, float16, greedy decoding for sycophancy and temperature 1.0 for L-VASE sampling.

Datasets

BibTeX

@InProceedings{Aranya_2026_CVPRW,
  author    = {Aranya, OFM Riaz Rahman and Desai, Kevin},
  title     = {To Agree or To Be Right? The Grounding-Sycophancy
               Tradeoff in Medical Vision-Language Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer
               Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2026},
  pages     = {6874-6882}
}