0views0forks
All six configurations scored by one harness with identical questions and scorers, validated against LLaVA-1.5-7B's published numbers to within 1.6 points. POPE is shown as accuracy rather than F1: a blind model that answers “no” to everything scores F1 near zero on this balanced set, which inflates the apparent grounding gap.
| Benchmark | Gain from the image | vs LLaVA-1.5-7B | vs Qwen3-VL-8B |
|---|---|---|---|
| TextVQA | +54.51 | +15.40 | −19.96 |
| GQA | +26.91 | +4.261 | +3.291 |
| POPE (acc) | +41.80 | +2.244 | −0.844 |
| MMMU | +1.556 | +9.889 | −4.111 |
GQA beats the professionally-trained reference and POPE essentially ties it, so relational reasoning and object hallucination are already close to what this data budget supports. TextVQA is the outlier in both directions: the largest genuine vision win over LLaVA-1.5, and at nearly 20 points almost the entire remaining deficit against a model built on the same language backbone.
MMMU's margin is not a vision result. The same model with the image withheld already scores 43.0 of its 44.556, so the +9.889 over LLaVA-1.5 is Qwen3-8B being a stronger language model. The image itself contributes 1.556 points, which is why the pre-registered criterion comes back partial rather than supported.