A study from USC and CMU has revealed that vision-language models (VLMs) struggle with geometry problems when presented as diagrams, despite solving them when written in text. This cross-modal consistency gap poses a threat to production pipelines that parse design documents, architecture diagrams, or mixed-modal inputs. The study introduces a controlled geometry benchmark, ODA-Data, where each problem appears in three locked views—text-dominant, image-dominant, and combined image-plus-text—demonstrating that current VLMs struggle with the surface presentation of identical semantics. To address this, the authors propose a reinforcement learning method that uses the model's best-performing view on a given problem as a teacher and distills it into weaker views via a reverse-KL objective and on-policy rollouts, without requiring external annotation, using the model's own strongest-performing view as an internal teacher signal.
The inconsistency is not an edge case. MM-R³, a paper published in ACL Findings 2025 (Chou et al., pages 4762–4788), documents that VLM consistency does not track accuracy — higher-accuracy models are not necessarily more consistent — across question rephrasing, image restyling, and context reasoning tasks, achieving absolute consistency improvements of 5.7% and 12.5% on BLIP-2 and LLaVA 1.5M with a targeted adapter. A separate July 2025 arXiv study on modality-order sensitivity finds that simply swapping the position of the image and the textual query in the prompt produces accuracy gaps of 6 to 26 percentage points across three model families and three benchmarks, with image-first prompting consistently outperforming question-first prompting. The MIRROR authors view the cross-view asymmetry as a supervision signal: if the model finds a valid reasoning path under one view, that trajectory can be rescored and reinforced under the restricted student view, using the model's own multimodal ambiguity as training data.
As a training methodology paper, MIRROR reports geometry-benchmark accuracy and cross-view consistency improvements against standard RL baselines on ODA-Val; this is the expected scope for this class of research, and the study does not include inference-serving metrics such as p50 or p99 latency, GPU-hours per training run, cost-per-call figures, or throughput numbers from a deployed inference stack. Architects should treat these results as a proof of concept for the training approach, not a validated platform improvement. If you currently mitigate modality risk by sending the same content as both text and image and voting on the outputs, you are absorbing non-trivial token and latency overhead on every call; the research frames paired-view fine-tuning as a path to eliminate that overhead, but only for teams running their own RL infrastructure. Until then, every architecture diagram passed to a VLM should be assumed to carry a view-dependent error rate independent of the model's headline benchmark score.
The integration path for MIRROR is challenging. It demands paired multimodal datasets where problem identity is preserved across views—something most enterprise corpora lack—and requires on-policy distillation with reverse-KL optimization, adding complexity beyond standard supervised fine-tuning. For teams consuming VLMs via API, the modality-order findings from the July 2025 arXiv study imply that even trivial prompt engineering, such as placing the image before versus after the text query, can shift accuracy by double-digit percentage points; without access to model weights, the only defense is redundant querying and an eval harness that explicitly tests for view-dependent regressions. The underlying failure is view-dependent, not difficulty-dependent, which means existing accuracy benchmarks are blind to the risk if they only test single-view inputs. The geometry setting is also a controlled testbed; whether MIRROR's gains transfer to messy real-world documents with overlapping text and visual noise is an open question.
Cross-modal inconsistency should be treated as a trainable supervision signal by using the strongest modality view on each sample as a reverse-KL teacher for the weaker ones, rather than dismissing it as a robustness bug to be prompt-engineered away.
Written and edited by AI agents · Methodology