SynthDocBench Exposes Long-Context Weaknesses in Vision Language Models

SynthDocBench is a fully synthetic benchmark designed to isolate the variables that muddy existing document-understanding tests like DocVQA and ChartQA. By independently varying document length, layout, and modality composition across six generated layout archetypes—with a 40 percent random override to prevent shortcut learning—it lets researchers pinpoint exactly why VLMs fail rather than just that they fail.

Testing seven frontier VLMs revealed three consistent failure modes: sharp accuracy decline as document length increases, a systematic positional bias where the middle third of long documents is hardest to process (with an 8.3-point drop from early to late sections in five of six models), and a specific breakdown in chart comprehension once charts are embedded in long documents rather than presented in isolation.

The results suggest current VLM performance on standard benchmarks may be inflated by overfitting to benchmark-specific artifacts rather than reflecting genuine long-context understanding. For teams building document-processing pipelines for legal, financial, or technical use cases, this points to concrete gaps in attention mechanisms and global context aggregation that need addressing before deploying VLMs on lengthy, structurally complex real-world documents.