A Decade of Vision-Language Models: Why Easy Benchmarks Mask Real Progress
A decade-long study finds vision-language model progress is real but hidden by easy benchmarks; only spatial reasoning errors remain unsolved.
New research challenges claims that image captioning models have reached 'human-level' performance, showing those claims rest almost entirely on easy, clean benchmarks like MS-COCO. Researchers built a new Complex Social Behavior (CSB) dataset from movie frames requiring multi-person social reasoning, then tested nine models spanning 2017 to 2025.
The results are stark: older CNN+LSTM captioners perform near human level on MS-COCO but collapse on CSB. Modern multimodal LLMs—GPT-4, Gemini, GPT-5.1—remain statistically indistinguishable from top-ranked human descriptions on both datasets. Of five defined error types (object detection, recognition, hallucination, scene understanding, spatial dependence), four have been essentially eliminated by current MLLMs.
The one persistent failure is 'spatial dependence'—models sometimes rely on different image regions than humans do when generating descriptions. Notably, this error hurts overall semantic similarity the least; detection and hallucination errors remain far more damaging. For engineers, the findings underscore how benchmark choice can inflate perceived capability, and point to where evaluation efforts should now focus.