Are AI Labs Gaming the Pelican-on-a-Bicycle Benchmark?

Simon Willison's long-running informal test — asking LLMs to draw an SVG of a pelican riding a bicycle — has become a widely cited proxy for model capability. Its fame raised a natural question: are AI labs quietly optimizing (or 'benchmaxxing') for this specific prompt? A researcher set out to test this by generating 1,008 SVGs across seven frontier models using a grid of 8 animals and 6 vehicles, then scoring them with an LLM judge and a feature-extraction pipeline.

The results show no clear evidence of benchmark gaming. Pelicans ranked 6th out of 8 animals by average score, and bicycles came in near the bottom of the vehicle rankings. A difficulty-adjusted fixed-effects regression across all 1,008 images found no statistically significant boost for any lab on pelicans, bicycles, or the specific pelican-bicycle combination — the one borderline result (Gemini 3.5 Flash on bicycles) failed to survive multiple-comparisons correction.

For engineers, the takeaway is a useful reminder that visible strength on a famous, publicly discussed benchmark doesn't automatically imply overfitting to it — and that rigorous statistical testing can debunk such suspicions. The methodology itself, combining LLM-as-judge scoring with automated feature extraction, offers a reusable template for probing benchmark contamination claims.