Qwen 3.8-Max and Claude Opus 5 show why benchmark scores don't predict cost

Alibaba's Qwen 3.8-Max launched with claims of near-top coding benchmark performance, but an independent test using much tighter time budgets ranked its default setting last and its best-effort configuration only mid-pack. The gap traces to timeout allowances: Alibaba's own footnotes permit up to 12 hours per run on some tests, while the independent harness, VulcanBench, capped attempts at 45-60 minutes — a five-to-sixteen-times difference large enough to flip rankings on its own.

The episode exposes a deeper problem: price-per-token comparisons no longer predict real spend for reasoning models. Verbose thinking can consume a token budget before an answer is produced, turning a paid run into an unrecoverable empty result. Separate benchmarks — Long-Horizon-Terminal-Bench, VulcanBench, TestEvo-Bench — show that budget exhaustion, not wrong answers, drives most failures, and that higher 'effort' settings sometimes score worse simply because they run out of time rather than fail on merit.

The practical fix gaining traction is cost per successful task: total spend, including failed attempts, divided by tasks that actually pass. Vendors like HubSpot, Zendesk and Fin already bill this way. Teams building routing ladders that escalate to 'smarter' models on failure should verify whether that escalation buys better results or just a timeout, and should audit default effort settings, since an unset parameter can quietly default to the costliest, worst-performing configuration.