Databricks Benchmarks AI Coding Agents on Massive Codebase

Databricks built an internal benchmark from real pull requests to evaluate coding agents against its multi-million line, multi-language codebase. The results revealed three distinct capability tiers: top-tier models handle any task but at high cost, while mid- and lower-tier models are just as effective for everyday work at a fraction of the price. Notably, the open-weight GLM 5.2 matched Anthropic's Opus 4.8 on quality while costing significantly less per task, positioning it as a viable daily-driver model.

The analysis also found that per-token pricing is a poor proxy for real task cost, since larger models can be far more token-efficient overall. Just as importantly, running the same model through different harnesses (like Claude Code/Codex versus Pi) produced up to 2x cost differences at equal quality, driven mainly by how much context each harness feeds the model per turn. Databricks argues that public benchmarks like SWE-Bench don't reflect their codebase or avoid data leakage, which is why they built their own task set from carefully filtered, human-written PRs — giving engineering teams a more reliable basis for choosing which models and harnesses to deploy.