AI Coding Agents Are Absorbing the Leaves, Not the Whole Tree
SWE-bench and METR data track fast AI coding gains, but verification cost — not raw difficulty — still defines where agents stop.
Benchmark trends make the capability curve hard to dispute: SWE-bench scores climbed from 1.96 percent in late 2023 to 74.9 percent by mid-2025, while METR's measured task horizon is now doubling roughly every three months. Yet contamination-resistant SWE-bench Pro shows the same frontier models dropping to about 23 percent on larger, unfamiliar-repo tasks, and Stack Overflow's 2025 survey finds 84 percent of developers using AI tools while only 3 percent highly trust the output.
The mechanism behind this split is reinforcement learning against cheap, checkable feedback — compilers and failing tests — which makes code an unusually good substrate for the loop. That explains why prominent skeptics like François Chollet and Andrej Karpathy have conceded ground specifically on routine implementation work, while remaining cautious about custom architecture and system-level design.
The underlying thesis: agent capability tracks verification cost, not task difficulty per se. Leaf-level work has a cheap verifier (a test goes green or red); root-level work only gets verified by an incident review or the market itself. Benchmarks like PerfCodeBench and SWE-Perf show models still lag on system-level performance work such as parallelism and GPU optimization — knowing the vocabulary of optimization isn't the same as owning its cost model. Research on model collapse further suggests that rejected diffs paired with good explanations may be more valuable training data than passing-but-flawed code, making human judgment traces the scarcer resource going forward.