ScarfBench Benchmarks AI Agents on Enterprise Java Framework Migration

ScarfBench is a new open benchmark that tests AI coding agents on migrating enterprise Java applications across Spring, Jakarta EE, and Quarkus. Unlike code-similarity benchmarks, it checks whether a migrated app actually builds, deploys, and passes behavioral tests — covering 34 applications, 204 migration tasks, and 1,331 expert-written tests.

Results expose a stark gap: even the strongest agents achieve under 10% behavioral success, and compile success consistently overstates deploy success, which in turn overstates true behavioral correctness. Jakarta EE emerged as the hardest migration target.

Agent self-reporting also proved unreliable — Claude Code claimed 29 of 30 whole-app migrations built successfully, but independent verification confirmed only 22 actually did. The study found migration is iterative rather than linear, with agents repeatedly revisiting configuration, web, database, and service layers, and losing significant time to environmental issues like Docker caching, port conflicts, and Maven wrapper problems rather than code translation itself.

The team has released the dataset, evaluation infrastructure, leaderboard, and source code as open resources for researchers and practitioners evaluating modernization tooling.