data-eng-bench: Snowflake's dbt benchmark for coding agents
Snowflake's data-eng-bench tests coding agents on 103 realistic dbt data-engineering tasks across DuckDB and Snowflake, via the open-source Harbor framework.
Snowflake has released data-eng-bench, a benchmark that measures how well coding agents perform real dbt data-engineering work on a large, realistic retail warehouse. Each of its 103 tasks drops an agent into a containerized dbt project with a ticket-style instruction; the agent edits or creates dbt models, runs dbt, and a hidden pytest verifier checks the output tables row by row against a reference solution.
Tasks span analytics marts, bug-fixes in broken models, dimensional modeling with snapshots, and engineering-heavy transforms like incremental models, spread across easy to very-hard difficulty tiers. Because the benchmark runs on Harbor, any Harbor-supported agent — Claude Code, Codex, Cortex Code, Terminus, and others — can be evaluated with a single command.
The same task set runs against either DuckDB (fully hermetic, no account needed) or a real Snowflake account, selected via the DB_TYPE variable. Running both and comparing isolates whether a failure stems from incorrect dbt SQL logic or from a Snowflake-specific gap in dialect, warehouse, or role handling — a distinction that matters for teams evaluating agents for production data pipelines.
The project is Apache-2.0 licensed, ships a balanced 30-task fast subset, and maintains a public leaderboard with a submission process via Harbor uploads and PRs.