ast-grep's Rust Rewrite of Tree-sitter Boosts Parsing 30%, App 22%

ast-grep, a structural code-search tool built on Tree-sitter, has replaced Tree-sitter's C parsing core with a Rust implementation, with much of the code written by ChatGPT under human direction. Benchmarks show raw parsing throughput up roughly 30%, tree traversal up about 10%, and the full ast-grep outline workflow (parsing plus traversal) running around 22% faster in CPU time, though peak memory (RSS) rose in most scenarios.

The gains came with tradeoffs: RSS increased by about 30% in the full outline benchmark. On the upside, the project's largest stress test — the TypeScript compiler's test suite — previously peaked above 1 GiB in memory and now peaks around 91 MiB.

The new Rust core is not a drop-in replacement for upstream Tree-sitter: binary interface (ABI) compatibility was preserved so existing generated grammars keep working, but incremental re-parsing and native WebAssembly grammar loading were removed, and unsafe code with raw pointers is still required for compatibility. The target workload is AI coding agents that analyze full file snapshots, not editors needing incremental updates.

The rewrite process itself stands out: rather than a from-scratch redesign, existing C tests served as the behavioral oracle, and code was translated section by section — utilities, tree storage, lexer, parser stack, navigation, and the parser loop — before optimization began. This approach reflects how AI-assisted coding made an ambitious rewrite feasible for a single engineer.