TaxCalcBench: Open Source Test for AI Tax Filing

TaxCalcBench is an open source evaluation framework built to measure how well large language models can actually prepare tax returns. It feeds models realistic PDF inputs like W-2s and 1099s and tests them against increasingly complex federal and state tax scenarios.

In the latest Tax Year 2025 results, GPT-5.5 with web search tops the leaderboard, hitting 84.44% accuracy on a per-line basis, yet its rate of producing a fully correct return (strict criterion) is only 54%. Claude and Gemini models trail further behind, with strict scores ranging from as low as 2% to 34% depending on the model and thinking configuration. The gap shows that even when models get most individual line items right, a single error can still invalidate an entire return.

The project ships as a command-line toolkit supporting multiple providers (OpenAI, Anthropic, Google), configurable thinking budgets, and optional web search tools, making it reproducible for ongoing model comparisons. For engineers, it offers concrete, repeatable evidence that LLMs still struggle with high-stakes, rule-heavy, precision-critical real-world tasks — underscoring the need for robust verification layers before deploying AI in financial automation workflows.