Token Governor: The Smart Gateway That Cuts LLM Costs

Token Governor is both a cost-analysis tool and a real gateway layer that re-engineers LLM prompts to use the fewest tokens necessary. In single mode, users build a constrained prompt with settings like succinctness, output shape, and temperature, then see a projected savings gauge and a verdict on whether the constraints are actually worth it. In batch mode, multiple prompts can be scored at once to flag ones that are deterministic and never needed a model call, with results exportable as CSV or Markdown.

The system is provider-agnostic, supporting Anthropic and any OpenAI-compatible backend—including Gemini, Groq, OpenRouter, and local runtimes like Ollama or vLLM—through a single adapter file that makes adding new providers straightforward. A live A/B panel runs the same prompt both raw and governed through a server-side proxy, reporting real token counts for direct comparison.

What sets it apart is that it doubles as a runtime gateway for production apps: pointing an existing SDK's base_url at it means every call is automatically governed, with max_tokens capped, deterministic prompts optionally blocked, and streaming passed through unmodified. Named policies (default, terse, strict, passthrough) can be selected per request via headers, giving engineers fine-grained control over cost behavior in live systems. Built-in health checks that plug into CI further simplify pre-deployment verification.