How I Cut Agent Token Costs 60% With a 4-Layer Infrastructure Fix

An engineer describes reducing token spend on multi-step agent workflows by 60% not by changing the agents themselves, but by rebuilding the infrastructure underneath them. The starting point was per-span instrumentation — logging model ID, token counts, cache status, latency, and a complexity score for every call — which revealed that a large share of tokens were wasted on redundant context and unnecessary frontier-model calls.

A three-tier cache (exact-match, semantic near-match, and in-process LRU) eliminates 30-35% of calls from ever reaching a model, with TTLs varied by task type to avoid serving stale reasoning. A deterministic prompt-compression pipeline strips redundant instructions, trims tool schemas, and summarizes older conversation turns, saving 20-30% of input tokens per call, with savings compounding on longer chains.

The largest single gain comes from complexity-based model routing: a lightweight heuristic score sends roughly 60% of calls to a cheap small model at a fraction of frontier cost, with a feedback loop that adjusts scoring based on downstream failures. The author notes that speculative execution and a learned semantic-similarity function could push total savings past 70%, underscoring that infrastructure discipline — not just model choice — drives agent cost efficiency.