Persistent AI Agent Memory Is a Write-Path Cost Problem
AI agent memory costs are driven by write-path LLM calls, not retrieval. Engineering levers to cut cost and latency without sacrificing quality.
Adding memory to an AI agent is usually treated as a quality challenge — remembering preferences, staying consistent across sessions. But the real operational burden lies elsewhere: agent memory writes on every turn, and each write is an LLM call. Unlike RAG systems that index once, agent memory re-runs extraction, deduplication, and conflict resolution per user, per turn, for the life of the product.
Profiling production memory systems shows a consistent cost shape: extraction dominates at roughly 60-75% of spend, consolidation (a second LLM call triggered by collisions) adds another 10-20%, while embeddings and vector search remain minor line items. Latency behaves inversely — the read path is per-user and narrow in scope (searching one user's namespace, not a global index), so ANN search stays in the low-millisecond range regardless of total user count, while the expensive write work can run asynchronously outside the critical path.
The practical levers follow directly from this asymmetry: make writes async and batch them, gate which turns even reach the extraction model, downgrade extraction and reconciliation to smaller or fine-tuned models, bound the per-user working set with decay/TTL policies, quantize the vector index, and reserve reranking for ambiguous cases only.
Ignoring this asymmetry produces three predictable failure modes: cost that scales with how long a user stays active, extraction cost tied directly to raw traffic volume, and write-path latency leaking into the read path under load when background workers contend for the same LLM capacity. The fix is straightforward capacity planning for a system built on language models instead of CPUs: size the write path deliberately, cap per-user memory, and keep heavy computation off the read path.