Stop Editing Prompts, Build a Context Compiler Instead

Many LLM teams hand-edit the final rendered prompt string like source code — when the model misbehaves, they tweak a sentence in the debug panel and redeploy. The source argues this is the wrong abstraction: the prompt is actually a compiled artifact assembled from templates, retrieved chunks, tool schemas, history, and a token budget, not the source of truth itself. The real sources — templates, retrieval logic, ranking, and truncation/eviction rules — should be versioned, since the rendered string isn't meaningfully diffable; the same template can produce wildly different output depending on what retrieval returns.

The context window should be treated as a scarce, priced resource rather than unlimited space. Teams keep adding retrieval calls, examples, and instructions just because there's room, until the model silently degrades — missing buried instructions, anchoring on the wrong example, or ignoring the system message under attention pressure from too many chunks. This is a resource-allocation problem requiring an explicit eviction and priority policy, similar to how cache systems work, rather than ad hoc decisions made whenever a prompt gets 'too long.'

The core warning is that context-assembly bugs are indistinguishable from model failures unless specifically checked for. A retriever returning stale embeddings, a template misinterpolating an empty field, or truncation cutting mid-instruction all look like model unreliability, prompting more defensive prompt engineering layered on a broken pipeline. The real fix is build discipline: version control for templates and retrieval logic, unit tests for the assembly step itself (correct sections, order, and budget compliance), and token-level tracing of what was actually sent in production. Crucially, teams need to separate two eval questions — did the context compiler produce correct, budget-respecting input, versus did the model produce a good output from it — otherwise every bug gets misdiagnosed as a model problem and the real upstream defect never gets fixed.