Inside OpenAI's Agent Loop: How Harness, API, and Inference Cut Costs

OpenAI engineers behind Codex and ChatGPT Work explained the layered architecture that turns a user request into a running agent task. A request never hits the LLM directly — it passes through a harness layer (which owns conversation history and runs the tool-execution loop), an API layer (authentication, rate limits, tokenization, safety checks), and an inference layer (the GPU fleet running the model). Heavy tasks can loop through this cycle over a hundred times, so even small per-iteration overhead compounds into real cost and latency.

Reducing that repeated work across iterations is the core optimization target. At the harness level, techniques include persistent WebSockets, stable prompt prefixes, deferred tool discovery, and Code Mode. The API layer tokenizes only the delta between turns and runs safety classifiers in parallel with inference instead of blocking on them. At the inference layer, cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode squeeze more throughput out of the same GPUs.

The stakes are concrete: frontier labs are now competing as much on cost-per-successful-task as on raw capability. GPT-5.6 Sol, for instance, reportedly outscores Fable 5 on the Artificial Analysis Coding Agent Index while costing less than half as much. For engineers building their own agentic systems, these layer-by-layer tricks offer a practical blueprint for cutting latency and inference spend without sacrificing capability.