Relay: A Self-Hosted LLM Gateway With Eval-Gated Smart Routing

Relay is a self-hosted LLM gateway and model router shipped as a single static Go binary. It accepts both OpenAI and Anthropic API dialects inbound and routes outbound to any provider using your own keys — OpenAI, Anthropic, Gemini, Groq, DeepSeek, Mistral, xAI, or a local Ollama instance — with routing policies ranging from static aliases to a learned smart tier.

What sets it apart is its built-in eval harness, which gates the smart routing feature rather than promoting it by default. On a held-out, live-judged 49-prompt benchmark, a plain 'cheapest model' policy matched or beat an always-frontier baseline at roughly 98% lower cost, while a lexical-based smart routing tier failed to clear its own quality bar — so smart routing ships off by default until it demonstrably earns its keep on a team's own traffic.

For engineers, the practical selling points are zero telemetry (not even opt-in pings), an Apache-2.0 license, and measured gateway overhead far under its own CI-enforced 5ms budget (about 0.18ms observed). Prompts never transit a third party unless explicitly opted into remote embeddings, positioning Relay as a privacy- and latency-conscious alternative to hosted aggregators like OpenRouter or proxy frameworks like LiteLLM.