LLMrPro: Open-Source LLM Balancer Pools Local Machines With Cloud Fallback

LLMrPro is a self-hosted LLM router that exposes a single OpenAI-compatible API and dispatches requests across a pool of machines you own — desktops, workstations, servers — each running a local inference engine such as LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS. When no local worker can serve a request, it transparently falls back to a cloud provider configured for that tier.

The system organizes traffic into three tiers — frontier, mini, nano — each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI) and configurable retry/backoff behavior. Desktop agents connect outbound to the balancer over websocket to pull jobs, avoiding inbound firewall rules, while a built-in admin UI handles device pairing, agent revocation, tier configuration, and real-time dispatch/health monitoring.

What sets LLMrPro apart is combining pieces found separately elsewhere: distributed-inference tools (exo, Petals), LLM gateways (LiteLLM, Bifrost), and GPU fleet managers (GPUStack, Kalavai) each solve part of the problem, but LLMrPro federates end-user desktops via a signed agent, adds per-tier cloud fallback, and keeps everything single-tenant and self-hosted behind one OpenAI-compatible surface. For engineers, that means putting existing hardware — including Apple Silicon via MLX — to work on production traffic while retaining control over keys and data, with cloud capacity as a safety net rather than the default path.