» Tag

load-balancing

3 posts

Jul 21, 20268.01
LLMrPro: Open-Source LLM Balancer Pools Local Machines With Cloud Fallback
LLMrPro is an open-source, self-hosted LLM balancer that unifies your own machines and cloud providers behind one OpenAI-compatible API.
cloud-fallback inference llm load-balancing
Aug 17, 20267.30
Solving Moe Load Imbalance in LLM Training via Optimal Transport
TAOT method improves MoE training speed by 43% while reducing communication costs by 74%.
llm load-balancing machine-learning moe
Jul 11, 20267.30
Designing a Rate Limiter: Token Bucket vs Sliding Window
Rate limiter design is crucial for load balancing and fair use across API servers.
algorithms api load-balancing rate-limiter
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com