» Tag

model-serving

2 posts

Jul 28, 20268.41
NVIDIA ModelExpress: P2P RDMA Cuts Model Startup From Minutes to Seconds
NVIDIA ModelExpress speeds up LLM weight loading with P2P GPU-to-GPU RDMA transfers, cutting model startup time from minutes to seconds.
gpu inference mlops model-serving
Jul 26, 20268.10
Randomized KV-Cache Eviction Enables Verifiable Error Certificates
A new paper shows randomized KV-cache eviction, unlike deterministic top-k, enables verifiable per-step error certificates for LLM serving.
efficient-inference kv-cache llm-inference model-serving
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com