» Tag
mlops
20 postsIf 30% of SWE-Bench Pro Tasks Are Broken, Add an Uncertainty Budget
OpenAI's SWE-Bench Pro audit found ~30% of tasks broken. Learn how to version task validity and report uncertainty intervals in benchmarks.
We gated CI on six LLM eval frameworks — only two survived
An eight-month CI test of six LLM eval frameworks found only Promptfoo and DeepEval reliable as merge-queue gates, thanks to deterministic checks.
Evaluation Debt: Why Offline Agent Evals Fail in Production
Why offline eval frameworks miss real production failures in AI agents, and why session-based observability is replacing static test suites.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comFixing False Positives in Prometheus Predictive Alerting
Why Prometheus-based predictive alerting models that ace offline tests turn noisy in production, and how query alignment, rate() handling and schema checks fix it.
NVIDIA ModelExpress: P2P RDMA Cuts Model Startup From Minutes to Seconds
NVIDIA ModelExpress speeds up LLM weight loading with P2P GPU-to-GPU RDMA transfers, cutting model startup time from minutes to seconds.
How Netflix Runs Its Own LLM Serving Stack with vLLM and Triton
Netflix engineers explain how they built an in-house LLM serving stack using vLLM, Triton, and an OpenAI-compatible API, with real production lessons.
How Airbnb Cut LLM Evaluation Time From Weeks to a Day
Airbnb's four-layer LLM evaluation stack uses deterministic caching and micro LoRA adapters to cut iteration time from weeks to a single day.
A 24-Test Readiness Checklist for Deploying AI Agents Safely
A 24-test, six-gate framework for verifying AI agents are safe for production, covering identity, tool safety, isolation, and observability.
Text Diffusion vs Autoregressive Models: The Real GPU Cost Math
A grounded comparison of text diffusion and autoregressive LLMs on GPU utilization, latency, batching economics, and real-world enterprise costs.
SIMURG: Real-Time Guard Stops LLM Decoding Corruption Mid-Stream
SIMURG is a numpy-only real-time monitor that detects and aborts LLM decoding corruption mid-stream, with zero training and conformal calibration.