» Tag
mlops
18 postsIf 30% of SWE-Bench Pro Tasks Are Broken, Add an Uncertainty Budget
OpenAI's SWE-Bench Pro audit found ~30% of tasks broken. Learn how to version task validity and report uncertainty intervals in benchmarks.
We gated CI on six LLM eval frameworks — only two survived
An eight-month CI test of six LLM eval frameworks found only Promptfoo and DeepEval reliable as merge-queue gates, thanks to deterministic checks.
Evaluation Debt: Why Offline Agent Evals Fail in Production
Why offline eval frameworks miss real production failures in AI agents, and why session-based observability is replacing static test suites.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comFixing False Positives in Prometheus Predictive Alerting
Why Prometheus-based predictive alerting models that ace offline tests turn noisy in production, and how query alignment, rate() handling and schema checks fix it.
NVIDIA ModelExpress: P2P RDMA Cuts Model Startup From Minutes to Seconds
NVIDIA ModelExpress speeds up LLM weight loading with P2P GPU-to-GPU RDMA transfers, cutting model startup time from minutes to seconds.
How Netflix Runs Its Own LLM Serving Stack with vLLM and Triton
Netflix engineers explain how they built an in-house LLM serving stack using vLLM, Triton, and an OpenAI-compatible API, with real production lessons.
How Airbnb Cut LLM Evaluation Time From Weeks to a Day
Airbnb's four-layer LLM evaluation stack uses deterministic caching and micro LoRA adapters to cut iteration time from weeks to a single day.
Text Diffusion vs Autoregressive Models: The Real GPU Cost Math
A grounded comparison of text diffusion and autoregressive LLMs on GPU utilization, latency, batching economics, and real-world enterprise costs.
nebius-actions: Fine-Tune and Deploy LLMs via GitHub Actions on Nebius
nebius-actions is a GitHub Actions toolkit for fine-tuning and deploying LLMs end-to-end on Nebius GPU infrastructure using Axolotl and vLLM.
PromptLedger v0.7 adds CI regression gates for prompt evaluation
PromptLedger v0.7 adds evaluation runs, metric comparisons, and policy-based regression gates for CI-safe prompt releases.