» Tag
llm-evaluation
10 postsIf 30% of SWE-Bench Pro Tasks Are Broken, Add an Uncertainty Budget
OpenAI's SWE-Bench Pro audit found ~30% of tasks broken. Learn how to version task validity and report uncertainty intervals in benchmarks.
We gated CI on six LLM eval frameworks — only two survived
An eight-month CI test of six LLM eval frameworks found only Promptfoo and DeepEval reliable as merge-queue gates, thanks to deterministic checks.
Evaluation Debt: Why Offline Agent Evals Fail in Production
Why offline eval frameworks miss real production failures in AI agents, and why session-based observability is replacing static test suites.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comAudit Finds LLM Agent Credit Signals No Better Than Random Chance
A causal audit in ALFWorld shows judge- and confidence-based credit signals for training LLM agents perform no better than chance at identifying key steps.
LLM Judges Verify Presence, Not Absence: Blind Spot in AI Clinical Notes
New research shows LLM judges struggle to detect omissions in AI-generated clinical notes; a restructured fact-checking method significantly improves detection.
SciCode-Verified: Flawed Benchmark Masked True LLM Coding Skill
Fixing 263 defects in the SciCode benchmark lifted LLM scientific-coding accuracy from 60% to 98%, revealing the test—not the models—was flawed.
data-eng-bench: Snowflake's dbt benchmark for coding agents
Snowflake's data-eng-bench tests coding agents on 103 realistic dbt data-engineering tasks across DuckDB and Snowflake, via the open-source Harbor framework.
Generating the database from the answer key in text-to-SQL benchmarks
A UIUC audit found over half of BIRD and Spider 2.0 annotations wrong. One developer inverted the process: declare the answer first, then generate a database that satisfies it.
OpenAI Retracts Its Own SWE-Bench Pro Recommendation
OpenAI's audit found roughly 30% of SWE-Bench Pro's tasks are flawed, prompting the company to retract its earlier endorsement of the agentic coding benchmark.
Why LLM-as-judge scores flip between identical runs
An LLM judge gate flipped between 0.79 and 0.82 on identical inputs. Sampling temperature, model drift and vague rubrics cause the jitter; k-sampling and a noise band fix it.