» Tag
evaluation
46 postsHandbook.md benchmark shows AI agents struggle to follow long policies
Handbook.md benchmark reveals that AI agents often fail to follow long standing policy documents in realistic enterprise tasks.
The 75% False-Negative Wall LLM Reviewers Can't Move
Why LLM-based reviewers hit a fixed 75% false-negative rate, and why voting, reruns, and prompt calibration all fail to move it.
LLMs Are Honest in Prose but Hallucinate Under JSON Schemas
Study finds LLMs admit uncertainty in prose but fabricate data under required JSON schemas, with 10 of 13 models hallucinating 100% of the time.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comAre AI Labs Gaming the Pelican-on-a-Bicycle Benchmark?
A statistical study generates 1,008 SVGs across 7 LLMs to test whether AI labs are secretly optimizing for the famous pelican-on-a-bicycle benchmark.
Testing AI Agents: The Bug Hides in the Answer, Not the Trace
An AI agent passed eight straight safety tests, then silently broke one - a case study in why traces, not answers, reveal agent failures.
SDABench: A New Benchmark Testing LLMs on Scientific Discovery
SDABench evaluates LLMs on six scientific capabilities beyond code execution, exposing major gaps in assumption selection and mechanistic reasoning.
How Airbnb Cut LLM Evaluation Time From Weeks to a Day
Airbnb's four-layer LLM evaluation stack uses deterministic caching and micro LoRA adapters to cut iteration time from weeks to a single day.
An Open-Source, Assessment-First Course: doerkit and rubric-bench
doerkit and rubric-bench: an open-source course using an LLM as rubric judge, plus a reusable regression-testing framework for any LLM judge.
LLM-as-a-Verifier Turns Verification Into a New Scaling Axis
New research scales LLM verification without extra training, introducing continuous scoring that hits state-of-the-art on SWE-Bench, Terminal-Bench and more.
Lessons from a Week Evaluating an AI PR Reviewer
Lessons from evaluating an AI PR review plugin: fixing the wrong skill, why risk classification matters, and how upstream evidence quality shapes review accuracy.