» Tag
benchmarks
32 postsOpen benchmark makes AI models' political bias measurable
The Neutrality Project launches an open, reproducible benchmark measuring political bias across six axes in leading AI language models.
ScarfBench Benchmarks AI Agents on Enterprise Java Framework Migration
ScarfBench is an open benchmark measuring whether AI agents can truly build, deploy and preserve behavior when migrating enterprise Java apps.
The Sustained Performance Gap: Why Laptop Boost Clocks Don't Hold
Why laptop boost clocks fade under sustained load: thermal throttling, PL1/PL2 power limits, and GPU TGP tricks explained for engineers and buyers.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com600 AI Architectures: What LLMs Default to When Designing Systems
Six LLM families produced 600 system architectures from identical briefs, exposing default tech choices, low consensus in key layers, and constraint-driven shifts.
SynthDocBench Exposes Long-Context Weaknesses in Vision Language Models
New synthetic benchmark SynthDocBench reveals systematic VLM failures in long-context document understanding, including positional bias and chart errors.
LLM-as-a-Verifier Turns Verification Into a New Scaling Axis
New research scales LLM verification without extra training, introducing continuous scoring that hits state-of-the-art on SWE-Bench, Terminal-Bench and more.
AI Agent Faked a Test Log, Then Trusted It: The Provenance Gap
Lilian Weng's new survey on self-optimizing agent harnesses shows fake test logs and how provenance vanishes when trajectories get compressed into summaries.
TutorMoments: Testing If AI Tutors Know When to Help or Hold Back
Allen AI's TutorMoments benchmark tests whether LLM tutors know when to scaffold and when to push students toward harder reasoning.
Generating the database from the answer key in text-to-SQL benchmarks
A UIUC audit found over half of BIRD and Spider 2.0 annotations wrong. One developer inverted the process: declare the answer first, then generate a database that satisfies it.
Distilling DeepSeek into GPT-OSS Doesn't Transfer Its Censorship
Research shows distilling DeepSeek into GPT-OSS-120B boosts financial reasoning without inheriting the teacher model's political censorship.