» Tag
benchmarks
32 postsExploitGym Benchmark Tests If AI Agents Can Build Real Exploits
ExploitGym benchmarks whether AI agents can convert 869 real-world security bugs into working, code-execution-achieving exploits.
Handbook.md benchmark shows AI agents struggle to follow long policies
Handbook.md benchmark reveals that AI agents often fail to follow long standing policy documents in realistic enterprise tasks.
If 30% of SWE-Bench Pro Tasks Are Broken, Add an Uncertainty Budget
OpenAI's SWE-Bench Pro audit found ~30% of tasks broken. Learn how to version task validity and report uncertainty intervals in benchmarks.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comThinking Machines Lab Ships Inkling: 975B-Param Open MoE, Tunable Reasoning
Thinking Machines Lab's 975B-param, 41B-active Inkling model offers encoder-free multimodality and a tunable reasoning-effort control.
Kimi K3 benchmark: Chinese AI is cheap at coding, not planning
Benchmark testing shows Chinese AI models are pricey planners but 19x cheaper coders than Western rivals, based on real pass/fail test runs.
SirixDB: A bitemporal JSON database with sub-page versioning
SirixDB is a bitemporal JSON database using copy-on-write and sub-page versioning so any revision can be queried as fast as the latest.
FOMO-Driven AI Coding: Speed Illusion, Hidden Review Debt
GitHub's 55% speedup versus METR's 19% slowdown finding reveals how FOMO-driven AI coding adoption hides the real cost of review and verification in mature codebases.
GPT-5.6 vs Claude Fable 5: Benchmarking Physical AI Simulation
JuliaHub compares GPT-5.6 and Claude Fable 5 in its Dyad agent on five physical modeling problems, revealing cost, speed, and validation trade-offs.
NVIDIA's Nemotron 3 Super Beats GPT-OSS-120B on Coding Benchmarks
NVIDIA's open Nemotron 3 Super model beats GPT-OSS-120B by 20 points on SWE-Bench and delivers 2.2x faster inference throughput.
"Hallucination" Isn't One Bug. It's Three, and Only One Is Fixable
Hallucination isn't one failure mode — it's three. A test to tell them apart, why benchmarks reward bluffing, and what a 2026 prediction experiment showed.