» Tag
ai-evaluation
7 postsEvery Frontier AI Model Tested Attempted to Cheat, AISI Finds
AISI finds every tested frontier AI model attempted to cheat in cyber evaluations; self-report and chain-of-thought monitoring proved unreliable.
A Decade of Vision-Language Models: Why Easy Benchmarks Mask Real Progress
A decade-long study finds vision-language model progress is real but hidden by easy benchmarks; only spatial reasoning errors remain unsolved.
SciCode-Verified: Flawed Benchmark Masked True LLM Coding Skill
Fixing 263 defects in the SciCode benchmark lifted LLM scientific-coding accuracy from 60% to 98%, revealing the test—not the models—was flawed.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comSynthDocBench Exposes Long-Context Weaknesses in Vision Language Models
New synthetic benchmark SynthDocBench reveals systematic VLM failures in long-context document understanding, including positional bias and chart errors.
Testing LLMs Like Software: A Promptfoo Deep Dive for QA
A production-level guide to testing LLM systems with Promptfoo: versioned evals, deterministic checks, judge-model bias, and provider benchmarking.
Self-Play LLM Judges Reward Persuasion, Not Correctness
Self-rewarding LLM judges score plausibility, not correctness; GSM8K experiments show judge approval climbing while true accuracy stays flat, revealing reward hacking.
Why Your LLM-as-Judge Gate Flips Between Runs
Same code, same input, different LLM judge score: why CI gates flap between runs, and how temperature, model pinning, k-sampling and quantization fix it.