» Tag
benchmarking
55 postsNew Verifier Exposes Hidden Bugs in LLM-Generated GPU Kernels
A 12-gate contract verifier finds that 39.5% of LLM-generated GPU kernels are broken despite passing standard correctness benchmarks.
MCP Sessions Moved Into the Payload: Sticky, Redis, Continuations Compared
MCP dropped server sessions for MRTR; a Rust rmcp v3 benchmark measures sticky, Redis, and sealed-continuation state strategies.
DuckDB vs SQLite: 100x more rows on the same $16.49 server
On the same $16.49/month server, DuckDB writes 4x-15x faster than SQLite and serves dashboards at 100x the row count. Full Traceway benchmark results.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comHow do you actually test if an AI understands your codebase
A hands-on benchmark for testing whether an AI agent truly understands codebase structure, tested across thirteen real Ruby projects.
Rethinking MCP Security: A Large-Scale Study of 64K Runtime Servers
MCPZoo tests 64,611 MCP servers, revealing that most security scanner alerts are false positives and scanners disagree widely.
nibble: Diagnose and Fix Redis Memory Waste, No Install Required
nibble diagnoses and fixes Redis memory waste with zero install, claiming an 8.6x memory reduction backed by transparent, documented benchmarks.
Toolgz cuts LLM tool-definition tokens by ~80% without accuracy loss
Toolgz reduces LLM tool-definition token usage by ~80% while preserving accuracy, validated across 420 runs on four frontier models.
Are AI Labs Gaming the Pelican-on-a-Bicycle Benchmark?
A statistical study generates 1,008 SVGs across 7 LLMs to test whether AI labs are secretly optimizing for the famous pelican-on-a-bicycle benchmark.
Fable 5 vs GPT-5.6 Sol on an NP-Hard Problem: Does /goal Help?
Claude Fable 5 and GPT-5.6 Sol tackle an NP-hard fiber-network problem, testing whether the /goal persistence feature actually improves results.
How a Benchmark Turned an MCP Security Proxy From 9% to 63%
An open benchmark shows how the mcp-bastion MCP security proxy grew from 9% to 63% attack-surface coverage through iterative testing.