» Tag
benchmarking
55 posts10 AI Coding Models, 5 Tasks: Price Doesn't Predict Quality
Benchmarking 10 LLMs across 5 coding tasks reveals price and code quality barely correlate, with budget models rivaling premium ones.
VetoBench Tests Whether AI Agent Memory Retains Rejected Decisions
VetoBench is an open benchmark testing whether AI agent memory systems retain and surface previously rejected technical decisions.
Benchmarking a Markdown Knowledge Graph as AI Agent Memory
IWE tested markdown knowledge graphs as AI agent memory using the LOCOMO benchmark, reaching 96% of a hand-built ceiling with a cheap curator model.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comA Decade of Vision-Language Models: Why Easy Benchmarks Mask Real Progress
A decade-long study finds vision-language model progress is real but hidden by easy benchmarks; only spatial reasoning errors remain unsolved.
Quantprobe Runs a 110B-Parameter LLM on a 16GB RAM 2016 Desktop
Quantprobe shows how careful memory-tier placement lets a 2016 desktop with 16GB RAM run a 110B-parameter LLM, validated with pre-registered predictions.
PyTorch-style training loop teaches the LLM harness, not the model
An open-source framework trains an LLM's prompts, tools, and repair loop with a PyTorch-style loop while freezing the model itself, using git as the ledger.
Local Model Showdown Round 9: Qwen 3.6, Nemotron, Qwythos Coding Test
Five LLMs face off on a real coding task via llama.cpp on an RTX 5090 in Round 9 of the Local Model Showdown series.
apitap: Rust-Based ELT Engine Moves 10M Rows in 9.9s
Open-source Rust engine apitap moves 10M database rows in 9.9 seconds via a single function call, benchmarked against dlt and ingestr.
VetoBench Tests Whether AI Agents Forget Rejected Decisions
VetoBench is an open benchmark asking whether AI memory systems re-propose previously rejected engineering decisions, not just whether retrieval works. Results are striking.
Prefill/Decode Disaggregation Can Worsen Tail Latency, Not Fix It
Splitting prefill and decode across GPU pools adds queues and KV transfer overhead that can worsen tail latency without careful control-loop design.