» Tag
llm
535 postsAn Open-Source, Assessment-First Course: doerkit and rubric-bench
doerkit and rubric-bench: an open-source course using an LLM as rubric judge, plus a reusable regression-testing framework for any LLM judge.
When Agent Tools Return Empty: Who Decides What It Means
Why empty results from agent tool calls hide three distinct failure states, and how typed states, evidence, and retry semantics fix the gap.
Testing LLMs Like Software: A Promptfoo Deep Dive for QA
A production-level guide to testing LLM systems with Promptfoo: versioned evals, deterministic checks, judge-model bias, and provider benchmarking.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comMemory Scarcity and Open Models Are Reshaping the AI Industry
A new paper models how the DRAM/HBM price crisis, open-weight models, and inference efficiency gains will reshape the AI industry through 2026-2030.
LLM-as-a-Verifier Turns Verification Into a New Scaling Axis
New research scales LLM verification without extra training, introducing continuous scoring that hits state-of-the-art on SWE-Bench, Terminal-Bench and more.
Probelock: A Lockfile for LLM Tool-Calling Capabilities
Probelock measures an LLM's tool-calling capabilities with deterministic tests and halts CI when a model, quantization, or runtime swap causes a regression.
HalluSquatting: How AI Coding Agents Turn Into a Botnet
AI coding tools like Cursor, Copilot, and Gemini CLI can hallucinate package names that attackers pre-register with malware, turning normal agent use into silent compromise.
AI Agent Memory Is Just a Smarter Filing Cabinet
Today's AI agent 'memory' is retrieval in disguise—it can't tell if past answers were right. Real memory needs judgment that learns from outcomes.
VetoBench Tests Whether AI Agents Forget Rejected Decisions
VetoBench is an open benchmark asking whether AI memory systems re-propose previously rejected engineering decisions, not just whether retrieval works. Results are striking.
Stop Editing Prompts, Build a Context Compiler Instead
Instead of hand-editing prompt strings, treat context assembly as a compiler: versioned templates, testable rules, and full traces of what shipped.