» Tag
evaluation
42 postsField Guide to LLM-as-a-Judge: AI-Powered Evaluation Process
LLM-as-a-judge is an AI method that accelerates evaluation processes. Discover our detailed guide.
I built an LLM eval framework from scratch: Lessons learned
Sharing my journey of building an LLM evaluation framework and key lessons learned.
What If the Model Knows It's Being Tested?
If a model knows it's being tested, it poses a serious challenge for AI safety. This article explores evaluation gaming and its implications.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comEvaluations Are Measuring the Wrong Thing
Most evaluations do not measure the reliability of AI models in production. Here are suggestions for accurate assessment.
TaxCalcBench: Open Source Test for AI Tax Filing
TaxCalcBench is an open source benchmark testing whether LLMs can accurately file real tax returns; even the best model only reaches 54% full accuracy.
Your LLM Judge Needs a Regression Test Suite
rubric-bench v0.1 is an open-source tool that regression-tests LLM judges against golden sets, catching grading drift in CI before it reaches production.
Evaluating General-Purpose Robot Policies for Real-World Use
Explore the challenges in evaluating robot policies and the RoboLab solution.
Harvey LAB-AA: Evaluating AI Agents on Real-World Legal Work
Harvey LAB-AA evaluates AI models on real-world legal tasks, providing critical insights for engineers.
Your AI Benchmark Might Be Measuring the Harness, Not the Model
Discover how software bugs can impact AI model evaluation. Understanding the role of the measurement system is crucial for assessing true model performance.
Your RAG Eval Is Checking the Receipt, Not the Patient
Explore the risks of entity attribution failures in clinical RAG systems and how to improve evaluations.