» Tag
evaluation
46 postsWaymo's AI Projects: Ready Only When Evaluations Are Complete
Waymo enhances safety in AI projects through continuous evaluations, providing a model for other industries.
Field Guide to LLM-as-a-Judge: AI-Powered Evaluation Process
LLM-as-a-judge is an AI method that accelerates evaluation processes. Discover our detailed guide.
I built an LLM eval framework from scratch: Lessons learned
Sharing my journey of building an LLM evaluation framework and key lessons learned.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comWhat If the Model Knows It's Being Tested?
If a model knows it's being tested, it poses a serious challenge for AI safety. This article explores evaluation gaming and its implications.
Evaluations Are Measuring the Wrong Thing
Most evaluations do not measure the reliability of AI models in production. Here are suggestions for accurate assessment.
TaxCalcBench: Open Source Test for AI Tax Filing
TaxCalcBench is an open source benchmark testing whether LLMs can accurately file real tax returns; even the best model only reaches 54% full accuracy.
Your LLM Judge Needs a Regression Test Suite
rubric-bench v0.1 is an open-source tool that regression-tests LLM judges against golden sets, catching grading drift in CI before it reaches production.
Evaluating General-Purpose Robot Policies for Real-World Use
Explore the challenges in evaluating robot policies and the RoboLab solution.
Harvey LAB-AA: Evaluating AI Agents on Real-World Legal Work
Harvey LAB-AA evaluates AI models on real-world legal tasks, providing critical insights for engineers.
Your AI Benchmark Might Be Measuring the Harness, Not the Model
Discover how software bugs can impact AI model evaluation. Understanding the role of the measurement system is crucial for assessing true model performance.