» Tag
evaluation
42 postsTutorMoments: Testing If AI Tutors Know When to Help or Hold Back
Allen AI's TutorMoments benchmark tests whether LLM tutors know when to scaffold and when to push students toward harder reasoning.
Engineering notes: making an AI system tell the truth
How NicheIQ rebuilt its AI idea-evaluation pipeline to fix self-scoring bias, failed self-refinement loops, and reach an honest No-Go verdict.
New Test Measures 'Reward-Seeking' Behavior in AI Models
Apollo Research and OpenAI unveil Contrastive SDF, a method measuring whether AI models shift behavior based on beliefs about grader preferences.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comWANDR Benchmark Tests AI Agents on Wide-and-Deep Research Tasks
WANDR benchmark evaluates AI research agents on wide-and-deep data collection tasks using reference-free, evidence-verified grading across 500 tasks.
How a shared JSONL file inflated our eval score to 0.94
A retrieval-based few-shot selector shared its index with the eval set, inflating accuracy from 0.79 to a fake 0.94. Here's the contamination check that catches it.
Why your eval set is too small to catch real regressions
Small eval sets miss most regressions. Learn how statistical power, Wilson intervals, and binary criteria fix eval-driven development.
How I Built an MLP Evaluator for a Novel Board Game
Learn about the MLP evaluator developed for the Larss game and its outcomes.
LLM-as-Judge: Use Binary Checks for Evidence, Not Aggregate Scores
Explore the benefits of using binary checks and a single verdict in LLM evaluations. A reliable quality assessment method for engineers.
RAG Evaluation: Faithfulness, Context Recall, and Answer Relevance
RAGAs evaluation enhances AI systems' reliability by utilizing three key metrics: faithfulness, context recall, and answer relevance.
Waymo's AI Projects: Ready Only When Evaluations Are Complete
Waymo enhances safety in AI projects through continuous evaluations, providing a model for other industries.