An Open-Source, Assessment-First Course: doerkit and rubric-bench
doerkit and rubric-bench: an open-source course using an LLM as rubric judge, plus a reusable regression-testing framework for any LLM judge.
After eight posts arguing that AI education is building the wrong product, there's now a working system instead of just a claim: doerkit, an open-source statistics course where an LLM never chats but grades constructed responses against a rubric as booleans, while code computes the rest. Its sibling repo, rubric-bench, provides regression testing, adversarial cases, and tone metrics for any LLM judge — a pattern useful well beyond education.
The project's findings are notable: resistance to prompt injection varies by model, grading severity and warmth are separable knobs (so a cold grader is a defect, not rigor), and the largest effect size in the underlying study came from the unglamorous cumulative spaced-review feature — not the AI grader or the chatbot everyone demos.
The author is explicit about what's missing for real deployment: LMS integration, auth and multi-tenancy, a FERPA data agreement, human-rater validation, and a randomized controlled trial. The current effect size rests on one observational pilot and could be inflated by selection (bracketed between 0.71 and 1.30 SD). The real takeaway for engineers: the reusable artifact isn't the education product, it's the eval-suite pattern — golden sets, drift diffs, and adversarial checks — that every team shipping an LLM judge into production needs and mostly doesn't have yet.