Audit Finds Community LLM Fine-Tunes Often Perform Worse, Not Better

A contamination-controlled audit set out to answer a question nobody had systematically checked: do community fine-tunes published on HuggingFace actually improve on their base models, or do they just score higher because benchmark questions leaked into training data ("benchmaxxing")? The study ran 150 matched base-model/fine-tune pairs across 164 unique models on three task families — grade-school math (GSM8K), general knowledge (MMLU), and instruction following (IFEval) — testing each item both as the standard published benchmark question and as a freshly generated, never-published equivalent testing the same skill.

The results were striking: fine-tune accuracy on the fresh, unseen items dropped significantly across all three task families (GSM8K -6.9pp, MMLU -6.0pp, IFEval -6.5pp). On average, community fine-tuning didn't improve the underlying skill — it made models worse at it. A second question, whether the contamination gap itself grows after fine-tuning, failed the study's own pre-registered validity check for MMLU and IFEval and isn't reported as a finding; only the GSM8K result passed that check, and it wasn't statistically significant.

The methodology is notably rigorous: a pre-registered analysis plan, fully deterministic grading with no LLM-as-judge, paired bootstrap and McNemar tests with Benjamini-Hochberg correction, and a label-shuffle null check the effect had to survive. Across 128,857 collected responses and ten real production bugs fixed along the way, 429 usable matched pairs made it into the final analysis. For engineers, the takeaway is concrete: a fine-tune's leaderboard score is not reliable evidence it actually learned the skill better than its base model.