Engineering notes: making an AI system tell the truth
How NicheIQ rebuilt its AI idea-evaluation pipeline to fix self-scoring bias, failed self-refinement loops, and reach an honest No-Go verdict.
When NicheIQ's idea generator scored every one of its own outputs at roughly the same inflated 0.88, the team confronted a familiar failure mode: a model grading its own homework always gives itself an A. Fixing it meant restructuring how the whole pipeline generates and judges ideas.
The rebuild split one large generation prompt into small per-cell tournaments, then added an independent, blind critic model to re-score outputs against fixed rubrics. A 61-idea benchmark checked against a neutral reference judge found the critic itself was still +0.13 too generous on market fit — a bias pattern consistent with published findings on LLM self-enhancement and overconfidence.
A self-refinement loop meant to improve ideas through iterative critique failed four times before working, because without external verification the model rewarded its own fabricated claims, including nonexistent APIs. The fifth version, which reframed the reviewer as a mentor constrained from expanding scope, finally improved scores — but the effect held for only one specific model; a comparable model produced the opposite result under identical prompts.
The clearest validation came from a real test case: home-baking businesses under cottage food laws. The pipeline generated five ideas, the best scoring 0.46, and returned a verdict of No-Go. For engineers building evaluation systems, the lesson is that the most valuable output an AI system can produce is sometimes an honest refusal.