When Green Tests Lie: Ten Patterns Behind False Positives
Ten real examples from a single audit day show how passing tests and checks can mask broken software, and how to actually test the checkers themselves.
In one working day on a live site, a page gave away a paid product for free, a contact form silently dropped every submission, a sitemap rendered blank, and a text metric labeled 'coherence' was mostly measuring sentence count. Every one of these had already passed a check before it broke. This piece catalogs ten distinct ways an automated or manual check can stay green while the thing it verifies is actually broken: scripts whose failure flag can never be set, checkers speaking a different syntax dialect than the system they inspect, audits that look at source code instead of the rendered output users actually see, metrics that are reliable but validate the wrong property, and error-handling code that swallows failures into silence.
The recurring lesson is that checking again only re-runs the same flawed instrument, while checking differently — feeding a known-good case to a search, testing at the layer where consequences actually occur, applying a transformation a metric should be indifferent to, or using data that predates the hypothesis — actually tests the instrument itself. A related failure mode shows documentation quietly drifting from the underlying data it once cited correctly, since no test ever throws when a sentence, not the code, becomes false.
For engineers, the practical takeaway is concrete: a check that has never failed is unproven, not trustworthy. Deliberately breaking the system once to confirm the check goes red is one of the cheapest and most neglected forms of testing in production environments.