A Fair Coin Isn't Enough: Why a Randomized Experiment Can Become Unanalyzable

An engineering team built a randomized experiment to justify a causal claim about a system decision, routing tasks to one of two arms via a coin flip. The randomizer itself was cryptographically sound and statistically fair — that part worked flawlessly. But a later audit revealed the experiment was unanalyzable from the start: the dispatch function never recorded which task each draw applied to, so there was no join key linking assignments to outcomes. Without that key, the core intention-to-treat comparison could never be computed, no matter how clean the coin was.

The audit also found three compounding failures: no safeguard against re-drawing assignments for the same unit, test draws mixed into the production log with only a documentation note (not a data flag) separating them, and a pre-registered stopping rule that depended on a mismatch rate which could never actually be measured. Even if all of this were fixed, the team discovered the experiment had never been power-calculated — projections showed it would land at under half the target sample size, guaranteeing an inconclusive result regardless of data hygiene.

The broader lesson for engineers is that randomization itself is the easy, library-provided part; making it usable requires deliberate, unglamorous instrumentation — stamping a stable unit ID at assignment time, propagating it to the outcome record, guarding against silent re-randomization, flagging test data explicitly, and computing a minimum detectable effect before collecting any data. Skip any one of these and even a provably fair coin produces an experiment that cannot be analyzed.