Audit Finds Most Distributional RL Risk Claims Are False
A new audit framework shows most risk claims from distributional RL agents like QR-DQN and C51 are training artifacts, not genuine environment risk signals.
Distributional reinforcement learning agents such as QR-DQN, C51, and IQN learn full return distributions that are increasingly trusted for interpretability, risk-sensitive control, and safety monitoring. A new study asks whether these learned risk signals are actually true, and audits them directly against ground truth.
Using a decision-relevant screening metric (the excess Wasserstein gap between top actions, which captures violations of stochastic dominance), snapshot-restart Monte Carlo ground truth, and a rigorous statistical harness (permutation nulls, bootstrap refutation, FDR control), the researchers tested 33 runs across MinAtar. Between 40% and 95% of the strongest claimed risk trade-offs were refuted at 95% confidence, and the placement of top claims was statistically indistinguishable from random. The learned 'risk' signal turned out to be a structural training artifact—forming early, uncorrelated with final performance, and idiosyncratic per seed—rather than a genuine reflection of environment stochasticity. The same pattern held at full Atari scale, where every top risk claim from a near-state-of-the-art pretrained QR-DQN on Breakout was refuted.
Positive controls of known magnitude confirmed 96-100% of genuine claims, validating the audit methodology itself. Yet acting on the risk heads' CVaR advice ranged from mildly beneficial to significantly worse than chance. Neither risk-aware training nor ensembling removed the artifact, and recalibration only passed the audit by nullifying the claims altogether—showing the risk head is fundamentally uninformative, not simply miscalibrated.