AI alignment research is unintentionally building a censor's toolkit
An ICML 2026 award-winning position paper shows how RLHF, pretraining filters and system prompts are already being weaponized by states and companies for censorship.
A position paper by Sarah Ball and Phil Hackemann, awarded Outstanding Position Paper at ICML 2026, argues that AI alignment techniques built to prevent harmful outputs are dual-use technologies. Unlike classical misalignment concerns, the authors focus on humans deliberately weaponizing alignment pipelines — pretraining filters, RLHF datasets, and system prompts — to control information at scale. The paper maps documented cases across three alignment layers: Chinese models like DeepSeek and Ernie Bot refusing politically sensitive queries under state-mandated refusal datasets, and Grok's system-prompt edits that pushed a political agenda and triggered antisemitic outputs, showing how easily inference-time control can be misused without oversight.
The authors also describe a 'global contamination' effect, where Western models trained on Simplified Chinese text inherit self-censorship patterns from years of state filtering of the public internet — spreading misuse invisibly across the ecosystem without deliberate action by other developers. Citing fewer than ten dominant foundation model providers and democratic backsliding to 1985-era levels, the paper does not call for halting alignment research but urges verifiable alignment benchmarks, genuine model pluralism, and greater researcher transparency about misuse risks in publications. For engineers, the takeaway is that alignment methods are not neutral — their impact depends entirely on who controls them.