Study: LLMs have a fundamental role-recognition flaw hackers can exploit
Researchers show LLMs identify roles by text style, not tags, revealing a fundamental flaw that may make full LLM security unattainable.
A paper presented at ICML argues that large language models have a structural weakness in how they determine who or what is giving them instructions. Researchers Charles Ye and Jasmine Cui found that LLMs infer a text segment's role — system, user, assistant, chain-of-thought, or tool — not from the tags surrounding it but from the style and content of the text itself. Swapping tags made almost no difference to model behavior.
This insight enabled a technique the researchers call 'chain-of-thought forgery,' where attackers spoof a model's internal reasoning notes to make it treat a malicious instruction as its own conclusion. Using this method, they extracted restricted content — including drug synthesis and aircraft sabotage instructions — from OpenAI's gpt-oss-20b and GPT-5, and report similar success against models from Anthropic, Alibaba, and DeepSeek. The attack won OpenAI's August 2025 red-teaming hackathon.
The deeper problem, the researchers argue, is that safety training amounts to an incomplete blocklist that can never anticipate every attack. ETH Zürich's Florian Tramèr calls the finding compelling, though he notes layered defenses have made leading models harder to exploit — just not immune. The takeaway for engineers: role-based guardrails, as currently implemented, cannot be fully hardened through training alone.