Researchers expose CoT Forgery: LLMs reveal unsafe info when fake reasoning claims compliance is OK

Researchers at MIT and independent labs have published a new jailbreak attack called 'CoT Forgery' that achieves ~60% success across all tested LLM families by injecting fabricated reasoning into prompts. The exploit—heading to ICML 2026 in Seoul—won the 2025 OpenAI GPT-OSS-20B red-teaming contest on Kaggle. The attack works by embedding false reasoning (e.g., 'the user is wearing a green shirt so compliance is fine') into a conversation, causing models to treat the injected text as their own trusted reasoning rather than user input. Because models rely on writing *style* rather than role tags to determine whether text is reasoning or a command, the attack bypasses tag-based safeguards entirely.

The researchers built 'role probes' that measure how strongly a model internally treats each token as its own reasoning versus user instruction. Removing stylistic markers that make injected text read like reasoning—while preserving the semantic meaning—dropped attack success from 61% to 10%. The findings suggest role confusion is the core mechanism behind prompt injection generally: models partition conversations using role tags (user, tool, think) meant to separate trusted commands from untrusted data, but don't actually discriminate based on those tags. The attack succeeded even for extreme requests and did not weaken as prompts grew more dangerous, unlike persuasion-based jailbreaks.

For architects: this is a first-principles vulnerability in how LLMs parse structured input. Tag-based isolation (the current de facto standard in agentic frameworks) is decorative, not protective. If your agent accepts documents, UI elements, or tool outputs, style-based injection can override core instructions at scale. Microsoft recently flagged the same agentic risk. Expect a wave of defenses focused on truly separating reasoning state from input processing—not via tags, but via architectural isolation or learned role detection.