Soft-Prefix Attacks Flip LLM Reasoning at 90% on Hidden Vector Injection

Learned soft-prefix attacks can compel production large language models (LLMs) to reverse correct logical judgments at rates up to 90% on unseen task variations. These prefixes, being continuous embedding-space vectors rather than text tokens, bypass standard input filters and web application firewall (WAF) rules.

Brian K. Chen's research, published on arXiv, assessed Qwen3.6-35B-A3B MoE, Qwen3-8B, and Gemma 4 31B on a syllogistic reasoning benchmark with precisely labeled logical forms. The attack involves prepending a learned soft prefix—a trainable continuous vector injected at the embedding layer—to hijack judgment while keeping model weights frozen. As the prefix never passes through the tokenizer, it remains invisible to standard text-based input filters, regex guardrails, and WAFs.

In all 16 model–direction–split comparisons, learned prefixes outperformed paired random controls by 37 to 99 percentage points. Qwen3.6-35B-A3B MoE exhibited flip rates of 72%–90% across wording and prompt variations, while Gemma 4 31B flipped 54%–56% of validity judgments, compared to less than 1% for matched random prefixes. The prefix induces a broad semantic preference for one answer, generalizing across unseen logical forms and interface changes. Simple score models often predict which Qwen judgments will flip but not the extent of margin movement, whereas Gemma's overall response distribution is more uniformly approximated by the same predictors.

Prompt injection has held the OWASP LLM01 slot for three consecutive years, with only 34.7% of organizations reporting deployed defenses. Production attack success rates range from 50% to 84% depending on system configuration, with recent critical CVEs including EchoLeak in Microsoft 365 Copilot (CVSS 9.3) and a GitHub Copilot RCE (CVSS 9.6). On February 13, OpenAI introduced ChatGPT Lockdown Mode, acknowledging that prompt injection "may never be fully patched," indicating a structural threat. The UK NCSC concluded in December that prompt injection "may be a problem that is never fully fixed," and Bruce Schneier and Barath Raghavan noted in IEEE Spectrum that unlike SQL injection, no code/data separation exists inside the model to mitigate this class of attack.

LLMs treat every position in the context window identically, lacking an architectural boundary separating user data from system instructions. However, standard commercial APIs do not expose embedding-layer inputs to end users, so an external attacker cannot directly inject a soft prefix through a chat completion endpoint. The risk is concentrated in self-hosted open-weight deployments using prefix tuning, adapter layers, or any learned context mechanism, and in supply-chain compromises where a malicious prefix rides inside a downloaded artifact. Text-based mitigations—input filtering, dual-LLM patterns, RLHF, and constitutional AI—are designed for natural-language adversaries, not embedding-space vectors. EU AI Act Articles 53 and 55 obligations and NIST AI 600-1 already name prompt injection as a formal risk, yet none of the mandated frameworks address continuous-vector manipulation specifically.

Architects should add adversarial soft-prefix test suites to pre-production eval harnesses and measure flip rates on structured reasoning tasks, as text-level guardrails provide no coverage against embedding-level context manipulation. If an eval pipeline cannot detect when a continuous vector overrides correct logical judgments, production monitoring is auditing the wrong attack surface.