Every guard model deployed to enforce GDPR, HIPAA, or SOX is failing the same test: swap out the governing rule and the verdict barely moves. A new paper from Lexsi Labs, published August 17, audited seven deployed guard systems and thirteen external benchmarks across 20 regulatory domains. The finding: deleting, permuting, or substituting the rule a detector is supposed to enforce leaves detection accuracy unchanged. The authors call this failure mode rule blindness.
The audit divides current guards into two families. Fixed-taxonomy classifiers — Llama Guard 3 and Qwen3Guard — never receive the governing rule at all. Their verdict cannot track rule changes because the rule is not part of the input. Policy-conditioned guards are the more damaging case. Latent Policy Guard (LPG) accepts a custom-policy channel and cites the governing clause in its output, yet barely changes its verdict when that clause is swapped for a permissive counterpart. The detector reads the rule; it doesn't use it. WildGuard, SIREN, GLiGuard, and an 8B zero-shot judge show the same pattern.
| Guard System | Family | Receives Rule as Input | Cites Rule in Output | Rule-Blind |
|---|---|---|---|---|
| Llama Guard 3 | Fixed-taxonomy classifier | No | No | Yes |
| Qwen3Guard | Fixed-taxonomy classifier | No | No | Yes |
| Latent Policy Guard (LPG) | Policy-conditioned | Yes (custom-policy channel) | Yes | Yes |
| WildGuard | Policy-conditioned | — | — | Yes |
| SIREN | Policy-conditioned | — | — | Yes |
| GLiGuard | Policy-conditioned | — | — | Yes |
| 8B zero-shot judge | Policy-conditioned | — | — | Yes |
Four of seven public compliance benchmarks are solvable by a policy-blind bag-of-words model. The existing evaluation infrastructure cannot distinguish a rule-conditioned detector from one ignoring the rule entirely. Teams that validated their guard stacks against these benchmarks have no signal on rule blindness. The paper introduces a crossed-rule benchmark — two rules crossed with two scenarios so neither alone predicts the label. Under that design, only step-by-step reasoning escapes rule blindness. No fast detector tested does.
A worked example clarifies the practical exposure. A GDPR storage-limitation rule requires deletion within 90 days. In a scenario where data was retained for 400 days, a deployed guard correctly flags the violation. It also flags the identical scenario when the rule is replaced with an unrelated one, or removed. The verdict tracks surface features, not the rule. This pattern holds across 20 regulatory domains: data protection, healthcare privacy, financial regulation, platform content policy.
The authors introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from 10 labelled pairs. It reads activations directly from the monitored model rather than running a separate network and recalibrates to a changed model at negligible cost. The result is honest: ICS does not meet its pre-registered criterion for beating baselines, and a bag-of-words model matches its pooled generalisation. It does raise the mechanically verified pass rate when used to rank candidate responses — but an adaptive white-box attack removes that gain.
The structural contribution is the counterfactual protocol. By publishing the crossed-rule benchmark and test suite, the authors give teams a way to falsify rule-conditioned claims in future guard releases. Any guard vendor claiming policy-conditioned detection should publish counterfactual accuracy; absent that, the claim is unverifiable.
For architects running compliance pipelines today, the fast-detector layer is not doing what the audit report says it is. Step-by-step reasoning is the only method that escapes rule blindness, but carries latency costs that rule it out for high-throughput paths. The practical near-term position: treat current guard verdicts as scenario-level heuristics, not rule-specific assurance, and run the counterfactual protocol against any guard before citing its output in a compliance filing.