RCI Framework Cuts Constraint Violations in Offline Safe RL Training

Researchers at the University of the Witwatersrand have published Redistribution-based Cost Inference (RCI), a framework that enables safe offline reinforcement learning without per-step cost annotations. The paper, released 12 August 2026, solves a critical bottleneck: standard safe RL methods require dense cost labels c(s, a) at every timestep, but production datasets—highway autonomy logs, clinical decision traces, trading histories—rarely contain them. Instead, safety monitors emit a single binary signal at the first unsafe transition. RCI converts that sparse trajectory-level feedback into dense per-step cost estimates before feeding the augmented dataset to any standard constrained offline RL solver.

RCI treats the gap as a temporal credit assignment problem. A return-decomposition module—instantiated with RUDDER, though GRD is a direct substitute—redistributes the terminal stop signal backward across the trajectory, producing a cost sequence return-equivalent to the original sparse label. The key guarantee: return-equivalent redistribution preserves both the feasible policy set and the optimal Lagrangian in the Constrained Markov Decision Process. After redistribution, any constrained offline RL algorithm trains on the enriched dataset; the paper uses BCQ-Lagrangian as default, with CPQ and CDT as drop-in replacements.

Experiments spanned highway driving and robotic manipulation across three dataset regimes: unsafe-policy trajectories, random-policy trajectories, and mixed behavior policies. RCI produced substantially lower constraint violation rates than two baselines—sparse cost training and classifier-based cost inference—while maintaining task reward comparable to an unconstrained offline baseline. The authors report robustness to label noise and heterogeneous dataset compositions, the failure modes most likely when assembling offline corpora from multiple sources. The paper does not publish a single summary violation figure; teams should reproduce domain-specific experiments before deployment.

RCI's modularity is its strength. The framework decouples annotation from policy learning, letting teams swap the redistribution backend (RUDDER vs. GRD) and downstream solver (BCQ-Lagrangian, CPQ, CDT) independently. Existing offline RL infrastructure doesn't need replacement—only a preprocessing stage inserts RCI before dataset ingestion. For teams running conservative Q-learning or constrained decision transformers, adoption is a pipeline addition.

RCI addresses why safe RL remains out of production control loops in robotics, autonomous vehicles, and clinical decision support: the annotation cost of labeling every transition. Trading systems and medical recommendation pipelines share the same constraint—human reviewers flag episodes, not steps. RCI's guarantee that redistribution is lossless under CMDP Lagrangian optimization offers a principled alternative to heuristic cost shaping.

Code is available. The key constraint: RCI's guarantees assume return-equivalent redistribution; decomposition quality depends on the RUDDER or GRD model fitting the actual causal structure of violations. Noisy or deceptively simple datasets may degrade redistribution fidelity. Run ablations on dataset composition before deploying in safety-critical loops.