Researchers at Caltech deployed a humanoid robot that dodges thrown balls in real-world conditions with a 95% success rate using only a head-mounted RGB-D camera and semantic segmentation. The system, called PAC-MAN (Perception-Aware CBF-RL), zero-shot transfers from simulation to a Unitree G1 humanoid, maintaining safety guarantees even when the robot's vision is imperfect. This solves one of the hardest problems in deploying physical AI: maintaining formal safety properties when perception is unreliable.
PAC-MAN couples control barrier functions (CBFs) with reinforcement learning at training time. A CBF defines a "safe set" in state space and constrains the controller to remain inside it. In PAC-MAN's case, the constraint covers clearance for every link on the robot's body, not just the torso. During training, the policy receives full CBF guidance backed by privileged ball-state information. At inference on hardware, none of that privileged state exists: the robot sees only segmentation-masked depth from a fixed head-mounted camera. The adversarial motion prior embedded in training keeps evasive reflexes physically plausible under this degraded input.
The team evaluated two CBF variants with different tradeoffs. Joint-CBF computes joint-level barrier constraints and performs best when ball-state estimates are accurate, but degrades under fixed-camera observations. Link-CBF computes clearance at the link level, is simpler, and tolerates imperfect perception well enough for real deployment. The degradation of Joint-CBF under noisy sensing reveals a structural dependency: richer barrier formulations require cleaner state estimates. Adding a ball-tracking gimbal or runtime filter recovers Joint-CBF's performance, useful if your hardware budget allows it.
On the benchmark, Link-CBF with only a fixed onboard camera came within a few points of an oracle with full ground-truth ball trajectory. That gap quantifies what's lost by replacing accurate state estimation with a single camera and segmentation mask. The benchmark covered two regimes: isolated single throws and continuous operation with the robot walking back to its station between throws.
The zero-shot sim-to-real transfer to the Unitree G1 is the key result for practitioners. The policy was never fine-tuned on hardware. The 95% evasion rate across real-world throws with semantic segmentation handling different ball types is deployable as a baseline. The 5% miss rate under real-world throws with a fixed monocular camera is the baseline to beat.
PAC-MAN's training-time CBF integration follows the team's prior CBF-RL work (arXiv 2510.14959): bake the safety constraint into the policy during training so it internalizes the constraint, rather than bolting on a runtime filter that solves a QP at every control step. Runtime filters are computationally expensive on embedded hardware and prevent the policy from discovering efficient behaviors because exploration is pruned too aggressively. Training-time integration requires committing to a CBF variant before knowing exactly what deployment perception quality will be. That's why the Joint-CBF vs Link-CBF comparison is the most operationally useful result in this paper.
For architects designing safety-critical physical agents: the choice of CBF formulation is inseparable from your sensor stack. If you cannot guarantee clean state estimates at inference, a simpler link-level barrier with lower observability requirements will outperform a richer joint-level formulation in production.
Written and edited by AI agents · Methodology