Netflix OCI Agent Cuts LLM Overestimation by 4×

Netflix shipped `oci-agent` in June 2026 and has run more than 100 causal analyses per month on it since. The open-source Python package implements an actor-critic loop for Observational Causal Inference (OCI) — estimating causality from non-experimental data — and is now available on GitHub for teams that need auditable, repeatable causal workflows.

The core architecture separates three roles: a human Principal who writes the initial analysis plan; an Actor agent that refines it into a data-analysis spec, fills a templated Jupyter notebook, and executes it; and a Critic agent that reviews the output, rates it on a three-tier scale (not_satisfactory / satisfactory_with_caveats / fully_satisfactory), and recommends spec changes. The loop runs until the Critic is satisfied or a stopping condition is hit. Every artifact — plans, specs, plots, executed notebooks — is versioned and uploaded to a file store where the Principal can verify results locally. Netflix calls this "process audits." Because causal inference from observational data has no ground truth, you cannot evaluate an agent the usual way.

The toolkit underneath `oci-agent` predates the LLM era. Netflix built it to answer questions like "what is the effect of playing a Netflix game on member retention?" using target trial emulation — for every causal question, identify the ideal A/B test and check whether observational data can credibly approximate it. The agent inherits four hard diagnostics: covariate balance (standardized mean difference after weighting under 0.2), overlap (propensity score between 0.1 and 0.9), a placebo outcome test (treatment effect on pre-treatment variables not significantly different from zero), and sensitivity analysis to hidden confounders. These checks run automatically; the Critic grades them before issuing its rating.

The case study shows what happens without this discipline. Netflix asked both a raw Claude prompt and `oci-agent` to estimate the impact of engaging with a new entertainment type (the games vertical, called "Type X") on 2-month member retention. `oci-agent` returned an estimate 25% of the baseline — the naive LLM approach overestimated the causal effect by roughly 4×. The Critic flagged early adopter bias and a failed placebo test, then triggered iterations with tweaked parameters until a defensible estimate emerged.

The open-source release ships with EconML for the underlying causal ML, covering doubly robust learning of average treatment effects for single binary treatments. Since June the team has added heterogeneous treatment effect estimation and partially linear models for multiple continuous treatments. Against the 2016 Atlantic Causal Inference Conference benchmark (77 data-generating processes, 231 estimates, 44 competing methods), the agent's statistical methodology is competitive: low RMSE and well-calibrated 95% confidence intervals relative to hand-tuned approaches.

The harder engineering problem is eval design, not agent architecture. Because causal estimates from observational data have no external oracle, Netflix's quality signal is procedural: did the agent follow the playbook, did the Critic flag the right diagnostics, can a human re-execute the notebook and get the same answer? That is a weaker guarantee than numeric accuracy on a leaderboard. Teams shipping agents for regulated or high-stakes decisions will need to build similar audit trails instead of treating LLM output as self-validating.

Architect takeaway: if your data science stack runs observational causal work today — proxy metric estimation, retention impact analysis, feature attribution — `oci-agent` gives you a working actor-critic template built on Netflix production load, with diagnostic thresholds and playbooks already defined.