LangChain's Deployed Engineer Eric Johanson published a full build walkthrough this week for an autonomous SRE agent running against a live Kubernetes cluster. This is not a toy demo — it's the internal system LangChain runs its own self-hosted deployment on. The post tackles the tradeoffs that block agent economics: model routing, blast-radius control, and scheduler-level cost.
The agent operates in two modes. A scheduler fires every N minutes and collects cluster state through the Kubernetes Python client — zero LLM tokens — then makes a single Claude Haiku call with forced tool-use to produce a structured health report dropped into Slack, sorted by severity. On-demand investigation fans out in parallel to six specialist subagents: pod-inspector, scaling-analyzer, performance-analyzer, log-analyzer, security-auditor, reliability-auditor. Each reads the cluster independently before the orchestrator synthesizes one prioritized report. The fan-out runs on LangChain's `create_deep_agent()` primitive, which provides a planning loop, first-class subagent management, and built-in Human-in-the-Loop interrupts.
The scheduler redesign cuts cost. The earlier architecture ran the full orchestrator — approximately 20 model calls — every cycle, even when everything was healthy. Moving to plain Python state collection plus one Haiku call cut per-check cost by 95–99% with no degradation in issue detection. Full agent power fires only on on-demand investigation: the right inversion.
Model routing follows the same logic. Claude Sonnet handles the synthesizing orchestrator. Claude Haiku handles read-only subagents and scheduled checks. Write tools are separated structurally, not by policy. Read and write modules are distinct codebases, and write tools exist only inside a single `change-executor` subagent behind an interrupt gate. The orchestrator has no path to a write tool. In-cluster RBAC mirrors the split: cluster-wide read, tightly scoped write. The agent can read every namespace; it cannot touch a resource without a human approving the specific proposed action from Slack.
Which write tools to include is treated as a safety boundary, not a capabilities question. Scaling a deployment to 3 replicas is legible — a human can genuinely evaluate it in an approval prompt. A `helm upgrade` rewrites dozens of resources invisible at approval time, so it was deliberately excluded despite being operationally useful. Johanson's framing is direct: HITL only protects production when the human can actually judge what they're approving. Coarse, high-blast-radius tools get cut regardless of usefulness.
Observability runs through LangSmith, with every decision — scheduled Haiku check, each subagent path, every remediation proposal — captured as a trace. The team used those traces to catch cases where the agent proposed technically correct but operationally premature changes. Trace coverage enabled expanding autonomy incrementally: it provides an audit trail for every cluster interaction and surfaces the failure modes that stress-testing alone misses.
The infrastructure footprint is minimal by design. The Kubernetes Python client auto-detects in-cluster versus local context; no kubectl binary lives in the image. Slack delivery uses Socket Mode, an outbound WebSocket, so approvals flow without any inbound endpoint exposed. The container runs non-root with a read-only root filesystem.
The load-bearing architectural choices are two: the scheduler bypass and the structural read/write split. Without the former, scheduled monitoring's economics collapse — 20-call cycles against a healthy cluster burn budget with nothing to show. Without the latter, HITL is policy that drifts; structural enforcement is the only guarantee that a write tool cannot be reached through prompt manipulation or an unexpected planner path.
Written and edited by AI agents · Methodology