Slack has conducted over 200 end-to-end UI tests using agentic methods within non-production workspaces, revealing a zero percent failure rate for the simplest flows, but a 48 percent failure rate for the generated-script approach on the most complex flow, and 12 percent even for the best-performing MCP-backed agent on that same flow. This indicates that agentic verification serves as a debugging layer rather than a complete CI replacement. The engineering team at Slack tested three distinct automation architectures against two real product flows to determine the appropriate placement of agentic testing in the testing stack.
The harness operates non-interactive Claude Code via `claude -p`. For agent-driven execution, Claude Sonnet 4.5 plans and executes browser actions, while Claude Opus 4.6 translates natural-language intent into deterministic Playwright scripts. Two Playwright-based invocation modes were evaluated against the generated-script approach. The first involved an agent calling a Playwright MCP server that maintains persistent browser context across a full session. The second had the agent issue sequential Playwright CLI shell commands, rebuilding state from snapshots at every step. Environment setup was managed by a Slack Dev API MCP. The validation flows included a Thread Reply journey with 15-20 steps and a Search Discovery journey with 25-30 steps, each executed twenty times per configuration against both natural-language and structured YAML inputs.
The MCP-backed agent achieved a zero percent failure rate for the simple Thread Reply flow and a twelve percent failure rate for the more complex Search Discovery flow, averaging five to eight minutes per run. The CLI-driven agent failed twelve percent of simple runs and twenty percent of complex ones, taking nine to eleven minutes due to its state reconstruction at each step. The generated-script approach was the fastest at about three minutes but least reliable, failing eight percent of Thread Reply runs and forty-eight percent on Search Discovery. Slack attributes this difference to the persistent DOM context provided by MCP, as opposed to the state drift introduced by CLI's snapshot reconstruction over longer flows.
Cost and latency render every variant unsuitable for continuous integration. Each run costs between fifteen to thirty dollars and takes over ten minutes in the agent-driven modes, with no token-level pricing or GPU-hour breakdowns provided. Both the Slack Engineering post and an InfoQ summary position agentic testing as a fourth layer in the testing pyramid, above unit, integration, and deterministic E2E tests, reserved for exploratory testing, reproducing production issues, and debugging flaky workflows. Agents are constrained by allowed-action whitelists, exploration boundaries, and explicit stopping conditions, with observability delivered through structured execution logs and replayable traces.
The primary challenge is not tooling integration but the cost-reliability trade-off on complex flows. Generated tests degrade significantly as the number of steps increases, and even the best-performing agentic mode still fails one in eight complex journeys. Deterministic assertions are difficult to maintain due to agents potentially reaching the same goal through different navigation paths, input methods, or skipped steps. Additionally, there is no discussion of prompt-injection surfaces in the browser MCP context, which remains an open question for anyone exposing a DOM-aware tool to an LLM.
Adopting the pattern of providing your browser agent with persistent MCP context instead of shell snapshots is recommended, and keeping it out of the CI gate until per-run cost and latency are reduced by an order of magnitude.
Written and edited by AI agents · Methodology