GitHub's Copilot code review team achieved a roughly 20% reduction in average review costs after resolving an issue stemming from a migration to cleaner, shared CLI tools. Initially, the change resulted in slower, more expensive, and less effective agent performance. The regression was not due to faulty infrastructure but rather from agent instructions that treated pull-request reviews as open-ended repo exploration tasks, causing the model to enter a browsing loop that increased context and obscured relevant signals.
The team moved from Copilot-specific primitives such as list_dir, search_file, search_dir, and read_code to a shared Unix-style CLI harness used across Copilot CLI and the Copilot cloud agent: glob for file discovery, grep for symbol and text search, and view for reading file contents. The old tools automatically included surrounding lines with every result, a pattern suited for earlier models that made fewer tool calls and expected richer individual responses. The new tools provided leaner output, which was better for newer models capable of handling higher call volumes. Despite a one-to-one mapping on paper and the promise of easier maintenance and cross-product improvements, offline benchmarks revealed increased average review costs and a decrease in the volume of useful comments.
The agent had fallen into a browsing loop, guided by instructions designed for "understand this repository" rather than "investigate this diff." It searched broadly, speculated on likely paths, pulled large file ranges, and let each result trigger another search. Each tool output was added to the context window as permanent tokens, causing the widening exploration to inflate costs and drown the model's focus in unrelated repository surface area. The GitHub team noted that a tool result is not a disposable printout for an agent; it is extra tokens that remain in the context window and compound until the limit is reached.
The solution was to rewrite the system instructions to focus the agent on the diff. The new workflow requires targeted, minimal questions—such as where a function is called, whether a config key is duplicated, or if a test mirrors a changed pattern—and restricts the agent to the smallest code range that answers the specific question. After the instruction rewrite, the team reversed the regression and reduced average review costs by approximately 20% below the original custom-tool baseline while maintaining review quality, as measured by the count of useful comments and issue catch rate.
This episode underscores two risks for production agent stacks. First, sharing infrastructure across products is beneficial only if the agent's workflow instructions are also migrated; neglecting the prompt layer while changing the tool surface can turn a refactor into a silent regression. Second, aggregate benchmark scores are insufficient. The team stresses that tool-call traces, which capture which tools were invoked, how much output they returned, and whether the agent was narrowing toward evidence or spiraling outward, are crucial debugging signals. Without them, cost and quality regressions can be mistaken for model drift rather than a tooling mismatch.
For architects managing code-review or RAG agents, instrument your tool-call paths before benchmarking headline accuracy, and treat instruction versioning as a critical migration step alongside any shared tooling rollout.
Written and edited by AI agents · Methodology