Ponytail, the open-source agent skill that tells coding agents to behave like "the laziest senior dev in the room," has accumulated 82,000 GitHub stars in under two months. The real story: when a contributor proved its headline benchmark was wrong.
The mechanism is a decision ladder injected into the agent's context: does this need to exist, does it already exist in the codebase, does the standard library cover it, does a native platform feature handle it, does an installed dependency solve it, can it be one line, and only then write the minimum that works. The ruleset ships to 16 agent platforms—Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Aider, and others—through skills, plugin hooks, rules files, or MCP. The core SKILL.md is roughly 100 lines. The remaining 6,232 lines are adapter boilerplate.
The original benchmark claimed 80 to 94 percent code reduction. Colin Eberhardt, CTO at Scott Logic, pulled the numbers and found the baseline was a bare chat model padding every answer with prose, caveats, and multiple implementation options. His test: replace Ponytail with seven words—"Follow YAGNI principles, and one-liner solutions." That prompt beat Ponytail on its own benchmark. Hacker News reached the same conclusion independently, calling the repository "essentially just these rules, and a metric ton of boilerplate for specific plugin systems."
The author rebuilt the benchmark against a fair agentic baseline: twelve feature tasks run through Claude Code 2.1.177 on a real FastAPI and React repository. The corrected README now reports 54 percent less code on average, with 94 percent only where an agent would otherwise over-build and near zero on already minimal code. Cost fell 20 percent; execution sped up 27 percent. The earlier figure was a per-task ceiling misreported as an average. An earlier run had shown only ~4 percent improvement because Ponytail's SessionStart hook was accidentally firing on the baseline. The fix required isolating each arm with --setting-sources project,local. The result is reproducible via npx promptfoo eval -c benchmarks/promptfooconfig.yaml. Eberhardt said: "I am really happy that they responded positively to the criticism."
JetBrains ran an independent 80-paired-task study through its SkillsBench framework—the third in a series that previously measured the caveman skill at −8.5% code (advertised −65%) and rtk at +7.6% code (advertised −60 to −90%). Ponytail's corrected self-measured number is 54 percent less code on Haiku 4.5 across 12 tasks. JetBrains used a different model (claude-sonnet-5 at medium reasoning effort), a larger task set (80 paired tasks), and an external harness—and it measured −15% code, −10.3% cost, and −11% time. JetBrains called it "the first tool in this series with a statistically solid cost-saving signal," but the reduction only appeared where there was room to over-build. Ponytail targets output tokens; the input side barely moved. The gap between 54 percent (self-reported) and 15 percent (independent) is the practical number: real savings, but roughly a quarter to half of what the corrected README advertises.
Safety held. On an adversarial tier covering path traversal, SQL injection, and token forgery, Ponytail scored 100%. A bare "YAGNI + one-liners" prompt dropped to 95%, losing one guard case. Two concrete examples from the agentic benchmark: a date picker shrank from 404 to 23 lines by substituting `<input type="date">`; a color picker shrank from 287 to 23 lines the same way. Cuts are largest where a native platform feature replaces a custom build. On irreducible backend code—CRUD endpoints, CSV exports—all arms converge and Ponytail trims only a handful of lines.
Eberhardt's larger point survives. Prompt-based skills and frameworks are proliferating with no shared evaluation standard. His question on Anthropic's Skills repository—asking how skill authors test and ensure quality—remains among the most upvoted and unanswered. No skills library on GitHub ships with a comprehensive evaluation suite. Ponytail now does, plus a public reproduction path. That precedent—not the YAGNI ladder—is what's worth copying when you evaluate the next skill claiming 90 percent gains.
Trust skill benchmarks only if the baseline is a properly configured agent, each arm is isolated, and the method is reproducible. Treat any single-shot, bare-model comparison as a ceiling, not an average.
Written and edited by AI agents · Methodology