The HERO'S JOURNEY benchmark from UT Austin reveals that state-of-the-art large language models (LLMs) can induce hidden rules from demonstrations in goal-directed text games, but this ability is limited and inconsistent. The benchmark includes eight task types—four attribute and four procedural—expressed across four structural rule forms with controllable lexical grounding.

In each episode, an agent plays a text game with some mechanics hidden, inferring the missing requirement from demonstrations, verbalizing the rule, and executing a multi-step plan against a novel entity. The released codebase, available on PyPI as herosjourney v0.1.0 and on GitHub under an MIT license, supports OpenAI-compatible APIs and local endpoints via vLLM, Ollama, and LM Studio. Custom tasks can be added via JSON or YAML rule files without writing Python, making it a drop-in eval tool for agent pipelines.

Evaluation focuses on ECSR, or Efficiency-Calibrated Success Rate: success rate multiplied by normalized efficiency, where efficiency equals reference episode length divided by the number of runs the model consumes, floored at one over n_tries. This metric penalizes agents that eventually succeed through brute-force retry loops. A secondary metric, RV (rule verbalization), uses an LLM judge to score the model's free-text description of the extracted pattern. The authors tested four steering strategies—standard prompting, ReAct, HR, and IDEA—to determine if induction-specific scaffolding closes the gap.

This research benchmark has no production deployment evidence yet; no GPU-hours, per-call latency, or token pricing is reported. Surface semantics—real words versus nonce words—has minimal effect, indicating the failure is structural, not vocabulary-level. Process execution is the confirmed bottleneck, and while steering methods lift performance on attribute induction tasks, they deliver no reliable gains on procedural induction, leaving that family as the open challenge.

The procedural gap is crucial for production agents. The TextQuests literature already established that LLMs hallucinate prior interactions and repeat actions in loops as context windows stretch past 100K tokens, with test-time compute yields flattening after a budget threshold. HERO'S JOURNEY sharpens that finding: even when models correctly infer a rule, they fail to translate it into reliable multi-step execution, and ReAct-style reasoning does not fix the procedural case. Architects should treat procedural rule induction as an unsolved primitive, not a sub-task to bury inside a broader agent framework.

For this benchmark to drive Monday-morning stack decisions, a cost-and-latency-calibrated leaderboard is needed: ECSR per dollar and per wall-clock minute, measured across concrete serving stacks. The transferable pattern to steal today is the ECSR metric itself—if your agent eval only tracks final accuracy, you are rewarding token-heavy retry loops that collapse under production budgets. And if you are building internal eval harnesses, copy the JSON/YAML rule-file interface; decoupling task definition from Python boilerplate is exactly how you keep benchmark velocity high as your agent surface area grows.

Treat procedural induction as a hard ceiling rather than a prompt-engineering gap, and bake efficiency-adjusted success into every agent eval you run.

Written and edited by AI agents · Methodology