Stanford Extracts Task Models from Screen Recordings

Stanford and CMU researchers published Task Model Induction (TMI) on August 20, a method that extracts structured task models from recorded computer-use traces without annotation. On controlled trajectories, TMI recovers interleaved tasks with 97.4% agreement against ground truth, reconstructs 74.9% of execution steps, and derives skills that improve held-out task accuracy by 30% over the strongest baseline.

The gap is real. Organizations with screen recordings of knowledge work have no practical way to convert footage into reusable task structure. Existing methods either compress sessions into prose summaries—losing control flow and concurrent tasks—or assume a single root task is given. Neither handles how people actually work: jumping between goals, nesting sub-objectives, and iterating within tasks.

TMI proceeds in three stages. First, low-level events (cursor, keys, pixels) are grounded into semantic actions using visual context. Second, latent task discovery disentangles concurrent activity without task labels—the system identifies which action spans belong to which task using only trace structure. Third, for each task, TMI induces an objective model (recursive goal hierarchy) and a procedure model (control flow using sequencing and iteration operators). The output is symbolic and auditable—what was done and how—not prose.

What matters for deployment: skills derived from TMI's models transferred to new task instances unseen during induction, lifting accuracy by 30% over baseline. The 74.9% execution step reconstruction rate shows how faithfully the procedure model captures actual control flow, making derived skills executable rather than decorative.

This work contrasts with online skill learning. CMU's Agent Skill Induction (ASI, from co-author Zora Zhiruo Wang) learns skills during live task execution, gaining 23.5% on WebArena. ASI needs a running agent and live environment. TMI needs only recordings. For enterprises where re-running work is impossible, passive induction is the practical path. Teams that can deploy agents iteratively can combine both: cold-start from historical traces, then online refinement.

Three constraints remain unresolved. Event grounding bottlenecks downstream—bad semantic action maps degrade the task model. The evaluation used controlled trajectories; real traces are noisier. The procedure model handles sequences and loops but not branching conditions, which are hard to infer from a single trace. Latent task discovery has no mechanism for genuinely ambiguous or user-idiosyncratic goals.

The codebase is public at github.com/Yucheng-Jiang/task-model-induction. For teams building computer-use agents over enterprise workflows, start by running TMI over existing screen recordings and auditing the generated models before using them in agent training. Symbolic output makes that audit tractable in ways embedding-based approaches cannot.