OctoLong improves code agents with cross-repository training

Researchers from TU Darmstadt and UKP Lab published OctoLong, a context engineering pipeline that fixes a blind spot in long-context training data: code corpora almost never capture cross-repository dependency chains. Released August 5, the paper introduces a three-tool stack—AST parser, language server backend, and package manager—that recursively traverses import graphs and symbol references to assemble training contexts of millions of tokens.

Standard long-context corpora rely on books, academic papers, and single-repository code dumps. None contain the cross-module, cross-package logic that coding agents encounter in production. When a function call threads through four upstream libraries, agents must resolve that logic in full. OctoLong's pipeline does this: start from an entry-point symbol, walk the language server's reference graph, pull in transitively imported packages, and repeat recursively until context budget is reached.

The resulting training mix totals roughly 50B tokens. OctoLong-specific cross-repository contexts account for 6.2B tokens—12% of the total. The OctoLong-Instruct model suite trains on this mixture via mid-training context extension, then fine-tunes on ~10B tokens of instruction data. Models span 600M to 14B parameters, making gains reproducible across inference budgets rather than confined to frontier-scale checkpoints.

Evaluations across 18 open-weight long-context LMs show that replacing 12% of the training corpus with OctoLong data improves long-range retrieval, long-term state tracking, repository-level code understanding, and agentic task performance. API usage accuracy in non-long-context coding scenarios also improves, suggesting the cross-repo training signal generalizes rather than overfits to needle-in-a-haystack recall.

OctoLong is not proprietary. The methodology orchestrates standard developer tooling. Any team with access to a language server (Pyright, rust-analyzer, clangd) and a package manager can replicate the pipeline against their own codebases. That matters for practitioners fine-tuning domain-specific code agents—financial infrastructure teams, compiler engineers, embedded systems developers—who need dependency structures representative of their own stacks.

Two limits warrant attention. First, contexts reach millions of tokens during curation, but the paper does not fully detail the lengths at which trained models are evaluated; production teams should benchmark at actual deployment context lengths. Second, recursive reference retrieval assumes a correctly configured language server, a non-trivial requirement for polyglot repositories or legacy codebases with incomplete type information.

For teams planning context-extension fine-tuning of repo-level coding agents: data substitution rate is the lever. Replacing 12% of a 50B-token mixture with AST/language-server-constructed cross-repository chains drives the gains. The pipeline is open methodology; the cost is language server infrastructure, not a proprietary license.