DataOrchestra cuts data-pipeline compute by skipping unnecessary rewrites

Researchers at Shanghai Jiao Tong University published DataOrchestra on July 27, a system that routes pretraining data chunks through a learned classifier to decide—per example—whether to drop, clean, or leave unprocessed each piece. Models trained from 0.5B to 7B parameters on DataOrchestra-processed web data achieved stable gains over baseline methods across 11 benchmarks.

The orchestrator performs three-way classification: drop, leave unprocessed, or clean. For chunks flagged for cleaning, it selects downstream operations—programmatic editing (rule normalization, deduplication, formatting fixes) or LLM-based rewriting. The rewriting path generates a natural-language instruction a tool model executes. Chunks requiring only programmatic cleanup skip LLM invocation entirely, saving compute proportional to frequency.

LLM-based rewriting is the bottleneck. Current pipelines apply it uniformly across all documents because identifying which chunks actually need it has no efficient solution at scale. DataOrchestra replaces uniform application with a learned classifier trained on how different operations affect downstream model quality. Skipping unnecessary LLM rewrites on chunks marked unprocessed or programmatic-only cuts total compute.

For domain-adaptive continued pretraining, teams typically hand-tune separate pipelines for math, code, or scientific text. DataOrchestra outperformed stronger baselines across distributions without domain-specific redesign. That matters for organizations running repeated continued pretraining on new domain data.

Stability across 0.5B-to-7B model scales is important. Data-processing gains often show diminishing returns at larger scales or interact unpredictably with model capacity. The paper reports consistent gains on web data across that range—the typical scale for continued pretraining (most teams don't train 70B+ models from scratch on custom corpora). An 11-benchmark evaluation is broad enough to rule out benchmark-specific overfitting.

One unresolved question: the orchestrator's cost to train and run at corpus scale. If it costs as much to invoke as LLM rewriting, the savings evaporate. The practical question for engineers is whether the orchestrator runs at inference speeds compatible with large web crawl pipelines, or requires separate compute budget. The paper has those numbers in the ablations.

If you run LLM-based rewriting on your pretraining corpus, DataOrchestra's per-example routing is the abstraction to evaluate. Uniform application is where computational waste lives.