New Tool Finds 1,060 Hidden Training Dependencies Across Major LLMs

Researchers from UC Berkeley and the Allen Institute for AI have introduced ModSleuth, an open-source system that reconstructs training-time dependency graphs from public artifacts. The tool was used to audit four recent large language model (LLM) releases, identifying 1,060 source-verified upstream links and revealing artifact chains not captured by traditional model cards and datasheets.

ModSleuth, a Python 3.11+ CLI package installable via pip, operates through an eight-stage pipeline—discover, extract, organize, audit, relate, reconcile, triage, merge—using Claude Opus 4.7 (planner) and Claude Sonnet 4.6 (subagent) for the paper's own audits. The system processes heterogeneous public releases, employing configurable strategies such as BFS, DFS, or beam search, and stores provenance in a local SQLite graph database and a content-addressed source store. It also includes a viewer for focused subgraphs on port 8102, and commands for monitoring token usage and system status.

The arXiv paper detailing the audits of DR Tulu, SmolLM3, Olmo 3, and Qwen3 32B exposed risks such as license issues, contamination, and circularity that standard decontamination suites overlook. For instance, DR Tulu's supervised fine-tuning data traces back to Claude Sonnet 3.7 through the ScholarQA pipeline. SmolLM3's FineMath dataset carries a transitive Llama license obligation via an upstream Llama-trained classifier, creating compliance exposure that flat datasheets miss. Olmo 3 trains on IFEval-derived synthetic data while benchmarking against IFEval, a train-eval coupling that standard decontamination misses because it crosses artifact boundaries. Qwen3 32B serves as both its own direct-preference-optimization generator and RL judge, forming a circular self-dependency.

ModSleuth faces operational challenges, requiring frontier 1M-context Claude models to reason across fragmented documentation, with a planner that enforces an 1,800-second silence timeout before auto-retry. The CLI exposes token spend tracking, though the paper does not report per-audit costs. The system is limited to public artifacts and cannot access private synthetic data pipelines, undocumented vendor API calls, or internal judge configurations—dependencies that pose significant enterprise liability. ModSleuth addresses the training-lineage gap ignored by traditional SBOMs and software composition analysis tools but does not mitigate runtime exposure.