Eduardo Valle and Fergal Reid published "Protoreasoning in Tiny Transformers" on August 5, 2026, showing that 1M-parameter models can learn chain-of-thought (CoT) reasoning. The finding breaks the assumed link between CoT capability and frontier-scale compute, enabling cheap, controlled mechanistic research.
The paper introduces protoreasoning: models at 1M parameters are trained to emit step-by-step traces before producing a final answer on Dyck-language tasks—formally nested brackets. Dyck languages provide a clean test bed: their structure is algorithmically transparent, out-of-distribution generalization is measurable, and correctness is unambiguous. This avoids the core frontier-research problem: models trained on terabytes of web text conflate memorization with reasoning in ways that are difficult to disentangle.
Two results anchor the work. First, protoreasoning traces measurably close the out-of-distribution generalization gap: models trained with step-by-step traces generalize better to bracket structures absent from training. Second, ablation studies confirm trace content drives the gain, not the extra tokens alone. A model trained to emit random or shuffled traces does not improve. CoT works through semantic content of intermediate steps, not token padding.
| Training Condition | Trace Content | OOD Generalization | Key Finding |
|---|---|---|---|
| Semantic step-by-step traces | Meaningful intermediate reasoning steps | Improved — gap measurably closed | CoT gains are content-driven |
| Random / shuffled traces | Non-semantic token padding | No improvement | Token count alone does not explain the gain |
This complements a broader theoretical debate. Merrill and Sabharwal (2024) proved that intermediate tokens extend transformer computational class—polynomial CoT steps let transformers solve polynomial-time problems that bounded-depth models without scratchpad cannot. Valle and Reid provide small-scale empirical confirmation: CoT operates at 1M parameters and is content-driven. Computational depth, not parameter count, drives multi-step reasoning.
For teams studying CoT in production systems, the practical implication is research-focused: controlled mechanistic experiments no longer require frontier-scale training runs. Prior work on CoT dynamics—including Pengmei et al. (2025), which found an "unfaithfulness phase" where early-training models produce correct answers while contradicting their reasoning—relied on larger models and opaque corpora. The Valle/Reid framework enables this analysis at negligible compute cost.
| Study | Year | Model Scale | Domain | Key Contribution |
|---|---|---|---|---|
| Valle & Reid | 2026 | 1M parameters | Dyck languages (nested brackets) | CoT works at tiny scale; trace content—not token count—drives OOD generalization gains |
| Merrill & Sabharwal | 2024 | Theoretical | Formal complexity theory | Polynomial CoT intermediate tokens extend transformer computational class to polynomial-time problems |
| Pengmei et al. | 2025 | Larger models | Natural language (opaque corpora) | Identified an "unfaithfulness phase" where models produce correct answers while contradicting their own reasoning traces |
The paper's limitation is scope: Dyck languages are a toy domain. The authors work "far below the threshold for natural-language competence" by design. Whether protoreasoning scales to natural-language tasks, multi-hop retrieval, or domains without known algorithmic structure remains untested. Teams understanding CoT failure modes in production LLMs must bridge this gap.
The core finding: CoT gains come from trace content, not token padding—now empirically verified at 1M-parameter scale without multi-GPU compute.