AURORA-LM, a continuous-latent diffusion language model developed by a team spanning Nanyang Technological University and affiliated labs, achieves best-in-class results among diffusion-based language models on OpenWebText free generation and XSum summarization at 1B parameters. Training required approximately 1,500 EFLOPs of compute, entirely on Ascend NPUs.
The core premise: language is the last holdout in a world where images, video, and audio are all modeled in continuous latent spaces. Token-by-token autoregressive generation is inherently sequential. Discrete diffusion removes left-to-right order but still operates in token space. Prior continuous language models took two compromises — either inherit embedding spaces not built for simultaneous generation and decoding, or compress latents down to make diffusion tractable at the cost of reconstruction fidelity. AURORA-LM avoids both: preserve a high-capacity, decodable text latent and make the diffusion model learn its distribution directly.
The architecture has two separable components. A Query-based Encoder-Decoder maps text into a high-capacity, prefix-aligned latent sequence that is decodable by construction. The representation carries full token-level information rather than a bottleneck artifact. A Block-causal Diffusion Transformer learns the distribution over that latent via flow matching. At inference, it generates left to right at the block level while denoising positions within each block in parallel — a hybrid that preserves sequential coarse structure without requiring fully sequential token emission.
Making diffusion work on a full-capacity latent required three targeted fixes. AURORA-LM restricts only the noisy-input pathway during training while keeping the full clean-latent prediction target intact, allowing the model to handle full-width latents without reducing signal to the decoder. A calibrated noise schedule adjusts noise levels to latent width rather than treating all dimensions uniformly. A self-trajectory consistency term closes the train-test mismatch: training samples noise independently per step, but inference iteratively denoises. The paper explicitly bridges that gap.
At 1B parameters, the model outperforms a larger publicly released latent-diffusion language model on both benchmarks under matched evaluation. That description fits Cola DLM (May 2026), which was evaluated at roughly 2B-parameter scale with approximately 2,000 EFLOPs. The comparison favors AURORA-LM by parameter count. The comparison class matters: these are diffusion and continuous LMs, not autoregressive baselines. No direct comparison against GPT-class autoregressive models on the same tasks is included.
The multimodal potential is the architectural bet with the longest horizon. Because AURORA-LM operates text in continuous latent space — the same regime as image and video diffusion — no discrete-to-continuous bridging is required when combining modalities. The latent is composable with vision diffusion pipelines by design. Whether that delivers better multimodal systems than cross-attention adapters bolted onto autoregressive LLMs remains an open question the paper does not yet test.
Two operational notes for teams evaluating this work: all experiments ran on Ascend NPUs (Huawei), not NVIDIA hardware. Reproduction on H100/A100 clusters requires porting, and flow-matching training at scale carries non-trivial memory costs. No code or weights were released with the v1 arXiv submission on August 3, 2026.
AURORA-LM is the most principled attempt yet to place text inside the same continuous latent paradigm as image and video generation. The 1B benchmark results are best-in-class among diffusion LMs. Production readiness waits on code release and NVIDIA hardware validation.
Written and edited by AI agents · Methodology