Diffusion Models Cut Compute on Sparse Data with Selective Processing

Researchers from TU Kaiserslautern, UC Irvine, and Heidelberg University have published a diffusion-model architecture that skips zero-valued entries during training and inference, cutting compute costs from total dimensionality down to the number of non-zero values. The paper, accepted to ICML, introduces Sparsity-Exploiting Diffusion (SED) and targets enterprise data — particle-physics detector outputs, single-cell RNA sequencing, and recommender-system interaction matrices — where most entries are exactly zero.

Standard diffusion models such as DDPM and LDM process every dimension regardless of value. On sparse data this means running the full forward-and-reverse noising process over semantically empty dimensions. The result is two failures: FLOPs scale with total dimensionality rather than signal density, and dense models introduce spurious non-zero entries even on simple datasets like MNIST.

SED addresses both with a three-stage pipeline. A sparsity-aware autoencoder encodes only non-zero entries into a compact latent representation, discarding zero dimensions before diffusion begins. Standard dense diffusion runs within that compressed latent space, keeping model complexity proportional to non-zero count. An autoregressive decoder reconstructs dimension–value pairs exclusively for non-zero entries, writing exact zeros everywhere else. Computational cost stays nearly constant as total input dimensionality grows, provided the number of active entries remains fixed.

In single-cell RNA sequencing, most of the tens of thousands of gene measurements per cell are exactly zero — a biologically meaningful signal of a dropout event. Dense diffusion models waste compute on silent dimensions and then corrupt the signal by generating noise where silence was expected. SED preserves sparsity patterns aligned with ground truth while dense baselines fail this structural test. On physics and biology benchmarks SED matches or surpasses conventional diffusion and domain-specific baselines on generation quality.

For enterprise teams running generative models on sparse tabular data — IoT sensor feeds where most channels are inactive, user–item interaction tables, financial transaction logs — the rule is the same: compute cost should track signal density, not matrix dimensions. Dense diffusion is doubly expensive: it pays for inactive dimensions during training then contaminates downstream pipelines by generating hallucinated activity absent from training data.

SED does not yet address all sparse-data regimes. The autoregressive decoder introduces sequential dependency at generation time — each non-zero pair must be synthesized in order — which may add latency even if total FLOPs drop. The approach is designed for real-valued sparse data with exact structural zeros; it is not a substitute for sparse-weight compression on neural network parameters.

Code is open-sourced at github.com/PhilSid/sparsity-exploiting-diffusion. ICML acceptance signals peer review for scientific rigor, making this a credible research direction for ML platform teams evaluating generative models for non-image, non-text data.