SparseDitto LLM Agents Cut Sparse Matrix Kernel Time 3.39x

Researchers from Northeastern University, the University of Minnesota, and the University of Connecticut published SparseDitto on August 5, 2026 — a system that deploys LLM agents to synthesize custom GPU kernels for sparse matrix operations. For the same sparse matrix operation, cuSPARSE shows a 350x performance gap between CSR and Blocked-ELL storage formats. That variance is not an edge case — it is the default state of sparse computation.

Sparse matrix kernels — SpMV, SpMM, SpGEMM — underpin GNN training, graph analytics, and scientific computing. Performance depends on how nonzeros distribute and how that maps to GPU memory access patterns. CSR works well for matrices with irregular row lengths. Blocked-ELL exploits regularity to hit Tensor Cores. COO, HYB, and half a dozen compiler-generated formats each have their own performance regimes. Before SparseDitto, the state of the art was pick one format for all inputs (wrong most of the time) or use a static ML classifier to select among a fixed menu of existing kernels (bounded by what is on the menu).

SparseDitto discards the menu. Its pipeline has three stages: a lightweight additive model ingests structural features of the input matrix — row length distribution, nonzero density, block regularity — and ranks candidate execution strategies. An architecture-aware planner proposes kernel designs tuned to the target GPU's memory hierarchy. Coding and verification agents implement each design, run it on the target hardware, and iterate using actual latency measurements as feedback. The loop closes on the GPU, not in simulation.

On an NVIDIA RTX PRO 6000, SparseDitto achieves a 2.68x geometric-mean speedup over cuSPARSE across three sparse operators and diverse matrix benchmarks, with a peak of 146.61x. On an H200, the geometric mean is 2.79x, peak 78.5x. The gap between geometric mean and peak reflects underlying variance: some matrices land where a generated kernel dramatically outperforms any fixed-format baseline; others see modest gains.

The most operationally significant result is 3.39x: the speedup on full-batch GCN training. GNN training is where sparse operations dominate total runtime — profiling of full-batch GraphSAGE training shows SpMM accounts for 83.6% of total training time. A 3.39x improvement on that kernel translates directly to training throughput.

Two practical constraints matter. First, SparseDitto generates a kernel per matrix-operator-GPU triple. For inference serving with fixed sparse weight matrices, generation is a one-time offline cost. For training with dynamically changing sparsity patterns — iterative pruning, dynamic sparsification — the amortization math changes. Second, the system was evaluated on RTX PRO 6000 and H200; behavior on other architectures (A100, B200, AMD MI300X) is not yet characterized.

The SparseDitto architecture — feature-based ranker → candidate planner → LLM coding agent → hardware-in-the-loop verification — is a general pattern for operations where performance is format-sensitive or input-distribution-sensitive. Sparse attention, custom quantized GEMM layouts, and sparsified MoE routing all have the same structure: performance varies 10–100x by configuration choice, static tuning is insufficient, and the solution space is too large for hand search. SparseDitto proves LLM agents can close that loop on real hardware.

If your GNN or sparse inference stack runs cuSPARSE with default format selection, you are running the wrong kernel for your specific matrices on your specific GPU. The gap may be hundreds of x, not percent.