SubFit Maintains 84.6% Accuracy While Pruning LLM Layers at 25% Sparsity

SubFit, a submodule-level compression technique from the University of Trento, maintains 84.6% downstream accuracy on instruction-tuned LLMs at 25% sparsity. It replaces individual attention and feed-forward submodules rather than pruning entire contiguous layers, achieving a 1.92× better perplexity degradation factor than the strongest layer-wise baseline at the same compression ratio. The method, detailed in "From Layers to Submodules" by Cunegatti et al., scores and removes attention and FFN submodules independently across non-contiguous layer indices. Each removed submodule is replaced by a fitted residual bypass tuned to the component type: attention submodules receive a low-rank approximation, while FFN submodules are replaced by a higher-rank map with a shared input basis across all selected FFN layers to limit the deployed parameter cost. The post-training pipeline requires only a forward-pass calibration dataset, similar to GPTQ or AWQ quantization workflows, and no backpropagation or retraining.

Evaluation across ten LLMs, including five base models and five instruction-tuned variants, at sparsity levels of 12.5, 18.75, 25, 31.25, and 37.5%, shows SubFit degrades perplexity by a factor of 2.42× at 25% sparsity, while the best baseline degrades by 4.34×. The gap accelerates with sparsity: at 12.5% sparsity, the margin is 0.11×, but it grows to 5.69× at 37.5%. SubFit is the only tested method to remain above 80% downstream accuracy at 25% sparsity and above 73% at 37.5%. The authors claim measurable inference speedup and KV-cache savings due to the elimination of key-value pairs from removed attention submodules, but they do not provide exact latency, throughput, or cache-reduction multipliers, preventing architects from modeling TCO.

Deployment is complex as non-contiguous submodule removal disrupts the regular layer topology optimized by inference engines like vLLM, TensorRT-LLM, and TGI. Implementing the bespoke low-rank and shared-basis bypasses efficiently would require custom kernels or at least graph-rewriting passes that do not exist in public repositories. The authors' code is not yet public, listed as cleanup-in-progress, halting independent verification and integration. Post-training calibration avoids the cost of fine-tuning but relies on the quality and domain-match of the calibration data; a shift from the calibration distribution can misalign the fitted residuals without the safety margin provided by retraining.

What an architect would steal: treat layers as non-monolithic compression units and apply submodule-specific replacement strategies with non-contiguous selection, as redundancy in pretrained transformers is distributed unevenly across attention and FFN blocks rather than clustered in contiguous depth ranges.