CARE Cuts Expert Activation in MoE-LoRA Fine-Tuning

A new paper from Saliencro, Desai, Nair, Lindqvist, and Whitmore introduces CARE — Confidence-Adaptive Routing of Experts — a replacement for fixed top-k routing in MoE-LoRA fine-tuning. Tested on LLaMA-3.1-8B and Qwen2.5-7B across eight commonsense benchmarks plus math, code, and knowledge tasks, CARE matches fixed-k=4 baseline accuracy while activating fewer experts per token. Code is public.

The problem is structural: standard MoE-LoRA routes each token to exactly k experts regardless of model confidence. The router's softmax already encodes uncertainty — a peaked distribution signals confidence; a flat one signals ambiguity — but fixed-k routing discards that signal.

CARE treats expert activation like nucleus sampling. Experts are ranked by decreasing router weight and admitted until cumulative probability mass clears a threshold. Ambiguous tokens spread mass across many candidates and pull in more experts automatically. High-confidence tokens concentrate mass in one or two and stop early. A disagreement extension adds one expert whenever the admitted set disagrees on output — a lightweight ensemble mechanism at zero cost on uncontested predictions.

A budget thermostat maintains average expert count at any target without manual threshold tuning. Set the target, calibrate on a small validation set, and CARE self-adjusts. The mechanism runs in a single forward pass with no additional parameters.

Benchmark results show two modes. At matched compute — same average active experts as fixed-k baseline — CARE improves accuracy across the suite. Matched against fixed-k=4, CARE reaches the same accuracy with fewer active experts. The paper does not state explicit wall-clock or token-throughput figures; the 20–30% latency estimates in the pitch are approximate until experimental tables are reviewed. The paper establishes that matched accuracy is reachable at lower expert-activation cost, where inference-time savings originate.

A secondary result: CARE's confidence and disagreement signals outperform Maximum Softmax Probability (MSP), entropy, and multi-pass proxies for out-of-distribution detection. When an MoE-LoRA model faces distribution drift, the same routing signal that saves compute functions as a lightweight anomaly detector at zero additional cost.

The method integrates into any MoE-LoRA stack exposing router weight distribution before top-k truncation. Architectures using MoA or LoRAMoE routing slot it in by replacing the top-k step. Budget thermostat calibration requires a small in-distribution validation set — standard in fine-tuning workflows but worth flagging for zero-shot deployment.

The key insight: the router's softmax already signals which tokens are hard. CARE is the first MoE-LoRA method to exploit that signal at inference time without additional parameters or forward passes.