From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Large language models (LLMs) continue to scale, making dense full-parameter adaptation increasingly expensive. Sparse and modular architectures such as Mixture-of-Experts (MoE) models have emerged as a promising direction to manage computational costs while maintaining high capacity. However, this shift raises a critical question for parameter-efficient fine-tuning (PEFT): at what granularity should parameters be selected and updated? Existing PEFT methods like LoRA operate on predefined weight matrices, while expert-level sparse tuning methods update entire selected experts. The authors observe that activated experts are internally sparse—only a small fraction of intermediate channels strongly respond to downstream tasks. This indicates that expert-level adaptation is still too coarse. To address this, they propose NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task-relevant sub-experts by combining routing importance with intra-expert activation saliency. To optimize sparse partial updates, NSFT introduces learning-rate scaling and dynamic gradient scaling to compensate for reduced effective update magnitude. Experiments on OLMoE and Ling-mini-2.0 across challenging domain-specific tasks and general benchmarks demonstrate that NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability. These results suggest that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.

NSFT operates by decomposing each expert in an MoE layer along the intermediate dimension into structured channel groups, termed sub-experts. For a given input, the router selects a set of experts, and within each selected expert, NSFT identifies task-relevant sub-experts by combining two signals: routing importance and intra-expert activation saliency. Routing importance reflects how strongly the router activates an expert for the current input, while activation saliency measures the response magnitude of intermediate channels within that expert. The combination yields a saliency score for each sub-expert, and only those above a threshold are updated during fine-tuning.