Ilmu Komputer & AI editorial
Open AccessOA2026
EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs
A single compiled expert graph plus UFS-backed expert virtualization enables dynamic MoE prefill on commercial Snapdragon smartphones, delivering up to 27.57× TTFT speedup and 12.29× memory reduction.
Junming Zhang; Zhenzhe Zheng; Fan Wu; Xiaoyao Huang; Jie Wu· 2026· DOI 10.48550/arXiv.2609.06551
The core problem
Mobile vendors and application developers increasingly deploy large language models (LLMs) on smartphones for diverse prefill-only services. Current systems, however, rely mainly on dense models whose regular computation maps efficiently to mobile neural processing units (NPUs), leaving more capable Mixture-of-Experts (MoE) models underused. MoE prefill does not fit mobile NPUs for two structural reasons. First, NPU graphs are fixed at compile time, yet MoE selects experts at runtime. Second, a single request touches most experts, exceeding the memory a phone can hold. EStream addresses both by separating what the NPU must fix from what MoE decides at runtime. The work targets prefill-only workloads on commercial Snapdragon smartphones and evaluates three 7B–16B MoEs across 256–4,096-token prompts, with scaling demonstrated up to 46.7B parameters.
Innovation
Across 18 comparative settings covering three 7B–16B MoEs and 256–4,096-token prompts, EStream is evaluated on a commercial Snapdragon smartphone. Compared to the fastest baseline at each setting, EStream achieves a 2.25–27.57× pure-prefill TTFT speedup and reduces peak physical memory by 1.19–12.29×. EStream further scales to MoE models with up to 46.7B parameters. These results demonstrate that dynamic MoE execution can be fully offloaded to the NPU without padding or CPU/GPU fallback, while memory remains bounded by the fixed-size arena rather than the model size.
Mobile vendors and application developers increasingly deploy large language models (LLMs) on smartphones for diverse prefill-only services. Current systems, however, rely mainly on dense models whose regular computation maps efficiently to mobile neural processing units (NPUs), leaving more capable Mixture-of-Experts (MoE) models underused. MoE prefill does not fit mobile NPUs for two structural reasons. First, NPU graphs are fixed at compile time, yet MoE selects experts at runtime. Second, a single request touches most experts, exceeding the memory a phone can hold. EStream addresses both by separating what the NPU must fix from what MoE decides at runtime. The work targets prefill-only workloads on commercial Snapdragon smartphones and evaluates three 7B–16B MoEs across 256–4,096-token prompts, with scaling demonstrated up to 46.7B parameters.
EStream introduces two core mechanisms. First, a single compiled expert graph serves every expert, with each expert's routed tokens and weight address bound at call time. This allows dynamic MoE execution to run entirely on the NPU without padding or CPU/GPU fallback. Second, expert virtualization keeps the expert pool in UFS flash storage and pages it through a fixed-size NPU-addressable arena, group by group, with loading hidden behind computation, so memory is bounded by the arena rather than by the model. A hardware-aware configuration algorithm automatically configures the UFS–NPU pipeline and maximizes loading–computation overlap.
Why it matters
The key insight of EStream is the separation of compile-time-fixed NPU graph structure from runtime MoE decisions. By compiling a single expert graph and binding routed tokens and weight addresses at call time, EStream eliminates the need for padding or fallback, which are common sources of overhead in prior systems. Expert virtualization further decouples memory usage from model size, enabling larger MoEs on memory-constrained smartphones. The hardware-aware configuration algorithm ensures that UFS loading is hidden behind NPU computation, maximizing overlap. This approach is particularly relevant for prefill-only services, where latency and memory are critical. The evaluation on a commercial Snapdragon smartphone confirms the practical viability of the design. Future work may extend EStream to decode phases or other mobile accelerators, and the taxonomy candidates (Architecture, Cybersecurity, Network, Cryptography) suggest potential cross-domain applications in secure and networked mobile AI.
Who should read this
CS practitioners and researchers
Opening member content…