DeepSeek-V4-Flash on AMD gfx90a: Correctness Recovery and Inference Performance Engineering

Enabling FP4 MoE inference on CDNA2 with numerical correctness repair and performance tuning

Siming Huang· 2026· DOI 10.48550/arXiv.2609.15627

The core problem

The deployment of large mixture-of-experts (MoE) models such as DeepSeek-V4-Flash on non-NVIDIA accelerators presents unique challenges. This work focuses on enabling efficient inference on AMD Instinct MI250 GPUs based on the gfx90a/CDNA2 architecture. The system integrates native safetensors loading, tensor and expert parallelism, FP4 routed MoE computation, FP8 dense projections, sparse attention, HIP graph execution, and OpenAI-compatible serving within SGLang. A critical issue emerged: an initially fast execution path produced numerically incorrect results due to a routed-expert W2 layout mismatch. This paper details the identification of the output permutation, repair of the weight layout at load time, and establishment of correctness checks before performance optimization. The corrected path then undergoes extensive performance engineering for both decode and prefill phases, demonstrating that efficient inference on CDNA2 is limited by factors beyond memory bandwidth, including FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs.

On four MI250 GCDs with TP4/EP1, the corrected and optimized system achieves approximately 74.5 tokens per second for native autoregressive decode. For a 4,604-token prompt, the time-to-first-token (TTFT) is 2.061-2.062 seconds, corresponding to an input throughput of approximately 2,234 tokens per second. These results are obtained after applying the full set of optimizations. The correctness checks confirm that the repaired weight layout eliminates the numerical errors present in the initial fast path. The performance improvements are attributed to the combination of packed FP4 weights, INT8 activation quantization, CDNA2-specific instructions, and topology-aware kernel geometry for decode, and MFMA kernels, improved weight reuse, and reduced sparse-attention overhead for prefill. The results highlight that memory bandwidth alone does not determine performance; FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs are significant limiting factors on CDNA2.

The deployment of large mixture-of-experts (MoE) models such as DeepSeek-V4-Flash on non-NVIDIA accelerators presents unique challenges. This work focuses on enabling efficient inference on AMD Instinct MI250 GPUs based on the gfx90a/CDNA2 architecture. The system integrates native safetensors loading, tensor and expert parallelism, FP4 routed MoE computation, FP8 dense projections, sparse attention, HIP graph execution, and OpenAI-compatible serving within SGLang. A critical issue emerged: an initially fast execution path produced numerically incorrect results due to a routed-expert W2 layout mismatch. This paper details the identification of the output permutation, repair of the weight layout at load time, and establishment of correctness checks before performance optimization. The corrected path then undergoes extensive performance engineering for both decode and prefill phases, demonstrating that efficient inference on CDNA2 is limited by factors beyond memory bandwidth, including FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs.

The methodology encompasses correctness recovery and performance optimization. First, the routed-expert W2 layout mismatch is addressed by identifying the output permutation and repairing the weight layout at load time. Fixed-token and hash-based correctness checks are established to validate numerical accuracy. For decode performance, several optimizations are applied: packed FP4 weights, INT8 activation quantization, CDNA2 dot-product instructions, peer-read all-reduce, and topology-aware kernel geometry. Prefill acceleration uses CDNA2 MFMA kernels, improved packed-weight reuse, reduced sparse-attention overhead, larger chunks, and retuned expert sorting. The system is evaluated on four MI250 GCDs with tensor parallelism (TP4) and expert parallelism (EP1). The key performance metrics are decode throughput (tokens per second) and time-to-first-token (TTFT) for a given prompt length. The architecture integrates multiple components as shown in the Mermaid diagram below.

The findings reveal that efficient DeepSeek-V4-Flash inference on CDNA2 requires careful attention to numerical correctness and hardware-specific optimizations. The routed-expert W2 layout mismatch underscores the importance of validating execution paths before performance tuning. The performance engineering demonstrates that leveraging CDNA2 features such as dot-product instructions and MFMA kernels can yield substantial gains. However, the remaining bottlenecks—FP4 execution-format mismatch, low-M utilization, and per-layer synchronization—suggest that further improvements may require architectural changes or more advanced scheduling. The achieved decode throughput of 74.5 tok/s and TTFT of ~2.06 s for a 4.6k-token prompt represent a significant step toward making large MoE models practical on AMD GPUs. Future work could explore alternative quantization schemes, better overlap of computation and communication, and optimizations for other CDNA architectures.

CS practitioners and researchers

Opening member content…