Ilmu Komputer & AI editorial
AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall
The core problem
Large language models (LLMs) place unprecedented and still-growing demands on the hardware that trains and serves them. As model scale, context length, and deployment breadth expand, the accelerator landscape has diversified far beyond general-purpose GPUs. This review surveys the full landscape of AI hardware accelerators for LLMs, including general-purpose GPUs, custom ASICs such as TPUs, Trainium, Groq, and Cerebras, reconfigurable FPGAs, processing-in-memory and near-memory architectures, and emerging neuromorphic and photonic approaches across cloud and edge deployment.
The central thesis is that the decisive constraint on LLM acceleration is not arithmetic but memory. Using the transformer's computational structure and roofline analysis as a common framework, the authors show that the autoregressive decode phase is bandwidth-bound, the key-value cache can rival the model weights in size, and data movement dominates energy. This reframes accelerator design around memory hierarchy, bandwidth, and data locality rather than peak FLOPs alone.
Innovation
The comparative analysis yields a clear pattern: no single architecture is optimal across workloads. GPUs remain the flexible default and the workhorse of training, owing to mature software ecosystems, high peak compute, and general programmability. Domain-specific ASICs win at scale for stable, high-volume workloads, where fixed dataflows and specialized memory hierarchies can be tuned to specific model families and serving patterns.
Processing-in-memory (PIM) emerges as the most promising near-term response to the memory wall, entering systems as a heterogeneous complement rather than a wholesale replacement. By placing computation near or within memory arrays, PIM reduces data movement, which the review identifies as the dominant energy cost. Reconfigurable FPGAs offer adaptability for evolving operators and edge constraints, though at lower peak throughput than leading GPUs and ASICs.
Neuromorphic and photonic computing, while promising, are not yet production-ready at frontier scale. Neuromorphic approaches offer event-driven, sparse computation that aligns conceptually with sparsity in LLMs, but toolchains and precision remain limiting. Photonic approaches promise high-band
Why it matters
The review's central analytical claim is that for large language models, the memory system has become the computer. Roofline analysis explains why: the decode phase is bandwidth-bound, the key-value cache can rival model weights in size, and data movement dominates energy. Consequently, architectural gains increasingly come from memory-centric design—higher bandwidth, larger and smarter caches, near-memory and in-memory compute, and reduced data movement—rather than from arithmetic throughput alone.
This reframing has direct implications for system design. Heterogeneous, memory-centric systems that combine GPUs, ASICs, and PIM accelerators can match different phases and workloads to the right substrate. Hardware-algorithm co-design becomes essential: quantization, sparsity, attention variants, and KV-cache management can shift operational intensity and change which roofline regime dominates.
The comparison across compute, memory, energy, programmability, and scalability suggests a portfolio strategy rather than a single winner. GPUs provide flexibility and training strength; ASICs provide scale efficiency for stable workloads; PIM provides a near-term memory-wall response as a complement; FPGAs provide reconfigurability; and neuromorphic and photonic approaches remain longer-horizon options. Future progress depends on hardware-algorithm co-design and heterogeneous, memory-centric systems, with the memory hierarchy—not the arithmetic unit—as the primary design target.
Who should read this
Opening member content…