Ilmu Komputer & AI editorial
Open AccessOA2026
OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows
A distributed orchestration layer that treats RAG stages as resource-aware operators, cutting non-model overhead by up to 20.7% without touching the LLM decoding kernel.
Arup Kumar Sarker; Mills Staylor; Aymen Alsaadi; Gregor von Laszewski; Shantenu Jha; Geoffrey Fox· 2026· DOI 10.48550/arXiv.2608.08340
The core problem
Agentic retrieval-augmented generation (RAG) systems chain together preprocessing, embedding, retrieval, memory access, context construction, generation, and vector-index updates. While LLM decoding is GPU-bound, the surrounding orchestration layer frequently becomes the bottleneck through serialization overhead, fragmented scheduling, inefficient batching, and CPU–GPU pipeline stalls. Existing frameworks such as LangChain, LangGraph, CrewAI, and AutoGen offer flexible control flow, and distributed runtimes like Ray provide scalable task parallelism, but neither exposes RAG stages as resource-aware operators with deterministic execution semantics. This gap motivates OpRAG, a resource-deterministic distributed runtime for GPU-backed multi-stage RAG workflows. The central hypothesis is that optimizing the distributed orchestration layer can substantially improve end-to-end performance without modifying the LLM decoding kernel. OpRAG is evaluated with Llama3-8B and Mistral-7B under FlashAttention 2, BF16 execution, and 32K RAG chunks, targeting both end-to-end GPU pipeline throughput and Higress-style query serving latency.
Innovation
In end-to-end GPU pipeline experiments, OpRAG improves over the nearest competitor by 16.16% for Llama3-8B and 15.66% for Mistral-7B. Against RayScalableRAG, the gains are 20.57% and 20.71%, respectively. Compared to the best framework baseline among LangChain, LangGraph, CrewAI, and AutoGen, OpRAG is 17.77% faster for Llama3-8B and 17.48% faster for Mistral-7B. In Higress-style query serving, OpRAG reduces hybrid retrieval latency by 59.20–59.62% and generation-scenario latency by 52.48–53.55%, while preserving 100% Recall@5. These results demonstrate that substantial performance improvements are achievable purely through orchestration-layer optimizations, without altering the LLM decoding kernel. The consistency of gains across two model families and two serving scenarios indicates that the bottlenecks addressed—serialization, scheduling fragmentation, batching inefficiency, and CPU–GPU stalls—are general rather than model-specific.
Agentic retrieval-augmented generation (RAG) systems chain together preprocessing, embedding, retrieval, memory access, context construction, generation, and vector-index updates. While LLM decoding is GPU-bound, the surrounding orchestration layer frequently becomes the bottleneck through serialization overhead, fragmented scheduling, inefficient batching, and CPU–GPU pipeline stalls. Existing frameworks such as LangChain, LangGraph, CrewAI, and AutoGen offer flexible control flow, and distributed runtimes like Ray provide scalable task parallelism, but neither exposes RAG stages as resource-aware operators with deterministic execution semantics. This gap motivates OpRAG, a resource-deterministic distributed runtime for GPU-backed multi-stage RAG workflows. The central hypothesis is that optimizing the distributed orchestration layer can substantially improve end-to-end performance without modifying the LLM decoding kernel. OpRAG is evaluated with Llama3-8B and Mistral-7B under FlashAttention 2, BF16 execution, and 32K RAG chunks, targeting both end-to-end GPU pipeline throughput and Higress-style query serving latency.
OpRAG models embedding, retrieval, reasoning, memory, and upsert as first-class operators and lowers them into communication-aware execution graphs. The runtime combines several systems techniques: an Arrow zero-copy data plane to eliminate serialization copies, persistent workers to amortize process startup, bounded queues for backpressure and memory safety, CPU tokenizer prefetching to overlap tokenization with GPU work, batched GPU embedding to maximize device utilization, and overlapped retrieval/generation execution to hide retrieval latency behind decoding. These mechanisms are orchestrated to reduce non-model overhead around LLM inference while preserving deterministic execution semantics. The design is evaluated against two classes of baselines: distributed RAG runtimes (RayScalableRAG) and agentic frameworks (LangChain, LangGraph, CrewAI, AutoGen). Experiments use Llama3-8B and Mistral-7B with FlashAttention 2, BF16 precision, and 32K RAG chunks. The evaluation covers end-to-end GPU pipeline throughput and Higress-style query serving, measuring hybrid retrieval latency, generation-scenario latency, and Recall@5. The operator graph can be represented as a directed acyclic graph where nodes are resource-annotated operators and edges carry Arrow-backed tensors:
Why it matters
The results validate the core thesis that the orchestration layer is a first-order performance concern in GPU-backed multi-stage RAG. By exposing RAG stages as resource-aware operators with deterministic execution semantics, OpRAG enables systematic optimization of communication and scheduling that is orthogonal to model-level improvements. The Arrow zero-copy data plane and persistent workers directly attack serialization and startup overhead, while bounded queues and CPU tokenizer prefetching mitigate pipeline stalls. Batched GPU embedding and overlapped retrieval/generation execution improve device utilization and hide latency. The 100% Recall@5 preservation confirms that these optimizations do not trade retrieval quality for speed. A key implication is that practitioners can adopt OpRAG as a drop-in runtime around existing LLM serving stacks, gaining double-digit throughput improvements without retraining or kernel modifications. Limitations include evaluation on two model families and a single chunk size (32K); future work should explore broader model coverage, dynamic resource allocation, and multi-tenant scheduling. The taxonomy candidates—Architecture, Cybersecurity, Network, Cryptography—suggest potential extensions to secure multi-party RAG and network-aware deployment, though the present work focuses on performance and determinism.
Who should read this
CS practitioners and researchers
Opening member content…