Ilmu Komputer & AI editorial
Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systems
The core problem
Enterprise Retrieval-Augmented Generation (RAG) deployments face a critical governance gap. While LLM generation cost is metered per token, the retrieval layer — vector memory, similarity compute, and embedding API calls — remains an unattributed shared cost. This opacity enables invisible cross-subsidization among tenants: heavy retrieval users are effectively subsidized by light ones, and no tenant can be billed or throttled on the basis of its true resource footprint.
The paper presents **Cost-Governed RAG**, an architecture that integrates a codebook-oblivious vector index (**TurboVec**) with a multi-tenant LLM governance gateway. The result is a unified observability stack in which embedding, retrieval, and generation costs are jointly attributable per tenant. The central enabling insight is that TurboVec's deterministic, closed-form memory formula permits near-exact per-tenant retrieval cost calculation — a property unavailable in graph-based indexes whose memory overhead is non-linear and therefore not cleanly separable across tenants.
The work formalizes a **three-layer cost model** spanning embedding, retrieval, and generation, and demonstrates that codebook-oblivious qu
Innovation
The system achieves **99.96% end-to-end cost attribution accuracy** across the 100 simulated tenants. This figure reflects the joint accuracy of embedding, retrieval, and generation attribution under the log-normal tenant size distribution at 10M vectors.
Telemetry overhead is reported at **below 0.04% of query latency**, indicating that unified observability is achievable without materially degrading serving performance.
On infrastructure economics, the architecture reduces retrieval infrastructure cost by **3.1–9.0x** compared to managed vector database services, under the pricing assumptions detailed in Section IV of the source. The range reflects variation across workload and deployment configurations rather than a single point estimate.
The accuracy result is the headline finding: near-exact per-tenant retrieval attribution is not merely approximated but derived from TurboVec's deterministic memory formula. Because is closed-form, the retrieval cost term can be computed directly rather than inferred from sampled telemetry, which is the mechanism underlying the 99.96% figure.
Why it matters
The paper's core argument is architectural rather than algorithmic: attribution quality is a property of the index design, not of the billing layer bolted on top. Codebook-oblivious quantization yields two distinct benefits. First, it makes per-tenant retrieval cost **deterministic**, because memory does not depend on a shared learned codebook whose occupancy is entangled across tenants. Second, it removes the **shared-codebook leakage surface** present in trained quantizers, where a common codebook can act as a side channel between tenants.
The authors are careful to bound the second claim. The leakage-removal observation is described as **exploratory** and subject to the limitations in Section VII. Readers should treat the security implication as a hypothesis motivating further work rather than a demonstrated guarantee.
Similarly, the 3.1–9.0x cost reduction is contingent on the pricing assumptions in Section IV; it is a comparison against managed vector database services under stated conditions, not a universal constant. The evaluation is simulation-based across 100 tenants and 10M vectors, so external validity to larger or differently skewed production fleets remains to be established.
Taken together, the work reframes multi-tenant RAG governance: the retrieval layer can be metered with the same rigor as token-based generation, provided the vector index exposes a separable, closed-form cost structure. This positions cost attribution as a first-class design constraint for enterprise RAG infrastructure.
Who should read this
Opening member content…