InSituANN: Revisiting IVF for PCIe-Efficient Billion-Scale Vector Search

An IVF-based ANNS engine that keeps base vectors in host memory and performs fine search in situ, eliminating PCIe transfers of high-dimensional vectors while using the GPU only for compact routing and optional pruning.

Yuemeng Xu; Zongxi Liu; Junyu Long; Yiming Huang; Jiarui Guo; Yangyujia Wang; Jiachen Xu; Dongyuan Yu; Zongwei Lv; Tong Yang· 2026· DOI 10.48550/arXiv.2608.08751

The core problem

Approximate nearest neighbor search (ANNS) over billion-scale vector datasets has become a foundational operator for modern retrieval systems, powering large-scale recommendation, semantic search, and LLM/RAG workloads. Although GPUs offer massive parallelism and high-bandwidth memory for batched vector search, their limited VRAM capacity makes fully GPU-resident billion-scale indexes difficult to deploy. In CPU-GPU heterogeneous designs, keeping the base vectors in host memory avoids this capacity limit, but naively offloading fine search to the GPU introduces a new bottleneck: large volumes of base-vector data must be streamed over PCIe. The paper presents InSituANN, an IVF-based ANNS engine that enables billion-scale vector search on a single commodity GPU. InSituANN keeps original base vectors in host memory, performs fine search in situ, and uses the GPU for compact routing and optional pruning. As a result, query processing avoids PCIe transfers of high-dimensional base vectors while retaining the simplicity of IVF. Beyond query performance, the authors further design an ultra-fast IVF construction path for InSituANN. On SIFT-1B, InSituANN builds the IVF index in 5.2 minutes,

Innovation

The paper evaluates InSituANN on billion-scale datasets, including SIFT-1B and DEEP-1B. At matched recall, InSituANN improves end-to-end throughput by 104.9x-4298.2x over the PCIe-bound Rummy baseline and by 2.4x-4.6x over DiskANN on SIFT-1B and DEEP-1B. The index construction time on SIFT-1B is 5.2 minutes, about 350x faster than the measured 30.4-hour HNSW build. In addition to throughput gains, InSituANN achieves strong recall-throughput trade-offs and requires lower index space than graph-based alternatives. These results demonstrate that billion-scale retrieval can be made practical on cost-efficient hardware by revisiting IVF and carefully orchestrating CPU-GPU collaboration to avoid PCIe transfers of high-dimensional base vectors.

Approximate nearest neighbor search (ANNS) over billion-scale vector datasets has become a foundational operator for modern retrieval systems, powering large-scale recommendation, semantic search, and LLM/RAG workloads. Although GPUs offer massive parallelism and high-bandwidth memory for batched vector search, their limited VRAM capacity makes fully GPU-resident billion-scale indexes difficult to deploy. In CPU-GPU heterogeneous designs, keeping the base vectors in host memory avoids this capacity limit, but naively offloading fine search to the GPU introduces a new bottleneck: large volumes of base-vector data must be streamed over PCIe. The paper presents InSituANN, an IVF-based ANNS engine that enables billion-scale vector search on a single commodity GPU. InSituANN keeps original base vectors in host memory, performs fine search in situ, and uses the GPU for compact routing and optional pruning. As a result, query processing avoids PCIe transfers of high-dimensional base vectors while retaining the simplicity of IVF. Beyond query performance, the authors further design an ultra-fast IVF construction path for InSituANN. On SIFT-1B, InSituANN builds the IVF index in 5.2 minutes, about 350x faster than the measured 30.4-hour HNSW build. At matched recall on billion-scale datasets, InSituANN improves end-to-end throughput by 104.9x-4298.2x over the PCIe-bound Rummy baseline and by 2.4x-4.6x over DiskANN on SIFT-1B and DEEP-1B. Together with strong recall-throughput trade-offs and lower index space than graph-based alternatives, these gains make billion-scale retrieval practical on cost-efficient hardware. The code is open-sourced at https://github.com/mindtravel/InSituANN-OpenSource.

InSituANN is built on the inverted file (IVF) index, which partitions the vector space into clusters via a coarse quantizer and stores base vectors in inverted lists. The key design principle is to keep the original base vectors in host memory and perform fine search in situ, i.e., on the CPU side, while using the GPU only for compact routing and optional pruning. This avoids the PCIe transfer of high-dimensional base vectors that would otherwise dominate the query latency in CPU-GPU heterogeneous designs.

Why it matters

The central insight of InSituANN is that the PCIe bottleneck in CPU-GPU heterogeneous ANNS is not fundamental but a consequence of naively offloading fine search to the GPU. By keeping base vectors in host memory and performing fine search in situ, InSituANN eliminates the transfer of high-dimensional vectors over PCIe, which is the dominant cost in prior systems. The GPU is then used only for compact routing and optional pruning, where its parallelism and high-bandwidth memory provide the most benefit without incurring large data movement. This design retains the simplicity of IVF, making the system easy to deploy and tune. The ultra-fast construction path further lowers the barrier to adopting billion-scale indexes, as building an HNSW index can take tens of hours. The reported throughput improvements over Rummy and DiskANN, combined with lower index space than graph-based alternatives, suggest that IVF-based designs remain highly competitive when the data movement is properly managed. The open-source release at https://github.com/mindtravel/InSituANN-OpenSource enables reproducibility and further research. Future work may explore adaptive pruning strategies and integration with other index structures to further improve recall-throughput trade-offs.

Who should read this

CS practitioners and researchers

Opening member content…