Ilmu Komputer & AI editorial
Open AccessOA2026
Decoupled I/O-Dominant Pipelines for Large-Scale Whole-Slide Image Embedding Extraction
A three-stage, I/O-aware architecture that reframes WSI embedding extraction as a data-centric systems problem
Mayanka Chandrashekar; Xi Zhang; Ethan Seefried; Tirthankar Ghosal; John Gounley; Heidi Hanson· 2026· DOI 10.48550/arXiv.2608.27278
The core problem
Whole-slide images (WSIs) are central to computational pathology, yet their gigapixel scale makes direct foundation model inference impractical. Patch-based processing has therefore become the practical unit of computation, but at scale the generation and handling of massive patch sets introduces significant I/O and orchestration overhead that often dominates end-to-end performance. The authors argue that WSI embedding extraction is not a purely compute-bound workload; rather, it is a data-centric systems problem in which data movement, staging, and persistence are first-order concerns. This motivates a decoupled, I/O-aware pipeline that isolates data movement from compute, enabling efficient patch delivery and scalable multi-node inference with minimal communication. The resulting system produces a distributed vector database where embeddings are persistently coupled with rich metadata (patient, slide, and patch attributes), enabling efficient filtering, retrieval, and downstream reuse for tasks such as retrieval, classification, and few-shot learning, particularly in low-resource environments.
Innovation
The authors demonstrate that decoupling I/O, computation, and ingestion enables high-throughput WSI embedding extraction at scale. By characterizing the scaling envelope, they show that storage dominates beyond moderate concurrency. This finding is significant because it contradicts the common assumption that embedding extraction is compute-bound. Instead, the bottleneck shifts to data movement and persistence as concurrency increases. The resulting distributed vector database persistently couples embeddings with rich metadata (patient, slide, and patch attributes), enabling efficient filtering, retrieval, and downstream reuse. The representation database is compact and reusable for tasks such as retrieval, classification, and few-shot learning, particularly benefiting low-resource environments where recomputation is costly. The system achieves scalable multi-node inference with minimal communication, validating the decoupled design.
Whole-slide images (WSIs) are central to computational pathology, yet their gigapixel scale makes direct foundation model inference impractical. Patch-based processing has therefore become the practical unit of computation, but at scale the generation and handling of massive patch sets introduces significant I/O and orchestration overhead that often dominates end-to-end performance. The authors argue that WSI embedding extraction is not a purely compute-bound workload; rather, it is a data-centric systems problem in which data movement, staging, and persistence are first-order concerns. This motivates a decoupled, I/O-aware pipeline that isolates data movement from compute, enabling efficient patch delivery and scalable multi-node inference with minimal communication. The resulting system produces a distributed vector database where embeddings are persistently coupled with rich metadata (patient, slide, and patch attributes), enabling efficient filtering, retrieval, and downstream reuse for tasks such as retrieval, classification, and few-shot learning, particularly in low-resource environments.
The proposed pipeline decomposes the workflow into three distinct stages:
Why it matters
The key insight is that WSI embedding extraction should be treated as a data-centric systems problem rather than a purely compute-bound workload. The decoupled pipeline isolates data movement from compute, allowing each stage to be optimized independently. This architectural choice enables efficient patch delivery and scalable multi-node inference with minimal communication. The persistent coupling of embeddings with metadata in a sharded vector database creates a reusable representation database that supports retrieval, classification, and few-shot learning without recomputation. This is particularly valuable in low-resource environments where compute and storage are constrained. The scaling analysis reveals that storage dominates beyond moderate concurrency, suggesting that future optimizations should focus on I/O-aware staging, compression, and sharding strategies. The work also highlights the importance of orchestration overhead in large-scale pipelines. By reframing the problem, the authors provide a blueprint for building efficient, scalable WSI embedding extraction systems that can be deployed in distributed environments. The taxonomy candidates (Architecture, Cybersecurity, Network, Cryptography) are not directly addressed in the source, but the systems architecture aspects align with distributed computing and data engineering.
Who should read this
CS practitioners and researchers
Opening member content…