Ilmu Komputer & AI editorial
Scout: Scalable Document Extraction via Data Similarity
The core problem
Extracting values from large document collections underpins data analysis across many domains, from financial filings to scientific literature. Frontier LLMs extract such values accurately, but processing an entire collection with one is prohibitively costly. This cost is largely avoidable: real-world collections exhibit rich similarity, so for the same query over similar documents, the answer tends to recur in similar locations. An LLM need only read that small span, not the whole document.
Prior methods that exploit this similarity fall short. They either assume a rigid document structure, or assume the answer is a set of substrings of the input and use an LLM-generated program to return it directly. Even a frontier agent fails to generate effective programs to directly locate the answer's span, because the search space is large and programs learned from a small sample tend to overfit.
The authors present Scout, a tool that generates accurate and cost-effective programs—called rules—to extract data at scale. From a few sampled documents, Scout generates a broad rule set and refines it by selecting a Pareto-optimal subset with low cost without sacrificing accuracy. The paper pro
Innovation
Experiments on six real-world datasets show that Scout matches the accuracy of the strongest baseline—a frontier LLM agent that reads each full document—while being 61x to over 1000x cheaper on a collection of 1,000 documents. Scout is also 61% more accurate than the strongest prior program-based approach.
The cost savings stem from Scout's ability to read only small spans of each document rather than the full text. The accuracy parity with the frontier LLM agent demonstrates that the refined rule set captures the necessary extraction logic without sacrificing correctness.
The 61% accuracy improvement over prior program-based methods highlights the effectiveness of the refinement and cascade strategies, which overcome the overfitting and rigidity issues of earlier approaches.
Why it matters
Scout's key insight is that document similarity can be exploited to reduce LLM calls dramatically. By generating a broad rule set from a few samples and refining it to a Pareto-optimal subset, Scout achieves a favorable accuracy-cost trade-off. The NP-hardness proof for rule refinement justifies the use of a greedy approximation, which provides a provable guarantee.
The cascade strategy addresses the realistic scenario of partially similar collections. By clustering documents without an LLM and falling back to the unrefined rule set when needed, Scout maintains accuracy even when similarity is not global. This makes Scout applicable to heterogeneous document collections where a single rigid structure cannot be assumed.
Compared to prior work, Scout avoids the pitfalls of assuming rigid document structure or relying on LLM-generated programs that overfit. The empirical results—61x to over 1000x cost reduction with equal accuracy—suggest that Scout is a practical solution for large-scale document extraction. Future work could explore adaptive clustering and rule generation for streaming collections.
Who should read this
Opening member content…