Ilmu Komputer & AI editorial
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
The core problem
Semantic filtering—evaluating a natural-language yes/no predicate over a document corpus under an accuracy target—is a cornerstone of LLM-based data processing. The naive approach of calling the LLM on every document (the oracle) is prohibitively expensive. Cascades pair the oracle with a fast proxy to reduce cost, but current deployments suffer from four key limitations:
1. **Single-representation commitment**: Each cascade family (model-free clustering, prebuilt small-LLM proxies, online-trained proxies) commits to one representation and pipeline, winning only on a narrow query regime.
2. **Weak online proxies**: The strongest online proxy uses a custom training scheme on a bi-encoder over dense embeddings, missing token-level evidence needed for richer predicates.
3. **Binary label waste**: Proxies are trained against binary yes/no labels, discarding the LLM's per-document confidence at boundary documents where learning matters most.
4. **Uniform calibration**: Existing calibrations add a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost.
This work addresses these limitations by adaptively composing families, replaci
Innovation
The authors evaluate their method on three 10K-document corpora at a 90% accuracy target. Key results:
- **Speedup**: The proposed method is 1.6–2.0× faster than the best prior method per corpus.
- **Accuracy**: It meets the 90% accuracy target on 95% of queries.
- **Headroom**: The BER-derived lower bound indicates a further ~4–20× of headroom for future work.
These results demonstrate that adaptive composition and soft-label training significantly reduce oracle calls while maintaining accuracy. The speedup varies by corpus, reflecting the method's ability to adapt to different query regimes.
2. **Weak online proxies**: The strongest online proxy uses a custom training scheme on a bi-encoder over dense embeddings, missing token-level evidence needed for richer predicates.
3. **Binary label waste**: Proxies are trained against binary yes/no labels, discarding the LLM's per-document confidence at boundary documents where learning matters most.
4. **Uniform calibration**: Existing calibrations add a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost.
Why it matters
The paper's contributions are threefold:
1. **Unified framework**: It provides a unified view of cascade families, showing that each is a special case of the proposed adaptive composition.
2. **Novel use of oracle confidence**: The oracle's per-document confidence is used as a difficulty compass, a lower bound estimator, and a soft training label—three distinct purposes that were previously unexplored.
3. **Practical impact**: The method achieves substantial speedups without sacrificing accuracy, making LLM-based semantic filtering more practical for large-scale data processing.
Limitations include the reliance on a small probe set for difficulty estimation and the assumption that the oracle's confidence is well-calibrated. Future work could explore more sophisticated calibration techniques and extend the framework to multi-label predicates. The BER-derived lower bound suggests that further optimizations could yield up to 20× additional speedup, motivating continued research in this area.
Who should read this
Opening member content…