Larch: Learned Query Optimization for Semantic Predicates

The integration of Large Language Models (LLMs) into database systems has enabled semantic operators that allow analytical queries over unstructured data such as text, images, and videos. However, these semantic operators incur high inference costs and latencies, making AI SQL queries challenging to apply on large-scale datasets. Their semantic nature leads database engines to treat them as black boxes, hindering optimization. This paper introduces Larch, a framework for optimizing the execution of semantic filters in AI SQL queries. Larch is inspired by two key observations: (i) the high latency of semantic operators leaves significant room for computationally-heavy runtime optimization techniques, and (ii) unstructured data are typically accompanied by semantic information in the form of embeddings, allowing for efficient semantic comparisons between AI_FILTER prompts and data values. Based on these observations, the authors present two Larch variants: Larch-A2C and Larch-Sel.

The authors evaluated Larch across diverse real-world datasets and comprehensive synthetic workloads. Both Larch variants consistently outperformed existing semantic filter optimization techniques in terms of token usage. Specifically, Larch reduced total token cost overhead by 3x to 19x compared to Palimpzest and Quest. The results demonstrate that Larch is robust across diverse workloads, effectively optimizing semantic filter execution. The experiments included various filter types, data modalities, and query complexities, showing that Larch adapts to different scenarios. The token cost reduction is attributed to the learned evaluation order that prioritizes filters with high selectivity and low cost, minimizing expensive LLM calls.