Ilmu Komputer & AI editorial
Discovery-Driven Integration of Disjoint Tables via Text
The core problem
Integrating heterogeneous datasets within data lakes is a critical challenge, particularly for semantically related tables that lack the explicit attributes needed to be joined. The authors study **Discovery-Driven Integration**, where the relevant sources and their missing relational structure must be discovered before integration. In this setting, unstructured text provides the evidence that connects otherwise disjoint tables.
The fundamental challenge is to discover the relationships at a fine-grained level that connect individual rows from different tables through specific sentences. The paper formalizes this task as **Text-Mediated Join Path Discovery** and proposes a horizontal bidirectional cross-attention architecture called **LOKI** (Latent-space Optimization for Knowledge Integration) that learns contextualized representations of table rows and sentences.
Existing multi-modal discovery methods largely retrieve coarse-grained column-text associations, whereas integration systems assume supplied row-text links, schemas, or queries. LOKI instead transforms these implicit associations into explicit, interpretable join paths, organizes them into relation-consistent groups, a
Innovation
Comprehensive evaluations on real-world benchmarks demonstrate that LOKI consistently outperforms state-of-the-art multi-modal data discovery approaches. The key quantitative results are:
- **0.982 macro typed-pair precision** when materializing typed integrated tables.
- **Up to 40× cheaper in LLM API cost** than direct prompting.
The authors report that LOKI's fine-grained row-sentence associations enable it to discover join paths that coarse-grained column-text methods miss. The materialized tables include sentence-level provenance, allowing users to trace each integrated row back to the specific sentences that justified the join.
Why it matters
The paper positions LOKI as a bridge between multi-modal data discovery and integration systems. While existing discovery methods retrieve coarse-grained column-text associations, and integration systems assume supplied row-text links, schemas, or queries, LOKI discovers fine-grained row-sentence links and turns them into explicit, interpretable join paths.
The use of a global contrastive objective is a notable design choice: it avoids the need for expensive local supervision (e.g., manually labeled row-sentence pairs) while still yielding fine-grained associations. This makes the approach scalable and cost-effective, as evidenced by the 40× reduction in LLM API cost compared to direct prompting.
The 0.982 macro typed-pair precision indicates that the materialized integrated tables are highly accurate. The sentence-level provenance adds interpretability, which is crucial for data lake integration where trust and traceability matter.
Potential limitations include reliance on the quality of the text evidence and the assumption that semantically related tables can be connected through sentences. Future work could explore extending LOKI to handle noisy or sparse text, and to integrate with downstream analytics pipelines.
Who should read this
Opening member content…