Lightning Fast Matching Dependency Discovery with Desbordante

Optimization techniques for HyMD achieve over 40x average speedup, reaching 170x in some cases, and are now available in the open-source Desbordante profiler with Python integration.

Alexey Shlyonskikh; Michael Sinelnikov; Daniil Nikolaev; Yurii Litvinov; George Chernishev· 2026· DOI 10.48550/arXiv.2607.10771

The core problem

Matching dependencies (MDs) generalize functional dependencies by allowing custom similarity functions to match attribute values. They are essential for data quality tasks such as entity resolution, deduplication, data integration, and schema matching. However, discovering MDs is computationally intensive, which has limited their practical adoption. This work addresses the performance bottleneck by optimizing HyMD, the current state-of-the-art MD discovery algorithm. The authors introduce three key optimizations: a new sampling technique to improve inference from record pairs, a faster generalization lookup technique, and an improved dependency representation. These optimizations target both the inference phase and lattice-related operations. The improved HyMD is implemented in Desbordante, an open-source high-performance data profiler, and is made available with bidirectional Python integration, allowing users to call the C++ implementation from Python and supply custom matching functions.

Experiments demonstrate that the optimizations yield a speedup of more than 40x on average over the state-of-the-art implementation, with some cases reaching a speedup greater than 170x. The evaluation was conducted using the Desbordante implementation. The paper reports these results as evidence of the effectiveness of the proposed techniques. The improved version of HyMD is ready for use and comes with bidirectional Python integration, allowing users to call the C++ algorithm from Python programs while supplying their own custom matching functions.

Matching dependencies (MDs) generalize functional dependencies by allowing custom similarity functions to match attribute values. They are essential for data quality tasks such as entity resolution, deduplication, data integration, and schema matching. However, discovering MDs is computationally intensive, which has limited their practical adoption. This work addresses the performance bottleneck by optimizing HyMD, the current state-of-the-art MD discovery algorithm. The authors introduce three key optimizations: a new sampling technique to improve inference from record pairs, a faster generalization lookup technique, and an improved dependency representation. These optimizations target both the inference phase and lattice-related operations. The improved HyMD is implemented in Desbordante, an open-source high-performance data profiler, and is made available with bidirectional Python integration, allowing users to call the C++ implementation from Python and supply custom matching functions.

The paper describes three main optimizations for HyMD:

The significant speedups achieved by the optimizations make matching dependency discovery more practical for real-world data quality tasks. The combination of a new sampling technique, faster generalization lookup, and improved dependency representation addresses both the inference and lattice operation bottlenecks. The integration into Desbordante, an open-source profiler, and the bidirectional Python interface lower the barrier to adoption, enabling users to leverage custom similarity functions. This work contributes to the broader applicability of MDs in entity resolution, deduplication, data integration, and schema matching. Future work may explore further optimizations or extensions to other dependency types.

CS practitioners and researchers

Opening member content…