LyndHub
Jadwal Sholat

Memuat jadwal sholat…

← All journalsSearch index

Ilmu Komputer & AI editorial

Open AccessOA2026

The Nuts and Bolts of Natural Language to SQL Translation: A Systematic Analysis of Model Pipeline Optimisation Approaches and their Interactions

An ablation and Shapley analysis of NatSQL, synthetic data fine-tuning, preprocessing, and reranking across SmBoP and RASAT backbones
Filip Klubicka; Vasudevan Nedumpozhimana; Sneha Rautmare; Bora Caglayan; Mingxue Wang; John D. Kelleher· 2026· DOI 10.48550/arXiv.2607.10911

The core problem

Natural Language to SQL (NL2SQL) translation remains an open problem despite the rise of large language models. The authors aim to inspire more lightweight models by exploring interactions between several NL2SQL pipeline extensions. They integrate the NatSQL intermediate representation, add a preprocessing step, apply fine-tuning with synthetic data, and develop a novel reranker model to improve SQL selection in the final beam. The core research question is how these components interact with each other and with two backbone architectures: SmBoP and RASAT. The study performs an ablation study supplemented by a Shapley analysis to quantify the contribution of each component and their interactions. The key finding is that simply combining all extensions does not lead to the best results; instead, the impact of each component depends on its interactions with the baseline system and with other components.

Innovation

The ablation study and Shapley analysis reveal that the impact of each component is highly dependent on the backbone architecture and on the presence of other components. Key findings include:

- **No universal best combination**: Combining all four extensions does not yield the best performance. Some components may hurt performance when combined with certain backbones or other components.
- **Interaction effects**: The Shapley analysis shows significant interaction effects between components. For example, the benefit of the reranker may depend on whether NatSQL is used, or the preprocessing step may only help when synthetic data fine-tuning is applied.
- **Backbone dependency**: The optimal set of extensions differs between SmBoP and RASAT. For instance, a component that improves SmBoP may degrade RASAT, and vice versa.
- **Lightweight models**: The authors suggest that understanding these interactions can guide the development of more lightweight models by selecting only the most beneficial components for a given backbone.

The abstract does not provide specific numbers, but the qualitative conclusion is that the interactions are complex and must be considered when designing NL2S

Natural Language to SQL (NL2SQL) translation remains an open problem despite the rise of large language models. The authors aim to inspire more lightweight models by exploring interactions between several NL2SQL pipeline extensions. They integrate the NatSQL intermediate representation, add a preprocessing step, apply fine-tuning with synthetic data, and develop a novel reranker model to improve SQL selection in the final beam. The core research question is how these components interact with each other and with two backbone architectures: SmBoP and RASAT. The study performs an ablation study supplemented by a Shapley analysis to quantify the contribution of each component and their interactions. The key finding is that simply combining all extensions does not lead to the best results; instead, the impact of each component depends on its interactions with the baseline system and with other components.
The authors evaluate four pipeline extensions on two backbone architectures, SmBoP and RASAT. The extensions are:

Why it matters

The study highlights the importance of systematic analysis of component interactions in NL2SQL pipelines. The finding that combining all extensions is suboptimal challenges the common assumption that more components always lead to better performance. Instead, the authors advocate for a nuanced approach that considers the backbone architecture and the specific interactions between components.

The Shapley analysis provides a principled way to attribute performance gains to individual components and their interactions. This can inform the design of more efficient pipelines by identifying redundant or harmful components. For example, if a component has a negative Shapley value in the presence of another component, it may be omitted to reduce computational cost without sacrificing performance.

The authors also emphasize the development of lightweight models. By understanding which components are most beneficial for a given backbone, practitioners can avoid unnecessary complexity. This is particularly relevant in resource-constrained environments where large language models may not be feasible.

Future work could extend this analysis to other backbones and datasets, and explore additional pipeline extensions. The methodology of combining ablation with Shapley analysis is general and could be applied to other NLP tasks.

A Mermaid diagram illustrating the pipeline and its components is shown below:

This diagram shows how the components interact: preprocessing feeds into the backbone, NatSQL is used as an intermediate representation, synthetic data is used for fine-tuning, and the reranker selects from the beam.

Who should read this

CS practitioners and researchers

Opening member content…