PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

A local pipeline for turning live property graphs into balanced, executable NL-to-Cypher benchmarks

Suraj Ranganath; Anish Raghavendra· 2026· DOI 10.48550/arXiv.2606.08481

The core problem

Enterprise property graphs are highly heterogeneous: they differ in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. Consequently, a deployment-relevant Text2Cypher benchmark must reflect the questions that users and agents actually ask of that specific graph. However, creating such a benchmark is challenging because schemas and values are unique to each graph, and graph structure evolves over time. Each natural language (NL) query paired with a Cypher query must be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. PIPE-Cypher addresses this gap by providing a local pipeline that automatically generates balanced NL-to-Cypher benchmarks from a live property graph and optional seed queries derived from customer questions, analyst logs, or agent tool calls.

Using local Qwen3.5-9B generation and judging, PIPE-Cypher exported **3,000 accepted FinBench/SNB examples**. The pipeline completed **three audited ablation suites** to isolate the contribution of each component. Judge behavior was calibrated with human labels, and **11 local downstream models** were evaluated on the generated benchmark. The benchmark is deliberately discriminative: zero-shot transfer performance is weak, indicating that models struggle without schema-specific examples. However, a few-shot control shows that schema-specific example banks can substantially help compatible model families. These results demonstrate that PIPE-Cypher produces a challenging and realistic benchmark that reflects the target graph and its workloads.

Enterprise property graphs are highly heterogeneous: they differ in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. Consequently, a deployment-relevant Text2Cypher benchmark must reflect the questions that users and agents actually ask of that specific graph. However, creating such a benchmark is challenging because schemas and values are unique to each graph, and graph structure evolves over time. Each natural language (NL) query paired with a Cypher query must be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. PIPE-Cypher addresses this gap by providing a local pipeline that automatically generates balanced NL-to-Cypher benchmarks from a live property graph and optional seed queries derived from customer questions, analyst logs, or agent tool calls.

PIPE-Cypher integrates several components into a single pipeline. First, **schema profiling** extracts structural and statistical summaries of the property graph. Second, **reverse-query grounding** uses seed queries to anchor generation in real usage patterns. Third, **constrained generation** employs a local LLM (Qwen3.5-9B) to produce candidate NL-query/Cypher pairs under schema and value constraints. Fourth, **deterministic Cypher governance** enforces syntactic and semantic rules. Fifth, **execution validation** runs each Cypher query against the live graph to ensure executability and correctness. Sixth, **redaction** removes sensitive information. Seventh, **diversity controls** balance query types and difficulty levels. Finally, a **calibrated local LLM judge** filters and scores examples. The pipeline is illustrated below:

The weak zero-shot transfer highlights the difficulty of generalizing across enterprise property graphs, where schemas and terminology are unique. The few-shot improvement suggests that providing in-context examples from the same schema can bridge the gap for models that can leverage them. PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads. By automating benchmark generation, it reduces manual effort and enables continuous evaluation as graphs change. The pipeline's local nature ensures data privacy and governance compliance. Future work may extend the approach to other graph query languages and larger model families.

CS practitioners and researchers

Opening member content…