Ilmu Komputer & AI editorial
Open AccessOA2026
PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs
A four-stage LangGraph pipeline with a YAML Corpus Adapter converts policy PDFs into deontic logic and SHACL constraints, achieving 86.9% classification accuracy and registry-swappable domain retargeting.
Ponkrit Kaewsawee; Chaklam Silpasuwanchai; Chutiporn Anutariya· 2026· DOI 10.48550/arXiv.2608.09028
The core problem
Institutional policies remain locked in natural language, while the systems that check compliance demand machine-readable constraints. Bridging this gap is still performed by hand, creating a bottleneck for automated compliance checking. PolicyKG addresses this by closing the loop: it is an LLM pipeline that reads a policy PDF, classifies each sentence as an obligation, permission, or prohibition, lifts the label into first-order deontic logic, and emits SHACL constraints. The pipeline is designed to be retargetable across domains through a Corpus Adapter, a YAML vocabulary registry that grounds LLM predicates in a target ontology. This work evaluates PolicyKG on the Asian Institute of Technology (AIT) Policies and Procedures corpus, comprising 1,663 sentences and 443 rules, and tests domain transfer to GDPR and lease-contract benchmarks.
Innovation
On the AIT corpus, PolicyKG achieves 86.9% deontic classification accuracy (Cohen's kappa = .709). Three annotators independently re-label a 50-item sample and agree at Fleiss' kappa = .844. SHACL shape correctness on a 69-shape subset yields = .866. The FOL path handles 79.2% of rules, with the remainder using the direct NL-to-SHACL fallback. The audit for higher-order constructs found none: an automated regex checklist flagged none, and a first-author pass on the 92 FOL-fallback cases confirmed the same. The exact upper 95% Clopper-Pearson bound on the true higher-order logic rate is 0.67%. Swapping the AIT registry for a GDPR registry raises exact property alignment from 1/15 to 11/15 (Fisher's exact p < .001; Cohen's h = 1.53). On the LexDeMod lease-contract benchmark (N = 200), Macro drops to .370 because lease English uses "shall be entitled" for permission, a vocabulary mismatch that registry swap is meant to fix. Repeated runs produce hash-identical SHACL outputs.
Institutional policies remain locked in natural language, while the systems that check compliance demand machine-readable constraints. Bridging this gap is still performed by hand, creating a bottleneck for automated compliance checking. PolicyKG addresses this by closing the loop: it is an LLM pipeline that reads a policy PDF, classifies each sentence as an obligation, permission, or prohibition, lifts the label into first-order deontic logic, and emits SHACL constraints. The pipeline is designed to be retargetable across domains through a Corpus Adapter, a YAML vocabulary registry that grounds LLM predicates in a target ontology. This work evaluates PolicyKG on the Asian Institute of Technology (AIT) Policies and Procedures corpus, comprising 1,663 sentences and 443 rules, and tests domain transfer to GDPR and lease-contract benchmarks.
PolicyKG runs four stages on a LangGraph state machine with per-stage validators. The stages are: (1) sentence classification into deontic categories, (2) lifting to first-order deontic logic (FOL), (3) SHACL constraint generation, and (4) validation. The Corpus Adapter is a YAML vocabulary registry that maps LLM predicates to a target ontology, enabling retargeting by swapping the registry rather than retraining a model. The FOL path handles 79.2% of rules; the remaining rules go through a direct NL-to-SHACL fallback. The pipeline's architecture is illustrated below:
Why it matters
The results demonstrate that PolicyKG effectively bridges the gap between natural language policies and machine-readable constraints, with high deontic classification accuracy and reliable SHACL generation. The Corpus Adapter's registry-swap mechanism enables domain retargeting without model retraining, as evidenced by the significant improvement in property alignment when moving from AIT to GDPR. However, the drop in Macro on the LexDeMod benchmark highlights the sensitivity of the pipeline to vocabulary mismatches, underscoring the importance of registry alignment. The audit finding that no higher-order constructs were detected in the AIT corpus is specific to this corpus and does not prove FOL sufficiency for institutional policy in general. The exact upper bound of 0.67% on the true HOL rate provides a quantitative measure of uncertainty. The hash-identical outputs across repeated runs indicate deterministic behavior, which is crucial for compliance applications. Future work should extend the audit to more diverse corpora and explore automated registry generation to reduce manual effort in domain adaptation.
Who should read this
CS practitioners and researchers
Opening member content…