Towards Behavior Tree-Guided Vulnerability Detection with Lightweight LLMs

A compact intermediate representation for token-constrained local models

Enna Basic; Alberto Giaretta· 2026· DOI 10.48550/arXiv.2609.01758

The core problem

Large Language Models (LLMs) are increasingly applied to software vulnerability detection, yet their effectiveness hinges on how source code is represented in the input. Most prompting approaches feed source code in its original form, while some works propose structured representations. Abstract Syntax Trees (ASTs) are popular but verbose: they increase input size relative to source code, making them hard to fit within some LLMs' context windows. This paper investigates Behavior Trees (BTs) as an alternative intermediate representation. BTs encode control flow, conditions, and executable actions more compactly than ASTs, making them a natural candidate when token count is a constraint. The authors propose a preprocessing stage that parses Java source code into ASTs and then converts them into BT representations. They compare vulnerability detection performance across 460 Java samples from the Juliet Java test suite using three input representations: raw source code, AST, and BT. All experiments use a single quantized local LLM, Mistral Small 3.2 24B (Q4_K_M). The central research question is whether BTs can provide a compact and useful structured representation for vulnerability de

The experimental results show distinct trade-offs among the three representations. On short code samples, BT representations improve recall compared to raw source code and ASTs. However, raw source code achieves higher precision. On longer samples, BTs improve overall performance over the original representation and fit within the context window, whereas many ASTs exceed the context limit. Specifically, the authors report that ASTs are the most verbose, often exceeding the context window of the quantized Mistral Small 3.2 24B model. BTs are more compact than ASTs and, in many cases, more compact than raw source code, allowing them to fit within the context window. The following table summarizes the qualitative findings: | Representation | Precision | Recall | Context Fit | |----------------|-----------|--------|-------------| | Raw Source | High | Low | Good | | AST | Medium | Medium | Poor (long samples) | | BT | Medium | High | Good | These results suggest that BTs are particularly beneficial when token count is a constraint, such as in locally deployable, quantized LLMs.

Large Language Models (LLMs) are increasingly applied to software vulnerability detection, yet their effectiveness hinges on how source code is represented in the input. Most prompting approaches feed source code in its original form, while some works propose structured representations. Abstract Syntax Trees (ASTs) are popular but verbose: they increase input size relative to source code, making them hard to fit within some LLMs' context windows. This paper investigates Behavior Trees (BTs) as an alternative intermediate representation. BTs encode control flow, conditions, and executable actions more compactly than ASTs, making them a natural candidate when token count is a constraint. The authors propose a preprocessing stage that parses Java source code into ASTs and then converts them into BT representations. They compare vulnerability detection performance across 460 Java samples from the Juliet Java test suite using three input representations: raw source code, AST, and BT. All experiments use a single quantized local LLM, Mistral Small 3.2 24B (Q4_K_M). The central research question is whether BTs can provide a compact and useful structured representation for vulnerability detection with quantized, locally deployable LLMs.

The proposed pipeline consists of two main stages: (1) parsing Java source code into an Abstract Syntax Tree (AST), and (2) converting the AST into a Behavior Tree (BT) representation. The BT encodes control flow, conditions, and executable actions in a hierarchical, compact format. For evaluation, the authors use 460 Java samples from the Juliet Java test suite, a standard benchmark for vulnerability detection. Each sample is processed into three input representations: raw source code, AST, and BT. All experiments are conducted with a single quantized local LLM: Mistral Small 3.2 24B (Q4_K_M). The model is prompted to classify each sample as vulnerable or non-vulnerable. Performance is measured using precision, recall, and overall accuracy. The token count of each representation is also recorded to assess context window fit. The conversion process can be summarized as:

The findings indicate that Behavior Trees offer a promising intermediate representation for vulnerability detection with lightweight LLMs. The compactness of BTs addresses the context window limitation of ASTs, enabling the model to process longer code samples without truncation. The improved recall on short samples suggests that BTs help the model identify vulnerable patterns that might be missed when using raw source code. However, the higher precision of raw source code implies that BTs may introduce some noise or lose fine-grained details that are important for avoiding false positives. The trade-off between precision and recall should be considered based on the application: in security-critical contexts, higher recall may be preferred to avoid missing vulnerabilities, while in other contexts, precision may be more important to reduce false alarms. The authors note that the conversion from AST to BT is lossy, which may explain the precision drop. Future work could explore hybrid representations or adaptive selection of representation based on code length. The study is limited to Java and a single LLM; generalization to other languages and models remains to be tested. Overall, BTs provide a compact and useful structured representation for vulnerability detection with quantized, locally deployable LLMs.

CS practitioners and researchers

Opening member content…