TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

Evaluating open-weight LLMs served locally via vLLM against DeepLog across four log datasets, with zero-shot, few-shot, and RAG prompting

Rohit Patel; Susil Kumar Mohanty; Jeenal Chaudhary· 2026· DOI 10.48550/arXiv.2609.14762

The core problem

Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk, network latency, and per-query cost that scale poorly with production log volumes. This motivates on-premise deployment of open-weight LLMs. TriCalRAG addresses this by providing a benchmark that evaluates open-weight LLMs served locally via vLLM on a single high-memory workstation GPU (NVIDIA RTX PRO 6000, 96GB) against a classical LSTM-based log anomaly detector (DeepLog). The benchmark spans four real, publicly available log datasets: BGL, HDFS, Thunderbird, and OpenStack. The study evaluates two open-weight models (Qwen2.5-14B, Mistral-Small) under three prompting strategies: zero-shot, few-shot, and retrieval-augmented generation (RAG) over a labeled incident history. Reported metrics include accuracy, precision/recall, and F_{1} with bootstrap 95% confidence intervals across 3 random seeds, alongside throughput and VRAM footprint. The central research question is whether RAG can improve both accuracy and calibration stability for on-premise LLM-based RCA, and how model choice interacts with deployment priorities such as peak accu

The results show that RAG not only improves mean F_{1} by 0.10-0.27 over zero-shot prompting but, more importantly, substantially stabilizes model calibration. Zero-shot prompting drives both models toward near-degenerate behavior, predicting "anomaly" on up to 100% of incidents on some datasets. In contrast, RAG keeps predicted-positive rates close to the true class balance in the majority of configurations. Mistral-Small achieves higher macro-averaged F_{1} than Qwen2.5-14B (0.644 vs. 0.560) but exhibits calibration failures in more configurations (7 vs. 5 of 12), while running at roughly half the throughput. This indicates that the better model choice depends on whether a deployment prioritizes peak accuracy or predictable behavior across prompting conditions. Ablations show that batching scales throughput 41 times on a single card and that 4-bit quantization reduces latency by 20% with no measurable accuracy loss. The following table summarizes key findings: | Model | Strategy | Mean F_{1} | Calibration Failures | Throughput (relative) | |-------|----------|---------|----------------------|-----------------------| | Qwen2.5-14B | Zero-shot | Lower | 5/12 | Higher | | Qwen2.5-14B | RAG

Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk, network latency, and per-query cost that scale poorly with production log volumes. This motivates on-premise deployment of open-weight LLMs. TriCalRAG addresses this by providing a benchmark that evaluates open-weight LLMs served locally via vLLM on a single high-memory workstation GPU (NVIDIA RTX PRO 6000, 96GB) against a classical LSTM-based log anomaly detector (DeepLog). The benchmark spans four real, publicly available log datasets: BGL, HDFS, Thunderbird, and OpenStack. The study evaluates two open-weight models (Qwen2.5-14B, Mistral-Small) under three prompting strategies: zero-shot, few-shot, and retrieval-augmented generation (RAG) over a labeled incident history. Reported metrics include accuracy, precision/recall, and F_{1} with bootstrap 95% confidence intervals across 3 random seeds, alongside throughput and VRAM footprint. The central research question is whether RAG can improve both accuracy and calibration stability for on-premise LLM-based RCA, and how model choice interacts with deployment priorities such as peak accuracy versus predictable behavior.

The TriCalRAG benchmark is designed to systematically compare prompting strategies and models under controlled conditions. The experimental setup uses a single NVIDIA RTX PRO 6000 GPU with 96GB memory, serving open-weight models locally via vLLM. Two models are evaluated: Qwen2.5-14B and Mistral-Small. Three prompting strategies are applied: zero-shot, few-shot, and retrieval-augmented generation (RAG) over a labeled incident history. The RAG strategy retrieves similar past incidents from a labeled history to augment the prompt. The baseline is DeepLog, a classical LSTM-based log anomaly detector. Four real, publicly available log datasets are used: BGL, HDFS, Thunderbird, and OpenStack. Evaluation metrics include accuracy, precision, recall, and F_{1} score, with bootstrap 95% confidence intervals computed across 3 random seeds. Additionally, throughput (queries per second) and VRAM footprint are measured. Ablations study the effect of batching and 4-bit quantization on throughput and latency. The benchmark harness, dataset splits, and evaluation code are released to support reproducible on-premise AIOps research. The overall architecture can be represented as follows:

The findings highlight a critical trade-off in on-premise LLM-based RCA: while Mistral-Small achieves higher peak F_{1}, its higher rate of calibration failures makes it less reliable across diverse prompting conditions. Qwen2.5-14B, despite lower mean F_{1}, exhibits more stable calibration, suggesting it may be preferable for deployments requiring consistent behavior. The substantial improvement in calibration stability with RAG is a key contribution, as it mitigates the near-degenerate behavior observed in zero-shot prompting. The ablation results demonstrate that batching and quantization can significantly enhance throughput and reduce latency without sacrificing accuracy, making on-premise deployment more feasible. However, the study is limited to two models and four datasets; future work should explore a broader range of models and log types. The release of the benchmark harness and evaluation code supports reproducibility and further research. Overall, TriCalRAG provides a rigorous framework for evaluating on-premise LLM-based RCA, emphasizing the importance of calibration alongside accuracy. The trade-off between accuracy and calibration stability can be formalized as a multi-objective optimization problem: θ max (α \cdot F 1 (θ) - β \cdot CalibrationError (θ)) where α and β weight the relative importance of accuracy and calibration, and θ represents the model and prompting strategy configuration.

CS practitioners and researchers

Opening member content…