Agentic Autoscaling through Worker-Pool Orchestration for LLM-driven Text Classification in Cloud Computing Environments

A classifier-agnostic framework that dynamically scales agent workers to balance accuracy, latency, and resource efficiency for bursty LLM inference workloads.

Bablu Kumar; Anshul Verma; Rajkumar Buyya· 2026· DOI 10.48550/arXiv.2609.14898

The core problem

Large language models (LLMs) are increasingly deployed for large-scale text classification, yet their high inference latency, bursty arrival patterns, and computational intensity create significant challenges for static resource allocation in cloud environments. Static provisioning leads to either over-provisioning—wasting cost and energy—or under-provisioning—causing latency violations and degraded throughput. This paper addresses the gap by proposing an **agentic autoscaling framework** that dynamically orchestrates a pool of worker agents to serve LLM-based text classification tasks. The framework is designed to be classifier-agnostic, supporting both zero-shot and fine-tuned models without altering the autoscaling logic. The authors evaluate the framework on two datasets (AG News and SMS Spam Collection) using BART and DeBERTa classifiers, comparing against static allocation and standalone RoBERTa/DistilBERT baselines. The core research question is: *Can elastic worker-pool orchestration improve resource efficiency while maintaining high classification accuracy for LLM-driven text classification?*

The experimental results demonstrate that the proposed framework consistently outperforms baseline approaches in resource efficiency while maintaining high classification performance. On the **AG News dataset**, Autoscaling+BART achieves **84.5% accuracy**, while Autoscaling+DeBERTa improves it to **90.5%**. On the **SMS Spam Collection dataset**, Autoscaling+DeBERTa achieves **99.5% accuracy**, whereas Autoscaling+BART attains **84.5% accuracy** with lower execution time. Compared to static allocation, the autoscaling framework reduces resource consumption by dynamically adjusting the worker pool to match workload intensity. Standalone RoBERTa and DistilBERT baselines, which do not benefit from autoscaling, exhibit either higher latency or lower accuracy under bursty workloads. The results indicate that elastic worker-pool orchestration can effectively handle high-latency, bursty LLM inference tasks without sacrificing classification quality. Notably, the classifier-agnostic design allows seamless substitution of the underlying model, as evidenced by the different accuracy profiles of BART and DeBERTa on the two datasets.

Large language models (LLMs) are increasingly deployed for large-scale text classification, yet their high inference latency, bursty arrival patterns, and computational intensity create significant challenges for static resource allocation in cloud environments. Static provisioning leads to either over-provisioning—wasting cost and energy—or under-provisioning—causing latency violations and degraded throughput. This paper addresses the gap by proposing an **agentic autoscaling framework** that dynamically orchestrates a pool of worker agents to serve LLM-based text classification tasks. The framework is designed to be classifier-agnostic, supporting both zero-shot and fine-tuned models without altering the autoscaling logic. The authors evaluate the framework on two datasets (AG News and SMS Spam Collection) using BART and DeBERTa classifiers, comparing against static allocation and standalone RoBERTa/DistilBERT baselines. The core research question is: *Can elastic worker-pool orchestration improve resource efficiency while maintaining high classification accuracy for LLM-driven text classification?*

The proposed framework comprises four key components: (1) a **priority task queue** that buffers incoming classification requests and orders them by priority; (2) a **dynamic pool of agent workers** that execute LLM inference; (3) a **real-time metrics collector** that monitors queue length, worker utilization, and latency; and (4) an **application-layer autoscaler** that adjusts the number of active workers based on collected metrics. The autoscaler employs a control policy that scales out when queue length exceeds a threshold and scales in when utilization drops below a threshold, with hysteresis to avoid oscillation. The framework is classifier-agnostic: it treats the LLM as a black box, so switching from BART to DeBERTa requires no changes to the autoscaling logic. The architecture is illustrated below:

The findings highlight several key insights. First, **autoscaling is essential** for LLM-driven text classification because static allocation cannot cope with bursty workloads: it either wastes resources during idle periods or violates latency SLAs during peaks. Second, the **classifier-agnostic design** enables practitioners to choose the best model for their accuracy-latency trade-off without re-engineering the autoscaling logic. For instance, DeBERTa yields higher accuracy on both datasets, but BART offers lower execution time on SMS Spam, making it suitable for latency-sensitive applications. Third, the **agentic approach**—where workers are autonomous agents managed by a central autoscaler—provides a clean separation of concerns and facilitates scalability. However, the paper does not explore the impact of different autoscaling policies (e.g., reinforcement learning-based) or the cost implications in multi-tenant cloud environments. Future work could investigate adaptive threshold tuning and integration with serverless platforms. Overall, the framework demonstrates that elastic worker-pool orchestration is a cost-efficient solution for scalable LLM-driven text classification, with potential extensions to other NLP tasks such as summarization and translation.

CS practitioners and researchers

Opening member content…