PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

A communication-efficient framework that separates global-shared and local-private LoRA adapters for federated multimodal large language models at the network edge

Jing Liu; Kun Yang; Yan Wang; Dingkang Yang; Xiaoshuai Hao; Wei Zhang; Yang Liu; Wei Zhou· 2026· DOI 10.48550/arXiv.2607.12111

The core problem

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents that can learn collaboratively while preserving data privacy at network edges. Within these distributed environments, Multimodal Large Language Models (MLLMs) act as cognitive engines for edge devices, yet federated fine-tuning struggles to balance global knowledge aggregation against local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, producing suboptimal personalization and excessive communication overhead. The authors (Jing Liu, Kun Yang, Yan Wang, Dingkang Yang, Xiaoshuai Hao, Wei Zhang, Yang Liu, and Wei Zhou) propose PFAdapter to address this tension directly. The core insight is that not all adapter parameters should be treated equally: some encode universal multimodal semantics worth sharing across the network, while others encode edge-specific expertise that must remain private. By making this distinction explicit in the architecture rather than hoping uniform averaging will sort it out, PFAdapter targets both pe

Innovation

Extensive experiments on four datasets—VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD—demonstrate that PFAdapter consistently outperforms state-of-the-art baselines. Accuracy improvements range from 2.4% to 4.8% across diverse edge intelligence tasks, spanning medical visual question answering (VQA-RAD, SLAKE), harmful content detection (Hateful Memes), and crisis-related multimodal classification (CrisisMMD). The gains hold across this heterogeneous task mix, which is notable because the datasets differ substantially in domain, modality balance, and label semantics. Alongside accuracy, the selective aggregation protocol reduces communication costs by nearly 50% relative to conventional uniform aggregation, since only the global-shared query/key adapters are synchronized each round. The combination of higher accuracy and lower bandwidth suggests that the global/local split is not merely a compression trick but a genuinely better inductive bias for federated multimodal learning. The authors report that PFAdapter establishes an efficient solution for agentic AI deployment in resource-constrained communication networks, positioning it as a practical candidate for edge deployments where

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents that can learn collaboratively while preserving data privacy at network edges. Within these distributed environments, Multimodal Large Language Models (MLLMs) act as cognitive engines for edge devices, yet federated fine-tuning struggles to balance global knowledge aggregation against local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, producing suboptimal personalization and excessive communication overhead. The authors (Jing Liu, Kun Yang, Yan Wang, Dingkang Yang, Xiaoshuai Hao, Wei Zhang, Yang Liu, and Wei Zhou) propose PFAdapter to address this tension directly. The core insight is that not all adapter parameters should be treated equally: some encode universal multimodal semantics worth sharing across the network, while others encode edge-specific expertise that must remain private. By making this distinction explicit in the architecture rather than hoping uniform averaging will sort it out, PFAdapter targets both personalization quality and communication efficiency in resource-constrained communication networks.

PFAdapter introduces hierarchical LoRA decomposition that explicitly separates adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization to capture universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. This split is motivated by the role each projection plays in attention: query/key interactions determine what the model attends to and are therefore more transferable, whereas value/output transformations shape how information is written back and are more sensitive to local data distributions.

Why it matters

The central contribution of PFAdapter is conceptual: it reframes federated fine-tuning of MLLMs as a problem of parameter role assignment rather than uniform averaging. By mapping query/key projections to the global-shared component and value/output projections to the local-private component, the framework encodes a prior about which attention operations generalize across clients. The Frobenius-norm orthogonality regularizer operationalizes this prior by actively preventing the two adapter sets from collapsing into redundant representations, which is a common failure mode when shared and private parameters are trained jointly without constraints. The selective aggregation protocol then follows naturally: if only global-shared parameters are meant to be universal, only they need to cross the network, yielding the observed ~50% communication reduction. The 2.4%-4.8% accuracy gains across VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD indicate that the separation does not sacrifice personalization for efficiency; instead, the two objectives reinforce each other. Limitations are not detailed in the abstract, but the approach assumes that the query/key versus value/output split is a robust proxy for transferability across domains, an assumption that may need revisiting for architectures with different attention variants. Overall, PFAdapter offers a principled template for personalized federated MLLMs in agentic AI systems, where privacy, bandwidth, and heterogeneous edge conditions must be balanced simultaneously.

Who should read this

CS practitioners and researchers

Opening member content…