FISGuard: Defending Against Membership Inference via Fixed Input Subspaces

A lightweight defense that constrains gradient geometry to defeat ProjRes-style membership inference in federated parameter-efficient fine-tuning

Haocheng Jiang; Hua Shen· 2026· DOI 10.48550/arXiv.2608.27836

The core problem

Large language models (LLMs) are increasingly deployed in federated learning (FL) settings, where clients perform parameter-efficient fine-tuning (PEFT) on distributed private data. Although clients share only gradients rather than raw data, these gradients can still leak membership information about training samples. ProjRes (S&P, 2026) amplifies this risk: with less information and without accessing model outputs, an attacker can distinguish members from non-members by exploiting the projection residual between a candidate representation and the subspace induced by server-observable gradients. Existing defenses against membership inference (MI) typically rely on gradient perturbation or regularization. These approaches can degrade model utility and, more critically, fail to defend against ProjRes, which exploits the geometric structure of gradients rather than their magnitude or noise. This paper introduces FISGuard, a lightweight defense that constructs and fixes a low-dimensional representation subspace using independent public data. By restricting the space through which private representations are exposed via gradients, FISGuard substantially reduces the projection-residual

The authors evaluate FISGuard against five representative defense methods across three NLP datasets, two LLMs, and two fine-tuning strategies: Adapter and LoRA. The primary metric is the AUC of the ProjRes membership inference attack, where 0.5 indicates random guessing. Results show that FISGuard reduces the ProjRes attack AUC to near 0.5 in most settings, effectively neutralizing the attack. For example, on the evaluated datasets and models, the AUC drops from significantly above 0.5 (under undefended or baseline defenses) to values close to 0.5. Meanwhile, downstream task performance remains close to that of the undefended model, indicating a favorable privacy-utility trade-off. The computational overhead introduced by FISGuard is limited, making it practical for federated learning deployments.

Large language models (LLMs) are increasingly deployed in federated learning (FL) settings, where clients perform parameter-efficient fine-tuning (PEFT) on distributed private data. Although clients share only gradients rather than raw data, these gradients can still leak membership information about training samples. ProjRes (S&P, 2026) amplifies this risk: with less information and without accessing model outputs, an attacker can distinguish members from non-members by exploiting the projection residual between a candidate representation and the subspace induced by server-observable gradients.

Existing defenses against membership inference (MI) typically rely on gradient perturbation or regularization. These approaches can degrade model utility and, more critically, fail to defend against ProjRes, which exploits the geometric structure of gradients rather than their magnitude or noise. This paper introduces FISGuard, a lightweight defense that constructs and fixes a low-dimensional representation subspace using independent public data. By restricting the space through which private representations are exposed via gradients, FISGuard substantially reduces the projection-residual discrepancy between members and non-members while preserving the primary information required for downstream tasks.

The key insight of FISGuard is that ProjRes exploits the geometric structure of gradients—specifically, the projection residual between a candidate representation and the subspace induced by server-observable gradients. By fixing a low-dimensional subspace constructed from public data, FISGuard restricts the variability of this residual across members and non-members, thereby reducing the attack's discriminative power. Unlike gradient perturbation or regularization, which add noise or penalize certain parameters, FISGuard directly constrains the information exposed through gradients without significantly harming the primary information needed for downstream tasks. This explains why it maintains utility while achieving strong privacy. The evaluation across multiple datasets, models, and PEFT strategies demonstrates the robustness of FISGuard. However, the defense assumes the availability of independent public data that is representative of the private data distribution. Future work could explore adaptive subspace construction and defenses against other geometry-aware attacks.

CS practitioners and researchers

Opening member content…