Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration

A Byzantine-resilient DFL framework that filters, weights, and validates cross-client knowledge across class, boundary, and correlation modalities

Xiao Ma; Hong Shen; Hui Tian; Wei Ke; Wenqi Lyu· 2026· DOI 10.48550/arXiv.2609.07230

The core problem

Decentralized federated learning (DFL) is increasingly demanded in edge computing and mission-oriented UAV collaborations, where clients operate without a central aggregator and often possess heterogeneous model architectures. In such unreliable real-world scenarios, clients are exposed to receiver-specific Byzantine messages containing malicious predictions. The authors (Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu) address this challenge by proposing a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. The core problem is that in DFL, knowledge exchanged between peers can be corrupted by Byzantine clients, and the absence of a central server removes a natural aggregation point for defense. The paper's contribution is a multi-modality knowledge collaboration scheme that evaluates received predictions in three modalities—class prediction, boundary decision, and prediction correlation—before constructing teachers and validating distillation gradients against private supervised gradients.

Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. The authors report that the method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clients under Byzantine distillation. The convergence proof shows stable local optimization for honest clients, and the empirical results confirm robustness across the two benchmark datasets. The paper does not provide specific numerical accuracy values in the abstract, but the qualitative claim is consistent improvement over baselines under adversarial conditions.

Decentralized federated learning (DFL) is increasingly demanded in edge computing and mission-oriented UAV collaborations, where clients operate without a central aggregator and often possess heterogeneous model architectures. In such unreliable real-world scenarios, clients are exposed to receiver-specific Byzantine messages containing malicious predictions. The authors (Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu) address this challenge by proposing a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data.

The core problem is that in DFL, knowledge exchanged between peers can be corrupted by Byzantine clients, and the absence of a central server removes a natural aggregation point for defense. The paper's contribution is a multi-modality knowledge collaboration scheme that evaluates received predictions in three modalities—class prediction, boundary decision, and prediction correlation—before constructing teachers and validating distillation gradients against private supervised gradients.

The key insight of this work is that robustness in decentralized federated distillation can be achieved by exploiting multiple complementary views of peer predictions. Class prediction, boundary decision, and prediction correlation each capture different aspects of model behavior, and cross-modality fusion makes it harder for Byzantine clients to corrupt the learning process undetected. The theoretical analysis establishes bounded Byzantine influence, which is a stronger guarantee than simple outlier rejection. The method's reliance on shared unlabeled public data is both a strength and a potential limitation: it enables collaboration among heterogeneous models without sharing parameters, but assumes the availability of a public dataset. The authors position the approach as having great potential for adoption of DFL in unreliable real-world scenarios, particularly edge computing and UAV collaborations. Future work could explore adaptive weighting schemes and extension to other modalities beyond the three considered.

CS practitioners and researchers

Opening member content…