Ilmu Komputer & AI editorial
Open AccessOA2026
Scalability and Performance Evaluation of Federated Learning Frameworks: A Comparative Analysis
A systematic experimental comparison of FedML, Flower, Substra, and OpenFL across varying client counts
Bassel Soudan; Sohail Abbas; Ahmed Kubba; Manar Wasif Abu Talib; Qassim Nasir· 2026· DOI 10.48550/arXiv.2609.15681
The core problem
Federated Learning (FL) enables collaborative model training across decentralized clients without sharing raw data, making it attractive for privacy-sensitive applications. However, selecting an appropriate FL framework for real-world deployment requires understanding how each scales with the number of clients. This paper presents a systematic examination and experimental comparison of four prominent FL frameworks: FedML, Flower, Substra, and OpenFL. The study emphasizes scalability analysis and key performance metrics—total training time, loss, accuracy, CPU usage, and RAM usage—as the number of clients increases. The goal is to provide guidance for effective deployment across diverse user bases.
Innovation
The results indicate distinct performance characteristics among the frameworks:
- **Flower**: Displays an unusually high loss.
- **FedML**: Achieves a notably low accuracy range of 66% to 79%.
- **Substra**: Demonstrates good resource efficiency, but total training time grows exponentially with client count.
- **OpenFL**: Emerges as the most scalable platform, maintaining consistent accuracy, loss, and training time across different client counts. Its CPU and RAM usage remain stable, underscoring reliability in real-world scenarios.
These findings highlight trade-offs between scalability, accuracy, and resource consumption.
Federated Learning (FL) enables collaborative model training across decentralized clients without sharing raw data, making it attractive for privacy-sensitive applications. However, selecting an appropriate FL framework for real-world deployment requires understanding how each scales with the number of clients. This paper presents a systematic examination and experimental comparison of four prominent FL frameworks: FedML, Flower, Substra, and OpenFL. The study emphasizes scalability analysis and key performance metrics—total training time, loss, accuracy, CPU usage, and RAM usage—as the number of clients increases. The goal is to provide guidance for effective deployment across diverse user bases.
The authors implement Federated Learning over a varying number of clients using each framework. They measure the impact of increasing client counts on:
Why it matters
The comparative analysis reveals that no single framework excels in all metrics. OpenFL's consistent performance across client counts makes it suitable for large-scale deployments where stability is critical. Substra's resource efficiency is offset by exponential training time growth, limiting its scalability. FedML's low accuracy suggests potential issues with convergence or model aggregation under increasing clients. Flower's high loss indicates possible instability or suboptimal hyperparameters. The study provides valuable insights into the relative capabilities of FL frameworks, guiding practitioners in selecting the right tool for their specific deployment needs. Future work could explore hybrid approaches or optimization strategies to mitigate observed weaknesses.
Who should read this
CS practitioners and researchers
Opening member content…