Federated learning is a distributed machine learning paradigm that preserves user privacy by only communicating model updates computed locally among clients to the central server. However, this significantly affects the training performance and user experience because the clients’ datasets are statistically heterogeneous and the computation and transmission of local model updates are costly for their resource-constrained devices. Prior art has addressed these issues by incorporating personalization with model compression schemes including quantization and pruning. The pruning nonetheless is computationally expensive as it is data-dependent and must be performed on the client-side. Furthermore, pruning commonly involves learning a binary supermask ∈ {0, 1} which restricts the model capac...[ Read more ]

Federated learning is a distributed machine learning paradigm that preserves user privacy by only communicating model updates computed locally among clients to the central server. However, this significantly affects the training performance and user experience because the clients’ datasets are statistically heterogeneous and the computation and transmission of local model updates are costly for their resource-constrained devices. Prior art has addressed these issues by incorporating personalization with model compression schemes including quantization and pruning. The pruning nonetheless is computationally expensive as it is data-dependent and must be performed on the client-side. Furthermore, pruning commonly involves learning a binary supermask ∈ {0, 1} which restricts the model capacity with no computational benefit. In this work, we propose HideNseek which performs one-shot pruning on a randomly initialized model on the server-side in a data-agnostic manner by selecting the most synaptically salient weights as the subnetwork. The clients then collectively learn a sign supermask ∈ {−1, +1} that is multiplied to the unpruned weights for faster convergence while maintaining the same model compression rate as the state-of-the-art. Experiments on three learning tasks reveals that HideNseek improves inference accuracies by up to 40.6% compared to the state-of-the-art while reducing the communication cost by up to 39.7% and training time by up to 46.8%.