THESIS
2023
1 online resource (ix, 55 pages) : illustrations (some color)
Abstract
Federated learning is a distributed machine learning paradigm that preserves user privacy
by only communicating model updates computed locally among clients to the central
server. However, this significantly affects the training performance and user experience because
the clients’ datasets are statistically heterogeneous and the computation and transmission
of local model updates are costly for their resource-constrained devices. Prior art
has addressed these issues by incorporating personalization with model compression schemes
including quantization and pruning. The pruning nonetheless is computationally expensive
as it is data-dependent and must be performed on the client-side. Furthermore, pruning
commonly involves learning a binary supermask ∈ {0, 1} which restricts the model capac...[
Read more ]
Federated learning is a distributed machine learning paradigm that preserves user privacy
by only communicating model updates computed locally among clients to the central
server. However, this significantly affects the training performance and user experience because
the clients’ datasets are statistically heterogeneous and the computation and transmission
of local model updates are costly for their resource-constrained devices. Prior art
has addressed these issues by incorporating personalization with model compression schemes
including quantization and pruning. The pruning nonetheless is computationally expensive
as it is data-dependent and must be performed on the client-side. Furthermore, pruning
commonly involves learning a binary supermask ∈ {0, 1} which restricts the model capacity
with no computational benefit. In this work, we propose HideNseek which performs one-shot
pruning on a randomly initialized model on the server-side in a data-agnostic manner by selecting
the most synaptically salient weights as the subnetwork. The clients then collectively
learn a sign supermask ∈ {−1, +1} that is multiplied to the unpruned weights for faster
convergence while maintaining the same model compression rate as the state-of-the-art. Experiments
on three learning tasks reveals that HideNseek improves inference accuracies by
up to 40.6% compared to the state-of-the-art while reducing the communication cost by up
to 39.7% and training time by up to 46.8%.
Post a Comment