Sparsification is a natural idea to boost the inference and training efficiency and generalization performance of neural networks. For inference efficiency, it could work on a small sparse model with much less parameter counts and computational time while preserving comparable or even better generalization performance. For training efficiency, it works on a small sparse model with constrained model size during the whole training process, with sparsified forward and backward propagations. For generalization performance, besides the already effective IID (Independently Identically Distributed) setting, we also give a novel view of ultilizing sparsity to boost the generalizaton performance in OOD (Out of Distribution) setting. We could also sparsity the dataset to speed-up the training pro...[ Read more ]

Sparsification is a natural idea to boost the inference and training efficiency and generalization performance of neural networks. For inference efficiency, it could work on a small sparse model with much less parameter counts and computational time while preserving comparable or even better generalization performance. For training efficiency, it works on a small sparse model with constrained model size during the whole training process, with sparsified forward and backward propagations. For generalization performance, besides the already effective IID (Independently Identically Distributed) setting, we also give a novel view of ultilizing sparsity to boost the generalizaton performance in OOD (Out of Distribution) setting. We could also sparsity the dataset to speed-up the training procedure and boost OOD performance.