1

What is the difference between network sparsification and model pruning? I watched USENIX ATC '21 - Octo: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny (at 01:29sec) where they state them as two different methods to simplify the model.

I have difficulties understanding what is the difference.

Mas A
  • 175
  • 9

1 Answers1

1

There is no operational difference. A pruned network is usually leads to a sparser network. However, the degree of sparsification, i.e., how much it is pruned, would determine if it is qualified as "sparse network". One could think of difference between a dense matrix and a sparse matrix, from weight matrices perspective. Probably, rule of thumb is maybe if 90% or more connections are pruned, it is a sparse network compare to the original one, see The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

msuzen
  • 1,709
  • 6
  • 27
  • So they are not two different things but related. Two of them is about having more zero weights in the weight matrix or removing a weight matrix completely can also be a solution to the sparse matrix? – Mas A Jan 18 '22 at 10:22
  • 1
    Yes, they are related. The idea is much more zeros in the weight matrix (recall connectivity matrix), i.e., removed connections means a sparse network. In implementations usually we don't assign zero weight but remove the corresponding connection completely. – msuzen Jan 18 '22 at 10:27
  • is this also feasible considering cnn types of networks? removing connection completely instead of assigning zero weights? – Mas A Jan 18 '22 at 11:08
  • 1
    Yes. CNN size will be smaller. Extended discussions in comments are discouraged. Please don't continue asking new questions in comments. – msuzen Jan 18 '22 at 11:12