Shapley Value as Principled Metric for Structured Network Pruning
  Structured pruning is a well-known technique to reduce the storage size and
inference cost of neural networks. The usual pruning pipeline consists of
ranking the network internal filters and activations with respect to their
contributions to the network performance, removing the units with the lowest
contribution, and fine-tuning the network to reduce the harm induced by
pruning. Recent results showed that random pruning performs on par with other
metrics, given enough fine-tuning resources. In this work, we show that this is
not true on a low-data regime when fine-tuning is either not possible or not
effective. In this case, reducing the harm caused by pruning becomes crucial to
retain the performance of the network. First, we analyze the problem of
estimating the contribution of hidden units with tools suggested by cooperative
game theory and propose Shapley values as a principled ranking metric for this
task. We compare with several alternatives proposed in the literature and
discuss how Shapley values are theoretically preferable. Finally, we compare
all ranking metrics on the challenging scenario of low-data pruning, where we
demonstrate how Shapley values outperform other heuristics.
