Filter Grafting for Deep Neural Networks: Reason, Method, and
  Cultivation
  Filter is the key component in modern convolutional neural networks (CNNs).
However, since CNNs are usually over-parameterized, a pre-trained network
always contain some invalid (unimportant) filters. These filters have
relatively small $l_{1}$ norm and contribute little to the output
(\textbf{Reason}). While filter pruning removes these invalid filters for
efficiency consideration, we tend to reactivate them to improve the
representation capability of CNNs. In this paper, we introduce filter grafting
(\textbf{Method}) to achieve this goal. The activation is processed by grafting
external information (weights) into invalid filters. To better perform the
grafting, we develop a novel criterion to measure the information of filters
and an adaptive weighting strategy to balance the grafted information among
networks. After the grafting operation, the network has fewer invalid filters
compared with its initial state, enpowering the model with more representation
capacity. Meanwhile, since grafting is operated reciprocally on all networks
involved, we find that grafting may lose the information of valid filters when
improving invalid filters. To gain a universal improvement on both valid and
invalid filters, we compensate grafting with distillation
(\textbf{Cultivation}) to overcome the drawback of grafting . Extensive
experiments are performed on the classification and recognition tasks to show
the superiority of our method. Code is available at
\textcolor{black}{\emph{https://github.com/fxmeng/filter-grafting}}.
