Learning Digital Circuits: A Journey Through Weight Invariant
  Self-Pruning Neural Networks
  Recently, in the paper "Weight Agnostic Neural Networks" Gaier & Ha utilized
architecture search to find networks where the topology completely encodes the
knowledge. However, architecture search in topology space is expensive. We use
the existing framework of binarized networks to find performant topologies by
constraining the weights to be either, zero or one. We show that such
topologies achieve performance similar to standard networks while pruning more
than 99% weights. We further demonstrate that these topologies can perform
tasks using constant weights without any explicit tuning. Finally, we discover
that in our setup each neuron acts like a NOR gate, virtually learning a
digital circuit. We demonstrate the efficacy of our approach on computer vision
datasets.
