Slot Machines: Discovering Winning Combinations of Random Weights in
  Neural Networks
  In contrast to traditional weight optimization in a continuous space, we
demonstrate the existence of effective random networks whose weights are never
updated. By selecting a weight among a fixed set of random values for each
individual connection, our method uncovers combinations of random weights that
match the performance of traditionally-trained networks of the same capacity.
We refer to our networks as "slot machines" where each reel (connection)
contains a fixed set of symbols (random values). Our backpropagation algorithm
"spins" the reels to seek "winning" combinations, i.e., selections of random
weight values that minimize the given loss. Quite surprisingly, we find that
allocating just a few random values to each connection (e.g., 8 values per
connection) yields highly competitive combinations despite being dramatically
more constrained compared to traditionally learned weights. Moreover,
finetuning these combinations often improves performance over the trained
baselines. A randomly initialized VGG-19 with 8 values per connection contains
a combination that achieves 91% test accuracy on CIFAR-10. Our method also
achieves an impressive performance of 98.2% on MNIST for neural networks
containing only random weights.
