Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained
  Parallelism
  Scaling CNN training is necessary to keep up with growing datasets and reduce
training time. We also see an emerging need to handle datasets with very large
samples, where memory requirements for training are large. Existing training
frameworks use a data-parallel approach that partitions samples within a
mini-batch, but limits to scaling the mini-batch size and memory consumption
makes this untenable for large samples. We describe and implement new
approaches to convolution, which parallelize using spatial decomposition or a
combination of sample and spatial decomposition. This introduces many
performance knobs for a network, so we develop a performance model for CNNs and
present a method for using it to automatically determine efficient
parallelization strategies.
  We evaluate our algorithms with microbenchmarks and image classification with
ResNet-50. Our algorithms allow us to prototype a model for a mesh-tangling
dataset, where sample sizes are very large. We show that our parallelization
achieves excellent strong and weak scaling and enables training for previously
unreachable datasets.
