Automatic Full Compilation of Julia Programs and ML Models to Cloud TPUs
  Google's Cloud TPUs are a promising new hardware architecture for machine
learning workloads. They have powered many of Google's milestone machine
learning achievements in recent years. Google has now made TPUs available for
general use on their cloud platform and as of very recently has opened them up
further to allow use by non-TensorFlow frontends. We describe a method and
implementation for offloading suitable sections of Julia programs to TPUs via
this new API and the Google XLA compiler. Our method is able to completely fuse
the forward pass of a VGG19 model expressed as a Julia program into a single
TPU executable to be offloaded to the device. Our method composes well with
existing compiler-based automatic differentiation techniques on Julia code, and
we are thus able to also automatically obtain the VGG19 backwards pass and
similarly offload it to the TPU. Targeting TPUs using our compiler, we are able
to evaluate the VGG19 forward pass on a batch of 100 images in 0.23s which
compares favorably to the 52.4s required for the original model on the CPU. Our
implementation is less than 1000 lines of Julia, with no TPU specific changes
made to the core Julia compiler or any other Julia packages.
