PACSET (Packed Serialized Trees): Reducing Inference Latency for Tree
  Ensemble Deployment
  We present methods to serialize and deserialize tree ensembles that optimize
inference latency when models are not already loaded into memory. This arises
whenever models are larger than memory, but also systematically when models are
deployed on low-resource devices, such as in the Internet of Things, or run as
Web micro-services where resources are allocated on demand. Our packed
serialized trees (PACSET) encode reference locality in the layout of a tree
ensemble using principles from external memory algorithms. The layout
interleaves correlated nodes across multiple trees, uses leaf cardinality to
collocate the nodes on the most popular paths and is optimized for the I/O
blocksize. The result is that each I/O yields a higher fraction of useful data,
leading to a 2-6 times reduction in classification latency for interactive
workloads.
