Phoebe: A Learning-based Checkpoint Optimizer
  Easy-to-use programming interfaces paired with cloud-scale processing engines
have enabled big data system users to author arbitrarily complex analytical
jobs over massive volumes of data. However, as the complexity and scale of
analytical jobs increase, they encounter a number of unforeseen problems,
hotspots with large intermediate data on temporary storage, longer job recovery
time after failures, and worse query optimizer estimates being examples of
issues that we are facing at Microsoft.
  To address these issues, we propose Phoebe, an efficient learning-based
checkpoint optimizer. Given a set of constraints and an objective function at
compile-time, Phoebe is able to determine the decomposition of job plans, and
the optimal set of checkpoints to preserve their outputs to durable global
storage. Phoebe consists of three machine learning predictors and one
optimization module. For each stage of a job, Phoebe makes accurate predictions
for: (1) the execution time, (2) the output size, and (3) the start/end time
taking into account the inter-stage dependencies. Using these predictions, we
formulate checkpoint optimization as an integer programming problem and propose
a scalable heuristic algorithm that meets the latency requirement of the
production environment.
  We demonstrate the effectiveness of Phoebe in production workloads, and show
that we can free the temporary storage on hotspots by more than 70% and restart
failed jobs 68% faster on average with minimum performance impact. Phoebe also
illustrates that adding multiple sets of checkpoints is not cost-efficient,
which dramatically reduces the complexity of the optimization.
