Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across
  Contexts
  Distributed dataflow systems enable the use of clusters for scalable data
analytics. However, selecting appropriate cluster resources for a processing
job is often not straightforward. Performance models trained on historical
executions of a concrete job are helpful in such situations, yet they are
usually bound to a specific job execution context (e.g. node type, software
versions, job parameters) due to the few considered input parameters. Even in
case of slight context changes, such supportive models need to be retrained and
cannot benefit from historical execution data from related contexts.
  This paper presents Bellamy, a novel modeling approach that combines
scale-outs, dataset sizes, and runtimes with additional descriptive properties
of a dataflow job. It is thereby able to capture the context of a job
execution. Moreover, Bellamy is realizing a two-step modeling approach. First,
a general model is trained on all the available data for a specific scalable
analytics algorithm, hereby incorporating data from different contexts.
Subsequently, the general model is optimized for the specific situation at
hand, based on the available data for the concrete context. We evaluate our
approach on two publicly available datasets consisting of execution data from
various dataflow jobs carried out in different environments, showing that
Bellamy outperforms state-of-the-art methods.
