Griffon: Reasoning about Job Anomalies with Unlabeled Data in
  Cloud-based Platforms
  Microsoft's internal big data analytics platform is comprised of hundreds of
thousands of machines, serving over half a million jobs daily, from thousands
of users. The majority of these jobs are recurring and are crucial for the
company's operation. Although administrators spend significant effort tuning
system performance, some jobs inevitably experience slowdowns, i.e., their
execution time degrades over previous runs. Currently, the investigation of
such slowdowns is a labor-intensive and error-prone process, which costs
Microsoft significant human and machine resources, and negatively impacts
several lines of businesses. In this work, we present Griffin, a system we
built and have deployed in production last year to automatically discover the
root cause of job slowdowns. Existing solutions either rely on labeled data
(i.e., resolved incidents with labeled reasons for job slowdowns), which is in
most cases non-existent or non-trivial to acquire, or on time-series analysis
of individual metrics that do not target specific jobs holistically. In
contrast, in Griffin we cast the problem to a corresponding regression one that
predicts the runtime of a job, and show how the relative contributions of the
features used to train our interpretable model can be exploited to rank the
potential causes of job slowdowns. Evaluated over historical incidents, we show
that Griffin discovers slowdown causes that are consistent with the ones
validated by domain-expert engineers, in a fraction of the time required by
them.
