Linnaeus: A highly reusable and adaptable ML based log classification
  pipeline
  Logs are a common way to record detailed run-time information in software. As
modern software systems evolve in scale and complexity, logs have become
indispensable to understanding the internal states of the system. At the same
time however, manually inspecting logs has become impractical. In recent times,
there has been more emphasis on statistical and machine learning (ML) based
methods for analyzing logs. While the results have shown promise, most of the
literature focuses on algorithms and state-of-the-art (SOTA), while largely
ignoring the practical aspects. In this paper we demonstrate our end-to-end log
classification pipeline, Linnaeus. Besides showing the more traditional ML
flow, we also demonstrate our solutions for adaptability and re-use,
integration towards large scale software development processes, and how we cope
with lack of labelled data. We hope Linnaeus can serve as a blueprint for, and
inspire the integration of, various ML based solutions in other large scale
industrial settings.
