Flexible Log File Parsing using Hidden Markov Models
  We aim to model unknown file processing. As the content of log files often
evolves over time, we established a dynamic statistical model which learns and
adapts processing and parsing rules. First, we limit the amount of unstructured
text by focusing only on those frequent patterns which lead to the desired
output table similar to Vaarandi [10]. Second, we transform the found frequent
patterns and the output stating the parsed table into a Hidden Markov Model
(HMM). We use this HMM as a specific, however, flexible representation of a
pattern for log file processing. With changes in the raw log file distorting
learned patterns, we aim the model to adapt automatically in order to maintain
high quality output. After training our model on one system type, applying the
model and the resulting parsing rule to a different system with slightly
different log file patterns, we achieve an accuracy over 99%.
