Given a number of application logs sharing the same HeaderLine and EndLine regular expressions we are trying out a xm_multiline with im_file config using wild cards. 

<Extension multi>
  Module      xm_multiline
    HeaderLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ batch_task\._init_logger : \[INFO\]\+ /
    EndLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ run_batch\.<module> : \[INFO\]- /

<Input inPython>
    Module    im_file
    File "C:\\data\\server1\\*.log"
    InputType multi
    Exec $FileName = file_name();

It works consistently without wildcards pointed at one file.
It works intermittently with wildcards pointed at multiple files being written to concurrenlty.

I'm wondering if this is a supported use case. i.e. multiline events from wildcarded files being written to concurrenlty. Or should we be specifiying each input file individually?




AskedJanuary 3, 2017 - 10:26pm

Answers (2)

The InputType directive ensures that the data is processed on the file level (and per connection for the networking modules) instead of combining data coming from all the files (*.log) and matching that as a whole, so yes, that's a supported use-case and there is no need to specify each file individually.

There was another report with a similar complaint recently, see this other forum post. Can you create a test case to reproduce and put it in a support ticket?

Seems like this is related to the auto-flush feature:


Until there is a new header read, the previous message is stored in the buffers because the module does not know where the message ends. The im_file module will forcibly flush this buffer after the configured PollInterval timeout. If this behavior is unacceptable, consider using some kind of an encapsulation method (JSON, XML, RFC5425, etc) or use an end marker with EndLine if possible.

For now a possible workaround is to increase the PollInterval value in im_file to minimize the chances of flushing the data in the middle of the record.

Auto-flushing should be disabled in the presence of EndLine, this is a bug that needs to be fixed.