title: proper set up of parsing custom logs with logstash to kibana, i see no errors and no data
body: <p>I'm playing a bit with kibana to see how it works.</p>
  <p>i was able to add nginx log data directly from the same server without logstash and it works properly. but using logstash to read log files from a different server doesn't show data. no error.. but no data.</p>
  <p>I have custom logs from <a href="https://pm2.keymetrics.io/" rel="nofollow noreferrer">PM2</a> that runs some PHP script for me and the format of the messages are:</p>
  <p><code>Timestamp [LogLevel]: msg</code></p>
  <p>example:</p>
  <pre><code>2021-02-21 21:34:17 [DEBUG]: file size matches written file size 1194179
  </code></pre>
  <p>so my gork filter is:</p>
  <pre><code>&quot;%{DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\]: %{GREEDYDATA:msg}&quot;
  </code></pre>
  <p>I checked with <a href="http://grokdebug.herokuapp.com/" rel="nofollow noreferrer">Gork Validator</a> and the syntax matches the file format.</p>
  <p>i've got files that contain the suffix <code>out</code> that are debug level, and files with suffix <code>error</code> for error level.</p>
  <p>so to configure logstash on the kibana server, i added the file <code>/etc/logstash/conf.d/pipeline.conf</code> with the following:</p>
  <pre><code>input {
      beats {
          port =&gt; 5544
      }
  }
  filter {
   grok {
     match =&gt; {&quot;message&quot;=&gt;&quot;%{DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\]: %{GREEDYDATA:msg}&quot;} 
   }
  mutate {
      rename =&gt; [&quot;host&quot;, &quot;server&quot;]
      convert =&gt; {&quot;server&quot; =&gt; &quot;string&quot;}
  }
  }
  
  output {
      elasticsearch {
          hosts =&gt; &quot;http://localhost:9200&quot;
          user =&gt; &quot;&lt;USER&gt;&quot;
          password =&gt; &quot;&lt;PASSWORD&gt;&quot;
      }
  }
  </code></pre>
  <p>I needed to rename the <code>host</code> variable to <code>server</code> or I would get errors like <code>Can't get text on a START_OBJECT</code> and <code>failed to parse field [host] of type [text]</code></p>
  <p>on the 2nd server where the pm2 logs reside I configure filebeat with the following:</p>
  <pre><code>- type: filestream
    enabled: true
    paths:
      - /home/ubuntu/.pm2/*-error-*log
    fields:
     level: error
  - type: filestream
    enabled: true
    paths:
      - /home/ubuntu/.pm2/logs/*-out-*log
    fields:
     level: debug
  </code></pre>
  <p>I tried to use <code>log</code> and not <code>filestream</code> the results are the same.
  but it makes sense to use <code>filestream</code> since the logs are updated constantly on ?</p>
  <p>so i have <code>logstash</code> running on one server and <code>filebeat</code> on the other, opened firewall ports, i can see they're connecting but i don't see any new data in the Kibana logs dashboard relevant to the files i fetch with logstash.</p>
  <p>filebeat log always shows this line <code>Feb 24 04:41:56 vcx-prod-backup-01 filebeat[3797286]: 2021-02-24T04:41:56.991Z        INFO        [file_watcher]        filestream/fswatch.go:131        Start next scan</code> and something about analytics metrics so it looks fine, and still no data.</p>
  <p>I tried to provide here as much information as I can, i'm new to kibana, i have no idea why data is not shown in kibana if there are no errors.</p>
  <p>I thought maybe i didn't escaped the square brackets properly in gork filter so I tried using <code>&quot;%{DATESTAMP:timestamp} \\[%{LOGLEVEL:loglevel}\\]: %{GREEDYDATA:msg}&quot;</code> which replaces <code>\[</code> with <code>\\[</code> but the results are the same.</p>
  <p>any information regarding this issue would be greatly appreciated.</p>
  <p>#update
  ֿ
  using stack version 7.11.1</p>
  <p>I changed back to <code>log</code> instead of <code>filestream</code> based on @leandrojmp recommendations.</p>
  <p>I checked for <code>harverser.go</code> related lines i filebeat and I found these:</p>
  <pre><code>Feb 24 14:16:36 SERVER filebeat[4128025]: 2021-02-24T14:16:36.566Z        INFO        log/harvester.go:302        Harvester started for file: /home/ubuntu/.pm2/logs/cdr-ssh-out-1.log
  Feb 24 14:16:36 SERVER filebeat[4128025]: 2021-02-24T14:16:36.567Z        INFO        log/harvester.go:302        Harvester started for file: /home/ubuntu/.pm2/logs/cdr-ftp-out-0.log
  </code></pre>
  <p>and I also noticed that when i configured the output to stdout, i do see the events that are coming from the other server. so logstash do receive them properly but for some reason i don't see them in kiban.</p>