title: Fluentd in Kubernetes DaemonSet selectively parsing different logs
body: <p>So the basic architecture is a Fluentd DaemonSet scrapping Docker logs from pods setup by following <a href="https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4" rel="nofollow noreferrer">this blog post</a>, which in the end makes use of <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch" rel="nofollow noreferrer">these resources</a>.</p>
  
  <p>I have multiple pods and services running in the Cluster and I can't control their log format. However I do have control over the services I'm working on and I would like to have them use a JSON output format that is compatible with Logstash (key-value).</p>
  
  <p>Now I don't know if there's a way to selectively parse certain logs through this extra step, because some of the services won't have these log format. Of course <strong>I want this data to be accessible to Elasticsearch/Kibana for data visualization</strong>.</p>
  
  <p>Example JSON formatted log (Rails app deployed):
  <a href="https://i.stack.imgur.com/ZihEu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZihEu.png" alt="enter image description here"></a></p>
  
  <p>Example non JSON formatted log (CI/CD service):
  <a href="https://i.stack.imgur.com/wxXdD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wxXdD.png" alt="enter image description here"></a></p>
  
  <p>The Fluentd specific configs are found <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml" rel="nofollow noreferrer">on this file</a></p>
  
  <p>Any help/suggestions are greatly appreciated</p>
  
  <h1>EDIT 1: More details provided</h1>
  
  <p>So the problem here is that a JSON log is extracted through the following steps</p>
  
  <pre><code># JSON Log Example
  {"log":"2014/09/25 21:15:03 Got request with path wombat\n", "stream":"stderr", "time":"2014-09-25T21:15:03.499185026Z"}
  </code></pre>
  
  <p>The logs are being read from the docker logs in each node using <code>tail</code> type:</p>
  
  <pre><code>&lt;source&gt;
    @id fluentd-containers.log
    @type tail
    path /var/log/containers/*.log
    pos_file /var/log/es-containers.log.pos
    tag raw.kubernetes.*
    read_from_head true
    &lt;parse&gt;
      @type multi_format
      &lt;pattern&gt;
        format json
        time_key time
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      &lt;/pattern&gt;
      &lt;pattern&gt;
        format /^(?&lt;time&gt;.+) (?&lt;stream&gt;stdout|stderr) [^ ]* (?&lt;log&gt;.*)$/
        time_format %Y-%m-%dT%H:%M:%S.%N%:z
      &lt;/pattern&gt;
    &lt;/parse&gt;
  &lt;/source&gt;
  </code></pre>
  
  <p><a href="https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter" rel="nofollow noreferrer">Kubernetes metadata plugin</a> adds extra data</p>
  
  <pre><code>{
    "log":"2014/09/25 21:15:03 Got request with path wombat\n",
    "stream":"stderr",
    "time":"2014-09-25T21:15:03.499185026Z",
    "kubernetes": {
      "namespace": "default",
      "pod_name": "synthetic-logger-0.25lps-pod",
      "container_name": "synth-lgr"
    },
    "docker": {
      "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"
    }
  }
  </code></pre>
  
  <p><strong>However, I have a log that looks like the following:</strong></p>
  
  <pre><code>{"method":"GET","path":"/","format":"html","controller":"HomeController","action":"index","status":200,"duration":1922.45,"view":1912.5,"db":883.41,"request_ip":"127.0.0.1","@timestamp":"2018-09-20T16:43:17.322Z","@version":"1","message":"[200] GET / (HomeController#index)"}
  </code></pre>
  
  <p>And the problem with this is that the entire log will be "jammed" inside the "log" field, as you can see in the first screenshot posted above. I would guess that with kubernetes metadata it would be something like:</p>
  
  <pre><code>{
    "log":"{"method":"GET","path":"/","format":"html","controller":"HomeController","action":"index","status":200,"duration":1922.45,"view":1912.5,"db":883.41,"request_ip":"127.0.0.1","@timestamp":"2018-09-20T16:43:17.322Z","@version":"1","message":"[200] GET / (HomeController#index)"}",
    "stream":"stdout",
    "time":"2014-09-25T21:15:03.499185026Z",
    "kubernetes": {
      "namespace": "default",
      "pod_name": "rails-pod",
      "container_name": "rails-app"
    },
    "docker": {
      "container_id": "801599971ee6366d4a5921f25b79286ad45ff37a74494f260c3bc98d909d0a7b"
    }
  }
  </code></pre>
  
  <p>I'm looking for a way to filter/parse/extract that data into something similar to this:</p>
  
  <pre><code>{
    "rails-log": {
      "method": "GET",
      "path": "/",
      "format": "html",
      "controller": "HomeController",
      "action": "index",
      ...
      "message": "[200] GET / (HomeController#index)"
    },
    "stream":"stdout",
    "time":"2014-09-25T21:15:03.499185026Z",
    "kubernetes": {
      "namespace": "default",
      "pod_name": "rails-pod",
      "container_name": "rails-app"
    },
    "docker": {
      "container_id": "801599971ee6366d4a5921f25b79286ad45ff37a74494f260c3bc98d909d0a7b"
    }
  }
  </code></pre>
  
  <p>I have a feeling this can be achieved by a parse/filtering, <strong>I'm just unsure of how to go about putting this inside the fluentd configs</strong> (<a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml#L113-L445" rel="nofollow noreferrer">complete current configs found here</a>)</p>
  
  <p>After the Kubernetes plugin executes (adds kubernetes specific metadata to the "payload") I can select (in Kibana) the logs from my specific service by doing a query with a filter: <code>kubernetes.labels.app is '&lt;service_name&gt;'</code>. Therefore this leads me to believe there's a way to make a "conditional extraction/transformation" of the "log" field into something I can work with.</p>