title: logstash grok, parse a line with json filter
body: <p>I am using ELK(elastic search, kibana, logstash, filebeat) to collect logs. I have a log file with following lines, every line has a json, my target is to using Logstash Grok to take out of key/value pair in the json and forward it to elastic search.</p>
  
  <pre><code>2018-03-28 13:23:01  charge:{"oldbalance":5000,"managefee":0,"afterbalance":"5001","cardid":"123456789","txamt":1}
  
  2018-03-28 13:23:01  manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
  </code></pre>
  
  <p>I am using <a href="http://grokdebug.herokuapp.com/" rel="nofollow noreferrer">Grok Debugger</a> to make regex pattern and see the result. My current regex is:</p>
  
  <pre><code>%{TIMESTAMP_ISO8601} %{SPACE} %{WORD:$:data}:{%{QUOTEDSTRING:key1}:%{BASE10NUM:value1}[,}]%{QUOTEDSTRING:key2}:%{BASE10NUM:value2}[,}]%{QUOTEDSTRING:key3}:%{QUOTEDSTRING:value3}[,}]%{QUOTEDSTRING:key4}:%{QUOTEDSTRING:value4}[,}]%{QUOTEDSTRING:key5}:%{BASE10NUM:value5}[,}]
  </code></pre>
  
  <p>As one could see it is hard coded since the keys in json in real log could be any word, the value could be integer, double or string, what's more, the length of the keys varies. so my solution is not acceptable. My solution result is shown as follows, just for reference. I am using <a href="https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns" rel="nofollow noreferrer">Grok patterns</a>.  </p>
  
  <p>My question is that trying to extract keys in json is wise or not since elastic search use json also? Second, if I try to take keys/values out of json, are there correct,concise Grok patterns? </p>
  
  <p>current result of Grok patterns give following output when parsing first line in  above lines.</p>
  
  <pre><code>{
    "TIMESTAMP_ISO8601": [
      [
        "2018-03-28 13:23:01"
      ]
    ],
    "YEAR": [
      [
        "2018"
      ]
    ],
    "MONTHNUM": [
      [
        "03"
      ]
    ],
    "MONTHDAY": [
      [
        "28"
      ]
    ],
    "HOUR": [
      [
        "13",
        null
      ]
    ],
    "MINUTE": [
      [
        "23",
        null
      ]
    ],
    "SECOND": [
      [
        "01"
      ]
    ],
    "ISO8601_TIMEZONE": [
      [
        null
      ]
    ],
    "SPACE": [
      [
        ""
      ]
    ],
    "WORD": [
      [
        "charge"
      ]
    ],
    "key1": [
      [
        ""oldbalance""
      ]
    ],
    "value1": [
      [
        "5000"
      ]
    ],
    "key2": [
      [
        ""managefee""
      ]
    ],
    "value2": [
      [
        "0"
      ]
    ],
    "key3": [
      [
        ""afterbalance""
      ]
    ],
    "value3": [
      [
        ""5001""
      ]
    ],
    "key4": [
      [
        ""cardid""
      ]
    ],
    "value4": [
      [
        ""123456789""
      ]
    ],
    "key5": [
      [
        ""txamt""
      ]
    ],
    "value5": [
      [
        "1"
      ]
    ]
  }
  </code></pre>
  
  <p><strong>second edit</strong></p>
  
  <p>Is it possible to use Json filter of Logstash? but in my case Json is part of line/event, not whole event is Json.</p>
  
  <p>===========================================================</p>
  
  <h1>      Third edition</h1>
  
  <p>I do not see updated solution functions well to parse json. My regex is as follows:</p>
  
  <pre><code>filter {
    grok {
      match =&gt; {
        "message" =&gt; [
             "%{TIMESTAMP_ISO8601}%{SPACE}%{GREEDYDATA:json_data}"
              ]
      }       
    }
  }
  
  
  filter {
    json{
      source =&gt; "json_data"
      target =&gt; "parsed_json"
    } 
  }
  </code></pre>
  
  <p>It does not have key:value pair, instead it is msg+json string. The parsed json is not parsed. </p>
  
  <p>Testing data is as below:</p>
  
  <pre><code>2018-03-28 13:23:01  manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
  2018-03-28 13:23:03  payment:{"cuurentValue":5001,"reload":0,"newbalance":"5002","posid":"987654321","something":"new3","additionalFields":2}
  2018-03-28 13:24:07  management:{"cuurentValue":5002,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
  
  [2018-06-04T15:01:30,017][WARN ][logstash.filters.json    ] Error parsing json {:source=&gt;"json_data", :raw=&gt;"manage:{\"cuurentValue\":5000,\"payment\":0,\"newbalance\":\"5001\",\"posid\":\"123456789\",\"something\":\"new2\",\"additionalFields\":1}", :exception=&gt;#&lt;LogStash::Json::ParserError: Unrecognized token 'manage': was expecting ('true', 'false' or 'null')
   at [Source: (byte[])"manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}"; line: 1, column: 8]&gt;}
  [2018-06-04T15:01:30,017][WARN ][logstash.filters.json    ] Error parsing json {:source=&gt;"json_data", :raw=&gt;"payment:{\"cuurentValue\":5001,\"reload\":0,\"newbalance\":\"5002\",\"posid\":\"987654321\",\"something\":\"new3\",\"additionalFields\":2}", :exception=&gt;#&lt;LogStash::Json::ParserError: Unrecognized token 'payment': was expecting ('true', 'false' or 'null')
   at [Source: (byte[])"payment:{"cuurentValue":5001,"reload":0,"newbalance":"5002","posid":"987654321","something":"new3","additionalFields":2}"; line: 1, column: 9]&gt;}
  [2018-06-04T15:01:34,986][WARN ][logstash.filters.json    ] Error parsing json {:source=&gt;"json_data", :raw=&gt;"management:{\"cuurentValue\":5002,\"payment\":0,\"newbalance\":\"5001\",\"posid\":\"123456789\",\"something\":\"new2\",\"additionalFields\":1}", :exception=&gt;#&lt;LogStash::Json::ParserError: Unrecognized token 'management': was expecting ('true', 'false' or 'null')
   at [Source: (byte[])"management:{"cuurentValue":5002,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}"; line: 1, column: 12]&gt;}
  </code></pre>
  
  <p>Please check the result:
  <a href="https://i.stack.imgur.com/ohDT8.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ohDT8.jpg" alt="enter image description here"></a></p>