{"action":"created","issue":{"url":"https://api.github.com/repos/ntop/ntopng/issues/1144","repository_url":"https://api.github.com/repos/ntop/ntopng","labels_url":"https://api.github.com/repos/ntop/ntopng/issues/1144/labels{/name}","comments_url":"https://api.github.com/repos/ntop/ntopng/issues/1144/comments","events_url":"https://api.github.com/repos/ntop/ntopng/issues/1144/events","html_url":"https://github.com/ntop/ntopng/issues/1144","id":220419471,"number":1144,"title":"Logstash errors when parsing data from ntopng","user":{"login":"jevonearth","id":1114943,"avatar_url":"https://avatars2.githubusercontent.com/u/1114943?v=3","gravatar_id":"","url":"https://api.github.com/users/jevonearth","html_url":"https://github.com/jevonearth","followers_url":"https://api.github.com/users/jevonearth/followers","following_url":"https://api.github.com/users/jevonearth/following{/other_user}","gists_url":"https://api.github.com/users/jevonearth/gists{/gist_id}","starred_url":"https://api.github.com/users/jevonearth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jevonearth/subscriptions","organizations_url":"https://api.github.com/users/jevonearth/orgs","repos_url":"https://api.github.com/users/jevonearth/repos","events_url":"https://api.github.com/users/jevonearth/events{/privacy}","received_events_url":"https://api.github.com/users/jevonearth/received_events","type":"User","site_admin":false},"labels":[{"id":265771516,"url":"https://api.github.com/repos/ntop/ntopng/labels/testing%20needed","name":"testing needed","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":{"login":"simonemainardi","id":3397663,"avatar_url":"https://avatars3.githubusercontent.com/u/3397663?v=3","gravatar_id":"","url":"https://api.github.com/users/simonemainardi","html_url":"https://github.com/simonemainardi","followers_url":"https://api.github.com/users/simonemainardi/followers","following_url":"https://api.github.com/users/simonemainardi/following{/other_user}","gists_url":"https://api.github.com/users/simonemainardi/gists{/gist_id}","starred_url":"https://api.github.com/users/simonemainardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonemainardi/subscriptions","organizations_url":"https://api.github.com/users/simonemainardi/orgs","repos_url":"https://api.github.com/users/simonemainardi/repos","events_url":"https://api.github.com/users/simonemainardi/events{/privacy}","received_events_url":"https://api.github.com/users/simonemainardi/received_events","type":"User","site_admin":false},"assignees":[{"login":"simonemainardi","id":3397663,"avatar_url":"https://avatars3.githubusercontent.com/u/3397663?v=3","gravatar_id":"","url":"https://api.github.com/users/simonemainardi","html_url":"https://github.com/simonemainardi","followers_url":"https://api.github.com/users/simonemainardi/followers","following_url":"https://api.github.com/users/simonemainardi/following{/other_user}","gists_url":"https://api.github.com/users/simonemainardi/gists{/gist_id}","starred_url":"https://api.github.com/users/simonemainardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonemainardi/subscriptions","organizations_url":"https://api.github.com/users/simonemainardi/orgs","repos_url":"https://api.github.com/users/simonemainardi/repos","events_url":"https://api.github.com/users/simonemainardi/events{/privacy}","received_events_url":"https://api.github.com/users/simonemainardi/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2017-04-08T18:53:28Z","updated_at":"2017-04-10T11:35:14Z","closed_at":null,"body":"Hi,\r\n\r\nI have been testing the ntopng to logstash feature, and I'm getting some good data passed from: mikrotik -> nprobe -> ntopng -> logstash -> elasticsearch\r\n\r\nBut I get regular json parse errors where there are spurious characters (such as `\\u00004` and `\\u0000` ) in the json. See two log samples below, along with version information and copies of my configs. \r\n\r\nPlease let me know if I can test anything specific in order to help you further. \r\n\r\nThanks,\r\n-Jev\r\n\r\n\r\n## Logstash errors on stdout:\r\n```[2017-04-08T18:41:10,831][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: 4, \"FIRST_SWITCHED\": 1491676739, \"LAST_SWITCHED\": 1491676739, \"json\": { \"15\": \"184.70.24.213\", \"10\": \"2\", \"14\": \"12\", \"5\": \"0\", \"16\": \"15169\", \"17\": \"0\" }, \"CLIENT_NW_LATENCY_MS\": 0.000000, \"SERVER_NW_LATENCY_MS\": 0.000000, \"DST_IP_COUNTRY\": \"US\", \"DST_IP_LOCATION\": [ -122.057404, 37.419201 ], \"NTOPNG_INSTANCE_NAME\": \"ntop-redder-dev\", \"INTERFACE\": \"tcp:\\/\\/127.0.0.1:5559c [probeIP: 10.60.58.1]\" }; line: 1, column: 2]>, :data=>\"\\u00004, \\\"FIRST_SWITCHED\\\": 1491676739, \\\"LAST_SWITCHED\\\": 1491676739, \\\"json\\\": { \\\"15\\\": \\\"184.70.24.213\\\", \\\"10\\\": \\\"2\\\", \\\"14\\\": \\\"12\\\", \\\"5\\\": \\\"0\\\", \\\"16\\\": \\\"15169\\\", \\\"17\\\": \\\"0\\\" }, \\\"CLIENT_NW_LATENCY_MS\\\": 0.000000, \\\"SERVER_NW_LATENCY_MS\\\": 0.000000, \\\"DST_IP_COUNTRY\\\": \\\"US\\\", \\\"DST_IP_LOCATION\\\": [ -122.057404, 37.419201 ], \\\"NTOPNG_INSTANCE_NAME\\\": \\\"ntop-redder-dev\\\", \\\"INTERFACE\\\": \\\"tcp:\\\\/\\\\/127.0.0.1:5559c [probeIP: 10.60.58.1]\\\" }\"}\r\n```\r\n```\r\n[2017-04-08T18:41:10,836][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: _LATENCY_MS\": 0.000000, \"SERVER_NW_LATENCY_MS\": 0.000000, \"DST_IP_COUNTRY\": \"US\", \"DST_IP_LOCATION\": [ -122.057404, 37.419201 ], \"NTOPNG_INSTANCE_NAME\": \"ntop-redder-dev\", \"INTERFACE\": \"tcp:\\/\\/127.0.0.1:5559c [probeIP: 10.60.58.1]\" }; line: 1, column: 2]>, :data=>\"\\u0000_LATENCY_MS\\\": 0.000000, \\\"SERVER_NW_LATENCY_MS\\\": 0.000000, \\\"DST_IP_COUNTRY\\\": \\\"US\\\", \\\"DST_IP_LOCATION\\\": [ -122.057404, 37.419201 ], \\\"NTOPNG_INSTANCE_NAME\\\": \\\"ntop-redder-dev\\\", \\\"INTERFACE\\\": \\\"tcp:\\\\/\\\\/127.0.0.1:5559c [probeIP: 10.60.58.1]\\\" }\"}\r\n```\r\n```\r\n[2017-04-08T18:58:21,387][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: 58.1]\" }; line: 1, column: 2]>, :data=>\"\\u000058.1]\\\" }\"}\r\n```\r\n```\r\n[2017-04-08T18:58:21,389][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: 58.1]\" }; line: 1, column: 2]>, :data=>\"\\u0000\\u000058.1]\\\" }\"}\r\n```\r\n```\r\n[2017-04-08T18:58:21,389][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: }; line: 1, column: 2]>, :data=>\"\\u0000}\"}\r\n```\r\n```\r\n[2017-04-08T18:58:21,389][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\\r, \\n, \\t) is allowed between tokens\r\n at [Source: 58.1]\" }; line: 1, column: 2]>, :data=>\"\\u000058.1]\\\" }\"}\r\n```\r\n\r\n\r\n## Logstash version: 5.2.2\r\n\r\n## % ntopng --version\r\n```\r\nv.2.5.170408 [Enterprise/Professional Edition]\r\nGIT rev:   dev:d5e73f423c1d1421feaf2d76c128f55452a27bdb:20170408\r\nPro rev:   r1036\r\nSystem Id: 39B3CB5200600F2D\r\nBuilt on:  CentOS Linux release 7.3.1611 (Core)\r\n```\r\n\r\n## % nprobe --version\r\n```\r\nWelcome to nProbe v.7.5.170408 (r5717) for x86_64-unknown-linux-gnu\r\n[snip]\r\nBuild OS:      CentOS Linux release 7.3.1611 (Core)\r\nSystemID:      39B3CB5200600F2D\r\nGIT rev:       dev:7f2b1b02fc599a445d0c5e62f732664ca9a64913:20170408\r\n```\r\n\r\n## ntopng config\r\n```\r\n-G=/var/run/ntopng.pid\r\n--community\r\n-i=tcp://127.0.0.1:5559c\r\n-F \"logstash;10.60.58.6;tcp;5000\"\r\n```\r\n\r\n## logstash config\r\n```\r\ninput { \r\n    tcp {\r\n        port => 5000\r\n        type => \"ntopng-ls\"\r\n        codec => json\r\n    }\r\n}\r\nfilter {\r\n   if [type] == \"ntopng-ls\" {\r\n     if \"\" not in [IPV4_SRC_ADDR] and \"\" not in [IPV6_SRC_ADDR] {\r\n        drop {}\r\n     }\r\n   }\r\n}\r\noutput {\r\n   if [type] == \"ntopng-ls\" {\r\n    elasticsearch { hosts => [\"elasticsearch-master:9200\"]  }\r\n    stdout { codec => rubydebug}\r\n   }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/ntop/ntopng/issues/comments/292924124","html_url":"https://github.com/ntop/ntopng/issues/1144#issuecomment-292924124","issue_url":"https://api.github.com/repos/ntop/ntopng/issues/1144","id":292924124,"user":{"login":"bperian","id":24822739,"avatar_url":"https://avatars0.githubusercontent.com/u/24822739?v=3","gravatar_id":"","url":"https://api.github.com/users/bperian","html_url":"https://github.com/bperian","followers_url":"https://api.github.com/users/bperian/followers","following_url":"https://api.github.com/users/bperian/following{/other_user}","gists_url":"https://api.github.com/users/bperian/gists{/gist_id}","starred_url":"https://api.github.com/users/bperian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bperian/subscriptions","organizations_url":"https://api.github.com/users/bperian/orgs","repos_url":"https://api.github.com/users/bperian/repos","events_url":"https://api.github.com/users/bperian/events{/privacy}","received_events_url":"https://api.github.com/users/bperian/received_events","type":"User","site_admin":false},"created_at":"2017-04-10T11:35:14Z","updated_at":"2017-04-10T11:35:14Z","body":"Issue should be fixed in https://github.com/ntop/ntopng/pull/1143 "}}
{"actor":{"display_login":"bperian"}}