{"action":"closed","issue":{"url":"https://api.github.com/repos/mozilla-services/hindsight/issues/120","repository_url":"https://api.github.com/repos/mozilla-services/hindsight","labels_url":"https://api.github.com/repos/mozilla-services/hindsight/issues/120/labels{/name}","comments_url":"https://api.github.com/repos/mozilla-services/hindsight/issues/120/comments","events_url":"https://api.github.com/repos/mozilla-services/hindsight/issues/120/events","html_url":"https://github.com/mozilla-services/hindsight/issues/120","id":249205877,"number":120,"title":"hindsight appears to lose a single message when recovering from failure","user":{"login":"joell","id":840599,"avatar_url":"https://avatars1.githubusercontent.com/u/840599?v=4","gravatar_id":"","url":"https://api.github.com/users/joell","html_url":"https://github.com/joell","followers_url":"https://api.github.com/users/joell/followers","following_url":"https://api.github.com/users/joell/following{/other_user}","gists_url":"https://api.github.com/users/joell/gists{/gist_id}","starred_url":"https://api.github.com/users/joell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joell/subscriptions","organizations_url":"https://api.github.com/users/joell/orgs","repos_url":"https://api.github.com/users/joell/repos","events_url":"https://api.github.com/users/joell/events{/privacy}","received_events_url":"https://api.github.com/users/joell/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-08-10T01:17:39Z","updated_at":"2017-08-21T15:12:21Z","closed_at":"2017-08-21T15:12:21Z","body":"Consider the following hindsight input and output (with a default configuration):\r\n```lua\r\n-- input/tail.cfg\r\nfilename              = \"tail.lua\"\r\ninstruction_limit     = 0\r\nticker_interval       = 1\r\nshutdown_on_terminate = true\r\n\r\ninput_filename        = \"/tmp/hindsight-in.txt\"\r\n\r\ndefault_headers       = {\r\n  Type = \"TEST\"\r\n}\r\n\r\n-- output/forward.cfg\r\nfilename              = \"heka_tcp.lua\"\r\ninstruction_limit     = 0\r\nshutdown_on_terminate = true\r\nmessage_matcher       = \"TRUE\"\r\n\r\naddress               = \"127.0.0.1\"\r\nport                  = 4242\r\ntimeout               = 1\r\n```\r\n\r\nNow consider the following sequence of commands:\r\n```sh\r\n### On terminal one ###\r\n\r\n$ hindsight defaults.cfg 7\r\n\r\n### On terminal two ###\r\n\r\n# start TCP receiver that spits frames to stdout\r\n$ socat TCP-L:4242,fork - &  socat_pid=$!\r\n# send a test event and watch it appear on stdout\r\n$ echo TEST1 >> /tmp/hindsight-in.txt\r\n\r\n# kill the TCP receiver and send a second test event\r\n$ kill $socat_pid\r\n$ echo TEST2 >> /tmp/hindsight-in.txt\r\n\r\n# bring the TCP receiver back up\r\n$ socat TCP-L:4242,fork - &\r\n# (the second test event never appears)\r\n\r\n# send a third test event and watch it appear on stdout\r\n$ echo TEST3 >> /tmp/hindsight-in.txt\r\n# (hindsight generates a \"retry message <N> err: closed\" error)\r\n```\r\n\r\nThe expected result would be that when the `TEST2` event is sent while the TCP receiver is down that Hindsight would fail to deliver it, retry, and eventually deliver it when the TCP receiver comes back up.  Looking at the luasandbox-extensions code for `heka_tcp.lua` I don't see any edge case where the TCP send failure wouldn't result in a proper `-3` (retry) error code.  Yet, though the `heka_tcp` output does re-establish the connection once the receiver is brought back up, and the `TEST3` event makes it across, the `TEST2` event disappears into the ether.\r\n\r\n(I used Hindsight 0.14.2, luasandbox 1.2.5, and luasandbox-extensions 2017.06.11 in testing this issue.)"}}
{"actor":{"display_login":"joell"}}