{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/6085","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/6085/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/6085/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/6085/events","html_url":"https://github.com/elastic/logstash/issues/6085","id":183703566,"number":6085,"title":"queue drainage options at shutdown","user":{"login":"colinsurprenant","id":2010,"avatar_url":"https://avatars.githubusercontent.com/u/2010?v=3","gravatar_id":"","url":"https://api.github.com/users/colinsurprenant","html_url":"https://github.com/colinsurprenant","followers_url":"https://api.github.com/users/colinsurprenant/followers","following_url":"https://api.github.com/users/colinsurprenant/following{/other_user}","gists_url":"https://api.github.com/users/colinsurprenant/gists{/gist_id}","starred_url":"https://api.github.com/users/colinsurprenant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinsurprenant/subscriptions","organizations_url":"https://api.github.com/users/colinsurprenant/orgs","repos_url":"https://api.github.com/users/colinsurprenant/repos","events_url":"https://api.github.com/users/colinsurprenant/events{/privacy}","received_events_url":"https://api.github.com/users/colinsurprenant/received_events","type":"User","site_admin":false},"labels":[{"id":140543350,"url":"https://api.github.com/repos/elastic/logstash/labels/discuss","name":"discuss","color":"cc317c","default":false},{"id":176735639,"url":"https://api.github.com/repos/elastic/logstash/labels/resiliency","name":"resiliency","color":"bfd4f2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-10-18T14:30:35Z","updated_at":"2017-02-23T22:19:47Z","closed_at":null,"body":"Historically logstash, for obvious reasons, always drained its internal in-memory queue(s) upon shutdown. \n\nNow with the persistent queue, this is not required anymore, a clean shutdown can happen at any time and the queue does not require to be drained as persisted events will be picked up where they were left upon restarting logstash. This is the current implementation in #5958. \n\nI think we should also support an option where logstash can wait until the queue is drained before completing its shutdown, similar to the legacy implementation. One of the problem is with the input plugins which can initiate a shutdown like the generator input: For a given configuration, if filter+output is slower, the input triggered shutdown will leave unprocessed events in the queue and this can be counter intuitive in a \"getting started\" scenario or create weird problems in specs like in #6084\n- What should be the default behaviour? In a \"getting started\" use-case it is probably expected that all events will be processed before completing a shutdown\n- In production use-case I believe a quick-clean shutdown is a better default as the queue could/will grow to a large number of events and waiting for it to drain could be a problem. \n- If we ever switch the in-memory queuing to use the in-memory PQ backend instead of the current queuing implementation, this option should definitely be enabled to respect the in-memory queuing behaviour to not loose inflight event upon shutdown.\n\nThoughts?\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/282140370","html_url":"https://github.com/elastic/logstash/issues/6085#issuecomment-282140370","issue_url":"https://api.github.com/repos/elastic/logstash/issues/6085","id":282140370,"user":{"login":"colinsurprenant","id":2010,"avatar_url":"https://avatars.githubusercontent.com/u/2010?v=3","gravatar_id":"","url":"https://api.github.com/users/colinsurprenant","html_url":"https://github.com/colinsurprenant","followers_url":"https://api.github.com/users/colinsurprenant/followers","following_url":"https://api.github.com/users/colinsurprenant/following{/other_user}","gists_url":"https://api.github.com/users/colinsurprenant/gists{/gist_id}","starred_url":"https://api.github.com/users/colinsurprenant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinsurprenant/subscriptions","organizations_url":"https://api.github.com/users/colinsurprenant/orgs","repos_url":"https://api.github.com/users/colinsurprenant/repos","events_url":"https://api.github.com/users/colinsurprenant/events{/privacy}","received_events_url":"https://api.github.com/users/colinsurprenant/received_events","type":"User","site_admin":false},"created_at":"2017-02-23T22:19:47Z","updated_at":"2017-02-23T22:19:47Z","body":"@guyboertje good point.\r\n\r\nI rebooted the drain PR #6433 and would like it to move forward. /cc @suyograo "}}
{"actor":{"display_login":"colinsurprenant"}}