{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/6700","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/6700/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/6700/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/6700/events","html_url":"https://github.com/elastic/logstash/issues/6700","id":207372849,"number":6700,"title":"Dead letter queue, resurrected","user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars2.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"labels":[{"id":90997545,"url":"https://api.github.com/repos/elastic/logstash/labels/enhancement","name":"enhancement","color":"009800","default":true},{"id":176735639,"url":"https://api.github.com/repos/elastic/logstash/labels/resiliency","name":"resiliency","color":"bfd4f2","default":false}],"state":"closed","locked":false,"assignee":{"login":"talevy","id":388837,"avatar_url":"https://avatars3.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"assignees":[{"login":"talevy","id":388837,"avatar_url":"https://avatars3.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2017-02-13T23:35:36Z","updated_at":"2017-05-15T17:29:36Z","closed_at":"2017-05-15T17:29:36Z","body":"## Dead Letter Queue (DLQ) Feature \r\n\r\nAbility to shunt poisoned or unsuccessful events in the running pipeline to a new destination for further processing. This feature allows the existing pipeline to continue processing events without getting stuck because of a bad event.\r\n\r\n## Phase 1 Goals:\r\n\r\n### Storing dead events\r\nGoals from the producer side, which is a Logstash instance creating dead letter queue events.\r\n\r\n* Production Logstash shouldn't be stopped to consume or manage the DLQ.\r\n* DLQ should be local to the Logstash instance. This means we should have a file based implementation where events get delivered.\r\n* The DLQ should rely on OS managed flush mechanism for durability. \r\n* DLQ has an inbuilt rotation policy to manage its file size. Without rotation, file size can get too big and would need manual curation. Once the threshold has been reached, the DLQ would behave like a ring buffer. Older events would be overwritten to accommodate new events.\r\n\r\n-----------------------------------------------------------------\r\n\r\n## Processing dead events\r\n\r\nOnce the DLQ is filled with events, users should be able to query, analyze and process events. What a user does with the dead event depends on the original error. For example, a mapping error in ES would create a DLQ entry. The user can then choose to remove the field causing the mapping issue and re-index the _cleaned_ event to ES.\r\n\r\n* A new DLQ input should be created to process events in DLQ. Multiple concurrent consumers (DLQ inputs) should be supported.\r\n* Users of DLQ input should only care about top-level DLQ path and not the underlying folder structure. This input should work even if the original Logstash that created the dead events is shutdown.\r\n* Each DLQ input should manage its own offsets (like SinceDB). It should support disabling offset management so users can stream events from the DLQ for a _dry run_, without keeping states. \r\n* DLQ input supports sequential access to read from DLQ for phase 1\r\n\r\n## Sample configuration\r\n\r\n### logstash.yml\r\n\r\n```yaml\r\nfeature.dead_letter_queue: true\r\n\r\n# retention\r\ndead_letter_queue.size.threshold:  24h\r\npath.dead_letter_queue: /path/to/dlq\r\n```\r\n\r\n### DLQ input\r\n\r\n```\r\ninput {\r\n   dlq {\r\n       path => /path/to/dlq\r\n       timestamp => \"2017-04-04T23:40:37\"\r\n       commit_offsets => false\r\n    }\r\n}\r\n\r\nfilter {}\r\noutput {\r\n   elasticsearch {}\r\n}\r\n```\r\n--------------------------------------------------------------------------------\r\n\r\n### Future Goals\r\n* Support range query in the DLQ input\r\n* Support a direct ES output\r\n* Plugins should be able to publish dead events to separate \"topics/tags\". This would allow for easy consumption later on, based on filtering by the type of plugin. Each plugin in Logstash will create a separate dead letter entry grouped by plugin name and stage. This physical separation on the file system allows for faster query/filtering when processing events from DLQ. Sequential access (having all events in one single file) would work but will make processing slower\r\non the consumer side. Rotation threshold is applied per tag, not globally.\r\n\r\n### Directory Structure (Future)\r\n\r\n```sh\r\npath.dlq\r\n   |\r\n   pipeline_id\r\n      | \r\n       outputs_elasticsearch\r\n       inputs_kafka\r\n```\r\n\r\nwhere `outputs_elasticsearch` contains events DLQ'd from ES outputs from the original production LS.\r\n\r\n"}}
{"actor":{"display_login":"talevy"}}