{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/7691","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/7691/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/7691/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/7691/events","html_url":"https://github.com/elastic/logstash/pull/7691","id":242929472,"number":7691,"title":"#7690 Fix needless waiting","user":{"login":"original-brownbear","id":6490959,"avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"labels":[{"id":227203799,"url":"https://api.github.com/repos/elastic/logstash/labels/performance%20improvements","name":"performance improvements","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-07-14T08:20:21Z","updated_at":"2017-07-19T21:55:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/logstash/pulls/7691","html_url":"https://github.com/elastic/logstash/pull/7691","diff_url":"https://github.com/elastic/logstash/pull/7691.diff","patch_url":"https://github.com/elastic/logstash/pull/7691.patch"},"body":"For #7690:\r\n\r\n* This also contains #7623 \r\n* Instead of the synchronous queue, use an array blocking queue of `2 * batch_size * worker_count` size to avoid needless sleeping\r\n* Use `drainTo` to require far less locking\r\n* Use a `HashSet` with appropriate sizing and load parameter to back the batch\r\n* Note how I didn't yet fix 4 specs that use some \"DummyQueue\" which are broken now, functionally this thing is 100% fine though as far as I can tell\r\n\r\n## Numbers:\r\n\r\nApache Dataset, command: `time cat ~/Downloads/apache_access_logs | bin/logstash -b 1024 -w 2 -f ~/tmp/logstash.cfg  | pv  | wc -c`\r\n\r\n### Master\r\n\r\n```sh\r\n➜  logstash git:(master) time cat ~/Downloads/apache_access_logs | bin/logstash -b 1024 -w 2 -f ~/tmp/logstash.cfg  | pv  | wc -c\r\n6.58MiB 0:10:17 [10.9KiB/s] [                     <=>                                                                                                                                   ]\r\n 6901231\r\ncat ~/Downloads/apache_access_logs  0.06s user 1.01s system 0% cpu 10:17.43 total\r\nbin/logstash -b 1024 -w 2 -f ~/tmp/logstash.cfg  1792.97s user 41.74s system 297% cpu 10:17.69 total\r\npv  0.98s user 6.64s system 1% cpu 10:17.69 total\r\nwc -c  0.09s user 0.60s system 0% cpu 10:17.69 total\r\n```\r\n\r\n** (2.97 * 10.3 ~ 31 min) **\r\n\r\n### This Branch\r\n\r\n```sh\r\n➜  logstash git:(fix-needless-waiting) time cat ~/Downloads/apache_access_logs | bin/logstash -b 1024 -w 2 -f ~/tmp/logstash.cfg  | pv  | wc -c\r\n6.58MiB 0:09:49 [11.4KiB/s] [                                                               <=>                                                                                         ]\r\n 6900962\r\ncat ~/Downloads/apache_access_logs  0.08s user 1.63s system 0% cpu 9:48.80 total\r\nbin/logstash -b 1024 -w 2 -f ~/tmp/logstash.cfg  1899.15s user 16.16s system 324% cpu 9:49.37 total\r\npv  1.10s user 6.99s system 1% cpu 9:49.37 total\r\nwc -c  0.11s user 0.85s system 0% cpu 9:49.37 total\r\n```\r\n\r\n** (3.24 * 9.8 ~ 31min) **\r\n\r\n=> Master takes `6%` more wall clock time and the same amount of CPU time overall.\r\n\r\nAlso visible when looking at the profiler timelines for the threads, lots more yellow for master's workers:\r\n\r\n<img width=\"1152\" alt=\"screen shot 2017-07-14 at 10 03 04\" src=\"https://user-images.githubusercontent.com/6490959/28204896-16158636-6880-11e7-8130-9333ce76d313.png\">\r\n\r\nwhen compared to this branch:\r\n\r\n<img width=\"1289\" alt=\"screen shot 2017-07-14 at 09 47 42\" src=\"https://user-images.githubusercontent.com/6490959/28204905-1cef60b2-6880-11e7-8ffa-7c83ef985183.png\">\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/316529886","html_url":"https://github.com/elastic/logstash/pull/7691#issuecomment-316529886","issue_url":"https://api.github.com/repos/elastic/logstash/issues/7691","id":316529886,"user":{"login":"jordansissel","id":131818,"avatar_url":"https://avatars1.githubusercontent.com/u/131818?v=4","gravatar_id":"","url":"https://api.github.com/users/jordansissel","html_url":"https://github.com/jordansissel","followers_url":"https://api.github.com/users/jordansissel/followers","following_url":"https://api.github.com/users/jordansissel/following{/other_user}","gists_url":"https://api.github.com/users/jordansissel/gists{/gist_id}","starred_url":"https://api.github.com/users/jordansissel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordansissel/subscriptions","organizations_url":"https://api.github.com/users/jordansissel/orgs","repos_url":"https://api.github.com/users/jordansissel/repos","events_url":"https://api.github.com/users/jordansissel/events{/privacy}","received_events_url":"https://api.github.com/users/jordansissel/received_events","type":"User","site_admin":false},"created_at":"2017-07-19T21:55:23Z","updated_at":"2017-07-19T21:55:23Z","body":"CI failed, but it seems unrelated.\r\n\r\n```\r\nFailures:\r\n\r\n  1) LogStash::Pipeline when collecting metrics in the pipeline global metric populates the different metrics\r\n     Failure/Error: end.to be_truthy\r\n\r\n       expected: truthy value\r\n            got: false\r\n     # /var/lib/jenkins/workspace/elastic+logstash+pull-request+multijob-linux-intake/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait/handler.rb:13:in `block in handle_matcher'\r\n\r\n...\r\n\r\nFinished in 12 minutes 53 seconds (files took 6.94 seconds to load)\r\n2885 examples, 1 failure, 3 pending\r\n```"}}
{"actor":{"display_login":"jordansissel"}}