{"action":"closed","number":8500,"pull_request":{"url":"https://api.github.com/repos/elastic/logstash/pulls/8500","id":147419484,"html_url":"https://github.com/elastic/logstash/pull/8500","diff_url":"https://github.com/elastic/logstash/pull/8500.diff","patch_url":"https://github.com/elastic/logstash/pull/8500.patch","issue_url":"https://api.github.com/repos/elastic/logstash/issues/8500","number":8500,"state":"closed","locked":false,"title":"[Performance] (Java exec beats Ruby exec inside!) Faster Java Execution Using CodeGen/Janino","user":{"login":"original-brownbear","id":6490959,"avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"body":"Will have to document this better so WIP for now, functionally this step should be fine though.\r\n\r\n### Changes\r\n\r\n* Compile `Dataset`s for outputs in a more efficient optimal fashion\r\n  * Cache `Callsite` for `multi_receive` instead of implementing Java interface from Ruby (this has quite a bit of overhead)\r\n   * Don't loop over `Dataset`, unroll parent `Dataset` iteration at compile time\r\n   * Monomorphic callsites only, use runtime compilation to either cast each field in `OutputDataset` to its type or at least make sure that there is only one type for each `invokevirtual` call to begin with (in the case of runtime compiled `Dataset` as explained in the Javadoc)\r\n* Avoid instantiating a `TerminalDataset` if we only have a single output\r\n* Compile an iteration free version of `TerminalDataset` if we have multiple outputs\r\n* Pass `RubyArray` instead of `Batch` from Ruby to Java to avoid the redundant `to_a` Java -> Ruby -> Java round trip since we only use the contents of a batch and not the `Batch` instance itself on the Java side (also makes instantiating read batches cheaper)\r\n \r\n### Some general benchmark results\r\n\r\n* Apache Example: Improves significantly relative to the previous Java exec and is measurably faster than Ruby exec now (58k/s vs. 52k/s over largeish sample size)\r\n* Baseline (4 generator threads, 1 worker thread): **biggest improvement here** as expected since only the outputs are optimized using code gen for now, `master` Java exec is at around 350k/s, Ruby is at 360k/s and this branch at 580k/s for Java exec! Workers are a lot faster and less memory hungry now.\r\n* Baseline (1 generator, 1 worker): This case is interesting, it actually gets slower with the Java exec. The reason for this is that the performance of the in-memory queue degrades as soon as outputs start to outpace inputs. This is simply a result of an empty queue likely not filling a batch on the initial `drainTo` call => you need to poll the Queue => you get contention on its lock => writers to the queue wait more often and get throttled. Not much to do here, this is simply the way an `ArrayblockingQueue` works (google \"false sharing\" for the why behind a faster consumer making things slower here overall) and would require some other data structure that comes with more coarse locking on writes. For now, this is a pretty academic example anyways :)\r\n* Optimal worker count and batch size both are higher for Java execution than for Ruby => this is kind of nice (imo), especially when looking at ES output bulk request sizes\r\n\r\nData:\r\n\r\n```sh\r\n➜  logstash git:(faster-java-exec-filters) time cat ~/Downloads/apache_access_logs ~/Downloads/apache_access_logs | bin/logstash -w 4 -b 128 -f ~/tmp/logstash.cfg --experimental-java-execution | pv | wc -c\r\n13.2MiB 0:04:16 [52.5KiB/s] [                                                                                                                            <=>                 ]\r\n 13801638\r\ncat ~/Downloads/apache_access_logs ~/Downloads/apache_access_logs  0.11s user 2.53s system 1% cpu 4:15.81 total\r\nbin/logstash -w 4 -b 128 -f ~/tmp/logstash.cfg --experimental-java-execution  1080.48s user 25.05s system 430% cpu 4:16.59 total\r\npv  2.95s user 18.76s system 8% cpu 4:16.59 total\r\nwc -c  0.37s user 3.08s system 1% cpu 4:16.59 total\r\n➜  logstash git:(faster-java-exec-filters) time cat ~/Downloads/apache_access_logs ~/Downloads/apache_access_logs | bin/logstash -w 4 -b 128 -f ~/tmp/logstash.cfg | pv | wc -c          \r\n13.2MiB 0:04:46 [47.1KiB/s] [                                                                      <=>                                                                       ]\r\n 13801638\r\ncat ~/Downloads/apache_access_logs ~/Downloads/apache_access_logs  0.11s user 2.20s system 0% cpu 4:45.78 total\r\nbin/logstash -w 4 -b 128 -f ~/tmp/logstash.cfg  1202.31s user 24.85s system 428% cpu 4:46.39 total\r\npv  3.02s user 19.37s system 7% cpu 4:46.39 total\r\nwc -c  0.38s user 3.18s system 1% cpu 4:46.39 total\r\n```\r\n\r\n### Next Steps (imo)\r\n\r\n* Do the same thing I did here and implement more optimal runtime compiled for filter and conditional `Dataset`\r\n* Extract a common abstraction for describing the inputs to compiling a `Dataset` (like what buffers do I need, what `Callsite`s do I invoke and such)\r\n* Use the abstraction to compile every pipeline into just a single method (and be super cache efficient and fast :))","created_at":"2017-10-18T21:51:38Z","updated_at":"2017-10-27T17:16:30Z","closed_at":"2017-10-27T17:16:30Z","merged_at":null,"merge_commit_sha":"22c201d4c94cf9f666361e7fe1a7468e3504a081","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/elastic/logstash/pulls/8500/commits","review_comments_url":"https://api.github.com/repos/elastic/logstash/pulls/8500/comments","review_comment_url":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/8500/comments","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/83f407ab3330e28851a34ae2b5289f5f4b52f934","head":{"label":"original-brownbear:faster-java-exec-filters","ref":"faster-java-exec-filters","sha":"83f407ab3330e28851a34ae2b5289f5f4b52f934","user":{"login":"original-brownbear","id":6490959,"avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"repo":{"id":88715786,"name":"logstash","full_name":"original-brownbear/logstash","owner":{"login":"original-brownbear","id":6490959,"avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/original-brownbear/logstash","description":"Logstash - transport and process your logs, events, or other data","fork":true,"url":"https://api.github.com/repos/original-brownbear/logstash","forks_url":"https://api.github.com/repos/original-brownbear/logstash/forks","keys_url":"https://api.github.com/repos/original-brownbear/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/original-brownbear/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/original-brownbear/logstash/teams","hooks_url":"https://api.github.com/repos/original-brownbear/logstash/hooks","issue_events_url":"https://api.github.com/repos/original-brownbear/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/original-brownbear/logstash/events","assignees_url":"https://api.github.com/repos/original-brownbear/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/original-brownbear/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/original-brownbear/logstash/tags","blobs_url":"https://api.github.com/repos/original-brownbear/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/original-brownbear/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/original-brownbear/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/original-brownbear/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/original-brownbear/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/original-brownbear/logstash/languages","stargazers_url":"https://api.github.com/repos/original-brownbear/logstash/stargazers","contributors_url":"https://api.github.com/repos/original-brownbear/logstash/contributors","subscribers_url":"https://api.github.com/repos/original-brownbear/logstash/subscribers","subscription_url":"https://api.github.com/repos/original-brownbear/logstash/subscription","commits_url":"https://api.github.com/repos/original-brownbear/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/original-brownbear/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/original-brownbear/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/original-brownbear/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/original-brownbear/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/original-brownbear/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/original-brownbear/logstash/merges","archive_url":"https://api.github.com/repos/original-brownbear/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/original-brownbear/logstash/downloads","issues_url":"https://api.github.com/repos/original-brownbear/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/original-brownbear/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/original-brownbear/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/original-brownbear/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/original-brownbear/logstash/labels{/name}","releases_url":"https://api.github.com/repos/original-brownbear/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/original-brownbear/logstash/deployments","created_at":"2017-04-19T07:40:07Z","updated_at":"2017-04-19T07:40:11Z","pushed_at":"2017-10-27T14:57:12Z","git_url":"git://github.com/original-brownbear/logstash.git","ssh_url":"git@github.com:original-brownbear/logstash.git","clone_url":"https://github.com/original-brownbear/logstash.git","svn_url":"https://github.com/original-brownbear/logstash","homepage":"https://www.elastic.co/products/logstash","size":98103,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"4bbdbca689ae4718d55f7ffe7d7a10f93ba1c083","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars0.githubusercontent.com/u/6764390?v=4","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":1090311,"name":"logstash","full_name":"elastic/logstash","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars0.githubusercontent.com/u/6764390?v=4","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/logstash","description":"Logstash - transport and process your logs, events, or other data","fork":false,"url":"https://api.github.com/repos/elastic/logstash","forks_url":"https://api.github.com/repos/elastic/logstash/forks","keys_url":"https://api.github.com/repos/elastic/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/logstash/teams","hooks_url":"https://api.github.com/repos/elastic/logstash/hooks","issue_events_url":"https://api.github.com/repos/elastic/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/logstash/events","assignees_url":"https://api.github.com/repos/elastic/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/logstash/tags","blobs_url":"https://api.github.com/repos/elastic/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/logstash/languages","stargazers_url":"https://api.github.com/repos/elastic/logstash/stargazers","contributors_url":"https://api.github.com/repos/elastic/logstash/contributors","subscribers_url":"https://api.github.com/repos/elastic/logstash/subscribers","subscription_url":"https://api.github.com/repos/elastic/logstash/subscription","commits_url":"https://api.github.com/repos/elastic/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/logstash/merges","archive_url":"https://api.github.com/repos/elastic/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/logstash/downloads","issues_url":"https://api.github.com/repos/elastic/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/logstash/labels{/name}","releases_url":"https://api.github.com/repos/elastic/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/logstash/deployments","created_at":"2010-11-18T01:31:50Z","updated_at":"2017-10-27T09:47:25Z","pushed_at":"2017-10-27T17:16:20Z","git_url":"git://github.com/elastic/logstash.git","ssh_url":"git@github.com:elastic/logstash.git","clone_url":"https://github.com/elastic/logstash.git","svn_url":"https://github.com/elastic/logstash","homepage":"https://www.elastic.co/products/logstash","size":97342,"stargazers_count":8113,"watchers_count":8113,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2279,"mirror_url":null,"archived":false,"open_issues_count":1011,"forks":2279,"open_issues":1011,"watchers":8113,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/logstash/pulls/8500"},"html":{"href":"https://github.com/elastic/logstash/pull/8500"},"issue":{"href":"https://api.github.com/repos/elastic/logstash/issues/8500"},"comments":{"href":"https://api.github.com/repos/elastic/logstash/issues/8500/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/logstash/pulls/8500/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/logstash/pulls/8500/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/logstash/statuses/83f407ab3330e28851a34ae2b5289f5f4b52f934"}},"author_association":"MEMBER","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":9,"review_comments":0,"maintainer_can_modify":false,"commits":1,"additions":370,"deletions":187,"changed_files":12}}
{"actor":{"display_login":"elasticsearch-bot"}}