{"action":"created","issue":{"url":"https://api.github.com/repos/apache/storm/issues/2241","repository_url":"https://api.github.com/repos/apache/storm","labels_url":"https://api.github.com/repos/apache/storm/issues/2241/labels{/name}","comments_url":"https://api.github.com/repos/apache/storm/issues/2241/comments","events_url":"https://api.github.com/repos/apache/storm/issues/2241/events","html_url":"https://github.com/apache/storm/pull/2241","id":245277979,"number":2241,"title":"STORM-2306 : Messaging subsystem redesign. ","user":{"login":"roshannaik","id":2366541,"avatar_url":"https://avatars1.githubusercontent.com/u/2366541?v=4","gravatar_id":"","url":"https://api.github.com/users/roshannaik","html_url":"https://github.com/roshannaik","followers_url":"https://api.github.com/users/roshannaik/followers","following_url":"https://api.github.com/users/roshannaik/following{/other_user}","gists_url":"https://api.github.com/users/roshannaik/gists{/gist_id}","starred_url":"https://api.github.com/users/roshannaik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roshannaik/subscriptions","organizations_url":"https://api.github.com/users/roshannaik/orgs","repos_url":"https://api.github.com/users/roshannaik/repos","events_url":"https://api.github.com/users/roshannaik/events{/privacy}","received_events_url":"https://api.github.com/users/roshannaik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":67,"created_at":"2017-07-25T03:48:50Z","updated_at":"2017-08-31T11:17:06Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/2241","html_url":"https://github.com/apache/storm/pull/2241","diff_url":"https://github.com/apache/storm/pull/2241.diff","patch_url":"https://github.com/apache/storm/pull/2241.patch"},"body":"Having spent a lot of time on this, I am happy to share some good news and some even better news with you. \r\n\r\nBefore venturing further, I must add, to limit the scope of this PR, no attempt was made to improve ACK-ing mode perf. Although there are some big latency gains seen in ACK mode, these are a side effect of the new messaging design and work remains to be done to improve ACK mode.   \r\n\r\nPlease see the design docs posted on the STORM-2306 jira for details on what is being done\r\n\r\nSo, first the good news .. a quick competitive evaluation:\r\n\r\n# 1) Competitive Perf evaluation : \r\n\r\nHere are some quick comparison of Storm numbers taken on my laptop against numbers for similar/identical topologies published by Heron, Flink and Apex. Shall provide just rolled up summary here and leave the detailed analysis for later.\r\n\r\nStorm numbers here were run on my MacBook Pro (2015) with 16GB ram and a single 4 core Intel i7 chip.\r\n\r\n\r\n### A) Compared To Heron and Flink:\r\n------------------------------\r\n Heron recently published this blog about the big perf improvements (~4-6x) they achieved.\r\nhttps://blog.twitter.com/engineering/en_us/topics/open-source/2017/optimizing-twitter-heron.html\r\nThey ran it on dual 12-core Intel Xeon chips (didn't say how many machines).\r\n\r\nThey use a simplified word count topology that I have emulated for comparison purposes and included it as part of this PR (SimplifiedWordCountTopo).\r\n\r\nFlink also publishes numbers for a similar setup here \r\nhttps://flink.apache.org/features.html#streaming\r\n\r\nBelow are per core throughput numbers.\r\n\r\n**[:HERON:]**\r\nAcking Disabled: per core = **~475 k/sec**.\r\nAcking Enabled:  per core = **~150 k/sec**. Latency = **30ms**\r\n\r\n**[:FLINK:]**\r\nPer core: **~1 mill/sec**\r\n\r\n**[:STORM:]**\r\nAcking Disabled: per core =   **2 mill/sec.**    (1 spout & 1 counter bolt)\r\nAcking Enabled:  per core = **0.6 mill/sec**, Latency = **0.73 ms**  (+1 acker)\r\n\r\n\r\n**Takeaways:** \r\n- Storm's with-ACK throughput is better than Heron's no-ACK throughput. \r\n- Without ACKs, Storm is far ahead of Heron and also better than Flink. \r\n- Storm's Latency (in microseconds) is also significantly better than both (although this metric is better to compared with multiple machines in the run). AFAIKT, Flink is generally not known for having good latency (as per Flink's own comparison with Storm -  https://data-artisans.com/blog/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink).\r\n\r\n\r\n\r\n### B) Compared to Apex:\r\n-----------------------------------\r\nApex appears to be the best performer among the opensource lot.. by a reasonably good margin. Some numbers they published in their early days (when it was called DataTorrent) were misleading/dubious IMO, but the newer numbers appear credible.\r\n\r\nHere we look at how fast inter spout/bolt communication can be achieved using an ultra minimalist topology.\r\nA ConstSpout emits a short string to a DevNull bolt that discards the tuples it receives. This topo has been in storm-perf for sometime now.\r\n\r\nApex provides numbers for a identical setup ... what they call \"container local\" performance here:\r\nhttps://www.datatorrent.com/blog/blog-apex-performance-benchmark/\r\n\r\nOther than the fact that Storm numbers were on my laptop, these numbers are a good apples to apples comparison.\r\n\r\n**[:APEX:]**\r\nContainer local Throughput : **~4.2 mill/sec**\r\n\r\n**[:STORM:]**\r\nWorker local throughput : **8.1 mill/sec**\r\n\r\n\r\n\r\n# 2) Core messaging Performance\r\n\r\nNow for the better news. The redesigned messaging system is actually much faster and able to move messages between threads at an astounding rate .... :\r\n\r\n- **120 mill/sec**  (batchSz=1000, 2 producers writing to 1 consumer).\r\n- **67 mill/sec**   (batchSz=1000, 1 producers writing to 1 consumer).\r\n\r\n\r\nI have included JCQueuePerfTest.java in this PR to help get quick measurements from within the IDE.\r\n\r\nThat naturally begs the question .. why is Storm pushing only 8.1 mill/sec between a ConstSpout and DevNullBolt ? The short answer is ... there are big bottlenecks in other parts of the code. In this PR I have tackled some such bottlenecks but many still remain. We are faster than the competition, but still have room to be much much faster. If anyone is interested in pursuing these to push Storm's perf to the next level, I am happy to point them in the right direction.\r\n\r\n\r\nAgain, please refer to the design docs in the JIRA for details on the new design and the rationale behind them.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/storm/issues/comments/326265560","html_url":"https://github.com/apache/storm/pull/2241#issuecomment-326265560","issue_url":"https://api.github.com/repos/apache/storm/issues/2241","id":326265560,"user":{"login":"roshannaik","id":2366541,"avatar_url":"https://avatars1.githubusercontent.com/u/2366541?v=4","gravatar_id":"","url":"https://api.github.com/users/roshannaik","html_url":"https://github.com/roshannaik","followers_url":"https://api.github.com/users/roshannaik/followers","following_url":"https://api.github.com/users/roshannaik/following{/other_user}","gists_url":"https://api.github.com/users/roshannaik/gists{/gist_id}","starred_url":"https://api.github.com/users/roshannaik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roshannaik/subscriptions","organizations_url":"https://api.github.com/users/roshannaik/orgs","repos_url":"https://api.github.com/users/roshannaik/repos","events_url":"https://api.github.com/users/roshannaik/events{/privacy}","received_events_url":"https://api.github.com/users/roshannaik/received_events","type":"User","site_admin":false},"created_at":"2017-08-31T11:17:06Z","updated_at":"2017-08-31T11:17:06Z","author_association":"CONTRIBUTOR","body":"@HeartSaVioR  : Agree with your observations on being able to process ACKs under BP. My latest update allows that in ACK mode. I have updated section 3.6 in the design doc with info on that. It has one limitation ....  in ACK=0 mode, it wont try to process metrics under BP. Will try to fix it in this PR if possible (or defer it to a followup PR if thats ok). \r\nFirst i would like to fix this one multi-worker issue that is remaining and finish collecting the numbers for the test plan to share with community.... As it also helps me identify any important problems."}}
{"actor":{"display_login":"roshannaik"}}