{"action":"created","issue":{"url":"https://api.github.com/repos/eclipse/vert.x/issues/1594","repository_url":"https://api.github.com/repos/eclipse/vert.x","labels_url":"https://api.github.com/repos/eclipse/vert.x/issues/1594/labels{/name}","comments_url":"https://api.github.com/repos/eclipse/vert.x/issues/1594/comments","events_url":"https://api.github.com/repos/eclipse/vert.x/issues/1594/events","html_url":"https://github.com/eclipse/vert.x/pull/1594","id":174005571,"number":1594,"title":"Implement High Fault Tolerance: Vertx Cluster is fault tolerant when several nodes are killed","user":{"login":"polipodi","id":21309845,"avatar_url":"https://avatars1.githubusercontent.com/u/21309845?v=3","gravatar_id":"","url":"https://api.github.com/users/polipodi","html_url":"https://github.com/polipodi","followers_url":"https://api.github.com/users/polipodi/followers","following_url":"https://api.github.com/users/polipodi/following{/other_user}","gists_url":"https://api.github.com/users/polipodi/gists{/gist_id}","starred_url":"https://api.github.com/users/polipodi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polipodi/subscriptions","organizations_url":"https://api.github.com/users/polipodi/orgs","repos_url":"https://api.github.com/users/polipodi/repos","events_url":"https://api.github.com/users/polipodi/events{/privacy}","received_events_url":"https://api.github.com/users/polipodi/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2016-08-30T12:26:38Z","updated_at":"2017-03-10T07:21:41Z","closed_at":"2017-03-03T14:18:02Z","pull_request":{"url":"https://api.github.com/repos/eclipse/vert.x/pulls/1594","html_url":"https://github.com/eclipse/vert.x/pull/1594","diff_url":"https://github.com/eclipse/vert.x/pull/1594.diff","patch_url":"https://github.com/eclipse/vert.x/pull/1594.patch"},"body":"Hi all,\n\nWe are big fans of VertX! We are working on setuping a cluster of at least 20 Vertx JVM in a single cluster. When starting playing killing nodes, more than one node in few seconds, we found that the vertx event bus cluster becomes unstable. After drilling down into the code, we found that the clusterMap of HAManager, and subs multimap of the ClusteredEventBus are often corrupted:\n- sometimes, references on nodes killed are still present in clusterMap and subs multimap\n- sometimes, references on nodes still running are missing in clusterMap and subs multimap\n\nSo we suggest to make sure to clean the maps from each node and not only from an elligible node. And we also suggest that each node is sending every 10 seconds their key/value into the clusterMap and subs multimap. \n\nThe main benefits are that when the nodes are killed, after few seconds, the map are correctly cleaned-up, and running nodes are populating again the maps. Even under load (we tested it), the cluster becomes available again and 100% operational. We tested up to 20 running nodes and 80% of nodes loss, while the system were under load.\nThe main drawback is that each node is responsible for sending (every 10s) their key/value constantly in the clusterMap and the subs multimap.\n\nSo we propose to enable this behavior when the HFT is enabled from VertxOptions (High Fault Tolerance), since it will add additional traffic on the cluster which is not always necessary depending the deployment. (by default set to false). We have also the capability to change the delay of sending data to maps (HFTInterval) which is 10s by default.\n\nPlease comment, accept, deny or anything else about this pull request. We are strongly considering Vertx at the heart of our new infra for deploying micro-services, and the fault tolerance is only thing that we see as a showstopper.\n\nSame issue already raised by other users: \n- https://github.com/eclipse/vert.x/pull/1593\n- https://github.com/vert-x3/vertx-hazelcast/issues/13\n"},"comment":{"url":"https://api.github.com/repos/eclipse/vert.x/issues/comments/285596798","html_url":"https://github.com/eclipse/vert.x/pull/1594#issuecomment-285596798","issue_url":"https://api.github.com/repos/eclipse/vert.x/issues/1594","id":285596798,"user":{"login":"tsegismont","id":1500598,"avatar_url":"https://avatars2.githubusercontent.com/u/1500598?v=3","gravatar_id":"","url":"https://api.github.com/users/tsegismont","html_url":"https://github.com/tsegismont","followers_url":"https://api.github.com/users/tsegismont/followers","following_url":"https://api.github.com/users/tsegismont/following{/other_user}","gists_url":"https://api.github.com/users/tsegismont/gists{/gist_id}","starred_url":"https://api.github.com/users/tsegismont/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsegismont/subscriptions","organizations_url":"https://api.github.com/users/tsegismont/orgs","repos_url":"https://api.github.com/users/tsegismont/repos","events_url":"https://api.github.com/users/tsegismont/events{/privacy}","received_events_url":"https://api.github.com/users/tsegismont/received_events","type":"User","site_admin":false},"created_at":"2017-03-10T07:21:41Z","updated_at":"2017-03-10T07:21:41Z","body":"An important part of the credit goes to Luc and Rikard who did great with\ntheir investigations and patches.\n\nLe 9 mars 2017 16:16, \"Jeff Long\" <notifications@github.com> a écrit :\n\n> Thanks so much for all the hard work, guys!\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/eclipse/vert.x/pull/1594#issuecomment-285379486>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe-auth/ABbltnq_UMylgFv-3a72Go1l1pJZAF6Hks5rkBfCgaJpZM4Jwd99>\n> .\n>\n"}}
{"actor":{"display_login":"tsegismont"}}