{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/flannel/issues/808","repository_url":"https://api.github.com/repos/coreos/flannel","labels_url":"https://api.github.com/repos/coreos/flannel/issues/808/labels{/name}","comments_url":"https://api.github.com/repos/coreos/flannel/issues/808/comments","events_url":"https://api.github.com/repos/coreos/flannel/issues/808/events","html_url":"https://github.com/coreos/flannel/pull/808","id":255973330,"number":808,"title":"Ensure that the iptables masquerade rule is always installed","user":{"login":"julia-stripe","id":23065472,"avatar_url":"https://avatars3.githubusercontent.com/u/23065472?v=4","gravatar_id":"","url":"https://api.github.com/users/julia-stripe","html_url":"https://github.com/julia-stripe","followers_url":"https://api.github.com/users/julia-stripe/followers","following_url":"https://api.github.com/users/julia-stripe/following{/other_user}","gists_url":"https://api.github.com/users/julia-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/julia-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/julia-stripe/subscriptions","organizations_url":"https://api.github.com/users/julia-stripe/orgs","repos_url":"https://api.github.com/users/julia-stripe/repos","events_url":"https://api.github.com/users/julia-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/julia-stripe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-09-07T15:14:36Z","updated_at":"2017-09-12T09:17:48Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/coreos/flannel/pulls/808","html_url":"https://github.com/coreos/flannel/pull/808","diff_url":"https://github.com/coreos/flannel/pull/808.diff","patch_url":"https://github.com/coreos/flannel/pull/808.patch"},"body":"We ran into an issue where sometimes the iptables masquerade rules that flannel creates get deleted. Obviously ideally this wouldn't happen, but if it *does* happen it's really bad -- containers can't send packets to any hosts outside the cluster.\r\n\r\nThis PR makes flannel automatically recover from this situation by recreating the iptables rules, so that the cluster will remain healthy:\r\n\r\n* added a goroutine that automatically recreates the masquerade rules every second\r\n* added an alternate version of `AppendUnique` which returns whether or not the rule was appended. This allows flannel to only log when a rule was actually successfully added, instead of always logging (which, now that we're retrying every second, would add too much noise to the logs)\r\n\r\nWe're running this in our production cluster now"},"comment":{"url":"https://api.github.com/repos/coreos/flannel/issues/comments/328794134","html_url":"https://github.com/coreos/flannel/pull/808#issuecomment-328794134","issue_url":"https://api.github.com/repos/coreos/flannel/issues/808","id":328794134,"user":{"login":"fasaxc","id":469264,"avatar_url":"https://avatars0.githubusercontent.com/u/469264?v=4","gravatar_id":"","url":"https://api.github.com/users/fasaxc","html_url":"https://github.com/fasaxc","followers_url":"https://api.github.com/users/fasaxc/followers","following_url":"https://api.github.com/users/fasaxc/following{/other_user}","gists_url":"https://api.github.com/users/fasaxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fasaxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fasaxc/subscriptions","organizations_url":"https://api.github.com/users/fasaxc/orgs","repos_url":"https://api.github.com/users/fasaxc/repos","events_url":"https://api.github.com/users/fasaxc/events{/privacy}","received_events_url":"https://api.github.com/users/fasaxc/received_events","type":"User","site_admin":false},"created_at":"2017-09-12T09:17:48Z","updated_at":"2017-09-12T09:17:48Z","author_association":"NONE","body":"On (1), looks like the code checks for the existence of the rule first, so it's 4 reads per second.  That  could be moderately expensive if there are a lot of rules (since most iptables operations do a full table load from the kernel to userspace, even to check one rule).\r\n\r\nOn (2), as long as the other process isn't deleting these rules actively then we should be OK (and if it was actively trying to delete those rules then there's not much we can do.\r\n\r\nLonger term, you may want to migrate to having only a single rule inserted into the root chain and then jumping fro there to a flannel-owned chain.  That's the approach that we have in Calico; it reduces cloberring since there's only one rule and it avoids reordering issues."}}
{"actor":{"display_login":"fasaxc"}}