{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/kube-aws/issues/224","repository_url":"https://api.github.com/repos/coreos/kube-aws","labels_url":"https://api.github.com/repos/coreos/kube-aws/issues/224/labels{/name}","comments_url":"https://api.github.com/repos/coreos/kube-aws/issues/224/comments","events_url":"https://api.github.com/repos/coreos/kube-aws/issues/224/events","html_url":"https://github.com/coreos/kube-aws/issues/224","id":199735407,"number":224,"title":"Tainted but idle workers trend to 100% CPU and fail within a few hours","user":{"login":"whereisaaron","id":1109585,"avatar_url":"https://avatars.githubusercontent.com/u/1109585?v=3","gravatar_id":"","url":"https://api.github.com/users/whereisaaron","html_url":"https://github.com/whereisaaron","followers_url":"https://api.github.com/users/whereisaaron/followers","following_url":"https://api.github.com/users/whereisaaron/following{/other_user}","gists_url":"https://api.github.com/users/whereisaaron/gists{/gist_id}","starred_url":"https://api.github.com/users/whereisaaron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whereisaaron/subscriptions","organizations_url":"https://api.github.com/users/whereisaaron/orgs","repos_url":"https://api.github.com/users/whereisaaron/repos","events_url":"https://api.github.com/users/whereisaaron/events{/privacy}","received_events_url":"https://api.github.com/users/whereisaaron/received_events","type":"User","site_admin":false},"labels":[{"id":492983918,"url":"https://api.github.com/repos/coreos/kube-aws/labels/waiting%20for%20community%20feedback","name":"waiting for community feedback","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-01-10T05:23:12Z","updated_at":"2017-01-10T08:30:18Z","closed_at":null,"body":"So I created a test cluster with 0 workers and in the main pool, a regular pool of 2 workers, and a spot pool of 3 workers. The 3 spot pool workers were tainted and I scheduled nothing on them. When I came back the next day, all the spot instances had flipped to 'NotReady', two can't be pinged any more and the third I can ping but not ssh to (times out). Checking the monitoring, all three have their CPU pegged at 100%.\r\n```\r\nNAME                                        STATUS                     AGE        \r\nip-172-31-106-46.region.compute.internal    Ready,SchedulingDisabled   1d\r\nip-172-31-107-195.region.compute.internal   Ready                      1d\r\nip-172-31-107-39.region.compute.internal    Ready                      1d\r\nip-172-31-108-131.region.compute.internal   NotReady                   1d\r\nip-172-31-108-16.region.compute.internal    NotReady                   1d\r\nip-172-31-108-68.region.compute.internal    NotReady                   1d\r\n```\r\n\r\nThis is similar to the problem if you run the `untaint-and-cordon` task on the controller (discussed in #199). The task is logged as failed (even if it succeeds) as (`rkt rm` fails), so the task repeats continuously. Some sort of garbage/clean-up accumulates and the controller eventually hits 100% CPU and grinds itself to death. After patching out the `untaint-and-cordon` task on my controllers I no longer have this problem (on the controllers).\r\n\r\nIt seem like any workers might have the same problem if you use the `experimental.taints` option. This is what I used on my spot instances:\r\n```\r\nexperimental:\r\n  taints:\r\n    - key: lifecycle\r\n      value: spot\r\n      effect: NoSchedule\r\n```\r\n\r\nHere is the CPU history of the now unpingable tainted spot workers:\r\n\r\n![dead-worker](https://cloud.githubusercontent.com/assets/1109585/21794572/7d36078a-d6c9-11e6-8d51-6e4fe262db0e.png)\r\n\r\n![dead-m4](https://cloud.githubusercontent.com/assets/1109585/21794708/cff6d1e2-d6ca-11e6-9b0c-20d0f75e5796.png)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/kube-aws/issues/comments/271515100","html_url":"https://github.com/coreos/kube-aws/issues/224#issuecomment-271515100","issue_url":"https://api.github.com/repos/coreos/kube-aws/issues/224","id":271515100,"user":{"login":"redbaron","id":16624,"avatar_url":"https://avatars.githubusercontent.com/u/16624?v=3","gravatar_id":"","url":"https://api.github.com/users/redbaron","html_url":"https://github.com/redbaron","followers_url":"https://api.github.com/users/redbaron/followers","following_url":"https://api.github.com/users/redbaron/following{/other_user}","gists_url":"https://api.github.com/users/redbaron/gists{/gist_id}","starred_url":"https://api.github.com/users/redbaron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redbaron/subscriptions","organizations_url":"https://api.github.com/users/redbaron/orgs","repos_url":"https://api.github.com/users/redbaron/repos","events_url":"https://api.github.com/users/redbaron/events{/privacy}","received_events_url":"https://api.github.com/users/redbaron/received_events","type":"User","site_admin":false},"created_at":"2017-01-10T08:30:18Z","updated_at":"2017-01-10T08:30:18Z","body":"What about keeping `rkt` but ignoring `rkt rm` errors? I am doing that in https://github.com/coreos/kube-aws/pull/217 \r\n\r\nThis would not trigger tasks restart and therefore no garbage will be accumulated with some \"undeletable\" garbage left until rkt bug is fixed.\r\n\r\nI think ignoring `rkt rm` errors is fine as it is a cleanup task and main goal of the script already achieved."}}
{"actor":{"display_login":"redbaron"}}