{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/782","repository_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws","labels_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/782/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/782/comments","events_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/782/events","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/782","id":244059804,"number":782,"title":"Optimize usage of AWS AutoScaling API","user":{"login":"danielfm","id":29534,"avatar_url":"https://avatars2.githubusercontent.com/u/29534?v=4","gravatar_id":"","url":"https://api.github.com/users/danielfm","html_url":"https://github.com/danielfm","followers_url":"https://api.github.com/users/danielfm/followers","following_url":"https://api.github.com/users/danielfm/following{/other_user}","gists_url":"https://api.github.com/users/danielfm/gists{/gist_id}","starred_url":"https://api.github.com/users/danielfm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielfm/subscriptions","organizations_url":"https://api.github.com/users/danielfm/orgs","repos_url":"https://api.github.com/users/danielfm/repos","events_url":"https://api.github.com/users/danielfm/events{/privacy}","received_events_url":"https://api.github.com/users/danielfm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-07-19T14:26:10Z","updated_at":"2017-07-20T13:39:23Z","closed_at":null,"body":"I'm consistently being throttled on `DescribeAutoScalingInstances` and `DescribeAutoScalingGroups` requests to the AWS API, most likely due to having both the cluster autoscaler and node drainer features enabled.\r\n\r\nFor the cluster autoscaler, we can provide a way for users to customize the scan interval in order to compensate for larger clusters with a high number of auto scaling groups. I'm not familiar with the cluster autoscaler code to know if there are anything we can optimize there, so I'm not commenting on that.\r\n\r\nNow, regarding the node drainer, we can probably make the current implementation much more efficient by running it in a single pod (instead of as a DaemonSet), that does something like this:\r\n\r\n```bash\r\n# This would be the node draining loop:\r\n\r\n# 1. Take the externalID (which maps to the EC2 instance ID) from all nodes\r\n# (We need to return some `externalID -> node name` mapping though, so we know which node to drain based on its EC2 instance ID)\r\n$ kubectl get nodes -o jsonpath='{range .items[*]}\"{.spec.externalID}\" {end}'\r\n\r\n# 2. Use a single call to retrieve the current status for all nodes\r\n# (We can only specify 50 IDs at a time, so we need to be careful in order to handle large clusters properly)\r\n$ aws autoscaling describe-auto-scaling-instances --instance-ids <ids>\r\n\r\n# 3. For all nodes that are flagged as being terminated\r\n$ kubectl drain --force --ignore-daemonset --delete-local-data <node name>\r\n\r\n# 4. Sleep for a few seconds and repeat\r\n```\r\n\r\nNote this is not the whole program, just some snippets to exemplify the idea. We can maybe take this opportunity to rewrite this thing (again) in Go, with some unit tests and everything.\r\n\r\nWhat do you guys think?"},"comment":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/comments/316705815","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/782#issuecomment-316705815","issue_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/782","id":316705815,"user":{"login":"danielfm","id":29534,"avatar_url":"https://avatars2.githubusercontent.com/u/29534?v=4","gravatar_id":"","url":"https://api.github.com/users/danielfm","html_url":"https://github.com/danielfm","followers_url":"https://api.github.com/users/danielfm/followers","following_url":"https://api.github.com/users/danielfm/following{/other_user}","gists_url":"https://api.github.com/users/danielfm/gists{/gist_id}","starred_url":"https://api.github.com/users/danielfm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielfm/subscriptions","organizations_url":"https://api.github.com/users/danielfm/orgs","repos_url":"https://api.github.com/users/danielfm/repos","events_url":"https://api.github.com/users/danielfm/events{/privacy}","received_events_url":"https://api.github.com/users/danielfm/received_events","type":"User","site_admin":false},"created_at":"2017-07-20T13:39:23Z","updated_at":"2017-07-20T13:39:23Z","body":"> That is a very nice way to do it! Do pods have access to node labels/annotations? Not without querying apiserver right?\r\n\r\nYes.\r\n\r\nI'm running a modified version of the node drainer, just for testing purposes, and it seems to be working fine.\r\n\r\n> Going with annotations, we can also utilise already existing reboot controllers instead of custom daemonset if they can do node draining, but if no such one exist then I'd prefer simple shell while true loop :)\r\n\r\nI don't know if we can use any of the reboot controllers, because in our case we need to call `aws autoscaling complete-lifecycle-action` when the draining is complete."}}
{"actor":{"display_login":"danielfm"}}